FlacLikeABoss

From FarmShare

(Difference between revisions)
Jump to: navigation, search
Line 1: Line 1:
-
Flac like a boss
+
=== Flac like a boss ===
-
Introduction:
+
'''Introduction:'''
-
Flac is a lossless audio codec designed to compress a .wav file. As
+
Flac is a lossless audio codec designed to compress a .wav file. As such the only tradeoff when creating flac files is the eventual size of the content. A worthwhile experiment before us then is:  
-
such the only tradeoff when creating flac files is the eventual size
+
-
of the content. A worthwhile experiment before us then is:
+
-
What is the best combination of options to create the smallest flac
+
What is the best combination of options to create the smallest flac file for a given .wav?  
-
file for a given .wav?
+
 +
'''Executive summary:'''
-
Executive summary:
+
In the experiment detailed below we determine that this flac command line generates the smallest flac file, in 15 minutes, using the barley cluster:  
 +
<pre>--lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac
 +
</pre>
 +
Running this sequentially (IE: on a corn) running flac directly would have taken some 1 day 14hrs to arrive at the same result.
-
In the experiment detailed below we determine that this flac command
+
'''Methodology:'''
-
line generates the smallest flac file, in 15 minutes, using the barley
+
-
cluster:
+
-
--lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac
+
Looking at the flac command line options, --best as described in the man page is synonymous with these options: -l 12 -b 4096 -m -e -r 6
-
Running this sequentially (IE: on a corn) running flac directly would
+
But are these options really the best for my .wav file?
-
have taken some 1 day 14hrs to arrive at the same result.
+
-
 
+
Looking at the set of all possible options it is not obvious to me at all that these are the exact options which give the best results for my .wav file, or any wave file for that matter.  
-
Methodology:
+
<pre>      -m, --mid-side
-
 
+
            Try mid-side coding for each frame (stereo input only)
-
Looking at the flac command line options, --best as described in the man page is syonymous with these options:
+
-
-l 12 -b 4096 -m -e -r 6
+
-
 
+
-
But are these options really the best for my .wav file?
+
-
 
+
-
Looking at the set of all possible options it is not obvious to me at
+
-
all that these are the exact options which give the best results for
+
-
my .wav file, or any wave file for that matter.
+
-
 
+
-
      -m, --mid-side
+
-
              Try mid-side coding for each frame (stereo input only)
+
       -M, --adaptive-mid-side
       -M, --adaptive-mid-side
-
              Adaptive mid-side coding for all frames (stereo input only)
+
            Adaptive mid-side coding for all frames (stereo input only)
       -0..-8, --compression-level-0..--compression-level-8
       -0..-8, --compression-level-0..--compression-level-8
-
              Fastest compression..highest compression (default is -5).  These
+
            Fastest compression..highest compression (default is -5).  These
-
              are synonyms for other options:
+
            are synonyms for other options:
               -0, --compression-level-0
               -0, --compression-level-0
-
                    Synonymous with -l 0 -b 1152 -r 3
+
                    Synonymous with -l 0 -b 1152 -r 3
               -1, --compression-level-1
               -1, --compression-level-1
-
                    Synonymous with -l 0 -b 1152 -M -r 3
+
                    Synonymous with -l 0 -b 1152 -M -r 3
               -2, --compression-level-2
               -2, --compression-level-2
-
                    Synonymous with -l 0 -b 1152 -m -r 3
+
                    Synonymous with -l 0 -b 1152 -m -r 3
               -3, --compression-level-3
               -3, --compression-level-3
-
                    Synonymous with -l 6 -b 4096 -r 4
+
                    Synonymous with -l 6 -b 4096 -r 4
               -4, --compression-level-4
               -4, --compression-level-4
-
                    Synonymous with -l 8 -b 4096 -M -r 4
+
                    Synonymous with -l 8 -b 4096 -M -r 4
               -5, --compression-level-5
               -5, --compression-level-5
-
                    Synonymous with -l 8 -b 4096 -m -r 5
+
                    Synonymous with -l 8 -b 4096 -m -r 5
               -6, --compression-level-6
               -6, --compression-level-6
-
                    Synonymous with -l 8 -b 4096 -m -r 6
+
                    Synonymous with -l 8 -b 4096 -m -r 6
               -7, --compression-level-7
               -7, --compression-level-7
-
                    Synonymous with -l 8 -b 4096 -m -e -r 6
+
                    Synonymous with -l 8 -b 4096 -m -e -r 6
               -8, --compression-level-8
               -8, --compression-level-8
-
                    Synonymous with -l 12 -b 4096 -m -e -r 6
+
                    Synonymous with -l 12 -b 4096 -m -e -r 6
       --fast Fastest compression.  Currently synonymous with -0.
       --fast Fastest compression.  Currently synonymous with -0.
Line 76: Line 63:
       -e, --exhaustive-model-search
       -e, --exhaustive-model-search
-
              Do exhaustive model search (expensive!)
+
            Do exhaustive model search (expensive!)
       -l #, --max-lpc-order=#
       -l #, --max-lpc-order=#
-
              Specifies  the maximum LPC order. This number must be <= 32. For
+
            Specifies  the maximum LPC order. This number must be &lt;= 32. For
-
              Subset streams, it must be <=12 if the sample rate  is  <=48kHz.
+
            Subset streams, it must be &lt;=12 if the sample rate  is  &lt;=48kHz.
-
              If  0,  the  encoder will not attempt generic linear prediction,
+
            If  0,  the  encoder will not attempt generic linear prediction,
-
              and use only fixed predictors. Using fixed predictors is  faster
+
            and use only fixed predictors. Using fixed predictors is  faster
-
              but usually results in files being 5-10% larger.
+
            but usually results in files being 5-10% larger.
       -p, --qlp-coeff-precision-search
       -p, --qlp-coeff-precision-search
-
              Do  exhaustive  search  of  LP  coefficient quantization (expen-
+
            Do  exhaustive  search  of  LP  coefficient quantization (expen-
-
              sive!).  Overrides -q; does nothing if using -l 0
+
            sive!).  Overrides -q; does nothing if using -l 0
       -q #, --qlp-coeff-precision=#
       -q #, --qlp-coeff-precision=#
-
              Precision of the quantized linear-predictor coefficients,  0  =>
+
            Precision of the quantized linear-predictor coefficients,  0  =&gt;
-
              let encoder decide (min is 5, default is 0)
+
            let encoder decide (min is 5, default is 0)
       -r [#,]#, --rice-partition-order=[#,]#
       -r [#,]#, --rice-partition-order=[#,]#
-
              Set the [min,]max residual partition order (0..16). min defaults
+
            Set the [min,]max residual partition order (0..16). min defaults
-
              to 0 if unspecified.  Default is -r 5.
+
            to 0 if unspecified.  Default is -r 5.
 +
</pre>
 +
Let us see if the barley cluster can help us answer this question.  
 +
The first task before us is to write a script which will generate all possible invocations of flac. This will help us characterize the cardinality of the problem. We do this with the following python script:
 +
<pre>#!/usr/bin/python
-
Let us see if the barley cluster can help us answer this question.
+
import os import datetime
-
The first task before us is to write a script which will generate all
+
startjob = datetime.datetime.now()
-
possible invocations of flac. This will help us characterize the
+
-
cardinality of the problem. We do this with the following python
+
-
script:
+
-
#!/usr/bin/python
+
#--lax
 +
#-b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384
 +
#-l &lt;= 32
 +
#-p
 +
#-e
 +
#-r 0..16,0..16
 +
#-m, -M
 +
#
-
import os
+
numjobs = 0
-
import datetime
+
-
startjob = datetime.datetime.now()
+
for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 8192, 16384 ]:
-
# --lax
 
-
# -b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384
 
-
# -l <= 32
 
-
# -p
 
-
# -e
 
-
# -r 0..16,0..16
 
-
# -m, -M
 
-
#
 
-
 
-
numjobs = 0
 
-
 
-
for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 8192, 16384 ]:
 
     for l in [ 0, 1, 4, 8, 16, 32 ]:
     for l in [ 0, 1, 4, 8, 16, 32 ]:
-
        for p in [ '-p', '' ]:
+
      for p in [ '-p', '' ]:''
-
            for e in [ '-e', '' ]:
+
          for e in [ '-e', '' ]:''
-
                for r in [ '%d,%d' % (x, y) for x in xrange(0,16+1,2) for y in xrange(0,16+1,2) if x < y  ]:
+
              for r in [ '%d,%d'&nbsp;% (x, y) for x in xrange(0,16+1,2) for y in xrange(0,16+1,2) if x &lt; y  ]:
-
                    for m in [ '-m', '' ]:
+
                  for m in [ '-m', '' ]:''
-
                        for M in [ '-M', '' ]:
+
                      for M in [ '-M', '' ]:''
-
                            numjobs += 1
+
                          numjobs += 1
-
                            jobout = 'out%04d.flac' % numjobs
+
                          jobout = 'out%04d.flac'&nbsp;% numjobs
-
                            jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
+
                          jobargs = '--lax -b&nbsp;%d -l&nbsp;%d&nbsp;%s&nbsp;%s -r&nbsp;%s&nbsp;%s&nbsp;%s foo.wav -o&nbsp;%s'&nbsp;% ( b, l, p, e, r, m, M, jobout )
-
                            print jobargs
+
                          print jobargs
-
print 'numjobs',numjobs
+
print 'numjobs',numjobs  
-
endjob = datetime.datetime.now()
+
endjob = datetime.datetime.now()  
-
print 'time to submit all jobs: ', endjob-startjob
+
print 'time to submit all jobs: ', endjob-startjob  
 +
</pre>
 +
Running the script says there are 41472 individual flac invocations to run through all possible of options. This is quite a few, and while possible, I see an easy way to reduce cardinality by performing the experiments with a constant block size of 4096.
 +
<pre>#!/usr/bin/python
 +
import os import datetime
-
Running the script says there are 41472 individual flac invocations to
+
startjob = datetime.datetime.now()
-
run through all possible of options. This is quite a few, and while
+
-
possible, I see an easy way to reduce cardinality by performing the
+
-
experiments with a constant block size of 4096.
+
 +
#--lax
 +
#-b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384
 +
#-l &lt;= 32
 +
#-p
 +
#-e
 +
#-r 0..16,0..16
 +
#-m, -M
 +
#
-
#!/usr/bin/python
+
numjobs = 0 for b in [ 4096 ]:
-
import os
 
-
import datetime
 
-
 
-
startjob = datetime.datetime.now()
 
-
 
-
# --lax
 
-
# -b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384
 
-
# -l <= 32
 
-
# -p
 
-
# -e
 
-
# -r 0..16,0..16
 
-
# -m, -M
 
-
#
 
-
 
-
numjobs = 0
 
-
for b in [ 4096 ]:
 
     for l in [ 0, 1, 4, 8, 16, 32 ]:
     for l in [ 0, 1, 4, 8, 16, 32 ]:
-
        for p in [ '-p', '' ]:
+
      for p in [ '-p', '' ]:''
-
            for e in [ '-e', '' ]:
+
          for e in [ '-e', '' ]:''
-
                for r in [ '%d,%d' % (x, y) for x in xrange(0,16,2) for y in xrange(0,16,2) if x < y  ]:
+
              for r in [ '%d,%d'&nbsp;% (x, y) for x in xrange(0,16,2) for y in xrange(0,16,2) if x &lt; y  ]:
-
                    for m in [ '-m', '' ]:
+
                  for m in [ '-m', '' ]:''
-
                        for M in [ '-M', '' ]:
+
                      for M in [ '-M', '' ]:''
-
                            numjobs += 1
+
                          numjobs += 1
-
                            jobout = 'out%04d.flac' % numjobs
+
                          jobout = 'out%04d.flac'&nbsp;% numjobs
-
                            jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
+
                          jobargs = '--lax -b&nbsp;%d -l&nbsp;%d&nbsp;%s&nbsp;%s -r&nbsp;%s&nbsp;%s&nbsp;%s foo.wav -o&nbsp;%s'&nbsp;% ( b, l, p, e, r, m, M, jobout )
-
                            os.system("qsub ~/flac.submit %s" % jobargs)
+
                          os.system("qsub ~/flac.submit&nbsp;%s"&nbsp;% jobargs)
-
                            print jobargs
+
                          print jobargs
-
 
+
-
print 'numjobs',numjobs
+
-
 
+
-
endjob = datetime.datetime.now()
+
-
 
+
-
print 'time to submit all jobs: ', endjob-startjob
+
-
 
+
-
 
+
-
#
+
-
#$ -cwd
+
-
#$ -j y
+
-
#$ -S /bin/bash
+
-
# -sync y
+
-
 
+
-
 
+
-
echo "Got $NSLOTS slots"
+
-
cat $PE_HOSTFILE
+
-
 
+
-
echo "running with"
+
-
echo $*
+
-
time flac -s $*
+
print 'numjobs',numjobs
 +
endjob = datetime.datetime.now()
 +
print 'time to submit all jobs: ', endjob-startjob
 +
</pre> <pre>#
 +
#$ -cwd
 +
#$ -j y
 +
#$ -S /bin/bash
 +
#-sync y
-
This reduces our cardinality to 2688.  This is quite doable on the barley cluster.
+
echo "Got $NSLOTS slots" cat $PE_HOSTFILE
 +
echo "running with" echo $*
-
$ python ~/doflacsubmit.py
+
time flac -s $*
-
Your job 41540 ("flac.submit") has been submitted
+
</pre>
 +
<br> This reduces our cardinality to 2688. This is quite doable on the barley cluster.
 +
<pre>$ python ~/doflacsubmit.py Your job 41540 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -m -M foo.wav -o out0001.flac
--lax -b 4096 -l 0 -p -e -r 0,2 -m -M foo.wav -o out0001.flac
Your job 41541 ("flac.submit") has been submitted
Your job 41541 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,2 -m foo.wav -o out0002.flac
+
--lax -b 4096 -l 0 -p -e -r 0,2 -m foo.wav -o out0002.flac
Your job 41542 ("flac.submit") has been submitted
Your job 41542 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,2 -M foo.wav -o out0003.flac
+
--lax -b 4096 -l 0 -p -e -r 0,2 -M foo.wav -o out0003.flac
Your job 41543 ("flac.submit") has been submitted
Your job 41543 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,2   foo.wav -o out0004.flac
+
--lax -b 4096 -l 0 -p -e -r 0,2 foo.wav -o out0004.flac
Your job 41544 ("flac.submit") has been submitted
Your job 41544 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -m -M foo.wav -o out0005.flac
--lax -b 4096 -l 0 -p -e -r 0,4 -m -M foo.wav -o out0005.flac
Your job 41545 ("flac.submit") has been submitted
Your job 41545 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,4 -m foo.wav -o out0006.flac
+
--lax -b 4096 -l 0 -p -e -r 0,4 -m foo.wav -o out0006.flac
Your job 41546 ("flac.submit") has been submitted
Your job 41546 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,4 -M foo.wav -o out0007.flac
+
--lax -b 4096 -l 0 -p -e -r 0,4 -M foo.wav -o out0007.flac
Your job 41547 ("flac.submit") has been submitted
Your job 41547 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,4   foo.wav -o out0008.flac
+
--lax -b 4096 -l 0 -p -e -r 0,4 foo.wav -o out0008.flac
Your job 41548 ("flac.submit") has been submitted
Your job 41548 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -m -M foo.wav -o out0009.flac
--lax -b 4096 -l 0 -p -e -r 0,6 -m -M foo.wav -o out0009.flac
Your job 41549 ("flac.submit") has been submitted
Your job 41549 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,6 -m foo.wav -o out0010.flac
+
--lax -b 4096 -l 0 -p -e -r 0,6 -m foo.wav -o out0010.flac
Your job 41550 ("flac.submit") has been submitted
Your job 41550 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,6 -M foo.wav -o out0011.flac
+
--lax -b 4096 -l 0 -p -e -r 0,6 -M foo.wav -o out0011.flac
Your job 41551 ("flac.submit") has been submitted
Your job 41551 ("flac.submit") has been submitted
-
--lax -b 4096 -l 0 -p -e -r 0,6   foo.wav -o out0012.flac
+
--lax -b 4096 -l 0 -p -e -r 0,6 foo.wav -o out0012.flac
.
.
.
.
-
.
+
.  
-
 
+
</pre>
-
 
+
<br> Checking qstat shows that in a little bit over 10 minutes all the jobs are finished:  
-
Checking qstat shows that in a little bit over 10 minutes all the jobs are finished:
+
<pre>$ more start end  
-
 
+
-
 
+
-
$ more start end
+
::::::::::::::
::::::::::::::
-
start
+
start  
::::::::::::::
::::::::::::::
-
Wed Dec 28 13:13:08 PST 2011
+
Wed Dec 28 13:13:08 PST 2011  
::::::::::::::
::::::::::::::
-
end
+
end  
::::::::::::::
::::::::::::::
-
Wed Dec 28 13:23:34 PST 2011
+
Wed Dec 28 13:23:34 PST 2011  
-
 
+
</pre>
-
However, looking at the accumulated cpu time, this experiment consumed
+
However, looking at the accumulated cpu time, this experiment consumed 1 day and 13hrs cpu time.  
-
1 day and 13hrs cpu time.
+
<pre>$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py
-
 
+
total 1 day, 13:17:16.980000  
-
$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py      
+
</pre>
-
total 1 day, 13:17:16.980000
+
This shows that even for tasks which do not scream HPC, using the barley cluster is still win. If you had executed flac in the inner loop of the python script instead of qsub, you would have waited for over a day and a half to get the results.  
-
 
+
-
This shows that even for tasks which do not scream HPC, using the
+
-
barley cluster is still win. If you had executed flac in the inner
+
-
loop of the python script instead of qsub, you would have waited for
+
-
over a day and a half to get the results.
+
-
 
+
-
 
+
-
 
+
-
Looking at the .flac files produced, ls -lrS shows the smallest files are these:
+
-
$ ls -lrS *.flac | head
+
Looking at the .flac files produced, ls -lrS shows the smallest files are these:
 +
<pre>$ ls -lrS *.flac | head
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2268.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2268.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2266.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2266.flac
Line 273: Line 227:
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2254.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2254.flac
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2252.flac
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2252.flac
-
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2250.flac
+
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2250.flac  
-
 
+
</pre>
-
and the largest are these:
+
and the largest are these:  
-
 
+
<pre>-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0447.flac
-
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0447.flac
+
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0445.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0445.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0223.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0223.flac
-
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0221.flac
+
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0221.flac  
-
 
+
</pre>
-
 
+
This is over an 11% spread in size, even while keeping the block size (-b option) constant. While --best is decent, it did not give the best results.  
-
This is over an 11% spread in size, even while keeping the block size
+
<pre>$ flac --best foo.wav -o best.flac  
-
(-b option) constant. While --best is decent, it did not give the best results.
+
-
 
+
-
$ flac --best foo.wav -o best.flac
+
-
 
+
-
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007  Josh Coalson
+
-
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
+
-
welcome to redistribute it under certain conditions.  Type `flac' for details.
+
-
 
+
-
foo.wav: wrote 26454456 bytes, ratio=0.578
+
-
 
+
-
$ ls -l best*
+
-
-rw-r--r-- 1 bishopj root 26454456 Dec 21 12:58 best.flac
+
 +
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details.
-
So now lets take the options which performed the best and pop those
+
foo.wav: wrote 26454456 bytes, ratio=0.578
-
into a new submit script which will run through all the possible block
+
-
sizes.
+
-
$ for i in `ls -rS *.flac | head -10`; do grep $i *submit*; done
+
$ ls -l best* -rw-r--r-- 1 bishopj root 26454456 Dec 21 12:58 best.flac
 +
</pre>
 +
So now lets take the options which performed the best and pop those into a new submit script which will run through all the possible block sizes.
 +
<pre>$ for i in `ls -rS *.flac | head -10`; do grep $i *submit*; done
flac.submit.o43807:--lax -b 4096 -l 32 -p -e -r 0,14 foo.wav -o out2268.flac
flac.submit.o43807:--lax -b 4096 -l 32 -p -e -r 0,14 foo.wav -o out2268.flac
flac.submit.o43805:--lax -b 4096 -l 32 -p -e -r 0,14 -m foo.wav -o out2266.flac
flac.submit.o43805:--lax -b 4096 -l 32 -p -e -r 0,14 -m foo.wav -o out2266.flac
Line 312: Line 255:
flac.submit.o43793:--lax -b 4096 -l 32 -p -e -r 0,8 -m foo.wav -o out2254.flac
flac.submit.o43793:--lax -b 4096 -l 32 -p -e -r 0,8 -m foo.wav -o out2254.flac
flac.submit.o43791:--lax -b 4096 -l 32 -p -e -r 0,6 foo.wav -o out2252.flac
flac.submit.o43791:--lax -b 4096 -l 32 -p -e -r 0,6 foo.wav -o out2252.flac
-
flac.submit.o43789:--lax -b 4096 -l 32 -p -e -r 0,6 -m foo.wav -o out2250.flac
+
flac.submit.o43789:--lax -b 4096 -l 32 -p -e -r 0,6 -m foo.wav -o out2250.flac  
 +
</pre>
 +
Looks like --lax -b 4096 -l 32 -p -e -r 0,14 is what we want. (0,16 actually, as I had a booboo in my previous script which is fixed below).
 +
<pre>#!/usr/bin/python
 +
import os import datetime
-
Looks like --lax -b 4096 -l 32 -p -e -r 0,14 is what we want. (0,16
+
startjob = datetime.datetime.now()  
-
actually, as I had a booboo in my previous script which is fixed below).
+
 +
numjobs = 0
-
#!/usr/bin/python
+
for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 4096+1024, 4096+2048, 8192, 16384 ]:
-
import os
 
-
import datetime
 
-
 
-
startjob = datetime.datetime.now()
 
-
 
-
numjobs = 0
 
-
 
-
for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 4096+1024, 4096+2048, 8192, 16384 ]:
 
     for l in [ 32 ]:
     for l in [ 32 ]:
-
        for p in [ '-p' ]:
+
      for p in [ '-p' ]:
-
            for e in [ '-e' ]:
+
          for e in [ '-e' ]:
-
                for r in [ '%d,%d' % (x, y) for x in xrange(0,1) for y in xrange(16,16+1) if x < y  ]:
+
              for r in [ '%d,%d'&nbsp;% (x, y) for x in xrange(0,1) for y in xrange(16,16+1) if x &lt; y  ]:
-
                    for m in [ '' ]:
+
                  for m in [ '' ]:''
-
                        for M in [ '' ]:
+
                      for M in [ '' ]:''
-
                            numjobs += 1
+
                          numjobs += 1
-
                            jobout = 'out%04d.flac' % numjobs
+
                          jobout = 'out%04d.flac'&nbsp;% numjobs
-
                            jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
+
                          jobargs = '--lax -b&nbsp;%d -l&nbsp;%d&nbsp;%s&nbsp;%s -r&nbsp;%s&nbsp;%s&nbsp;%s foo.wav -o&nbsp;%s'&nbsp;% ( b, l, p, e, r, m, M, jobout )
-
                            os.system("qsub ~/flac.submit %s" % jobargs)
+
                          os.system("qsub ~/flac.submit&nbsp;%s"&nbsp;% jobargs)
-
                            print jobargs
+
                          print jobargs
-
 
+
-
print 'numjobs',numjobs
+
-
 
+
-
endjob = datetime.datetime.now()
+
-
 
+
-
print 'time to submit all jobs: ', endjob-startjob
+
 +
print 'numjobs',numjobs
-
Lets run it:
+
endjob = datetime.datetime.now()
 +
print 'time to submit all jobs: ', endjob-startjob
 +
</pre>
 +
<br> Lets run it:
 +
<pre>
$ python ~/doflacsubmit2.py
$ python ~/doflacsubmit2.py
Your job 44258 ("flac.submit") has been submitted
Your job 44258 ("flac.submit") has been submitted
Line 381: Line 319:
numjobs 14
numjobs 14
time to submit all jobs:  0:00:00.511025
time to submit all jobs:  0:00:00.511025
 +
</pre> <pre>$ qstat job-ID prior name user state submit/start at queue slots ja-task-ID
 +
 +
----
-
$ qstat
 
-
job-ID  prior  name      user        state submit/start at    queue                          slots ja-task-ID
 
-
-----------------------------------------------------------------------------------------------------------------
 
   44258 0.25001 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley13.stanford.edu      1         
   44258 0.25001 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley13.stanford.edu      1         
-
  44259 0.25001 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley13.stanford.edu      1         
+
44259 0.25001 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley13.stanford.edu      1         
-
  44260 0.25001 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley07.stanford.edu      1         
+
44260 0.25001 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley07.stanford.edu      1         
-
  44261 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley01.stanford.edu      1         
+
44261 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley01.stanford.edu      1         
-
  44262 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley08.stanford.edu      1         
+
44262 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley08.stanford.edu      1         
-
  44263 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley14.stanford.edu      1         
+
44263 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley14.stanford.edu      1         
-
  44264 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu    1         
+
44264 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu    1         
-
  44265 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley15.stanford.edu      1         
+
44265 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley15.stanford.edu      1         
-
  44266 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley13.stanford.edu      1         
+
44266 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley13.stanford.edu      1         
-
  44267 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley07.stanford.edu      1         
+
44267 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley07.stanford.edu      1         
-
  44268 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley01.stanford.edu      1         
+
44268 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley01.stanford.edu      1         
-
  44269 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley08.stanford.edu      1         
+
44269 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley08.stanford.edu      1         
-
  44270 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley14.stanford.edu      1         
+
44270 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 main.q@barley14.stanford.edu      1         
-
  44271 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu    1         
+
44271 0.25000 flac.submi bishopj      r    12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu    1         
 +
</pre>
 +
Wait 5 minutes for the jobs to finish....
 +
<pre>$ flac --best foo.wav -o best.flac
 +
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details.
-
Wait 5 minutes for the jobs to finish....
+
foo.wav: wrote 26454456 bytes, ratio=0.578  
-
 
+
-
$ flac --best foo.wav -o best.flac
+
-
 
+
-
flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007  Josh Coalson
+
-
flac comes with ABSOLUTELY NO WARRANTY.  This is free software, and you are
+
-
welcome to redistribute it under certain conditions.  Type `flac' for details.
+
-
 
+
-
foo.wav: wrote 26454456 bytes, ratio=0.578
+
-
 
+
$ ls -lrS *.flac
$ ls -lrS *.flac
Line 428: Line 361:
-rw-r--r-- 1 bishopj root 27469566 Dec 21 12:58 out0006.flac
-rw-r--r-- 1 bishopj root 27469566 Dec 21 12:58 out0006.flac
-rw-r--r-- 1 bishopj root 27796933 Dec 21 12:58 out0001.flac
-rw-r--r-- 1 bishopj root 27796933 Dec 21 12:58 out0001.flac
-
 
+
</pre>
-
This is our winning flac command line:
+
<br> This is our winning flac command line:  
-
 
+
<pre>--lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac  
-
--lax -b 4096 -l 32 -p -e -r 0,16   foo.wav -o out0010.flac
+
</pre>
-
 
+
Incidentally, this last set of jobs still took significant resources (just under an hour)  
-
 
+
<pre>$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py total 0:54:31.850000
-
Incidentally, this last set of jobs still took significant resources (just under an hour)
+
</pre>
-
 
+
-
$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py
+
-
total 0:54:31.850000
+

Revision as of 18:53, 3 January 2012

Flac like a boss

Introduction:

Flac is a lossless audio codec designed to compress a .wav file. As such the only tradeoff when creating flac files is the eventual size of the content. A worthwhile experiment before us then is:

What is the best combination of options to create the smallest flac file for a given .wav?

Executive summary:

In the experiment detailed below we determine that this flac command line generates the smallest flac file, in 15 minutes, using the barley cluster: 

--lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac

Running this sequentially (IE: on a corn) running flac directly would have taken some 1 day 14hrs to arrive at the same result.

Methodology:

Looking at the flac command line options, --best as described in the man page is synonymous with these options: -l 12 -b 4096 -m -e -r 6

But are these options really the best for my .wav file?

Looking at the set of all possible options it is not obvious to me at all that these are the exact options which give the best results for my .wav file, or any wave file for that matter. 

       -m, --mid-side
             Try mid-side coding for each frame (stereo input only)

       -M, --adaptive-mid-side
             Adaptive mid-side coding for all frames (stereo input only)

       -0..-8, --compression-level-0..--compression-level-8
             Fastest compression..highest compression (default is -5).  These
             are synonyms for other options:

              -0, --compression-level-0
                    Synonymous with -l 0 -b 1152 -r 3

              -1, --compression-level-1
                    Synonymous with -l 0 -b 1152 -M -r 3

              -2, --compression-level-2
                    Synonymous with -l 0 -b 1152 -m -r 3

              -3, --compression-level-3
                    Synonymous with -l 6 -b 4096 -r 4

              -4, --compression-level-4
                    Synonymous with -l 8 -b 4096 -M -r 4

              -5, --compression-level-5
                    Synonymous with -l 8 -b 4096 -m -r 5

              -6, --compression-level-6
                    Synonymous with -l 8 -b 4096 -m -r 6

              -7, --compression-level-7
                    Synonymous with -l 8 -b 4096 -m -e -r 6

              -8, --compression-level-8
                    Synonymous with -l 12 -b 4096 -m -e -r 6

       --fast Fastest compression.  Currently synonymous with -0.

       --best Highest compression.  Currently synonymous with -8.

       -e, --exhaustive-model-search
             Do exhaustive model search (expensive!)

       -l #, --max-lpc-order=#
             Specifies  the maximum LPC order. This number must be <= 32. For
             Subset streams, it must be <=12 if the sample rate  is  <=48kHz.
             If  0,  the  encoder will not attempt generic linear prediction,
             and use only fixed predictors. Using fixed predictors is  faster
             but usually results in files being 5-10% larger.

       -p, --qlp-coeff-precision-search
             Do  exhaustive  search  of  LP  coefficient quantization (expen-
             sive!).  Overrides -q; does nothing if using -l 0

       -q #, --qlp-coeff-precision=#
             Precision of the quantized linear-predictor coefficients,  0  =>
             let encoder decide (min is 5, default is 0)

       -r [#,]#, --rice-partition-order=[#,]#
             Set the [min,]max residual partition order (0..16). min defaults
             to 0 if unspecified.  Default is -r 5.

Let us see if the barley cluster can help us answer this question.

The first task before us is to write a script which will generate all possible invocations of flac. This will help us characterize the cardinality of the problem. We do this with the following python script: 

#!/usr/bin/python

import os import datetime 

startjob = datetime.datetime.now() 

#--lax 
#-b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384 
#-l <= 32 
#-p 
#-e 
#-r 0..16,0..16 
#-m, -M 
#

numjobs = 0 

for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 8192, 16384 ]: 

    for l in [ 0, 1, 4, 8, 16, 32 ]:
       for p in [ '-p', '' ]:''
           for e in [ '-e', '' ]:''
               for r in [ '%d,%d' % (x, y) for x in xrange(0,16+1,2) for y in xrange(0,16+1,2) if x < y  ]:
                   for m in [ '-m', '' ]:''
                       for M in [ '-M', '' ]:''
                           numjobs += 1
                           jobout = 'out%04d.flac' % numjobs
                           jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
                           print jobargs

print 'numjobs',numjobs 

endjob = datetime.datetime.now() 

print 'time to submit all jobs: ', endjob-startjob

Running the script says there are 41472 individual flac invocations to run through all possible of options. This is quite a few, and while possible, I see an easy way to reduce cardinality by performing the experiments with a constant block size of 4096. 

#!/usr/bin/python

import os import datetime 

startjob = datetime.datetime.now() 

#--lax 
#-b 192, 576, 1152, 2304, 4608 256 512 1024 2048 4096 8192 16384 
#-l <= 32 
#-p 
#-e 
#-r 0..16,0..16 
#-m, -M 
#

numjobs = 0 for b in [ 4096 ]: 

    for l in [ 0, 1, 4, 8, 16, 32 ]:
       for p in [ '-p', '' ]:''
           for e in [ '-e', '' ]:''
               for r in [ '%d,%d' % (x, y) for x in xrange(0,16,2) for y in xrange(0,16,2) if x < y  ]:
                   for m in [ '-m', '' ]:''
                       for M in [ '-M', '' ]:''
                           numjobs += 1
                           jobout = 'out%04d.flac' % numjobs
                           jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
                           os.system("qsub ~/flac.submit %s" % jobargs)
                           print jobargs

print 'numjobs',numjobs 

endjob = datetime.datetime.now() 

print 'time to submit all jobs: ', endjob-startjob 

#

  1. $ -cwd 

  2. $ -j y 

  3. $ -S /bin/bash 

  4. -sync y



echo "Got $NSLOTS slots" cat $PE_HOSTFILE 

echo "running with" echo $* 

time flac -s $*


This reduces our cardinality to 2688. This is quite doable on the barley cluster. 

$ python ~/doflacsubmit.py Your job 41540 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -m -M foo.wav -o out0001.flac
Your job 41541 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -m foo.wav -o out0002.flac
Your job 41542 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 -M foo.wav -o out0003.flac
Your job 41543 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,2 foo.wav -o out0004.flac
Your job 41544 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -m -M foo.wav -o out0005.flac
Your job 41545 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -m foo.wav -o out0006.flac
Your job 41546 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 -M foo.wav -o out0007.flac
Your job 41547 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,4 foo.wav -o out0008.flac
Your job 41548 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -m -M foo.wav -o out0009.flac
Your job 41549 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -m foo.wav -o out0010.flac
Your job 41550 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 -M foo.wav -o out0011.flac
Your job 41551 ("flac.submit") has been submitted
--lax -b 4096 -l 0 -p -e -r 0,6 foo.wav -o out0012.flac
.
.
.


Checking qstat shows that in a little bit over 10 minutes all the jobs are finished: 

$ more start end 
::::::::::::::
start 
::::::::::::::
Wed Dec 28 13:13:08 PST 2011 
::::::::::::::
end 
::::::::::::::
Wed Dec 28 13:23:34 PST 2011

However, looking at the accumulated cpu time, this experiment consumed 1 day and 13hrs cpu time. 

$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py
total 1 day, 13:17:16.980000

This shows that even for tasks which do not scream HPC, using the barley cluster is still win. If you had executed flac in the inner loop of the python script instead of qsub, you would have waited for over a day and a half to get the results.

Looking at the .flac files produced, ls -lrS shows the smallest files are these: 

$ ls -lrS *.flac | head
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2268.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2266.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2264.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2262.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2260.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2258.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2256.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out2254.flac
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2252.flac
-rw-r--r-- 1 bishopj root 26416224 Dec 21 12:58 out2250.flac

and the largest are these: 

-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0447.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0445.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0223.flac
-rw-r--r-- 1 bishopj root 29887331 Dec 21 12:58 out0221.flac

This is over an 11% spread in size, even while keeping the block size (-b option) constant. While --best is decent, it did not give the best results. 

$ flac --best foo.wav -o best.flac 

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details. 

foo.wav: wrote 26454456 bytes, ratio=0.578 

$ ls -l best* -rw-r--r-- 1 bishopj root 26454456 Dec 21 12:58 best.flac

So now lets take the options which performed the best and pop those into a new submit script which will run through all the possible block sizes. 

$ for i in `ls -rS *.flac | head -10`; do grep $i *submit*; done
flac.submit.o43807:--lax -b 4096 -l 32 -p -e -r 0,14 foo.wav -o out2268.flac
flac.submit.o43805:--lax -b 4096 -l 32 -p -e -r 0,14 -m foo.wav -o out2266.flac
flac.submit.o43803:--lax -b 4096 -l 32 -p -e -r 0,12 foo.wav -o out2264.flac
flac.submit.o43801:--lax -b 4096 -l 32 -p -e -r 0,12 -m foo.wav -o out2262.flac
flac.submit.o43799:--lax -b 4096 -l 32 -p -e -r 0,10 foo.wav -o out2260.flac
flac.submit.o43797:--lax -b 4096 -l 32 -p -e -r 0,10 -m foo.wav -o out2258.flac
flac.submit.o43795:--lax -b 4096 -l 32 -p -e -r 0,8 foo.wav -o out2256.flac
flac.submit.o43793:--lax -b 4096 -l 32 -p -e -r 0,8 -m foo.wav -o out2254.flac
flac.submit.o43791:--lax -b 4096 -l 32 -p -e -r 0,6 foo.wav -o out2252.flac
flac.submit.o43789:--lax -b 4096 -l 32 -p -e -r 0,6 -m foo.wav -o out2250.flac

Looks like --lax -b 4096 -l 32 -p -e -r 0,14 is what we want. (0,16 actually, as I had a booboo in my previous script which is fixed below). 

#!/usr/bin/python

import os import datetime 

startjob = datetime.datetime.now() 

numjobs = 0 

for b in [ 192, 576, 1152, 2304, 4608, 256, 512, 1024, 2048, 4096, 4096+1024, 4096+2048, 8192, 16384 ]: 

    for l in [ 32 ]:
       for p in [ '-p' ]:
           for e in [ '-e' ]:
               for r in [ '%d,%d' % (x, y) for x in xrange(0,1) for y in xrange(16,16+1) if x < y  ]:
                   for m in [ '' ]:''
                       for M in [ '' ]:''
                           numjobs += 1
                           jobout = 'out%04d.flac' % numjobs
                           jobargs = '--lax -b %d -l %d %s %s -r %s %s %s foo.wav -o %s' % ( b, l, p, e, r, m, M, jobout )
                           os.system("qsub ~/flac.submit %s" % jobargs)
                           print jobargs

print 'numjobs',numjobs 

endjob = datetime.datetime.now() 

print 'time to submit all jobs: ', endjob-startjob


Lets run it: 

$ python ~/doflacsubmit2.py
Your job 44258 ("flac.submit") has been submitted
--lax -b 192 -l 32 -p -e -r 0,16   foo.wav -o out0001.flac
Your job 44259 ("flac.submit") has been submitted
--lax -b 576 -l 32 -p -e -r 0,16   foo.wav -o out0002.flac
Your job 44260 ("flac.submit") has been submitted
--lax -b 1152 -l 32 -p -e -r 0,16   foo.wav -o out0003.flac
Your job 44261 ("flac.submit") has been submitted
--lax -b 2304 -l 32 -p -e -r 0,16   foo.wav -o out0004.flac
Your job 44262 ("flac.submit") has been submitted
--lax -b 4608 -l 32 -p -e -r 0,16   foo.wav -o out0005.flac
Your job 44263 ("flac.submit") has been submitted
--lax -b 256 -l 32 -p -e -r 0,16   foo.wav -o out0006.flac
Your job 44264 ("flac.submit") has been submitted
--lax -b 512 -l 32 -p -e -r 0,16   foo.wav -o out0007.flac
Your job 44265 ("flac.submit") has been submitted
--lax -b 1024 -l 32 -p -e -r 0,16   foo.wav -o out0008.flac
Your job 44266 ("flac.submit") has been submitted
--lax -b 2048 -l 32 -p -e -r 0,16   foo.wav -o out0009.flac
Your job 44267 ("flac.submit") has been submitted
--lax -b 4096 -l 32 -p -e -r 0,16   foo.wav -o out0010.flac
Your job 44268 ("flac.submit") has been submitted
--lax -b 5120 -l 32 -p -e -r 0,16   foo.wav -o out0011.flac
Your job 44269 ("flac.submit") has been submitted
--lax -b 6144 -l 32 -p -e -r 0,16   foo.wav -o out0012.flac
Your job 44270 ("flac.submit") has been submitted
--lax -b 8192 -l 32 -p -e -r 0,16   foo.wav -o out0013.flac
Your job 44271 ("flac.submit") has been submitted
--lax -b 16384 -l 32 -p -e -r 0,16   foo.wav -o out0014.flac
numjobs 14
time to submit all jobs:  0:00:00.511025

$ qstat job-ID prior name user state submit/start at queue slots ja-task-ID 

----



 44258 0.25001 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley13.stanford.edu       1        
44259 0.25001 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley13.stanford.edu       1        
44260 0.25001 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley07.stanford.edu       1        
44261 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley01.stanford.edu       1        
44262 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley08.stanford.edu       1        
44263 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley14.stanford.edu       1        
44264 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu     1        
44265 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley15.stanford.edu       1        
44266 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley13.stanford.edu       1        
44267 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley07.stanford.edu       1        
44268 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley01.stanford.edu       1        
44269 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley08.stanford.edu       1        
44270 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 main.q@barley14.stanford.edu       1        
44271 0.25000 flac.submi bishopj      r     12/29/2011 10:09:05 bigmem.q@barley05.stanford.edu     1

Wait 5 minutes for the jobs to finish.... 

$ flac --best foo.wav -o best.flac 

flac 1.2.1, Copyright (C) 2000,2001,2002,2003,2004,2005,2006,2007 Josh Coalson flac comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. Type `flac' for details. 

foo.wav: wrote 26454456 bytes, ratio=0.578 

$ ls -lrS *.flac
-rw-r--r-- 1 bishopj root 26416201 Dec 21 12:58 out0010.flac
-rw-r--r-- 1 bishopj root 26422887 Dec 21 12:58 out0005.flac
-rw-r--r-- 1 bishopj root 26433325 Dec 21 12:58 out0011.flac
-rw-r--r-- 1 bishopj root 26436989 Dec 21 12:58 out0004.flac
-rw-r--r-- 1 bishopj root 26451960 Dec 21 12:58 out0009.flac
-rw-r--r-- 1 bishopj root 26454456 Dec 21 12:58 best.flac
-rw-r--r-- 1 bishopj root 26454801 Dec 21 12:58 out0012.flac
-rw-r--r-- 1 bishopj root 26494744 Dec 21 12:58 out0013.flac
-rw-r--r-- 1 bishopj root 26575590 Dec 21 12:58 out0003.flac
-rw-r--r-- 1 bishopj root 26611006 Dec 21 12:58 out0008.flac
-rw-r--r-- 1 bishopj root 26612491 Dec 21 12:58 out0014.flac
-rw-r--r-- 1 bishopj root 26857694 Dec 21 12:58 out0002.flac
-rw-r--r-- 1 bishopj root 26923480 Dec 21 12:58 out0007.flac
-rw-r--r-- 1 bishopj root 27469566 Dec 21 12:58 out0006.flac
-rw-r--r-- 1 bishopj root 27796933 Dec 21 12:58 out0001.flac


This is our winning flac command line: 

--lax -b 4096 -l 32 -p -e -r 0,16 foo.wav -o out0010.flac

Incidentally, this last set of jobs still took significant resources (just under an hour) 

$ grep elap *.submit* | awk '{print $3}' | sed -e 's/\([0-9]*:[0-9]*\.[0-9]*\)elapsed/\1/' | python ~/parseelaspedtime.py total 0:54:31.850000
Retrieved from "http://stanford.edu/group/farmshare/cgi-bin/wiki/index.php/FlacLikeABoss"
Personal tools
Toolbox
LANGUAGES