[arch-general] lzma compression tests

[Dan McGee]

CCing to pacman-dev.

On Mon, Mar 24, 2008 at 3:25 PM, Jan de Groot <jan at jgc.homeip.net> wrote:
>
> On Mon, 2008-03-24 at 14:43 -0500, Dan McGee wrote:
>  > On Mon, Mar 24, 2008 at 2:24 PM, Mark Constable <markc at renta.net> wrote:
>  > > FWIW, possibly worth considering adding lzma to pacman...
>  > >
>  > >   http://www.kdedevelopers.org/node/3326
>  > >
>  > >  --markc
>  >
>  > Interesting. You may want ot take this to the pacman-dev ML. A good
>  > "real-world" look might be grabbing the openoffice package and
>  > uncompressing it to a folder, and then run the following tests:
>  >
>  > bsdtar czf package (gzip)
>  > bsdtar cjf package (bzip2)
>  > <lzma creation of package>
>  >
>  > and then the similar unzip costs, along with the size of the original
>  > folder structure (unzipped package) vs. their zipped counterparts.
>  >
>  > -Dan
>  >
>
>  I tested with OpenOffice.org, here are the results. Note that I had to
>  use a weird construction to compress lzma files, as tar has no native
>  support yet (1.19.90 has official lzma support I read on some debian
>  mailinglist).
>
>  Compression times take a huge increase in time, but in case of
>  openoffice, I think 30 seconds extra compression time will save on much
>  more upload time, as the resulting package would be 20MB smaller.
>  For the end user, it depends on their connection. Most users will have
>  <10Mb/s connections and will see a positive change in installation
>  speed: 10 seconds extra for extraction, 20 seconds less for downloading
>  the package.
>
>  [jan at server ~]$ time bsdtar czf oo.tar.gz oo/
>  real    0m24.244s
>  user    0m23.882s
>  sys     0m0.360s
>
>  [jan at server ~]$ time bsdtar cjf oo.tar.bz2 oo/
>  real    0m55.812s
>  user    0m55.310s
>  sys     0m0.377s
>
>  [jan at server ~]$ time tar c oo | lzma -2 > oo.tar.lzma
>  real    0m55.070s
>  user    0m54.583s
>  sys     0m0.740s
>
>  [jan at server ~]$ time tar c oo | lzma -1 > oo.tar.lzma1
>  real    0m38.995s
>  user    0m38.737s
>  sys     0m0.503s
>
>  [jan at server ~]$ du -h oo.tar.*
>  108M    oo.tar.bz2
>  115M    oo.tar.gz
>  95M     oo.tar.lzma
>  101M    oo.tar.lzma1
>
>  [jan at server u]$ time tar zxf ../oo.tar.gz
>  real    0m3.570s
>  user    0m3.383s
>  sys     0m0.967s
>
>  [jan at server u]$ time tar jxf ../oo.tar.bz2
>  real    0m19.431s
>  user    0m17.826s
>  sys     0m1.367s
>
>  [jan at server u]$ time tar --use-compress-program=lzma -xf ../oo.tar.lzma
>  real    0m13.211s
>  user    0m12.306s
>  sys     0m1.117s
>  (the lzma1 archive gives equal timings)

Did you try with bsdtar on the extraction as well? That may make a
difference over (gnu)tar. Should be:
time bsdtar xf ../oo.tar.gz
time bsdtar xf ../oo.tar.bz2
time bsdtar --use-compress-program=lzma -xf ../oo.tar.lzma

-Dan