[pacman-dev] tar based sync backend implemented!

[Allan McRae]

Hi,

I have implemented the reading directly from the db tarball for sync 
databases.  See the patchset on by "backend" branch 
(http://projects.archlinux.org/users/allan/pacman.git/log/?h=backend). 
Some of the changes are quite big (e.g. 9 files changed, 2184 
insertions, 1571 deletions), but that is mainly moving and duplicating 
code in the process of splitting the handling of local and sync dbs. 
There is still _a lot_ of clean-up to be done in the code, but it is 
fully functional.

This breaks pactests...  lots of pactests... but that is expected as the 
pactest suite does not understand a tar based sync db.  I would like 
help fixing that as I get lost in the pactest code.

Anyway, here is the timing for some operations.  First time is with 
dropped caches, the second running it again with the db cached.

   pacman -Syy  (download time ~15-20sec)
   old:  1m25.197s, 0m22.445s
   new: 0m19.391s, 0m22.084s   (download speed variation...)

   pacman -Sl extra
   old: 0m6.973s, 0m0.193s
   new: 0m3.974s, 0m0.390s

   pacman -Ss pacman
   old: 0m11.521s, 0m0.395s
   new: 0m4.171s, 0m0.477s

   pacman -Si pacman
   old: 0m6.067s, 0m0.101s
   new: 0m1.929s, 0m0.437s

   echo "n" | pacman -Su  (five packages to update)
   old: 0m17.823s, 0m0.306s
   new: 0m10.577s, 0m0.489s

So the tar backend gets some quite significant speed gains in the 
uncached case, and has marginal losses when the db is cached.

Note that the entire sync db is read and parsed no matter the (sync) 
operation.  That explains the difference in the cases with the db 
cached.  This could be adjusted to read the package list (pkgname, 
pkgver-pkgrel) first and read the desc/depends files for the entire db 
or a single package as needed but that is probably distant future stuff...

Allan