[pacman-dev] package loading scheme

[Dan McGee]

On 9/27/07, Aaron Griffin <aaronmgriffin at gmail.com> wrote:
> On 9/26/07, Xavier <shiningxc at gmail.com> wrote:
> > Last commit on Dan's pkgname_check branch :
> > http://code.toofishes.net/gitweb.cgi?p=pacman.git;a=commitdiff;h=b3d5764134bbbdadca9abe439f12aab174eae107
> >
> > > Partial cache cleaning was eliminated in a previous commit because it relied
> > > on package naming conventions. Re-add it the correct way- we actually open
> > > up each package in the cache and get a name and version out of it. If the
> > > name and version match that of an installed package, keep it. If the package
> > > is not installed or the version does not match the locally-installed version,
> > > get rid of it.
> > >
> > > This can easily be modified if some other heuristic of keeping and removing
> > > packages is desired, or if we should clean out the cache dir of any files
> > > that are not packages, etc.
> > >
> > > The biggest current problem with this new approach- speed. Here is one run
> > > on my local machine, going from 1643 to 729 packages in the cache (753 in
> > > the local DB):
> > > real    4m25.829s
> > > user    3m22.527s
> > > sys     0m6.713s
> > >
> > > This is likely best addressed by the package loading scheme, which may be
> > > loading the entirety of each package archive, which is a waste when we only
> > > need the .PKGINFO file read.
> >
> > That's a difference with frugalware I noticed earlier :
> > http://www.archlinux.org/pipermail/pacman-dev/2007-June/008524.html
> >
> > Back then, I thought fw added this feature. But actually, it's the other way
> > around, this feature was removed in arch, in this commit :
> > http://projects.archlinux.org/git/?p=pacman.git;a=commit;h=b2da4b42344444dc22f1e5b01fb4cd09033adc1d
> >
> > And this was caused by this bug report : http://bugs.archlinux.org/task/5120
> >
> > I actually have to ask the same as Aaron in one of his comment there :
> > "Ok, after re-looking at this, I'm fairly confused. If a gzip file is
> > corrupt, shouldn't the gunzipping algorithm/whatever KNOW that? How does it
> > even begin to extract itself, if it's corrupt? I don't know compression
> > algorithms that well, but I thought there was some sort of check for this?"
> >
> > I just started downloading a package with wget, cancelled it, then tried to
> > install it with pacman, and :
> > loading package data... error: error while reading package: Premature end of
> > gzip compressed data: Input/output error
> >
> > I restored the old code that prevented reading the whole archive :
> >  956     if(config && filelist && scriptcheck) {
> >  957       /* we have everything we need */
> >  958       break;
> >  959     }
> >
> > And after that, I got :
> > loading package data... error: error while reading package: (Empty error
> > message)
> >
> > So apparently, it still fails, which is good. However, it apparently doesn't
> > fail at the same point, because of this Empty error message.
> > So I didn't figure out yet where it failed exactly, and if it'll fail with
> > all corrupted archives.
> > At least, it doesn't work with any corrupted archives, which might be good
> > enough for restoring that feature.
> > The times showed by Dan are quite crazy :)
>
> Well, see here's the thing. The *reason* we run through once is just
> for verification that the archive isn't corrupt near the tail end.
>
> For instance, if pacman starts pulling out files, and fails on the
> 101st file, well, everything goes to hell.
>
> It's a valid check, in my opinion, BUT I think we should make it
> optional, for cases like Dan's cache cleaning.

We could implement a pkg_load_meta or pkg_load_info quite easily, I'm sure.

> That is, it might be worth it to throw some flag at it that will tell
> us if we need an integrity check or not. Packages in the cache are
> assumed to be functional, so we should be able to skip it at that
> point, right?

Well...do we really care if it is valid? If it isn't installed, its
getting deleted anyway. If it is installed and invalid...well, let's
hope that isn't the case. Regardless, if we can get a name and version
out of it we should be good to go.

-Dan