Hi,
I have been looking through the current delta implementation in libalpm and have put some thought into changing makepkg/repo-add to support delta creation. However, I'm running into some problems, mostly due to md5sums and gzip.
The current implementation works as follows. On a sync operation it is checked, whether a valid delta path exists and if the summed filesize of the deltas is smaller than the filesize of the whole download. When this is the case the deltas are downloaded and applied to the old file. After that the patched file is treated as if it was downloaded normally, this includes a check of the md5sum. Gzip files have a header, that has a timestamp, which will screw with this md5sum. When a patch is applied to a gzipped file by xdelta, xdelta will unzip the file, apply the patch and then rezip the file. The author of xdelta was obviously aware of the problems with the timestamp, because he decided to leave it empty. The same can be achieved by the -n option of gzip. But there comes the next problem, xdelta uses zlib for compression, gzip implements compression itself. And files created by gzip can differ from files created by zlib. Bsdtar uses zlib as well, but writes the timestamp and there is no option to prevent this (at least none that I can see).
First of all, our current delta implementation doesn't work at all atm, see FS#12000. So any maintainer are welcome ;-)
There are four ways around this, that I can think of:
1. create the package, then create the delta, apply the delta to the old version, remove the original new package and present the patched package as output
I think this sucks, this ties delta creation to makepkg (more about that later) and has an incredibly huge and useless overhead (countless unzips and rezips and applying the patch).
-1
2. create the package, but don't compress it with bsdtar, use gzip -n instead. This means we have to use gzip again, in libalpm, when we apply the delta.
Seems better than 1, but makes makepkg and libalpm rely on gzip. Not sure if this is a good thing, especially for libalpm.
maybe. But I don't see why we should use gzip in libalpm. Iirc we never compress things in alpm.
3. save the md5sums of the unzipped tars in the synchdb and change libalpm to check those
Seems reasonable, but I don't see a way to do this with libarchive, so this would require using zlib directly and pacman would lose the ability to handle to handle tar.bz2
-1
4. Skip checking the md5sum for deltas
OK during the initial synch, as long as we trust xdelta to do its job (the md5sums of both the old and the new file are in the delta file). But the created package will have the wrong md5sum and can't be used to reinstall, etc. which makes this look like a bad idea.
-1 Although, xdelta has its own md5sum mechanism, it won't help here, as you said.
In a previous mail Xavier toyed with the idea to put delta creation into repo-add, I have given this some thought, as it seems nice in principle, but there are drawbacks. For Arch this would mean creating deltas on Gerolde, which seems to be fairly strained already, according to the dev list. Furthermore this introduces some new variables to repo-add (at least repo location and an output location) this would be manageable, but doesn't look very nice.
I don't even understand why we create deltas in makepkg. But if we create deltas with repo-add, makepkg should be changed as well (the resultant pkg.tar.gz should not contain timestamp).
Delta creation in makepkg seems somehow ok (its already in there after all). But what I would really like is a separate tool for delta creation, which would allow the separation of building packages and creating deltas and setting up a separated delta server. This leaves us with options 2 and 3 and I am not really sure, which way to go.
Bye ------------------------------------------------------ SZTE Egyetemi Konyvtar - http://www.bibl.u-szeged.hu This message was sent using IMP: http://horde.org/imp/