On 7/12/22 03:51, Jeff Norden wrote: <snip>
The core of the idea is to change the ordering of the package archives so that the files required for an upgrade will be at the beginning. Ideally, they would ordered according to the package-version in which they were last changed. Each pkg.tar file would start with files that are different in the current version than in the previous one. The next section would be those that are the same in the last two versions but differed in the one previous to that, etc. This would turn the tar file into something like a multi-volume incremental backup tape, except that all of the outdated files have been expunged. There would be no redundant files (and no way to restore previous versions). If I'm correct about (A), these files would work fine with current pacman/alpm software and be completely backwards compatible. This would make it possible to do an incremental upgrade by retrieving just an initial part of the archive, which could be done by adding an 'HTTP range' header to the request.
A client that wants to upgrade foo from p.q-r to x.y-z, would need to have two additional pieces of information in order to try this. It would need the exact size N of an initial portion of the uncompressed foo-x.y-z.pkg.tar that contains all blocks for the x.y-z files that differ from the q.r-s ones. (This initial portion would, in fact, be a valid tar file.) It would also need a number Nz such that piping the first Nz bytes of the compressed pkg.tar.zst package file to zstd would produce at least N valid bytes of output before zstd failed upon receiving an early eof.
If the client decides that Nz is sufficiently smaller than the full download size, it would do something like the following:
curl --range 0-Nz https://server:/path/to/foo-x.y-z.pkg.tar.zsd \ | zstd -d 2>/dev/null | head -c N > foo-x.y-z.pkg.part.tar
The client could make its decision on whether this is worthwhile based on its network bandwidth, computing resources, etc.
<snip> Thanks for the very detailed suggestion. I think the downfall is how to calculate the size needed to download. Pacman/makepkg supports at least 9 compression methods, and I don't think this will be an easy calculation for any of them. Allan