[arch-devops] [arch-projects] [dbscripts] [PATCH 2/4] Add reproducible archive of packages.

[Florian Pritz]

On Tue, Dec 04, 2018 at 01:15:20PM -0500, Eli Schwartz via arch-devops <arch-devops at lists.archlinux.org> wrote:
> On 12/4/18 1:09 PM, Eli Schwartz wrote:
> > Whenever adding new package files to the pool of distributed packages,
> > hardlink a copy of every package it was built with, into a
> > "reproducible" pool, and log which file required it.
> 
> The question becomes, where can I store these? As-is, this will burden
> the mirror network as well. Unsure how to handle this. Could this be
> configurable by the mirror, as ISOs are now? Should we exclusively
> self-host this, and if so, where?

I'm not a fan of adding this pool to the mirror root for multiple
reasons:

 - Most mirrors would likely want to avoid mirroring it because it can
   become quite large and we told them that we only need around 100GB.
   If everyone wants to exclude it that requires action by every admin.
   Not ideal.

 - I'm not sure if all of our mirrors have hardlink support. We don't
   currently ask for it even though we suggest the -H rsync option. Also
   the current repos use symlinks for the packages instead of hardlinks.
   That said, I'm not even sure if rsync can detect hardlinks across
   directories. It can't even detect renames/moves across directories...

 - I don't expect that we need to mirror it because we don't even get
   that many requests to our current archive. If we ever need to mirror
   it, we can worry about that later I'd say since moving it to the
   mirror root should be rather simple.

I'd suggest to make the base path of the repro pool configurable so that
we can keep it out of the mirror root. For now I'd suggest something
like this:

REPRO_BASE="/srv/reproducible-archive/"
pkgname="foo"
pkgfile="foo-1.0-1.pkg.tar.xz"
dest="$REPRO_BASE/packages/${pkgname:0:1}/$pkgname/$pkgfile"
ln .. "$dest"

Also note that this does intentionally not include $PKGPOOL any more
even though you include it in your patch. The archive doesn't have it
and I don't think it really helps anyone. It will just cause confusion
if packages are moved between repos and it makes using the archive more
difficult because the user would have to check all possible pool names
or know which one to check.

Ideally I'd like to later extend this to also include the current
archive's features and from the looks of it, storing the packages like
this is the first step. Then we just need to copy the repos (dbs and pkg
symlinks) once a day and archive the ISOs.

Also thinking about this, it would be great if we could skip the pkg
symlinks for each day's archive and only copy the db itself. All we'd
need is to have a dedicated PackageServer= setting (like Server=, but
only for packages, not for the database) for pacman to find the
packages, but I'm not sure if Allan would like that. That setting would
also have to support the pkgname substring and the pkgname obviously.

Comments/thoughts/patches/... welcome.

Florian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20181205/6774f74a/attachment-0001.asc>