Em julho 5, 2017 15:57 Florian via arch-devops escreveu:

My gut says 6 months should be enough, but feel free to disagree.

Agree with 6 months. More than enough time for issues to appear and people to need the archive.

Once we have decided how long data should be kept, we also need something to actually delete it. Deleting files from the ./repos directory should be simple, but the ./packages directory is more complicated because it is not nicely separated into directories per day.

Sebastien, do you have an idea about how we could delete old data or do you have a script for that?

Well, find can do that. At least, I always used either atime/mtime to delete older backups with find. Don't know if there is a better tool for this. Cheers, Giancarlo Razzolini

Em julho 5, 2017 19:26 Levente Polyak escreveu:

Well it will be bit more complicated then that. One we finally get the reproducible builds patches live we will need the archives for the reproducer script to build such package again. If we simply clean by date we will loose the possibility to rebuild certain packages that don't require a rebuild of something but we're using a specific version with its specific behaviors during build time.

I'm not sure how to tackle this while not loosing reproducibility, but we certainly should think about this scenario before doing time based cleanups.

What about time based + keep the latest n versions of a package? Do you think that would be enough to satisfy the reproducibility needs? Cheers, Giancarlo Razzolini

Can't we use btrfs block level dedup for this as well so the disk space issue is not as pressing? On Thu, Jul 6, 2017, 00:36 Levente Polyak wrote:

On July 6, 2017 12:31:07 AM GMT+02:00, Giancarlo Razzolini < grazzolini@archlinux.org> wrote:

...

Em julho 5, 2017 19:26 Levente Polyak escreveu:

...

Well it will be bit more complicated then that. One we finally get

the reproducible builds patches live we will need the archives for the reproducer script to build such package again. If we simply clean by date we will loose the possibility to rebuild certain packages that don't require a rebuild of something but we're using a specific version with its specific behaviors during build time.

...

I'm not sure how to tackle this while not loosing reproducibility,

but we certainly should think about this scenario before doing time based cleanups.

...

What about time based + keep the latest n versions of a package? Do you think that would be enough to satisfy the reproducibility needs?

Not really, there are tons of packages not getting rebuild/bumped for a long time that used certain versions of packages. Those will never be able to get rebuild. If we want to safely keep reproducibility of f.e. at least the most current version of all packages, we will need to parse its used dependencies from the internal metadata file and keep all those packages as long as the last package that used it vanished. Right now I don't see any way around this while having both, cleanup and at least the possibility to be able to verify all most current versions of our packages are in fact reproducible.

Cheers Levente

On 2017-07-06 00:36, Levente Polyak wrote:

On July 6, 2017 12:31:07 AM GMT+02:00, Giancarlo Razzolini wrote:

...

Em julho 5, 2017 19:26 Levente Polyak escreveu:

...

Well it will be bit more complicated then that. One we finally get

the reproducible builds patches live we will need the archives for the reproducer script to build such package again. If we simply clean by date we will loose the possibility to rebuild certain packages that don't require a rebuild of something but we're using a specific version with its specific behaviors during build time.

...

I'm not sure how to tackle this while not loosing reproducibility,

but we certainly should think about this scenario before doing time based cleanups.

...

What about time based + keep the latest n versions of a package? Do you think that would be enough to satisfy the reproducibility needs?

Not really, there are tons of packages not getting rebuild/bumped for a long time that used certain versions of packages. Those will never be able to get rebuild. If we want to safely keep reproducibility of f.e. at least the most current version of all packages, we will need to parse its used dependencies from the internal metadata file and keep all those packages as long as the last package that used it vanished. Right now I don't see any way around this while having both, cleanup and at least the possibility to be able to verify all most current versions of our packages are in fact reproducible.

Cheers Levente

Given that at the moment our packages are not reproducible, I think it makes sense to store only the last 6-12 months of packages and exclude the archive from backups. I don't see it as critical loss if our disk explodes and it eats too much space on our backup box for no gain. Bartłomiej