[arch-devops] [arch-dev-public] Uploading old packages on archive.org (Was: Archive cleanup)
baptiste at bitsofnetworks.org
Thu Jan 24 08:27:23 UTC 2019
On 23-01-19, Florian Pritz wrote:
> Hi Baptiste,
> On Thu, Jun 14, 2018 at 10:28:17AM +0200, Florian Pritz via arch-dev-public <arch-dev-public at archlinux.org> wrote:
> > On 10.06.2018 01:35, Baptiste Jonglez wrote:
> > > Archival of all packages between September 2013 and December 2016 is finished:
> > >
> > > https://archive.org/details/archlinuxarchive
> > >
> > > Here is some documentation on this "Historical Archive" hosted on archive.org:
> > >
> > > https://wiki.archlinux.org/index.php/Arch_Linux_Archive#Historical_Archive
> So the archive is still growing and we'll need to clean it up again
> soonish. Where do you have the archive.org uploader and is it in a state
> where we (devops team) can also easily use it? If not, could you help us
> getting it to that point?
I have just pushed the script I wrote last time:
It's a bit hackish and requires some manual work to correctly upload all
packages for a given year, because archive.org rate-limits quite
aggressively when they are overloaded.
> We also have an archive cleanup script here. Maybe the uploader can
> be integrated there? I don't know how complicated it is.
>  https://github.com/archlinux/archivetools/blob/master/archive-cleaner
What about uploading to archive.org as soon as we archive packages on orion?
It would avoid hammering the archive.org server, because we would only
send one package at a time.
In any case, we need a retry mechanism to cope with the case where the
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 833 bytes
Desc: not available
More information about the arch-devops