[arch-devops] [arch-dev-public] Uploading old packages on archive.org (Was: Archive cleanup)

[Baptiste Jonglez]

Hi,

On 23-01-19, Florian Pritz wrote:
> Hi Baptiste,
> 
> On Thu, Jun 14, 2018 at 10:28:17AM +0200, Florian Pritz via arch-dev-public <arch-dev-public at archlinux.org> wrote:
> > On 10.06.2018 01:35, Baptiste Jonglez wrote:
> > > Archival of all packages between September 2013 and December 2016 is finished:
> > > 
> > >   https://archive.org/details/archlinuxarchive
> > > 
> > > Here is some documentation on this "Historical Archive" hosted on archive.org:
> > > 
> > >   https://wiki.archlinux.org/index.php/Arch_Linux_Archive#Historical_Archive
> 
> So the archive is still growing and we'll need to clean it up again
> soonish. Where do you have the archive.org uploader and is it in a state
> where we (devops team) can also easily use it? If not, could you help us
> getting it to that point?

I have just pushed the script I wrote last time:

  https://github.com/zorun/arch-historical-archive

It's a bit hackish and requires some manual work to correctly upload all
packages for a given year, because archive.org rate-limits quite
aggressively when they are overloaded.

> We also have an archive cleanup script here[1]. Maybe the uploader can
> be integrated there? I don't know how complicated it is.
> 
> [1] https://github.com/archlinux/archivetools/blob/master/archive-cleaner

What about uploading to archive.org as soon as we archive packages on orion?

  https://github.com/archlinux/archivetools/blob/master/archive.sh

It would avoid hammering the archive.org server, because we would only
send one package at a time.

In any case, we need a retry mechanism to cope with the case where the
upload fails.

Baptiste
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.archlinux.org/pipermail/arch-devops/attachments/20190124/a7303901/attachment.asc>