[arch-dev-public] Uploading old packages on archive.org (Was: Archive cleanup)

[Florian Pritz]

On 01.06.2018 19:24, Baptiste Jonglez wrote:
> - does the devops team have a place to store passwords?  I would like to
>   create an "Arch Linux" account, so that I'm not the only one to have
>   access. 

You can send me the password and I'll put it in our ansible repo in the
vault.

> I also need an email address for the account, maybe something
>   like internetarchive at archlinux.org or just the devops mailing list address?

We don't want to spam the ML with password rest mails or whatever else
we get so let's use a private address. I've set up
internetarchive at archlinux.org to forward to root at archlinux.org. Please
use that and if you want, I can also add you to the alias.

> - is that OK to upload ~2 TB from orion?  Is the server on an limited data
>   plan?

That's fine.

> The upload process itself is quite slow latency-wise, it takes 5-10
> seconds to upload a file whatever its size.  For packages from 2013 to
> 2015 there's 250k files to upload, I estimate it will take a few days if I
> run 32 upload threads in parallel.

Fine by me. I'll wait for that to complete before deleting anything.

> 
> By the way, we could even keep the year/month/day symlink hierarchy on
> orion for old packages, and redirect downloads to archive.org.  There is
> just a small issue with packages that have "+", "@" or "." in their name,
> because that's not allowed as identifiers in archive.org (see the second
> and third examples above, where my script replaced the "@" with "_")

I'd much prefer getting rid of the inodes because having this many
inodes around makes file system operations slow. It's also the reason
why the archive is not part of the backup.

Thanks for working on this!

Florian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://lists.archlinux.org/pipermail/arch-dev-public/attachments/20180601/1b061c25/attachment.asc>