[arch-mirrors] Possibility of adding debug repositories

[Tyler Dence]

I (arlm.tyzoid.com) would prefer an opt-in solution. Such a method would
allow us to evaluate in advance the storage requirements to make sure that
we don't run out of storage space on the virtual disk.

How it's implemented with rsync is an open question. Would it exist as a
new endpoint? If so, wouldn't it necessitate extra config on the
apache/nginx side?


On Sat, Jun 6, 2020, 6:37 PM Tails Hon1nbo <hon1nbo+mirror at hackingand.coffee>
wrote:

> Hacking & Coffee would be interested, but make it a separate "opt-in"
> dataset especially if it's unpredictable how much space it will take.
> If suddenly enabled, and set to opt-out, I imagine some mirror clusters
> would have a sudden, unexpected storage problem
>
> On Sat, 6 Jun 2020, 17:12 Erwin Bronkhorst - Studenten Net Twente, <
> erwin at snt.utwente.nl> wrote:
>
>> Hi Eli,
>>
>> > So, question to mirror admins: if Arch was to add debug repositories,
>> > would you be okay syncing them? And should it be opt-in or opt out?
>>
>> We (ftp.snt.utwente.nl) would be okay syncing them. I really don't mind
>> whether it will be an opt-in or opt-out solution. I'm okay with an increase
>> in size of the current mirror, but I also don't mind setting up an extra
>> rsync-cronjob if that is what it takes to opt-in.
>>
>> Kind regards,
>>
>> Erwin Bronkhorst
>> SNT FTPCom
>>
>> > -----Oorspronkelijk bericht-----
>> > Van: arch-mirrors <arch-mirrors-bounces at archlinux.org> Namens Eli
>> > Schwartz
>> > Verzonden: vrijdag 5 juni 2020 21:58
>> > Aan: arch-mirrors at archlinux.org
>> > Onderwerp: [arch-mirrors] Possibility of adding debug repositories
>> >
>> > If Arch Linux were to add repositories containing split debug packages
>> > for all our x86_64 packages, this would obviously add a fair amount of
>> > space to the mirror requirements. It could possibly double or triple
>> > the
>> > size taken by non-data packages. I don't have real-world numbers for
>> > how
>> > much space it would take up, but I do have some comparisons.
>> >
>> > For my custom repo as a sample, which I do upload debug packages for, I
>> > am using 2.9Gi of space, and 1.7Gi comes from debug packages.
>> >
>> > On the other side of things, our biggest 2 official packages are cuda,
>> > 4137.96 MiB of proprietary blobs that aren't currently stripped and
>> > therefore even if we did split it into a debug package it wouldn't
>> > increase space usage at all, and kicad-library-3d, 5171.97 MiB of pure
>> > data in /usr/share, which is not eligible for debug packages anyway.
>> >
>> > A naive list of packages which would probably generate debug packages:
>> >
>> > $ expac -H M '%m %n %a' | grep -v 'any$' | sort --human-numeric-sort
>> > [...]
>> > 1109.50 MiB emscripten x86_64
>> > 1136.27 MiB python-tensorflow x86_64
>> > 1145.23 MiB python-tensorflow-opt x86_64
>> > 1286.69 MiB python-pytorch-cuda x86_64
>> > 1289.44 MiB python-pytorch-opt-cuda x86_64
>> > 1518.42 MiB ghc-static x86_64
>> > 2589.23 MiB python-tensorflow-cuda x86_64
>> > 2597.38 MiB python-tensorflow-opt-cuda x86_64
>> > 3757.68 MiB tensorflow-cuda x86_64
>> > 3765.75 MiB tensorflow-opt-cuda x86_64
>> > 4137.96 MiB cuda x86_64
>> >
>> > (Basically all of the really big stuff is tensorflow/cuda/machine
>> > learning bits. We could selectively disable debug packages for two
>> > PKGBUILDs and avoid all the worst offenders, if we needed to. heh.)
>> >
>> > Packages which definitely would not (there's some big, high-profile
>> > packages here, and my custom repo doesn't reflect this sort of spread
>> > at
>> > all):
>> >
>> > $ expac -H M '%m %n %a' | grep -v 'any$' | sort --human-numeric-sort
>> > [...]
>> > 1202.02 MiB texlive-fontsextra any
>> > 1307.98 MiB texlive-fontsextra any
>> > 2006.67 MiB 0ad-data any
>> > 3112.39 MiB nltk-data any
>> > 5171.97 MiB kicad-library-3d any
>> >
>> > ...
>> >
>> > Anyway, providing these symbols would be generally desirable for users,
>> > and ideally it would work opt-out to make it easier for users to get
>> > access to them. It's something we've generally wanted to do, see for
>> > example https://bugs.archlinux.org/task/38755
>> > And it's possible we may actually, at long last, get around to
>> > implementing this.
>> >
>> > So, question to mirror admins: if Arch was to add debug repositories,
>> > would you be okay syncing them? And should it be opt-in or opt out?
>> >
>> > The answers to these questions will influence the direction I will take
>> > in trying to devise a satisfactory resolution to this outstanding
>> > infrastructure request. So I would love to get some input from the
>> > people who would be affected by such a change.
>> >
>> > --
>> > Eli Schwartz
>> > Bug Wrangler and Trusted User
>> >
>> >
>>
>
>  This message contains confidential information and is intended only for
> the individual named. If you are not the named addressee you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately by e-mail if you have received this e-mail by mistake and
> delete this e-mail from your system. If you are not the intended recipient
> you are notified that disclosing, copying, distributing or taking any
> action in reliance on the contents of this information is strictly
> prohibited.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.archlinux.org/pipermail/arch-mirrors/attachments/20200606/b6cb67a7/attachment-0001.htm>