On 24.03.2014 23:50, Dave Reisner wrote:

...

Current cost in byte is: # du -hcs 2013 2014 111G 2013 55G 2014 165G total

Does 2013 cover the whole year?

2013 starts on August 31th.

On Tue, Mar 25, 2014 at 12:55:51AM +0100, Sébastien Luttringer wrote:

On 24/03/2014 23:50, Dave Reisner wrote:

...

On Mon, Mar 24, 2014 at 11:35:36PM +0100, Sébastien Luttringer wrote: Does 2013 cover the whole year? Four full months. More details in listing this page[1]

...

You're suggesting that 2014 will occupy over 200GB. We'd need new infrastructure for this, and it comes with a monetary cost we'd need to accomodate (or are you proposing that you'll be paying for this forever?). I don't suggest to pay forever, one of the benefits of moving this, is to stop rely on one guy. We already suffer of a previous shutdown on this service.

Right, but throwing this responsibility on the arms of Arch developers isn't really a solution, either, without quiescence. You seem to be of the mindset of that adding this to the archlinux.org domain will magically lighten your own sysadmin load. While I think the idea is neat, I don't think is has a place underneath archlinux.org -- we'd be taking a step towards supporting partial upgrades (which we refuse to support everywhere else). Before you mention the AUR, remember that the AUR hosts source packages; not binary packages.

...

Have you looked into how much this would be per month with a potential provider? How much bandwidth is associated with this? The current server cost is 16.99€/month excl. VAT with 2x1TB HDD with 1Gbit/s connectivity, 200Mbit/s internet bandwidth and unlimited traffic.

A cheap solution based on a dedibox classic[2] with 2x1TB, 1 Gbit/s con, 150Mbit/s internet and unlimited traffic for 29.99€/month excl. VAT. should be sufficient.

A more future proof solution may based on a dedibox pro[3] with 2x3TB drives, with 400Mbit/s internet bandwidth and unlimited traffic for 69€/month excl. VAT.

And the SLA? This all seems reasonable at a glance, but I can't speak to how this would impact our budgeting.

Of course, any Hetzner alternative may fit, but traffic limitation may cause extra cost that we don't have with online.net offers.

...

How long would packages be retained? What about resource planning for future growth? My suggestion would to keep them as far as we can, until costs is not a showstopper. With a 1TB server and 300GB by year, we can keep ~3 years.

...

...

- add AUR

This doesn't make sense. We already have a git repo which does a far better job of compressing the deltas between versions of text files. Keeping the git backend or not, it makes sense to me to move this from the build server (pkgbuild.com) to the archive server (something like archive.archlinux.org).

Sure, I agree with this. Our serving structure is pretty strange and it doesn't make sense to host the repo on the build server for any other reason than: "this is where we had unallocated resources."

Git is an awesome SCM, but it was not think to backup stuff, especially

You lost me here. git is made up entirely of deltas and serves as a distributed backup. It's *really* *good* at this, even on wide trees.

with a big directory tree. My feeling trying to find a previous version of a PKGBUILD with aur-mirror.git is: - a very long time to fetch the repository - a space based dig into changelog to find the right commit for my date - a long checkout to get the tree at this commit

Seems like this is just three different ways to spin the same point. We can restructure to repo to make it more clone-friendly if that's all that's needed.

Having something with faster access and in a similar hierarchy may have use cases.

How often do you use the git repo?

...

...

- backup them (by mirroring or others)

There's going to be a large cost here and I doubt it's any potential benefit. Do you get complaints/requests from your users to mirror your implementation? No extra cost is expected for mirroring, which is a kind of distributed backup. I see at least 2 potentials benefits for mirroring:

Are you really suggesting that mirroring 300GB-3TB of data has no cost? Who's hosting the mirrors? Have they agreed to shoulder the additional storage and bandwidth requirements?

- Better latency around the world. - Get our data back in case of disaster recovery.

Of course, I suggest to keep these mirrors separate from our packages mirrors. Purpose and HW requirement are not the same.

The only complain I get is because this service is not official and rely on my health. I had 3 requests: - One guy for mirroring because of latency. - One from ArchARM asking to add ARM to their project :) - One from a guy asking to add AUR support.

Do you have any metrics to show how many 30 day users you have? Downloads per day? Week? Bandwidth consumed? I'm really interested in knowing how much impact this actually has.... d

[2014-03-25 00:55:51 +0100] Sébastien Luttringer:

On 24/03/2014 23:50, Dave Reisner wrote:

...

On Mon, Mar 24, 2014 at 11:35:36PM +0100, Sébastien Luttringer wrote: Does 2013 cover the whole year? Four full months. More details in listing this page[1]

Is there any use keeping packages older than, say, a year? That'd take 200 GB of disk space. Would you have an estimate of the average monthly bandwidth usage? Personally, although I think the AUR is a valuable service and that you shouldn't be assuming its costs and maintenance alone, I'm not sure if making it official would be a good thing (TM) in terms of encouraging users to downgrade packages rather than reporting bugs and upgrading cleanly when things break. -- Gaetan

[2014-03-25 02:46:23 +0100] Sébastien Luttringer:

On 25/03/2014 02:02, Gaetan Bisson wrote:

...

Personally, although I think the AUR is a valuable service and that you shouldn't be assuming its costs and maintenance alone,... I guess you are speaking of ARM (Arch Rollback Machine) not AUR here.

Indeed, that was a typo. Sorry for the confusion. -- Gaetan

On Fri, Mar 28, 2014 at 1:14 AM, Andrea Scarpino wrote:

On Mon 24, March 15:02:53 Gaetan Bisson wrote:

...

Personally, although I think the AUR is a valuable service and that you shouldn't be assuming its costs and maintenance alone, I'm not sure if making it official would be a good thing (TM) in terms of encouraging users to downgrade packages rather than reporting bugs and upgrading cleanly when things break.

I want to quote Gaetan and Dave here. I fear that making ARM "official" or "semi-official" we unconsciously boost single packages downgrade.

I absolutely agree that we must make it abundantly clear that partial upgrades/downgrades are not supported. Never have been, and never will be. Apart from keeping this in mind when making any public statements on this subject (to avoid giving false expectations) I don't think it is a show-stopper though. If people do stupid things they get to keep the pieces when stuff breaks. We shouldn't let the fear of stupid people being stupid hinder our development. Another important point is that downgrading your whole system means you don't get any security upgrades or bug fixes, so in that sense it is not supported. In precisely the same way not upgrading your system is not supported. That said, I do think the ARM (or whatever its new name will be) is a really crucial service given our development model. We are on the bleeding edge, and our QA is more or less nonexistent. Surprisingly the quality of our packages is still very high (at least in my experience), but we do of course occasionally let the some buggy package slip through to core/extra/community. When this happens, the reasonable thing for a user to do is to file a bug and downgrade the package until the bug has been fixed. However, we don't support partial downgrades (at all, it would be a really, really stupid thing to do), so the only reasonable thing to do is to downgrade the whole system to a previously known good state (which is usually the state of the archives just before the offending package was updated). Doing this now is actually a real pain, as the user may not have all the necessary packages in their local cache, and even if they do, they can't easily know which packages to downgrade to what version. Another concern I have seen raised in the past is that if we make it simple to downgrade to temporarily avoid buggy packages, we will get fewer bug reports. I don't think that is a valid argument, rather we'd discourage users from upgrading frequently, or from using Arch at all. The argument is similar to a filesystem developer refusing to ship a fsck tool as it would allow people to fix their broken filesystems without filing bugs first... Having the ARM service (or something like that) with all the necessary caveats and warnings, makes it simple to force-downgrade to a given point in time, allowing users a reasonable fallback-mode in case some crucial package breaks on upgrade (and I mean crucial in the sense that it is something the user urgently needs, not in the sense that it is in [core], it could easily be a text editor, music player or web browser). So, in conclusion, I fully support adding such a service under the archlinux.org domain, and hosted on Arch infrastructure. This is under the assumption that care is taken when communicating precisely what this does and does not entail and someone is volunteering to do the actual work (and that whoever is actually affected by it on the infrastructure side do not object). If hosting it on our infrastructure is a problem, I'd suggest we pick up the bill for the current hosting costs instead, but still assign it an arch domainname and consider it officially part of our project. I have no comments regarding the details when it comes to pricing or other resources, except to say that we don't necessarily need a very long history, if space poses a problem, and that the kind of expenses indicated seems well within what we can afford (though we should of course always be respectful of our donors and try to keep expenses down when possible). Cheers, Tom

On 25/03/2014 02:15, keenerd wrote:

On 3/24/14, Sébastien Luttringer wrote: The Rollback Machine has a convenient CLI wrapper to access packages. Have you tried using the Rollback Machine with just a browser? It is nearly impossible. The CLI wrapper makes it much nicer. I always use the web interface to get packages from ARM. No wrapper. The unique CLI tool I used was pacman to rollback a whole system.

The ARM server latency is pretty good where pkgbuild.com is a nightmare. $ time wget -qO/dev/null https://seblu.net/a/arm/2014 wget -qO/dev/null https://seblu.net/a/arm/2014 0,02s user 0,00s system 33% cpu 0,063 tot $ time wget -qO/dev/null http://pkgbuild.com/git/aur-mirror.git/tree/?id=0b2c85b0b252c9bab30df2c411d5... wget -qO/dev/null 0,17s user 0,74s system 18% cpu 4,970 total

The aur.git backup has no such convenient wrapper. Yes, it is painful to use. About as painful as trying to navigate the Rollback Machine with a web browser.

I'm not speaking of beauty, but speed and latency of answers.

Write a front end instead of this vague, nebulous proposal. I did not wish hurt you in any manner. Your aur-mirror.git is a good initiative. Please don't take it personally.

Just to clarify the Arch Linux ARM thing. No one asked to incorporate ALARM into the Rollback Machine. Someone did ask for assistance with setting up their own Rollback Machine. And this guy was not "from ArchARM", he was an ALARM mirror host. Sorry if that was not clear in my first mail.

Cheers, -- Sébastien "Seblu" Luttringer https://seblu.net | Twitter: @seblu42 GPG: 0x2072D77A

On 25/03/2014 08:04, Thomas Bächler wrote:

Am 24.03.2014 23:35, schrieb Sébastien Luttringer:

...

Hello,

As you may know I'm running ARM[1] since the last maintainer resign in August 2013. I would like to propose its addition into our official services.

As a side not, may I point out that the name 'ARM' is poorly chosen for this service?

I kept it for legacy reason back in 2013. I share your concern and that's why I suggest to rename it to archive / museum / rollback. Do you feel confident with this renaming proposal? -- Sébastien "Seblu" Luttringer https://seblu.net | Twitter: @seblu42 GPG: 0x2072D77A