Hi,
Some admins have told me that they are not interested in traffic regarding other mirrors. I tried to keep such traffic away from the mailing list, but I have the feeling that it would probably be better to simply split the list.
I propose the following two lists:
- arch-mirrors: List that mirror admins should subscribe to. This will only accept mails from Arch Linux staff and will only be used if we need to contact all mirror admins.
- arch-mirrors-discussion: General discussion list. Our users might be subscribed here. Mirror admins can use it to send mail regarding downtime or user facing changes. They will either be whitelisted or can subscribe and disable receiving mail (it's in the mailman user settings).
This should move approximately all of our current traffic on arch-mirrors to arch-mirrors-discussion, so mirror admins should get less boring mails. I am unsure if I should also split general discussion from downtime announcements, but it looks like there hardly was discussion in the past.
Given I do not know who actually reads this list and what they would like to see here, any feedback (from mirror admins or users) is welcome.
I plan to implement these changes next week or once the discussion is over.
Florian
On 02/01/2016 07:15 PM, Florian Pritz wrote:
Hi,
I propose the following two lists:
- arch-mirrors:
No need..
- arch-mirrors-discussion: General discussion list. Our users might be
subscribed here. Mirror admins can use it to send mail regarding downtime or user facing changes. They will either be whitelisted or can subscribe and disable receiving mail (it's in the mailman user settings).
Make one list called arch-mirrors and let only AL staff and whitelisted mirror admins post to it. Others interested in mirroring should be manually enabled to submit each time.
P.S. The company i work for mirrors AL at http://archlinux.mirrors.linux.ro/ if anyone wonders.
On 01/02/16 19:15, Florian Pritz wrote:
Some admins have told me that they are not interested in traffic regarding other mirrors. I tried to keep such traffic away from the mailing list, but I have the feeling that it would probably be better to simply split the list.
I propose the following two lists:
- arch-mirrors: List that mirror admins should subscribe to. This will
only accept mails from Arch Linux staff and will only be used if we need to contact all mirror admins.
- arch-mirrors-discussion: General discussion list. Our users might be
subscribed here. Mirror admins can use it to send mail regarding downtime or user facing changes. They will either be whitelisted or can subscribe and disable receiving mail (it's in the mailman user settings).
This should move approximately all of our current traffic on arch-mirrors to arch-mirrors-discussion, so mirror admins should get less boring mails. I am unsure if I should also split general discussion from downtime announcements, but it looks like there hardly was discussion in the past.
Given I do not know who actually reads this list and what they would like to see here, any feedback (from mirror admins or users) is welcome.
I plan to implement these changes next week or once the discussion is over.
Blame it on my FreeBSD past (and part present), but I actually prefer the short generic name to be used for discussions (i.e. arch-mirrors@), while the announcements-only list to be named accordingly (e.g. arch-mirrors-announce@). This would perhaps also be more in line with the general AL announcements list, arch-announce@.
Actually, trying to see the things from the users' perspective, I'd probably expect the "announcements" list to be exactly for announcements about the mirrors themselves (i.e. downtime notifications, etc.), while the list that is intended for the mirror administrators might be more aptly named arch-mirrors-admins@.
To summarize, if the number of mailing lists is not an issue on its own, I'd imagine the mirror-related lists the following way:
arch-mirrors@ General discussions about the AL mirrors Anyone can write to the list, anyone is free to subscribe if interested.
arch-mirrors-admins@ AL staff notifications to the mirror admins Only AL staff can write, mirror admins are _required_ to subscribe.
arch-mirrors-announce@ Announcements about the various mirrors status Only mirror admins can write, AL users are expected to subscribe.
Admittedly, three separate lists for an already quite a low traffic one is an overkill, but on the other hand there would be a good separation between the different types of messages with their different urgency.
Well, just my two cents, and thanks for raising the issue.
Cheers, Luchesar C.lab, University of Plovdiv mirrors.uni-plovdiv.net
On 01.02.2016 20:22, Luchesar V. ILIEV wrote:
Blame it on my FreeBSD past (and part present), but I actually prefer the short generic name to be used for discussions (i.e. arch-mirrors@), while the announcements-only list to be named accordingly (e.g. arch-mirrors-announce@). This would perhaps also be more in line with the general AL announcements list, arch-announce@.
Good point. I'll do that.
Admittedly, three separate lists for an already quite a low traffic one is an overkill, but on the other hand there would be a good separation between the different types of messages with their different urgency.
We don't care about the list count so I'll probably go with this. We really have 3 pretty distinct types of traffic here so we should just separate them properly.
I did see Daniel's reply about only having one list, but I feel that we should have a list for discussion. We also definitively need a list for Arch Linux staff to reach all mirror admins so 3 lists seems like the best way to go.
arch-mirrors mailing list arch-mirrors@archlinux.org https://lists.archlinux.org/listinfo/arch-mirrors
I'll also get rid of this footer because it is really annoying and none of our other lists have it. It also breaks DKIM signatures and I also think about dropping the subject prefix for this reason. That discussion will happen at a later date though.
Florian
Hi Florian,
thank you for working on this issue.
I like your proposal of splitting up the list. Having one list - no matter the name - where only AL admins post seems to be a good idea for certain announcements. Most projects have this, and I'm perfectly fine with that.
Having another list for mirror admins to reach the AL admins - maybe with some interested parties also reading those messages - also is necessary in most cases.
However, I don't really like mixing up the discussion and the mirror-admin-to-AL-admin cases. In the first case some kind of subscription is quite OK, but in the second case I as a mirror admin don't see the advantage of subscribing and changing settings. Furthermore, there seem to be issues if such mails are sent out to more projects as AL (awaiting moderation because list is only mentioned in BCC?). I'd strongly prefer a pure mail address where anyone (including spammers) send send to, and AL admins read those mails.
Any announcements resulting out of that could be forwarded to the list where all mirrors subscribe, or to a dedicated list where users subscribe. Or, and this is what I prefer, the AL admins re-configure their systems/load balancers/... so that specific outages are mitigated.
This is exactly how most other projects do it, Debian being the one I like the most. As a mirror admin I just wite a mail, and get a reply and/or a temporary DNS change.
Could you explain where you see the advantage of having mirror admins use a mailing list interface, with subscriptions, configuration changes, and possible moderation/filtering/acknowledgement issues (compared to a plain mail address that is forwarded to AL admins)?
Bye, Carsten
On Wed, Feb 03, 2016 at 05:47:47PM +0100, Carsten Otto wrote:
However, I don't really like mixing up the discussion and the mirror-admin-to-AL-admin cases. In the first case some kind of subscription is quite OK, but in the second case I as a mirror admin don't see the advantage of subscribing and changing settings. Furthermore, there seem to be issues if such mails are sent out to more projects as AL (awaiting moderation because list is only mentioned in BCC?). I'd strongly prefer a pure mail address where anyone (including spammers) send send to, and AL admins read those mails.
Just to avoid confusion: the AL setup did not expose any such issue, but my last "big" mail had/has troubles reaching the intended lists/project admins because of this.
On 03.02.2016 17:47, Carsten Otto wrote:
I like your proposal of splitting up the list. Having one list - no matter the name - where only AL admins post seems to be a good idea for certain announcements. Most projects have this, and I'm perfectly fine with that.
Having another list for mirror admins to reach the AL admins - maybe with some interested parties also reading those messages - also is necessary in most cases.
However, I don't really like mixing up the discussion and the mirror-admin-to-AL-admin cases. In the first case some kind of subscription is quite OK, but in the second case I as a mirror admin don't see the advantage of subscribing and changing settings.
I see I should have explained this more clearly. With whitelisting I mean that the list would be moderated and mails to the list can either be delivered because they are accepted by the mailing list admin or because the sender subscribed first.
However, this is somewhat moot given I now want to implement the 3 list scheme proposed by Luchesar:
- arch-mirrors would be a general discussion list - arch-mirrors-admins would be for mail from Arch Linux staff to mirror admins - arch-mirrors-announce would be for mail from you guys to our users (like downtime announcements)
The last 2 would be moderated and only selected senders would be allowed to post. None of these lists really cover the mirror-admin-to-AL-admin case though. (more on that below)
I'd also consider making the discussion list entirely open for posting from anyone, but this would not be in line with all our other lists so it would require some more discussion across all lists first. I dislike the idea of having different policies between lists so if I did this it should be for all lists.
As for the mirror-admin-to-AL-admin case I think it's probably best if I publish the direct admin email I gave you. I'm unsure as to why you think this is mixed with the discussion list. The discussion list is really just intended for discussion where the sender wants outside opinions, not for mail that is only meant for Arch Linux staff. I'm not sure if anyone else is really interested in such discussion though or if all traffic should be directly with the mirro. Feedback welcome!
We used to handle direct requests via our bug tracker, but I can see how that is inconvenient for mirror admins and I certainly see that email is better suited here. I will send an announcement with the direct email (mirrors@archlinux.org) once I split the lists.
Furthermore, there seem to be issues if such mails are sent out to more projects as AL (awaiting moderation because list is only mentioned in BCC?).
Either because of BCC or because the mail has too many recipients. Mailman dislike both of these.
Could you explain where you see the advantage of having mirror admins use a mailing list interface, with subscriptions, configuration changes, and possible moderation/filtering/acknowledgement issues (compared to a plain mail address that is forwarded to AL admins)?
If mirror admins send their mail to a list we don't have to intervene and thus users get their information quicker and we don't need additional manpower. I'm currently taking care of mirrors on my own and the work load is really minimal. This might change if I start forwardings mails.
Also pacman (the most important software that uses our mirror URLs) supports multiple URLs in it's mirror list. It will automatically try the next mirror if it is unable to connect to one or if a file is missing. Assuming users list multiple mirrors, the impact of downtime is thus rather small.
That being said I am not entirely happy with the current setup where users get a list of all direct mirror URLs and we can not control which ones they use. I believe it makes it really difficult for new mirrors to start being used and it is also practically impossible to properly balance traffic according to mirror bandwidth. I haven't thought about how this could be done better though because I don't really have the time to work on this issue at the moment.
Considering the points above, I'm not even sure if we need downtime notifications for users. When I took over as the Arch Linux mirror admin, I just noticed that some mirror admins send downtime notification to this list (most don't) and I didn't think about it. Any feedback by users that read such notification is welcome.
Does this mail address your concerns? I'm unsure if I have understood everything correctly.
Thanks for the feedback!
Florian
On 03/02/16 21:39, Florian Pritz wrote:
I'd also consider making the discussion list entirely open for posting from anyone, but this would not be in line with all our other lists so it would require some more discussion across all lists first. I dislike the idea of having different policies between lists so if I did this it should be for all lists.
I'm afraid I didn't understand that "open for posting from anyone": you mean also open for non-subscribers? Because I think most/all discussion lists are already open for posting for anyone who's subscribed, right?
As for the mirror-admin-to-AL-admin case I think it's probably best if I publish the direct admin email I gave you. I'm unsure as to why you think this is mixed with the discussion list. The discussion list is really just intended for discussion where the sender wants outside opinions, not for mail that is only meant for Arch Linux staff. I'm not sure if anyone else is really interested in such discussion though or if all traffic should be directly with the mirro. Feedback welcome!
I agree that a simple e-mail address to contact the AL staff is more than enough. And if it's something that should be discussed in public, that's what the general discussions list is about, after all.
Could you explain where you see the advantage of having mirror admins use a mailing list interface, with subscriptions, configuration changes, and possible moderation/filtering/acknowledgement issues (compared to a plain mail address that is forwarded to AL admins)?
If mirror admins send their mail to a list we don't have to intervene and thus users get their information quicker and we don't need additional manpower. I'm currently taking care of mirrors on my own and the work load is really minimal. This might change if I start forwardings mails.
As a person who has managed such forwarding aliases, I totally agree that a mailing list is a much more convenient and foolproof solution. And for me this is true even as a subscriber. It isn't more convenient at all when you have to write an e-mail just to update that address of yours in the list... and then you realize that you've sent the wrong address, and there goes another mail, and then the alias admin actually screws it all and then two dozens of people suddenly pop up several months later wondering why haven't they received any mail recently. ;)
That being said I am not entirely happy with the current setup where users get a list of all direct mirror URLs and we can not control which ones they use. I believe it makes it really difficult for new mirrors to start being used and it is also practically impossible to properly balance traffic according to mirror bandwidth. I haven't thought about how this could be done better though because I don't really have the time to work on this issue at the moment.
I don't have much experience with this, but a DNS pool akin to the NTP or PGP keyserver ones is the first thing that comes to mind. And just like NTP, for example, there could be several layers of pools, based on location: a "world" one, per-continent ones, perhaps even per-country.
I'm not really sure however how easy would it be to also prioritize the mirrors in a pool based on their bandwidth. Or should we also account for the other parameters, like latency, which can be pretty important, or https availability, or delay from master...
In the end, I'm not even sure if that would be really necessary. Before setting up our mirror, I've regularly used Reflector [1], which does the job of choosing the best mirror quite well -- and based on what the user considers most important. Not sure if it allowed prioritizing based on latency, for example, but I guess there are also other tools like it around that could be even better. Such tools have the strong advantage over a DNS pool that the _real_ performance of the mirror with respect _also_ to the user's location is taken into account.
Long story short: I wouldn't consider this to be much of an issue.
Considering the points above, I'm not even sure if we need downtime notifications for users. When I took over as the Arch Linux mirror admin, I just noticed that some mirror admins send downtime notification to this list (most don't) and I didn't think about it. Any feedback by users that read such notification is welcome.
From my standpoint as also being a user, I agree that the notifications
are not strictly necessarily, however:
1) I still would be happy to know if my preferred mirror (which happens to be close by, is fast, etc.) is down for just a couple of hours, or if it's something that's going to take awhile, or even if the mirror is actually being retired completely, and
2) In the end, if the mirror admins _do_ want to notify their users in some way, such mailing list is practically the only possible one. And I can imagine rather more important notifications needing to be sent to the users. Just think about a mirror that has been hacked and malicious files have been uploaded to it. Of course, there are checksums and all, but this is still something that is better communicated to the users for various reasons, and the faster the better, rather than notifying AL staff and hoping they can react quickly enough... Not to mention that the AL staff would also then have the problem of identifying the interested users -- if there isn't an announce@ list, that is. [2]
Well, hope this wasn't excessively TL;DR. ;)
Cheers, Luchesar C.lab, University of Plovdiv mirrors.uni-plovdiv.net
--- [1] https://wiki.archlinux.org/index.php/Reflector [2] That was probably not the best example, as such an incident might even need to be announced on the AL website, but I'm sure you can think of other situations that might also require notifying the users of a mirror, and which are more important than a downtime announcement.
On 03.02.2016 23:00, Luchesar V. ILIEV wrote:
On 03/02/16 21:39, Florian Pritz wrote: I'm afraid I didn't understand that "open for posting from anyone": you mean also open for non-subscribers? Because I think most/all discussion lists are already open for posting for anyone who's subscribed, right?
Indeed. I've used the wrong word there.
Also thanks for the rest of your mail. You've got some good points there.
Florian
I propose there are only two mailing lists: the general discussion one and the announcements, which is only AL => subscribers. This one could even be an internal one, like a list of addresses, and not a real mailing list. This means it's mandatory. It should include all mirror admins, whose addresses are put in when the mirror is registered. It may also send to the general discussion and whoever else the boss wants.
Of course all lists should publish the contact address of AL, as has already been said and seems to be a consensus.
Concerning mirror outages and other problems, I think important mirrors should warn the community if the problems last more than about 4h-6h not only because of the impact on users but also on downstream mirrors.
On Wed, Feb 03, 2016 at 08:39:32PM +0100, Florian Pritz wrote:
As for the mirror-admin-to-AL-admin case I think it's probably best if I publish the direct admin email I gave you. I'm unsure as to why you think this is mixed with the discussion list. The discussion list is really just intended for discussion where the sender wants outside opinions, not for mail that is only meant for Arch Linux staff. I'm not sure if anyone else is really interested in such discussion though or if all traffic should be directly with the mirro. Feedback welcome!
Does this mail address your concerns? I'm unsure if I have understood everything correctly.
Mostly, thanks for that. The mirrors@ address really helps. The only missing aspect:
I rarely want to reach out to AL users. But I often (by comparison, which still is very rare) need to get into touch with project admins in case the upstream rsync server fails, some IP needs to be whitelisted, or our contact information changes. This simple use case doesn't really fit into any of the mailing lists you mentioned, but the mirrors@ is a good approach.
Bye, Carsten
Hi,
Thanks for all the input. I have decided that a two list setup is sufficient and now created the arch-mirrors-announce list. The new lists work like this:
- arch-mirrors: Discussion about anything related to mirrors. Membership is not required for anyone any more (this includes mirror admins).
- arch-mirrors-announce: Announcements from mirror admins to users. This is currently open for posting from anyone, including non-subscribers. If this turns out to cause lots of non-announcements mails to be sent to the list I will consider turning on moderation, however I'd like to keep the delay as short as possible for admins that just want to notify users that their mirror broke down.
I am not creating a list for mails from us to all admins because there is really no need for you guys to keep the mailman subscription up to date when we have all mail addresses in our database. All important traffic (for mirror admins) will now be sent via direct mail. If you are currently subscribed to arch-mirrors and not interested in receiving discussion, feel free to unsubscribe.
If you want to subscribe to the new list, here's the direct link: https://lists.archlinux.org/listinfo/arch-mirrors-announce Also note that the posting address for this new list is arch-mirrors-announce@lists.archlinux.org only. There is no legacy forwarding for the archlinux.org domain like there is for arch-mirrors.
PS: I have also removed the mailing list footer.
Florian
Great, thanks, Florian! Just one question though...
On 13/02/16 11:01, Florian Pritz wrote:
I am not creating a list for mails from us to all admins because there is really no need for you guys to keep the mailman subscription up to date when we have all mail addresses in our database. All important traffic (for mirror admins) will now be sent via direct mail.
If we need to update this email address in your database, should we open a ticket or simply send a note to mirrors@archlinux.org?
Cheers, Luchesar C.lab, University of Plovdiv mirrors.uni-plovdiv.net
Am 2016-02-13 17:17, schrieb Luchesar V. ILIEV:
If we need to update this email address in your database, should we open a ticket or simply send a note to mirrors@archlinux.org?
Either is fine.
I'll probably make that the official policy because the bug tracker is really cumbersome for these kinds of requests since you can't just BCC every distro's tracker if you host multiple mirrors. I plan to send out a mail about this at some point.
Florian
arch-mirrors@lists.archlinux.org