[arch-dev-public] Arch mirror tracking woes
Whoopsies- wrong ML. Please reply here. On Tue, Jan 20, 2009 at 3:36 PM, Dusty Phillips <buchuki@gmail.com> wrote:
2009/1/20 Aaron Griffin <aaronmgriffin@gmail.com>:
On Tue, Jan 20, 2009 at 12:04 AM, Geoffroy Carrier <geoffroy.carrier@koon.fr> wrote:
On Tue, Jan 20, 2009 at 05:24, Dan McGee <dpmcgee@gmail.com> wrote:
When you wake up, feel free to hack at this- it should do something similar to what we want.
Neat!
The only thing I don't like... this is yet-another-mirror-list. We currently have: * pacman-mirrorlist * The list of IPs for rsyncing mirrors * A django DB of mirrors got the download page
Any way you can reuse one of these?
There's also a list of mirrors on the wiki. Its a pain to add a new mirror it has to be added in four places.
Let's get this stuff fixed. How does this look for a step in the right direction? class Mirror(models.Model): id = models.AutoField(primary_key=True) domain = models.CharField(max_length=255) rsyncip = models.IPAddressField() country = models.CharField(max_length=255) admin_email = models.EmailField(max_length=255, null=True, blank=True) def __unicode__(self): return self.domain class Meta: db_table = 'mirrors' class MirrorProtocol(models.Model): id = models.AutoField(primary_key=True) protocol = models.CharField(max_length=10, unique=True) class MirrorUrl(models.Model): id = models.AutoField(primary_key=True) url = models.CharField(max_length=255) protocol = models.ForeignKey(MirrorProtocol, related_name="urls") mirror = models.ForeignKey(Mirror, related_name="urls") 1. rsync IP address will be stored right with the mirror. We can easily run a DB query and script it to generate our list as needed for our rsync configuration. 2. Mirror <-> URL is no longer a one-to-one mapping. A mirror could have 1-4 URLs for any of the protocols it supports (FTP, HTTP, RSYNC, and maybe one day HTTPS). 3. More detailed storage could turn into a Django view that says "give me a HTTP and FTP mirrorlist for Canada" and boom! mirrorlist is made. 4. pacman-mirrorlist would be an automatically generated package if at all possible. The caveat would be the current continent/country-based sorting- any idea if we need to do something with the freeform Country field to make this easier? 5. (CCed on this email) Gerhard, does this model look flexible enough to perhaps incorporate mirror check status information in the future? I would not mind seeing that feature on the main Arch site. 6. It might also be useful to add a "has_isos" boolean field to those MirrorURLs (or Mirrors? not sure) that have ISOs mirrored. This would ensure the download page makes sense. Diff is below from the current model, although this may be quite a bit less helpful than the above model. diff --git a/main/models.py b/main/models.py index b5a2a42..0fe17d5 100644 --- a/main/models.py +++ b/main/models.py @@ -52,15 +52,24 @@ class PackageManager(models.Manager): class Mirror(models.Model): id = models.AutoField(primary_key=True) domain = models.CharField(max_length=255) + rsyncip = models.IPAddressField() country = models.CharField(max_length=255) - url = models.CharField(max_length=255) - protocol_list = models.CharField(max_length=255, null=True, blank=True) - admin_email = models.CharField(max_length=255, null=True, blank=True) + admin_email = models.EmailField(max_length=255, null=True, blank=True) def __unicode__(self): return self.domain class Meta: db_table = 'mirrors' +class MirrorProtocol(models.Model): + id = models.AutoField(primary_key=True) + protocol = models.CharField(max_length=10, unique=True) + +class MirrorUrl(models.Model): + id = models.AutoField(primary_key=True) + url = models.CharField(max_length=255) + protocol = models.ForeignKey(MirrorProtocol, related_name="urls") + mirror = models.ForeignKey(Mirror, related_name="urls") + class Press(models.Model): id = models.AutoField(primary_key=True) name = models.CharField(max_length=255)
Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
How does this look for a step in the right direction?
Looks good to me. I use a similar approach for the server stats page. Do you think it would be possible to store those data in the mirrorlist package somehow? -- Pierre Schmitz Clemens-August-Straße 76 53115 Bonn Telefon 0228 9716608 Mobil 0160 95269831 Jabber pierre@jabber.archlinux.de WWW http://www.archlinux.de
On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
How does this look for a step in the right direction?
Looks good to me. I use a similar approach for the server stats page. Do you think it would be possible to store those data in the mirrorlist package somehow?
Can you point me to the page you are talking about? And what do you mean by storing it in the mirrorlist package? The intent here is to make the Django DB the canonical source of information that will be disseminated out to each of the places that need it- the downloads page, a mirrors page, and a way to generate the pacman mirrorlist. -Dan
On Wed, Jan 21, 2009 at 7:37 PM, Dan McGee <dpmcgee@gmail.com> wrote:
On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
How does this look for a step in the right direction?
Looks good to me. I use a similar approach for the server stats page. Do you think it would be possible to store those data in the mirrorlist package somehow?
Can you point me to the page you are talking about?
And what do you mean by storing it in the mirrorlist package? The intent here is to make the Django DB the canonical source of information that will be disseminated out to each of the places that need it- the downloads page, a mirrors page, and a way to generate the pacman mirrorlist.
Some mostly-complete work on this stuff is on my branches: http://code.toofishes.net/gitweb.cgi?p=archweb_dev.git;a=shortlog;h=refs/hea... http://code.toofishes.net/gitweb.cgi?p=archweb_pub.git;a=shortlog;h=refs/hea... -Dan
2009/1/21 Dan McGee <dpmcgee@gmail.com>:
On Wed, Jan 21, 2009 at 7:37 PM, Dan McGee <dpmcgee@gmail.com> wrote:
On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
How does this look for a step in the right direction?
Looks good to me. I use a similar approach for the server stats page. Do you think it would be possible to store those data in the mirrorlist package somehow?
Can you point me to the page you are talking about?
And what do you mean by storing it in the mirrorlist package? The intent here is to make the Django DB the canonical source of information that will be disseminated out to each of the places that need it- the downloads page, a mirrors page, and a way to generate the pacman mirrorlist.
Some mostly-complete work on this stuff is on my branches: http://code.toofishes.net/gitweb.cgi?p=archweb_dev.git;a=shortlog;h=refs/hea... http://code.toofishes.net/gitweb.cgi?p=archweb_pub.git;a=shortlog;h=refs/hea...
Hey all, Sorry for dropping the ball on this one. I've merged Dan's changes and am in the process of manually adding mirrors. I have entered all the details into the admin except the rsync IPs. I would appreciate it if someone would add those manually at some point. Then somebody can write scripts to automatically generate the rsync whitelist and pacman mirrorlist from this data. (Or did you do that already, Dan?) In addition, I had to verify each of the mirrors to check if they have ISOs. In doing that I discovered some of the mirrors in the current database are broken. Specifically these ones: http://mirror.vmmatrix.net/ArchLinux/ (China) mirrors.jakimowicz.com (France) ftp://ftp.parrswood.net/Mirrors/ftp.archlinux.org/ (UK) ftp://saule.mintis.lt/pub/linux/ (Lithuania) They should be removed from the mirrorlist and rsync whitelist if they exist there, as I've already removed them from the DB. I suspect there may be mirrors on the wiki page and/or in the pacman mirror list that are not in the database. It would be nice to get them all synced up. Finally, there are a bunch of unofficial mirrors listed here: http://wiki.archlinux.org/index.php/Mirrors#Unofficial_mirrors Is there a reason we shouldn't make some/all of these official?? Some of them may have been done already.
On Fri, Feb 6, 2009 at 5:47 PM, Dusty Phillips <buchuki@gmail.com> wrote:
2009/1/21 Dan McGee <dpmcgee@gmail.com>:
On Wed, Jan 21, 2009 at 7:37 PM, Dan McGee <dpmcgee@gmail.com> wrote:
On Wed, Jan 21, 2009 at 12:57 AM, Pierre Schmitz <pierre@archlinux.de> wrote:
Am Mittwoch 21 Januar 2009 03:32:54 schrieb Dan McGee:
How does this look for a step in the right direction?
Looks good to me. I use a similar approach for the server stats page. Do you think it would be possible to store those data in the mirrorlist package somehow?
Can you point me to the page you are talking about?
And what do you mean by storing it in the mirrorlist package? The intent here is to make the Django DB the canonical source of information that will be disseminated out to each of the places that need it- the downloads page, a mirrors page, and a way to generate the pacman mirrorlist.
Some mostly-complete work on this stuff is on my branches: http://code.toofishes.net/gitweb.cgi?p=archweb_dev.git;a=shortlog;h=refs/hea... http://code.toofishes.net/gitweb.cgi?p=archweb_pub.git;a=shortlog;h=refs/hea...
Hey all,
Sorry for dropping the ball on this one. I've merged Dan's changes and am in the process of manually adding mirrors. I have entered all the details into the admin except the rsync IPs. I would appreciate it if someone would add those manually at some point. Then somebody can write scripts to automatically generate the rsync whitelist and pacman mirrorlist from this data. (Or did you do that already, Dan?)
I can try to do the rsync IPs this weekend, as well as making scripts/etc. that spit out what we need. For the mirrorlist, we can just make this a view on the public site- for the rsync IP list, we might be able to make it a view on the dev site.
In addition, I had to verify each of the mirrors to check if they have ISOs. In doing that I discovered some of the mirrors in the current database are broken. Specifically these ones:
http://mirror.vmmatrix.net/ArchLinux/ (China) mirrors.jakimowicz.com (France) ftp://ftp.parrswood.net/Mirrors/ftp.archlinux.org/ (UK) ftp://saule.mintis.lt/pub/linux/ (Lithuania)
They should be removed from the mirrorlist and rsync whitelist if they exist there, as I've already removed them from the DB.
Noted, and I'll try to remove them from the mirrorlist this weekend.
I suspect there may be mirrors on the wiki page and/or in the pacman mirror list that are not in the database. It would be nice to get them all synced up.
Agreed. That was the point of this whole exercise.
Finally, there are a bunch of unofficial mirrors listed here: http://wiki.archlinux.org/index.php/Mirrors#Unofficial_mirrors Is there a reason we shouldn't make some/all of these official?? Some of them may have been done already.
I think our only definition of "official" is "do they sync from gerolde?". Not sure if that answers the question, but yeah. -Dan
Am Donnerstag 22 Januar 2009 02:37:10 schrieb Dan McGee:
Can you point me to the page you are talking about?
http://www.archlinux.de/?page=MirrorStatus
And what do you mean by storing it in the mirrorlist package?
My intention was just something to make those information (support for http, ftp, rsync; country etc.) available for everyone. So my first idea was to store those additional information as comments in the mirrorlist. -- Pierre Schmitz Clemens-August-Straße 76 53115 Bonn Telefon 0228 9716608 Mobil 0160 95269831 Jabber pierre@jabber.archlinux.de WWW http://www.archlinux.de
participants (3)
-
Dan McGee
-
Dusty Phillips
-
Pierre Schmitz