On Tue, Aug 4, 2009 at 12:28 PM, Aaron Griffin wrote:

2009/8/4 Angel Velásquez :

...

Hi, I've syncing my local mirror today and seems to be duplicated packages of 'any' architecture on i686 directory

for example kde-l10n-de-4.3.0-1-any.pkg.tar.gz is respectively on any dir and i686 dir

See the log of pkgsync on [1]:

db-clean should fix that iirc, can others mirrors admin checks if this still happening? I reviewed mirror.archlinux.com.ve on 'any' [2] architecture and 'i686' [3], 'x86_64' [4] and as I said 'any' packages still existing on other repos.

The package should be in the any dir only and should be symlinked to both i686 and x86_64 dirs. Are you not syncing symlinks for some reason?

Hmm, I really don't know why are downloading the packages again and not symlinked against it. I am using the script of this wiki page [1] on both mirrors. I am seeing rsync manual atm to improve a solution, I filled a bug report anyway [2] Thanks [1] http://wiki.archlinux.org/index.php/Local_Mirror [2] http://bugs.archlinux.org/task/15788 -- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909

2009/8/4 Angel Velásquez :

2009/8/4 Angel Velásquez :

...

On Tue, Aug 4, 2009 at 12:28 PM, Aaron Griffin wrote:

...

2009/8/4 Angel Velásquez :

...

Hi, I've syncing my local mirror today and seems to be duplicated packages of 'any' architecture on i686 directory

for example kde-l10n-de-4.3.0-1-any.pkg.tar.gz is respectively on any dir and i686 dir

See the log of pkgsync on [1]:

db-clean should fix that iirc, can others mirrors admin checks if this still happening? I reviewed mirror.archlinux.com.ve on 'any' [2] architecture and 'i686' [3], 'x86_64' [4] and as I said 'any' packages still existing on other repos.

The package should be in the any dir only and should be symlinked to both i686 and x86_64 dirs. Are you not syncing symlinks for some reason?

Hmm, I really don't know why are downloading the packages again and not symlinked against it.

I am using the script of this wiki page [1] on both mirrors.

I am seeing rsync manual atm to improve a solution, I filled a bug report anyway [2]

Thanks

[1] http://wiki.archlinux.org/index.php/Local_Mirror

[2] http://bugs.archlinux.org/task/15788

-- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909

Seeing the manpage of rsync I should remove --safe-links and everything should work fine.

If this is true, the script on the wiki page and mirrors should be updated.

Cheers

-- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909

Removing safe-links option still downloading the files again.. I'm syncing against mirror.rit.edu anyway, any ideas? -- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909

2009/8/4 Angel Velásquez :

Ok I found a solution! (I swear this is my last quiote of myself in this thread hehe).

I did these steps:

1) entered on i686,x86_64 dirs under extra and and delete *-any.pkg.tar.gz 2) edited the mirrorsync script and deleted safe-links option 3) run the script again

And everything worked again.

In my local mirror I wasn't syncing testing, iso and other dirs, so I went to the second case of the script, .. the first case (which syncs everything) works fine, is respecting the symlinks, the second case should now be update on wiki page, can I do this? or I have to fill a bug report, can somebody confirm this?.

Thanks anyway and sorry for this issue

Full mirror sync in that script uses these rsync options: -rptLv --delete-after --delay-updates Per-repository sync uses these rsync options: -rptlv --safe-links --delete-after --delay-updates

From rsync --help: -l, --links copy symlinks as symlinks -L, --copy-links transform symlink into referent file/dir --safe-links ignore symlinks that point outside the source tree

So if I understood correctly we should change -L to -l and remove --safe-links which is what I'll do now on the wiki. -- Roman Kyrylych (Роман Кирилич)

2009/8/4 Angel Velásquez :

On Tue, Aug 4, 2009 at 2:03 PM, Roman Kyrylych wrote:

...

2009/8/4 Angel Velásquez :

...

Ok I found a solution! (I swear this is my last quiote of myself in this thread hehe).

I did these steps:

1) entered on i686,x86_64 dirs under extra and and delete *-any.pkg.tar.gz 2) edited the mirrorsync script and deleted safe-links option 3) run the script again

And everything worked again.

In my local mirror I wasn't syncing testing, iso and other dirs, so I went to the second case of the script, .. the first case (which syncs everything) works fine, is respecting the symlinks, the second case should now be update on wiki page, can I do this? or I have to fill a bug report, can somebody confirm this?.

Thanks anyway and sorry for this issue

Full mirror sync in that script uses these rsync options: -rptLv --delete-after --delay-updates

Per-repository sync uses these rsync options: -rptlv --safe-links --delete-after --delay-updates

...

From rsync --help:  -l, --links                 copy symlinks as symlinks  -L, --copy-links            transform symlink into referent file/dir     --safe-links            ignore symlinks that point outside the source tree

So if I understood correctly we should change -L to -l and remove --safe-links which is what I'll do now on the wiki.

Exactly I miss to mention about the -L <-> -l option, now I am testing if it works correctly, seems to be for now ...

-- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909

Searching on the man of rsync option -a do the same of rptl so .. I propose to change -rptl in favour of -a. now mirror.archlinux.com.ve is working fine [1]. I am getting the e-mail of the rest of the mirrors admins to let them know about the issue with the mirrorsync script Cheers [1] http://mirror.archlinux.com.ve -- Angel Velásquez angvp @ irc.freenode.net Linux Counter: #359909

On Tue, Aug 4, 2009 at 23:13, Aaron Griffin wrote:

2009/8/4 Angel Velásquez :

...

I am getting the e-mail of the rest of the mirrors admins to let them know about the issue with the mirrorsync script

Do you have a list of mirror admins? Our backend DB is missing contact info for a few of the mirrors.

This is what I was thinking to do a year ago before becoming inactive. Looks like a perfect time to actually start doing this. And this will help us to optimize our mirror network (by having more info recorded than just URLs and IP addresses). -- Roman Kyrylych (Роман Кирилич)

On Tue, Aug 4, 2009 at 23:22, Pierre Schmitz wrote:

Am Dienstag 04 August 2009 22:13:37 schrieb Aaron Griffin:

...

2009/8/4 Angel Velásquez :

...

I am getting the e-mail of the rest of the mirrors admins to let them know about the issue with the mirrorsync script

Do you have a list of mirror admins? Our backend DB is missing contact info for a few of the mirrors.

Afaik Gerhrad has such a list as he is allways in contact with some mirror admins.

On Tue, Aug 4, 2009 at 23:22, Pierre Schmitz wrote:

Am Dienstag 04 August 2009 22:13:37 schrieb Aaron Griffin:

...

2009/8/4 Angel Velásquez :

...

I am getting the e-mail of the rest of the mirrors admins to let them know about the issue with the mirrorsync script

Do you have a list of mirror admins? Our backend DB is missing contact info for a few of the mirrors.

Afaik Gerhrad has such a list as he is allways in contact with some mirror admins.

Could we place that lists on a private wiki (so no emails are revealed to spam bots), so we can collaborate on filling it? (I can provide info from bugtracker feature requests where people asked to be added as a mirror, since I know bugtracker users' email addresses). -- Roman Kyrylych (Роман Кирилич)

On Wed, Aug 5, 2009 at 00:09, Dan McGee wrote:

On Tue, Aug 4, 2009 at 3:57 PM, Roman Kyrylych wrote:

...

On Tue, Aug 4, 2009 at 23:22, Pierre Schmitz wrote:

...

Am Dienstag 04 August 2009 22:13:37 schrieb Aaron Griffin:

...

2009/8/4 Angel Velásquez :

...

I am getting the e-mail of the rest of the mirrors admins to let them know about the issue with the mirrorsync script

Do you have a list of mirror admins? Our backend DB is missing contact info for a few of the mirrors.

Afaik Gerhrad has such a list as he is allways in contact with some mirror admins.

On Tue, Aug 4, 2009 at 23:22, Pierre Schmitz wrote:

...

Am Dienstag 04 August 2009 22:13:37 schrieb Aaron Griffin:

...

2009/8/4 Angel Velásquez :

...

I am getting the e-mail of the rest of the mirrors admins to let them know about the issue with the mirrorsync script

Do you have a list of mirror admins? Our backend DB is missing contact info for a few of the mirrors.

Afaik Gerhrad has such a list as he is allways in contact with some mirror admins.

Could we place that lists on a private wiki (so no emails are revealed to spam bots), so we can collaborate on filling it? (I can provide info from bugtracker feature requests where people asked to be added as a mirror, since I know bugtracker users' email addresses).

Just missed this one with my last post (to the private list, sorry for those that can't see there).

Let's get this in the ***king database please. If the model has to change to get more info in there, so be it, I'd be glad to help. But I'm sick of scattered information everywhere and people not really giving back when they come up with cool tools and help gather this information.

Honestly I didn't know (at least that's what I remember) about the info in the database. I do agree that having list of mirrors in the package, the wiki, the download page etc. is a pain to sync, so autogenerating everything from a DB would be cool.

P.S. Roman, you have been the unfortunate author of a few emails I've responded to lately, please don't feel like I'm attempting to flame you or anything, these posts have just hit some of my hot buttons as of late. :) Thanks for all your effort here, I just want to make sure we don't direct effort in the wrong directions.

Ah, I don't have any problems with your responses. :-) P.S.: this thread is getting scattering through different MLs and private mails already. Let's move it to dev ML completely. -- Roman Kyrylych (Роман Кирилич)

Angel Velásquez wrote:

mirror.rit.edu

I don't see any symlinks there. I would assume, there sits real the problem. ftp://ftp.archlinux.org: lrwxrwxrwx 1 99 99 54 Jul 29 19:02 texlive-bibtexextra-2009.14400-1-any.pkg.tar.gz -> ../any/texlive-bibtexextra-2009.14400-1-any.pkg.tar.gz ftp://mirror.rit.edu -rw-rw-r-- 1 0 18446744073709551614 1410784 Jul 29 19:02 texlive-bibtexextra-2009.14400-1-any.pkg.tar.gz