[arch-dev-public] Reducing traffic usage during package move between repos
On Wed, Sep 2, 2009 at 11:41, Dieter Plaetinck<dieter@plaetinck.be> wrote:
I think a big improvemt will be when we fix it so that if we move packages from testing to extra/core, mirrors will not see as a "file remove and a new file", which causes a lot of unneeded traffic. if we make this system use symlinks or whatever this can be handled much better. but other then that, I think the current syncing system is fine. (the "getting" part is something different. see tiered setup,mirrorbrain etc. discussions)
Since we already use symlinks for -any packages this proposal looks okay. The layout could be like this then: <repo>/os/i686/foobar-1.2.3-1-i686.pkg.tar.gz -> ../all/foobar-1.2.3-1-i686.pkg.tar.gz <repo>/os/x86_64/foobar-1.2.3-1-x86_64.pkg.tar.gz -> ../all/foobar-1.2.3-1-x86_64.pkg.tar.gz <repo>/os/any/foobaz-4.5.6-1-any.pkg.tar.gz -> ../all/foobaz-4.5.6-1-any.pkg.tar.gz <repo>/os/all/foobar-1.2.3-1-i686.pkg.tar.gz -> ../all/foobar-1.2.3-1-i686.pkg.tar.gz <repo>/os/all/foobar-1.2.3-1-x86_64.pkg.tar.gz -> ../all/foobar-1.2.3-1-x86_64.pkg.tar.gz <repo>/os/all/foobaz-4.5.6-1-any.pkg.tar.gz -> ../all/foobaz-4.5.6-1-any.pkg.tar.gz As mentioned above - this will reduce the amount of traffic during package moves between repos. The negative side is that this will need one-time redownload of all packages by mirrors, though this can be avoided by manual moving of the files by mirror admin (need to be done in this way: wait until /all syncing starts, kill rsync, move packages manually, start rsync). BTW I always wondered why we have that silly "os" subdir. Would be nice to get rid of it during this big mirroring change (if it'll happen). -- Roman Kyrylych (Роман Кирилич)
On Wed, Sep 2, 2009 at 11:55, Roman Kyrylych<roman.kyrylych@gmail.com> wrote:
On Wed, Sep 2, 2009 at 11:41, Dieter Plaetinck<dieter@plaetinck.be> wrote:
I think a big improvemt will be when we fix it so that if we move packages from testing to extra/core, mirrors will not see as a "file remove and a new file", which causes a lot of unneeded traffic. if we make this system use symlinks or whatever this can be handled much better. but other then that, I think the current syncing system is fine. (the "getting" part is something different. see tiered setup,mirrorbrain etc. discussions)
Since we already use symlinks for -any packages this proposal looks okay. The layout could be like this then:
Oops, copy-paste error. Should be this way. i686/foobar-1.2.3-1-i686.pkg.tar.gz -> ../all/foobar-1.2.3-1-i686.pkg.tar.gz x86_64/foobar-1.2.3-1-x86_64.pkg.tar.gz -> ../all/foobar-1.2.3-1-x86_64.pkg.tar.gz any/foobaz-4.5.6-1-any.pkg.tar.gz -> ../all/foobaz-4.5.6-1-any.pkg.tar.gz all/foobar-1.2.3-1-i686.pkg.tar.gz all/foobar-1.2.3-1-x86_64.pkg.tar.gz all/foobaz-4.5.6-1-any.pkg.tar.gz
As mentioned above - this will reduce the amount of traffic during package moves between repos. The negative side is that this will need one-time redownload of all packages by mirrors, though this can be avoided by manual moving of the files by mirror admin (need to be done in this way: wait until /all syncing starts, kill rsync, move packages manually, start rsync).
BTW I always wondered why we have that silly "os" subdir. Would be nice to get rid of it during this big mirroring change (if it'll happen).
-- Roman Kyrylych (Роман Кирилич)
On Wed, Sep 2, 2009 at 11:58, Roman Kyrylych<roman.kyrylych@gmail.com> wrote:
On Wed, Sep 2, 2009 at 11:55, Roman Kyrylych<roman.kyrylych@gmail.com> wrote:
On Wed, Sep 2, 2009 at 11:41, Dieter Plaetinck<dieter@plaetinck.be> wrote:
I think a big improvemt will be when we fix it so that if we move packages from testing to extra/core, mirrors will not see as a "file remove and a new file", which causes a lot of unneeded traffic. if we make this system use symlinks or whatever this can be handled much better. but other then that, I think the current syncing system is fine. (the "getting" part is something different. see tiered setup,mirrorbrain etc. discussions)
Since we already use symlinks for -any packages this proposal looks okay. The layout could be like this then:
Oops, copy-paste error. Should be this way.
Double FAIL!!! core/os/i686/foobar-1.2.3-1-i686.pkg.tar.gz -> ../../../all-packages/os/i686/foobar-1.2.3-1-i686.pkg.tar.gz extra/os/i686/foobaz-4.5.6-1-i686.pkg.tar.gz -> ../../../all-packages/os/i686/foobar-4.5.6-1-i686.pkg.tar.gz all-packages/os/i686/foobar-4.5.6-1-i686.pkg.tar.gz This is just an idea. Feel free to shoot it if you don't like it. -- Roman Kyrylych (Роман Кирилич)
Roman Kyrylych schrieb:
The negative side is that this will need one-time redownload of all packages by mirrors,
Which will take several weeks!
though this can be avoided by manual moving of the files by mirror admin (need to be done in this way: wait until /all syncing starts, kill rsync, move packages manually, start rsync).
This can be avoided if we (instead of moving everything at once), just start doing this for new packages and leave the old ones where they are. There is no need to resync already synced packages.
BTW I always wondered why we have that silly "os" subdir. Would be nice to get rid of it during this big mirroring change (if it'll happen).
So, you also want to break user's configurations now? Why fix something that isn't broken?
On Wed, 2009-09-02 at 11:05 +0200, Thomas Bächler wrote:
This can be avoided if we (instead of moving everything at once), just start doing this for new packages and leave the old ones where they are. There is no need to resync already synced packages.
+1 Just place new packages in /all/ or /pool/ and symlink them to the repos. One issue is that mirrors need to synchronize this new directory also. This will reduce a lot of traffic when packages are moved from testing to core.
On Wed, Sep 2, 2009 at 4:21 AM, Jan de Groot<jan@jgc.homeip.net> wrote:
On Wed, 2009-09-02 at 11:05 +0200, Thomas Bächler wrote:
This can be avoided if we (instead of moving everything at once), just start doing this for new packages and leave the old ones where they are. There is no need to resync already synced packages.
+1
Just place new packages in /all/ or /pool/ and symlink them to the repos. One issue is that mirrors need to synchronize this new directory also. This will reduce a lot of traffic when packages are moved from testing to core.
Sounds good to me. If someone provides a patch for the dbscripts, we can get this in faster. I plan on tagging a new release soon anyway
On Wed, Sep 2, 2009 at 12:05, Thomas Bächler<thomas@archlinux.org> wrote:
Roman Kyrylych schrieb:
The negative side is that this will need one-time redownload of all packages by mirrors,
Which will take several weeks!
True
though this can be avoided by manual moving of the files by mirror admin (need to be done in this way: wait until /all syncing starts, kill rsync, move packages manually, start rsync).
This can be avoided if we (instead of moving everything at once), just start doing this for new packages and leave the old ones where they are. There is no need to resync already synced packages.
Good point!
BTW I always wondered why we have that silly "os" subdir. Would be nice to get rid of it during this big mirroring change (if it'll happen).
So, you also want to break user's configurations now? Why fix something that isn't broken?
Workaround by another symlink (<repo>/os -> ../)? :-P I agree that this is just cosmetics (extra /os/ is ugly IMO), so this is not that important. -- Roman Kyrylych (Роман Кирилич)
participants (4)
-
Aaron Griffin
-
Jan de Groot
-
Roman Kyrylych
-
Thomas Bächler