[arch-general] Syncing the mirrors
A lot of people have been getting bit by mirrors being out of sync. Fundamentally, this comes down to a mirror's database tarball being ahead or behind of the packages which actually exist on the mirror. Rsync is made for bulk updating, but it is not atomic, and bad things happen if you interact with a mirror while the rsync is running. Generally, you get package not found errors. This can be fixed, but it is a pain to deploy, because the mirrors require an update. Here's the what mirrors (probably) should be doing: wget ...../core.db.tar.gz -O core.db.tar.gz.new # and the others rsync --delete-after --exclude="*.db.tar.gz" rsync://...... mv core.db.tar.gz.new core.db.tar.gz # and the others The key differences here: put off deleting files as long as possible (rsync's default is to delete before transferring new files, causing all sorts of trouble), and do not release the new DB until all the new files have finished transferring. This is hardly the same type of guaranteed transaction you get with a ACID database, but it should eliminate most all of the current errors. There does not seem to be a documented "standard rsync" command for the mirrors to use, so I'm making all sorts of wild assumptions about what a mirror's rsync is doing. -Kyle
On Mon, Mar 8, 2010 at 7:09 PM, keenerd <keenerd@gmail.com> wrote:
There does not seem to be a documented "standard rsync" command for the mirrors to use, so I'm making all sorts of wild assumptions about what a mirror's rsync is doing.
http://wiki.archlinux.org/index.php/Local_Mirror#The_synchronization_script
participants (2)
-
Dan McGee
-
keenerd