[arch-general] 'Local mirror' page was removed from wiki

[David C. Rankin]

On 09/09/2010 07:40 AM, Fess wrote:
> Page "Local mirros" was removed from wiki by this reason:
>
> -----
> It is generally frowned upon to create a local mirror due the bandwidth that is required.
> There is not a good reason to create a local mirror, since one of the alternatives below will likely meet your needs.
> -----
>
> I think it's very useful ability, and why this page was removed... i don't know. It's silly.
> If anyone else think it must be return - maybe we should do it?
>
> P.S.
>
> A lot of guys agreed with me this morning, so i think many people want this article back.

I looked into an Arch local 'mirror/repo' when I started using Arch, similar to 
what I used to maintain for my SuSE installs. (the local mirror page was a mess 
and wrong then as well) However, since Arch will check for the presence of 
packages in /var/cache/pacman/pkg before/(instead of) re-downloading the 
packages, for local network boxes, it was just easier to use rsync to grab the 
packages from whatever box did the most recent update and transfer then to the 
nextbox you need to update (same architecture only).

The only caveat is you need a way to manage duplicate (old packages) in 
/var/cache/pacman/pkg so I came up with a script that does that.

So instead of worrying about a 'local mirror', just:

(1) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman

(2) grab the following 2 scripts:

(the wrapper script that calls the main script [twice - see why below])
http://www.3111skyline.com/dl/Archlinux/scripts/fduparch.sh

(the main duplicate identification and removal script)
http://www.3111skyline.com/dl/Archlinux/scripts/fduppkg

Put them both in /usr/local/bin (or link them there), edit fduparch.sh and 
change the directories you want the duplicates from /var/cache/pacman/pkg moved 
to (default is /home/backup/pkg-old and for the second pass to 
/home/backup/pkg-older). Then after updating one box, just call fduparch.sh (the 
wrapper script) as root and the duplicates in /var/cache/pacman/pkg are moved as 
follows:

Pass 1:
/var/cache/pacman/pkg => /home/backup/pkg-old

Pass 2:
/home/backup/pkg-old => /home/backup/pkg-older

Which leaves you with the current set of packages in:
/var/cache/pacman/pkg

The last used packages before update in:
/home/backup/pkg-old

And finally all older packages in:
/home/backup/pkg-older

which can be deleted or archived. (of course you can delete the packages in 
/home/backup/pkg-old if you like as well)

NOTE: the duplicate removal script uses the file ctime and/or mtime to determine 
which is the newer package, so when copying machine to machine with rsync, make 
sure you preserve the file attributes. (rsync's -a option works fine).

Once you set this up, maintaining a local set of packages is a breeze, just:

(1) pacman -Syu as normal on one box (works the same for new installs too)
(2) fduparch.sh (as root to remove the older versions of new packages)
(3) rsync -uav updatedbox:/var/cache/pacman/pkg nextbox:/var/cache/pacman
(only for boxes of the same architecture of course)
(4) pacman -Syu on the box you rsync'ed the packages to (nextbox)
(5) call fduparch.sh on the box you just updated (nextbox), and so on, and so on...

As long as your boxes have reasonably similar packages installed, this 
eliminates 90+% of re-downloading, takes no additional bandwidth because you 
have to download the updated packages once anyway, and is simple enough for me 
to manage so you guy will have no problems with it :p

And if you wanted to get even more creative, you could simply write a script on 
your main box that you update to automate the entire process for all your local 
boxes using nothing more than rsync and ssh calls of run the updates and dup 
removal scripts. (I haven't been that creative yet) Just remember to separate 
your boxes into groups/classes by architecture (i686 & x86_64)

Cheers.

-- 
David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
www.rankinlawfirm.com