[arch-general] Archweb package update script

[Dan McGee]

On Wed, Jul 21, 2010 at 5:45 AM, Netanel Shine <netanelshine at gmail.com> wrote:
>>
>> Date: Sun, 18 Jul 2010 21:36:42 -0500
>> From: Dan McGee <dpmcgee at gmail.com>
>> Subject: Re: [arch-general] RSS feed - missing feature|problem
>> On Mon, Jul 12, 2010 at 3:59 AM, Netanel Shine <netanelshine at gmail.com>
>> wrote:
>> > Hey all,
>> >
>> > My name is Netanel Shine, and a few days ago i began the process of
>> building
>> > an israeli community of archlinux. I used a git clone to set up the main
>> > site. (like some other great projects file that archlinux git have to
>> offer
>> > like the wiki and the forums)
>> > Everything went smoothly and just works great, except for one problem -
>> the
>> > package rss feeds.
>> >
>> > The site address is: http://archlinux.org.il - under heavy development.
>> >
>> > The problem: As you all can see, the left rss box with the latest updates
>> > dont work and the list is empty. - because the system comes with an
>> > integrated "packages" part (you can see it here:
>> > http://archlinux.org.il/packages - not gonna use it), so the rss list
>> > updates from here (i guess)
>> >
>> > After discussions with some great people (in the irc) we concluded that
>> the
>> > rss box pull`s the data from the 'models.Package' object - that connected
>> to
>> > archlinux mainsite DB (Package.objects=PackageManager()), so if i dont
>> have
>> > any access for the database - thats not gonna work.
>> >
>> > All sorts of ideas were came up like bulding a script that will do the
>> job
>> > or, a an iframe that will copy the box from the mainsite (but the problem
>> it
>> > will not be in hebrew) and an idea to replace the table that has for
>> update
>> > in pkg_updates. (with logic that pulls from the feed) - but thats not
>> easy
>> > because the logic is a wrapped up in django MVC.
>> >
>> > So dear friends, i turn to you for help - and I`m sure I would find it.
>>
>> Check out the reporead command baked into manage.py- you can very
>> easily populate the package DB yourself.
>>
>> I have a small script I can send you for this, but I don't have it
>> handy at the moment. Let me know if that would be useful to you.
>>
>> -Dan
>
> First of all, I sorry for seeing this message just now.
> Thanks Dan, I appreciate your comment, and i would like to have the script
> you're talking about , please send him to: netanelshine at gmail dot com ;)

Something like the following:

$ cat projects/archweb/fetch_update_package_db.sh
#!/bin/bash -e

# run at nice 3. it can churn quite a bit of cpu after all.
renice +3 -p $$ > /dev/null

# setup paths
DLURL="ftp://ftp.archlinux.org"
SPATH="/path/to/archweb"
SCRIPT="${SPATH}/scripts/reporead.py"
TMPPATH="/tmp/updaterepos"
ARCHES="i686 x86_64"
REPOS="core extra testing community community-testing"

startdir="$(pwd)"
[ ! -d ${TMPPATH} ] && mkdir ${TMPPATH}

for arch in ${ARCHES}; do
    [ ! -d ${TMPPATH}/${arch} ] && mkdir ${TMPPATH}/${arch}
    for repo in ${REPOS}; do
        #[ ! -d ${TMPPATH}/${repo} ] && mkdir ${TMPPATH}/${repo}
        #cd ${TMPPATH}/${repo}
        echo "Updating ${repo}-${arch}"
        cd $TMPPATH/${arch}
        wget -N ${DLURL}/${repo}/os/${arch}/${repo}.db.tar.gz
        #${SCRIPT} ${arch} ${TMPPATH}/${arch}/${repo}.db.tar.gz
        cd "$startdir"
        ./manage.py reporead ${arch} ${TMPPATH}/${arch}/${repo}.db.tar.gz
        echo ""
    done
done