[arch-dev-public] New DB scripts
Hey all, I'm comfortable enough with the new db scripts to throw them out in the wild. So, I've placed them at /arch-new for you to play with. Here are some caveats, however: * This will not update the website DB. Use at your discresion. Once these are "approved", then Eliott can move the new-model web changes out and the front end WILL be updated on a cron job. * There will be a warning about ~/staging/${REPO}64/ existing. This is expected. Because we have the arch in the filename, we don't need extra dirs anymore. Don't worry, the scripts handle it gracefully and a new devtools release will do away with pushing to the '64' dirs. Then we can clean up our staging dirs a bit. * I didn't fully test removal, because I didn't have anything to remove. If someone has a package to remove, let me know so I can test it properly. If you try, let me know if it fails. Neat things: * This can now _kinda_ create on-the-fly repos, but I locked it down a tad to prevent abuse. The following pattern should make a brand new repo with no problem: archrelease foobar-i686 ssh archlinux.org /arch-new/db-update foobar i686 Now, this actually requires someone with the right privileges to create /home/ftp/foobar, because I didn't want 400 repos being sent out to all our mirrors. * Should be much faster on the svn checkout and all that stuff * Uses repo-add which means we get the 5 or 6 DB values that we've been missing since pacman 3.0 was released. Now, when I "move these live", the following need to be done at the same time. * I need to rebuild ALL dbs with repo-add once, just to get some data for old packages in there. * We need to move the newmodel code so the front end is updated * We _should_ release a new devtools, for the staging dir changes Extra notes: * All cleanup scripts should be able to function off of a repo DB file, and be runnable as a cron job. I am going to be setting up Thomas' ftpdir-cleanup script as a cron job which will clean up all package files that are NOT in the db. No more dupe cleaning, yay! * packages.txt has been broken since we moved to svn. No one has noticed, I think. Should this be done away with, or should it be adapted as a cron-able task?
On 5/6/08, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
Hey all, I'm comfortable enough with the new db scripts to throw them out in the wild.
So, I've placed them at /arch-new for you to play with. Here are some caveats, however:
* This will not update the website DB. Use at your discresion. Once these are "approved", then Eliott can move the new-model web changes out and the front end WILL be updated on a cron job.
I just want to say that this is pretty fucking great. Decoupling for the win. This allows us WAY more flexibility on the web frontend side. 1. We can pull in the db when we need it, refactor, fix, change, whatever is needed. 2. We could host the website and the sql db on a different box from the pacman repo if we wanted/needed to.
* There will be a warning about ~/staging/${REPO}64/ existing. This is expected. Because we have the arch in the filename, we don't need extra dirs anymore. Don't worry, the scripts handle it gracefully and a new devtools release will do away with pushing to the '64' dirs. Then we can clean up our staging dirs a bit.
* I didn't fully test removal, because I didn't have anything to remove. If someone has a package to remove, let me know so I can test it properly. If you try, let me know if it fails.
Neat things:
* This can now _kinda_ create on-the-fly repos, but I locked it down a tad to prevent abuse. The following pattern should make a brand new repo with no problem: archrelease foobar-i686 ssh archlinux.org /arch-new/db-update foobar i686
Now, this actually requires someone with the right privileges to create /home/ftp/foobar, because I didn't want 400 repos being sent out to all our mirrors.
* Should be much faster on the svn checkout and all that stuff
* Uses repo-add which means we get the 5 or 6 DB values that we've been missing since pacman 3.0 was released.
w00t!
Now, when I "move these live", the following need to be done at the same time.
* I need to rebuild ALL dbs with repo-add once, just to get some data for old packages in there.
* We need to move the newmodel code so the front end is updated
* We _should_ release a new devtools, for the staging dir changes
Extra notes:
* All cleanup scripts should be able to function off of a repo DB file, and be runnable as a cron job. I am going to be setting up Thomas' ftpdir-cleanup script as a cron job which will clean up all package files that are NOT in the db. No more dupe cleaning, yay!
* packages.txt has been broken since we moved to svn. No one has noticed, I think. Should this be done away with, or should it be adapted as a cron-able task?
Awesome work Aaron. Progress is good. Thanks. :)
On 5/6/08, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
* packages.txt has been broken since we moved to svn. No one has noticed, I think. Should this be done away with, or should it be adapted as a cron-able task?
One quick note, for the installer people... or, well anyone: pacman -Sl core | cut -d' ' -f2,3 This wil give you all packages in core, by name, with versions - which is the entire reason we have this packages.txt anymore (we no longer have categories, so that part is out)
Aaron Griffin wrote:
* Uses repo-add which means we get the 5 or 6 DB values that we've been missing since pacman 3.0 was released.
Now, when I "move these live", the following need to be done at the same time.
* I need to rebuild ALL dbs with repo-add once, just to get some data for old packages in there.
That can't be true, it's too beautiful :) Seriously, that's amazing, thanks! I hope that part will go smoothly so that we get nice, clean and complete dbs again.
On Wed, May 7, 2008 at 1:45 AM, Xavier <shiningxc@gmail.com> wrote:
Aaron Griffin wrote:
* Uses repo-add which means we get the 5 or 6 DB values that we've been missing since pacman 3.0 was released.
Now, when I "move these live", the following need to be done at the same
time.
* I need to rebuild ALL dbs with repo-add once, just to get some data for old packages in there.
That can't be true, it's too beautiful :) Seriously, that's amazing, thanks! I hope that part will go smoothly so that we get nice, clean and complete dbs again.
And that means pacman 3.2 won't break people's systems! Jeff already fell prey to that little issue when running pacman-git 8)
On Wed, 7 May 2008, Aaron Griffin wrote:
Hey all, I'm comfortable enough with the new db scripts to throw them out in the wild.
So, I've placed them at /arch-new for you to play with. Here are some caveats, however:
* This will not update the website DB. Use at your discresion. Once these are "approved", then Eliott can move the new-model web changes out and the front end WILL be updated on a cron job.
* There will be a warning about ~/staging/${REPO}64/ existing. This is expected. Because we have the arch in the filename, we don't need extra dirs anymore. Don't worry, the scripts handle it gracefully and a new devtools release will do away with pushing to the '64' dirs. Then we can clean up our staging dirs a bit.
* I didn't fully test removal, because I didn't have anything to remove. If someone has a package to remove, let me know so I can test it properly. If you try, let me know if it fails.
There's 2 packages that you could remove to test the script. There's my proftpd package that noone seems to be interested in picking up. A package in community depends on it, so it'll need to be moved there. I have an updated PKGBUILD and fixed daemon script that I could send you so you could put the fixed package in community. Alternatively, I could build the packages but you'll need to test the x86_64 one because of reasons mentioned in the proftpd thread. There's also acroread. The discussion about it seems to have settled down. -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
On Wed, May 7, 2008 at 11:26 AM, Eric Belanger <belanger@astro.umontreal.ca> wrote:
On Wed, 7 May 2008, Aaron Griffin wrote:
Hey all, I'm comfortable enough with the new db scripts to throw them out in the wild.
So, I've placed them at /arch-new for you to play with. Here are some caveats, however:
* This will not update the website DB. Use at your discresion. Once these are "approved", then Eliott can move the new-model web changes out and the front end WILL be updated on a cron job.
* There will be a warning about ~/staging/${REPO}64/ existing. This is expected. Because we have the arch in the filename, we don't need extra dirs anymore. Don't worry, the scripts handle it gracefully and a new devtools release will do away with pushing to the '64' dirs. Then we can clean up our staging dirs a bit.
* I didn't fully test removal, because I didn't have anything to remove. If someone has a package to remove, let me know so I can test it properly. If you try, let me know if it fails.
There's 2 packages that you could remove to test the script.
There's my proftpd package that noone seems to be interested in picking up. A package in community depends on it, so it'll need to be moved there. I have an updated PKGBUILD and fixed daemon script that I could send you so you could put the fixed package in community. Alternatively, I could build the packages but you'll need to test the x86_64 one because of reasons mentioned in the proftpd thread.
There's also acroread. The discussion about it seems to have settled down.
Thanks Eric, I'll kill off acroread. Do you think we should announce this?
participants (4)
-
Aaron Griffin
-
eliott
-
Eric Belanger
-
Xavier