On Mon, Aug 16, 2010 at 12:49 PM, Pierre Schmitz wrote:

Hi all,

as the new dbscripts are quite done I'll send you a list with the most important changes. (If you are in a hurry read at least the next paragraph)

Incompatible changes: * db-update will now update all packages found in your staging dir from  all repositories * db-move has now a different syntax:  "<repo-from> <repo-to> ..."  You can move as many packages as you like at once. There is no need  to specify an architecture here. * The wrapper scripts like testing2core have been removed as they are  no longer needed; use db-move instead * testing2x now moves both arches at the same time (testing2x64 has  been removed) * I have also started a user documentation at  http://wiki.archlinux.org/index.php/DeveloperWiki:dbscripts

Important changes: * implementation of a package pool: all packages are now stored in  pool/packages or pool/community directories on the ftp server. The  current repository directories like extra/os/i686 only contain  symlinks.  As a results moving pacakges between repositories is now a matter of  altering symlinks and as such very cheap. This should solve the  problems with not fully synced mirrors we had in the past when a lot  of packages were moved from e.g. testing to extra. * Packages are only switched to this new layout if they are updated or  moved. This way the migration should be slow but smooth. * db-move can now handle multiple packages within one "transaction".

General improvements: * unit tests based on shunit2 * heavy refactoring to share common functions * Added more integrity checks which are run before the repositories are  touched. * rewritten functions for easier and smaller code * the scripts can handle other db compressions than gz (but there is no  migration path yet)

Note: even though the result is more or less a rewrite there are still things that can be improved or new useful features that can be added. The tests only cover some basic use cases so we should still have a look at it once the new version is deployed.

I think we should deploy these soon (before Allan's big python rebuild party) with CLEANUP_DRYRUN enabled during the first days.

Thanks a bunch for good work on this, Pierre. -Dan

Important changes: * implementation of a package pool: all packages are now stored in pool/packages or pool/community directories on the ftp server. The current repository directories like extra/os/i686 only contain symlinks. As a results moving pacakges between repositories is now a matter of altering symlinks and as such very cheap. This should solve the problems with not fully synced mirrors we had in the past when a lot of packages were moved from e.g. testing to extra.

Has anyone done any testing on this to make sure our mirrors are configured to rsync symlinks properly? There is a flag to copy them as the source file, one to copy the symlink as a symlink, and another to ignore them altogether. We don't want to make this change until we can confirm the majority of the mirrors are using the proper rsync flag(s). Dale

On Mon, Aug 16, 2010 at 1:49 PM, Pierre Schmitz wrote:

Hi all,

as the new dbscripts are quite done I'll send you a list with the most important changes. (If you are in a hurry read at least the next paragraph)

Incompatible changes: * db-update will now update all packages found in your staging dir from  all repositories * db-move has now a different syntax:  "<repo-from> <repo-to> ..."  You can move as many packages as you like at once. There is no need  to specify an architecture here. * The wrapper scripts like testing2core have been removed as they are  no longer needed; use db-move instead * testing2x now moves both arches at the same time (testing2x64 has  been removed) * I have also started a user documentation at  http://wiki.archlinux.org/index.php/DeveloperWiki:dbscripts

Important changes: * implementation of a package pool: all packages are now stored in  pool/packages or pool/community directories on the ftp server. The  current repository directories like extra/os/i686 only contain  symlinks.  As a results moving pacakges between repositories is now a matter of  altering symlinks and as such very cheap. This should solve the  problems with not fully synced mirrors we had in the past when a lot  of packages were moved from e.g. testing to extra. * Packages are only switched to this new layout if they are updated or  moved. This way the migration should be slow but smooth. * db-move can now handle multiple packages within one "transaction".

General improvements: * unit tests based on shunit2 * heavy refactoring to share common functions * Added more integrity checks which are run before the repositories are  touched. * rewritten functions for easier and smaller code * the scripts can handle other db compressions than gz (but there is no  migration path yet)

Note: even though the result is more or less a rewrite there are still things that can be improved or new useful features that can be added. The tests only cover some basic use cases so we should still have a look at it once the new version is deployed.

I think we should deploy these soon (before Allan's big python rebuild party) with CLEANUP_DRYRUN enabled during the first days.

Greetings,

Pierre

-- Pierre Schmitz, https://users.archlinux.de/~pierre

Hi, I just notice problems with the sourceballs script. First , it removes sourceballs for packages that it incorrectly thinks are no longer in svn. The packages are in svn. I suppose it is caused by svn seg faulting. The sourceballs cleanup script sends the svn output to /dev/null so I'm not sure but there is several seg fault in the logs when it creates sourceballs. As the cron job is about to run/running, the old logsare in http://dev.archlinux.org/~eric/sourceballslogs/ Another more important problem is that since the dbscripts update, the sourceballs script has stopped to download sourceballs and to log the makepkg output in /var/log/sourceballs. Maybe we should disable the sourceballs cleanup (comment out $dirname/../misc-scripts/sourceballs-cleanup in cron-jobs/sourceballs) until the script is fixed. Eric