[arch-dev-public] Important: upcoming dbscripts release
Hi all, as the new dbscripts are quite done I'll send you a list with the most important changes. (If you are in a hurry read at least the next paragraph) Incompatible changes: * db-update will now update all packages found in your staging dir from all repositories * db-move has now a different syntax: "<repo-from> <repo-to> <pkgname|pkgbase> ..." You can move as many packages as you like at once. There is no need to specify an architecture here. * The wrapper scripts like testing2core have been removed as they are no longer needed; use db-move instead * testing2x now moves both arches at the same time (testing2x64 has been removed) * I have also started a user documentation at http://wiki.archlinux.org/index.php/DeveloperWiki:dbscripts Important changes: * implementation of a package pool: all packages are now stored in pool/packages or pool/community directories on the ftp server. The current repository directories like extra/os/i686 only contain symlinks. As a results moving pacakges between repositories is now a matter of altering symlinks and as such very cheap. This should solve the problems with not fully synced mirrors we had in the past when a lot of packages were moved from e.g. testing to extra. * Packages are only switched to this new layout if they are updated or moved. This way the migration should be slow but smooth. * db-move can now handle multiple packages within one "transaction". General improvements: * unit tests based on shunit2 * heavy refactoring to share common functions * Added more integrity checks which are run before the repositories are touched. * rewritten functions for easier and smaller code * the scripts can handle other db compressions than gz (but there is no migration path yet) Note: even though the result is more or less a rewrite there are still things that can be improved or new useful features that can be added. The tests only cover some basic use cases so we should still have a look at it once the new version is deployed. I think we should deploy these soon (before Allan's big python rebuild party) with CLEANUP_DRYRUN enabled during the first days. Greetings, Pierre -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Mon, Aug 16, 2010 at 12:49 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Hi all,
as the new dbscripts are quite done I'll send you a list with the most important changes. (If you are in a hurry read at least the next paragraph)
Incompatible changes: * db-update will now update all packages found in your staging dir from all repositories * db-move has now a different syntax: "<repo-from> <repo-to> <pkgname|pkgbase> ..." You can move as many packages as you like at once. There is no need to specify an architecture here. * The wrapper scripts like testing2core have been removed as they are no longer needed; use db-move instead * testing2x now moves both arches at the same time (testing2x64 has been removed) * I have also started a user documentation at http://wiki.archlinux.org/index.php/DeveloperWiki:dbscripts
Important changes: * implementation of a package pool: all packages are now stored in pool/packages or pool/community directories on the ftp server. The current repository directories like extra/os/i686 only contain symlinks. As a results moving pacakges between repositories is now a matter of altering symlinks and as such very cheap. This should solve the problems with not fully synced mirrors we had in the past when a lot of packages were moved from e.g. testing to extra. * Packages are only switched to this new layout if they are updated or moved. This way the migration should be slow but smooth. * db-move can now handle multiple packages within one "transaction".
General improvements: * unit tests based on shunit2 * heavy refactoring to share common functions * Added more integrity checks which are run before the repositories are touched. * rewritten functions for easier and smaller code * the scripts can handle other db compressions than gz (but there is no migration path yet)
Note: even though the result is more or less a rewrite there are still things that can be improved or new useful features that can be added. The tests only cover some basic use cases so we should still have a look at it once the new version is deployed.
I think we should deploy these soon (before Allan's big python rebuild party) with CLEANUP_DRYRUN enabled during the first days.
Thanks a bunch for good work on this, Pierre. -Dan
On Mon, 16 Aug 2010 13:03:53 -0500 Dan McGee <dpmcgee@gmail.com> wrote:
Thanks a bunch for good work on this, Pierre.
-Dan
2nd that. cool stuff Dieter
Important changes: * implementation of a package pool: all packages are now stored in pool/packages or pool/community directories on the ftp server. The current repository directories like extra/os/i686 only contain symlinks. As a results moving pacakges between repositories is now a matter of altering symlinks and as such very cheap. This should solve the problems with not fully synced mirrors we had in the past when a lot of packages were moved from e.g. testing to extra.
Has anyone done any testing on this to make sure our mirrors are configured to rsync symlinks properly? There is a flag to copy them as the source file, one to copy the symlink as a symlink, and another to ignore them altogether. We don't want to make this change until we can confirm the majority of the mirrors are using the proper rsync flag(s). Dale
On Mon, 16 Aug 2010 14:09:40 -0400, Dale Blount <dale@archlinux.org> wrote:
Important changes: * implementation of a package pool: all packages are now stored in pool/packages or pool/community directories on the ftp server. The current repository directories like extra/os/i686 only contain symlinks. As a results moving pacakges between repositories is now a matter of altering symlinks and as such very cheap. This should solve the problems with not fully synced mirrors we had in the past when a lot of packages were moved from e.g. testing to extra.
Has anyone done any testing on this to make sure our mirrors are configured to rsync symlinks properly? There is a flag to copy them as the source file, one to copy the symlink as a symlink, and another to ignore them altogether. We don't want to make this change until we can confirm the majority of the mirrors are using the proper rsync flag(s).
Dale
We already use symlinks for our "any" packages which are implemented for a long time. Thomas already created the pool dirs and they are synced correctly on our tier1 servers. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On Mon, Aug 16, 2010 at 1:49 PM, Pierre Schmitz <pierre@archlinux.de> wrote:
Hi all,
as the new dbscripts are quite done I'll send you a list with the most important changes. (If you are in a hurry read at least the next paragraph)
Incompatible changes: * db-update will now update all packages found in your staging dir from all repositories * db-move has now a different syntax: "<repo-from> <repo-to> <pkgname|pkgbase> ..." You can move as many packages as you like at once. There is no need to specify an architecture here. * The wrapper scripts like testing2core have been removed as they are no longer needed; use db-move instead * testing2x now moves both arches at the same time (testing2x64 has been removed) * I have also started a user documentation at http://wiki.archlinux.org/index.php/DeveloperWiki:dbscripts
Important changes: * implementation of a package pool: all packages are now stored in pool/packages or pool/community directories on the ftp server. The current repository directories like extra/os/i686 only contain symlinks. As a results moving pacakges between repositories is now a matter of altering symlinks and as such very cheap. This should solve the problems with not fully synced mirrors we had in the past when a lot of packages were moved from e.g. testing to extra. * Packages are only switched to this new layout if they are updated or moved. This way the migration should be slow but smooth. * db-move can now handle multiple packages within one "transaction".
General improvements: * unit tests based on shunit2 * heavy refactoring to share common functions * Added more integrity checks which are run before the repositories are touched. * rewritten functions for easier and smaller code * the scripts can handle other db compressions than gz (but there is no migration path yet)
Note: even though the result is more or less a rewrite there are still things that can be improved or new useful features that can be added. The tests only cover some basic use cases so we should still have a look at it once the new version is deployed.
I think we should deploy these soon (before Allan's big python rebuild party) with CLEANUP_DRYRUN enabled during the first days.
Greetings,
Pierre
-- Pierre Schmitz, https://users.archlinux.de/~pierre
Hi, I just notice problems with the sourceballs script. First , it removes sourceballs for packages that it incorrectly thinks are no longer in svn. The packages are in svn. I suppose it is caused by svn seg faulting. The sourceballs cleanup script sends the svn output to /dev/null so I'm not sure but there is several seg fault in the logs when it creates sourceballs. As the cron job is about to run/running, the old logsare in http://dev.archlinux.org/~eric/sourceballslogs/ Another more important problem is that since the dbscripts update, the sourceballs script has stopped to download sourceballs and to log the makepkg output in /var/log/sourceballs. Maybe we should disable the sourceballs cleanup (comment out $dirname/../misc-scripts/sourceballs-cleanup in cron-jobs/sourceballs) until the script is fixed. Eric
participants (5)
-
Dale Blount
-
Dan McGee
-
Dieter Plaetinck
-
Eric Bélanger
-
Pierre Schmitz