[arch-dev-public] devtools and db-scripts plans
Hi all, There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far: devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?) db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing) - delta support... Any others? Allan
On 02/04/2010 02:23 PM, Allan McRae wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing) - delta support...
Any others?
db-scripts - support for moving packages from community to extra. single "package" directory it will help and is a matter of svn commit the proper files. -- Ionut
On Thu, 04 Feb 2010 22:23:56 +1000, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
Funny you mentioned this. I have thought about this recently and I think our tools, work flow, mirror setup etc. will require some improvements. I'd like to work on this, but I think I'll need to get rid of most of my packages first. I'll send a more detailed mail about this later today (hopefully, but cannot guarantee) Pierre -- Pierre Schmitz, https://users.archlinux.de/~pierre
Am 04.02.2010 13:23, schrieb Allan McRae:
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
Someone has to do it. I probably don't have enough free time, so I could claim I would do it, but it still wouldn't happen - therefore I will be honest to myself and simply say I won't do it.
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages)
-> Split packages in different repositories (only if they reside in the same SVN tree). This would help dbus and dbus-core, maybe more, like some of the gcc stuff.
- support for xz compression (nothing needed?)
I would make it more general and say support for different compressions. The dev/db scripts shouldn't care what the compression is and simply add any (supported) package, be it gz, bz2 or xz. While we should probably agree on one compression type for everyone, I think our scripts should also support everything without problems. And if there is ever a new compression, we simply add the file extension to the "supported compressions" list in devtools/dbscripts and need no further changes.
db-scripts - support for xz compression transition (e.g. in clean-up script)
See as above, the scripts should be entirely flexible here.
- support for single "package" directory with symlinks to repos
This should be one directory per SVN tree (pool-packages and pool-community), so core/extra/testing and community/community-testing will be separate - otherwise, the whole rsyncing stuff will be a huge nightmare. Also: -> Allow a smooth transition such that the cleanup script handles a mix of "single directory" and the old style gracefully - we don't want to move all packages at once, but only have this single directory for newly added packages.
- allow db-move to handle multiple packages (or add a script to help moving from community-testing)
We should kill these testing2whatever scripts and unify it completely in db-move (including the testing2x case where the right repository is automatically determined). Short-term it would probably suffice to create community-testing2community.
- delta support...
Personally, I would consider that low priority.
On Thu, Feb 4, 2010 at 7:23 AM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing)
There are already testing2community* scripts in git as well as a db-community-testing. They just need to be released on sigurd.
- delta support...
Any others?
Couple of bugs that could be fixed. FS#17058 - {dbscripts} ftpdir-cleanup marked new packages as out of date FS#17098 - sourceballs: Removal of old sourceball not efficient Probably sourceballs and sourceballs-cleanup should be merged. And the sourceball script is still lacking support for the community repos. I used to work a lot on the sourceball script but don't have much time lately. I also have no idea on how to access the svn for commynity from gerolde like the way the sourceballs-cleanup script currently do.
Allan
On Thu, Feb 4, 2010 at 12:26 PM, Eric Bélanger <snowmaniscool@gmail.com> wrote:
On Thu, Feb 4, 2010 at 7:23 AM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing)
There are already testing2community* scripts in git as well as a db-community-testing. They just need to be released on sigurd.
- delta support...
Any others?
Couple of bugs that could be fixed.
FS#17058 - {dbscripts} ftpdir-cleanup marked new packages as out of date
FS#17098 - sourceballs: Removal of old sourceball not efficient
Probably sourceballs and sourceballs-cleanup should be merged.
I made a patch for that. It's untested yet. As I can't test it on my system, I'll need to test it on gerolde. I could make my copy of the sourceballs directory in my home directory. It'll be the most convenient for me if fixes needs to be done but it'll use up (temporarily) 9.5-10 GB of the avaliable 11GB of disk space. Alternatively, an admin could start the new scripts while I'm able to keep an eye on it and stay available in case the script needs to be stopped. I could also just send a git patch of the untested script. Any idea on the best way to proceed? And
the sourceball script is still lacking support for the community repos. I used to work a lot on the sourceball script but don't have much time lately. I also have no idea on how to access the svn for commynity from gerolde like the way the sourceballs-cleanup script currently do.
Allan
On Fri, Feb 5, 2010 at 10:35 PM, Eric Bélanger <snowmaniscool@gmail.com> wrote:
On Thu, Feb 4, 2010 at 12:26 PM, Eric Bélanger <snowmaniscool@gmail.com> wrote:
On Thu, Feb 4, 2010 at 7:23 AM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing)
There are already testing2community* scripts in git as well as a db-community-testing. They just need to be released on sigurd.
- delta support...
Any others?
Couple of bugs that could be fixed.
FS#17058 - {dbscripts} ftpdir-cleanup marked new packages as out of date
FS#17098 - sourceballs: Removal of old sourceball not efficient
Probably sourceballs and sourceballs-cleanup should be merged.
I made a patch for that. It's untested yet. As I can't test it on my system, I'll need to test it on gerolde. I could make my copy of the sourceballs directory in my home directory. It'll be the most convenient for me if fixes needs to be done but it'll use up (temporarily) 9.5-10 GB of the avaliable 11GB of disk space. Alternatively, an admin could start the new scripts while I'm able to keep an eye on it and stay available in case the script needs to be stopped. I could also just send a git patch of the untested script. Any idea on the best way to proceed?
And
the sourceball script is still lacking support for the community repos. I used to work a lot on the sourceball script but don't have much time lately. I also have no idea on how to access the svn for commynity from gerolde like the way the sourceballs-cleanup script currently do.
I now also have an untested patch for community support.
Allan
On Fri, Feb 5, 2010 at 9:35 PM, Eric Bélanger <snowmaniscool@gmail.com> wrote:
On Thu, Feb 4, 2010 at 12:26 PM, Eric Bélanger <snowmaniscool@gmail.com> wrote:
On Thu, Feb 4, 2010 at 7:23 AM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing)
There are already testing2community* scripts in git as well as a db-community-testing. They just need to be released on sigurd.
- delta support...
Any others?
Couple of bugs that could be fixed.
FS#17058 - {dbscripts} ftpdir-cleanup marked new packages as out of date
FS#17098 - sourceballs: Removal of old sourceball not efficient
Probably sourceballs and sourceballs-cleanup should be merged.
I made a patch for that. It's untested yet. As I can't test it on my system, I'll need to test it on gerolde. I could make my copy of the sourceballs directory in my home directory. It'll be the most convenient for me if fixes needs to be done but it'll use up (temporarily) 9.5-10 GB of the avaliable 11GB of disk space. Alternatively, an admin could start the new scripts while I'm able to keep an eye on it and stay available in case the script needs to be stopped. I could also just send a git patch of the untested script. Any idea on the best way to proceed?
Don't copy, use hardlinks or symlinks please. -Dan
On Thu, Feb 4, 2010 at 6:23 AM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing) - delta support...
Any others?
I am more than willing to work on the package pooling part of the db-scripts code, I just need to find the time. And I agree with Thomas that the compression stuff should be made more generic.
On Thu, Feb 4, 2010 at 11:29 AM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On Thu, Feb 4, 2010 at 6:23 AM, Allan McRae <allan@archlinux.org> wrote:
Hi all,
There has been talk about what is needed/wanted for the future of devtools and db-scripts but not much done lately. So I thought I would get together a list of ideas so we could perhaps do a coding sprint to implement them. Here is what I have got so far:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages -> overridden pkgver -> overridden pkgrel (and only selected sub-packages) - support for xz compression (nothing needed?)
db-scripts - support for xz compression transition (e.g. in clean-up script) - support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing) - delta support...
Any others?
I am more than willing to work on the package pooling part of the db-scripts code, I just need to find the time.
And I agree with Thomas that the compression stuff should be made more generic.
The following branch contains some initial work: http://code.phraktured.net/cgit.cgi/dbscripts/log/?h=working * Merged Daniel's patch * Removed the testing2* scripts * Added package pool support to db-update The rest of the scripts still need support for the whole pooling thing, but here's a starting point. Please critique, it's not all that much: http://code.phraktured.net/cgit.cgi/dbscripts/commit/?h=working&id=5dc5fc0dff1632f834ac11d3bfcef87a512b1096 db-move and db-remove need adjusting, and then the cleanup scripts.
Am Donnerstag, 4. Februar 2010 13:23:56 schrieb Allan McRae:
devtools - split-PKGBUILDs with (supported in future pacman): -> both binary and arch=any packages
This should be doable. But I guess we need to check the cleanup-scripts, too. They might get confused.
-> overridden pkgver -> overridden pkgrel (and only selected sub-packages)
This is not easy and you would need a repo dir for every split package; otherwise the PKGBUILD in the abs tree wont match the packages in the repo.
- support for xz compression (nothing needed?)
Afaik I already committed everything for this.
db-scripts - support for xz compression transition (e.g. in clean-up script)
support for several compression methods is already there; but not at the same time. It's more complicated than it sounds. This is very likely the next issue I'll look at.
- support for single "package" directory with symlinks to repos - allow db-move to handle multiple packages (or add a script to help moving from community-testing)
Looks like Aaron has patches for this in his git repo
- delta support...
That should be the last thing we want to implement. We should have xz support, package pooling and a multi tier mirror setup first to handle the extra traffic that comes with delta packages. -- Pierre Schmitz, https://users.archlinux.de/~pierre
On 15/02/10 21:42, Pierre Schmitz wrote:
-> overridden pkgver -> overridden pkgrel (and only selected sub-packages)
This is not easy and you would need a repo dir for every split package; otherwise the PKGBUILD in the abs tree wont match the packages in the repo.
Why not? Say I have a split package making packages "foo", "bar" and "baz". I build the packages with main pkgrel=1 and upload. Later I find a pure packaging error in "foo", so I adjust the package_foo function, override its pkgrel to 2 and upload just foo. "bar" and "baz" remain untouched so the PKGBUILD should match the package. I'm sure they are cases where this may break something, but there are cases where this would be a good thing. Think of a game where there is a binary package and an "any" data package. A fix to the binary package should not require a user redownloading the data package with just a pkgrel bump. Allan
On 15/02/10 21:42, Pierre Schmitz wrote:
- delta support... That should be the last thing we want to implement. We should have xz support, package pooling and a multi tier mirror setup first to handle the extra traffic that comes with delta packages.
Fair enough. I just bring this up because it is all ready to go and has been for a while. repo-add can add and remove deltas, pacman handles them fine and there is even a clean-up script floating around to remove deltas that are no longer reachable. So all it needs is a very minor modification to the db-scripts to generate them on addition of a package to the repos. Allan
participants (7)
-
Aaron Griffin
-
Allan McRae
-
Dan McGee
-
Eric Bélanger
-
Ionut Biru
-
Pierre Schmitz
-
Thomas Bächler