[arch-dev-public] Rewriting dbscripts (packagers should read this!)
Hi, After having read the recent git thread I've come to the conclusion that we should rework dbscripts before we start working on the package backend (although my proposal works better with git because it uses tags). TLDR: You should care and read it. If you don't I won't care if you later miss features that contradicts my proposal. Note: The text below is more or less a cleaned up brain dump so it's probably not complete and possibly confusing. If so, please tell me. I'm also open for discussion via irc as long as the important points are later sent to the ML. # Goal Decouple dbscripts from svn to make it faster and simpler. Moving lots of packages is awefully slow right now. svn tracks the PKGBUILDS and tags refering to certain PKGBUILDs. dbscripts maintains an internal, separate database which tracks in which repo a given package is (identified by a pkgname and a tag). # Definitions - tag: Identifier for a particular pkgbuild + support files; git tag/svn repos subdir; "$epoch:$pkgver-$pkgrel" - package: A single .pkg.tar.xz file. -debug and split packages count as single packages. # dbscripts DB This is not important to packagers, it's internal to dbscripts and only tracks which packages are in a given repo for history purposes. The db contains files called "$repo/$arch/$pkgname" which contain <<EOF %PKGNAME% $pkgbase %VERSION% $tag EOF The whole tree will be versioned via git. This is just my idea, <https://wiki.archlinux.org/index.php/User:Pierre/pkgdbgit> might be talking about simply extracting the db repo-add creates for pacman and tracking that in git, I don't know. I believe dbscripts wouldn't need a db to perform the functions outlined below so we can talk about this later. # Executables Note: Since releasing/moving/removing to/from any (the arch) doesn't make sense any will be an alias that targets all repos. "db-remove extra any pkg1" will therefore remove pkg1 from i686 and x86_64, similar for "commitpkg -a any ..." which will release to i686 and x86_64. ## commitpkg [-a $arch] (from devtools) - create a tag and upload packages to the server - -any packages default to be released to all arches - if target arch is supplied we only release to one arch (also counts for -any packages) - do we want support for directly supplying the commit message? (commitpkg "update to version 1.2.3") ## commitpkg [-a $arch] pkg1.tar.xz pkg2.tar.xz - if called with arguments it will only release those, nothing else - -any packages will be release to all arches unless -a is used ## db-add $repo $arch pkg1.pkg.tar pkg2.pkg.tar - check if the tag exists in the package repo (extract pkgbase from package) - check if the user has permission to release to the repo - call repo-add on the package ## db-update - call db-add for every package in the user's staging directory ## db-move $srcrepo $dstrepo $arch pkg1 pkg2 - move packages between repos ## db-remove $repo $arch pkg1 pkg2 - call repo-remove # Example pacman, core repo pacman-debug (automatic debug split), core repo pacman-shared (split any arch package), core repo pacman-contrib (normal split), extra repo All these are create by building a single PKGBUILD. db-* scripts still run on the server, but simplicity I'll ignore that below ## Releasing Note: Just create a ./release script in your package repo to do that if a simple foopkg doesn't cut it. - corepkg pacman-xxx.pkg.tar.xz pacman-debug-xxx.tar.xz pacman-shared-xxx.tar.xz - extrapkg pacman-contrib-xxx.tar.xz - db-update ## Removing - db-remove core any pacman pacman-shared pacman-contrib pacman-debug - removes the packages from both arches ## moving from testing to core: - db-move testing core i686 pacman pacman-shared pacman-contrib pacman-debug - moves only i686, leaves x86_64 - to move both arches at once just use any instead of i686
On 01/10/13 03:36, Florian Pritz wrote:
Hi,
After having read the recent git thread I've come to the conclusion that we should rework dbscripts before we start working on the package backend (although my proposal works better with git because it uses tags).
TLDR: You should care and read it. If you don't I won't care if you later miss features that contradicts my proposal.
Two comments: 1) Please use "all" rather than "any" when referring to all architectures. "any" already has a defined use. 2) Does this involve stopping the use of repo-add? I know of at least two projects that aim at replacing it, although none have been proposed on pacman-dev yet. I think it is important for something like repo-add to be tied to pacman, but we could add an option to update an extracted directory (e.g. git repo) rather than a tarball if that would be useful. (Aside: please consider a separate repo for debug packages. The [extra] and [community] databases are already large enough.) Allan
On 01.10.2013 04:29, Allan McRae wrote:
1) Please use "all" rather than "any" when referring to all architectures. "any" already has a defined use.
Will do.
2) Does this involve stopping the use of repo-add?
No, I plan on using repo-add.
(Aside: please consider a separate repo for debug packages. The [extra] and [community] databases are already large enough.)
Noted. Thanks for the review.
On Mon, Sep 30, 2013 at 7:36 PM, Florian Pritz <bluewind@xinu.at> wrote:
## commitpkg [-a $arch] (from devtools) - create a tag and upload packages to the server
This will lead to having thousands of tags in the repo. Does anyone have any experience with git repositories with so many tags? I am thinking about possible performance issues since I don't think this is typical git usage.
On Tue, Oct 1, 2013 at 11:25 AM, Massimiliano Torromeo <massimiliano.torromeo@gmail.com> wrote:
This will lead to having thousands of tags in the repo. Does anyone have any experience with git repositories with so many tags? I am thinking about possible performance issues since I don't think this is typical git usage.
I assume we will use one git repo per PKGBUILD. The amount of tags should stay manageable. Putting all the packages into one repo would create too much merge/rebase contention. Using branches to separate the packages would mean we'd lose the use of branches to manage development, say adding a new linux stable release in [core] while the next major release cooks in [testing]. That has historically been rather painful in our setup, and making it easier would also help us with staging larger projects like GNOME releases without hindering [extra] maintenance. In any case, git got a lot better lately in dealing with many refs.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/01/2013 12:38 PM, Jan Alexander Steffens wrote:
On Tue, Oct 1, 2013 at 11:25 AM, Massimiliano Torromeo <massimiliano.torromeo@gmail.com> wrote:
This will lead to having thousands of tags in the repo. Does anyone have any experience with git repositories with so many tags? I am thinking about possible performance issues since I don't think this is typical git usage.
I assume we will use one git repo per PKGBUILD. The amount of tags should stay manageable.
I really hope this is exactly the case. That slightly complicates creation of new packages though, as server-side support for creating of new repositories is needed instead of just push access. Is this taken into consideration? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSSsuaAAoJEHYfrWm6BsapCYUIAL3eFdv2g13rx7un4VPRtWon XrRvmNwPnG7mqoicABDQuarJ54BnzwGnYu9PUUncfUPbAs1d5iXzF/6UtsyRTiPV Y1RUMv5IJMjVXZijmnLzv0lsDiwiEE7tUDtV+V+ddYjv7P6q6KHaq9+NxVWQ+9g3 66BQRpeFeFHydgwuNNULBaHATPZ6lozA0VFK7OTM7A9jqTolplRvuI0j6JlN2vZf M6RWlLqnPHTr0ZpIddmxaf0GWTekEGQ5mUBP/5oAiUncPCw7wy5gOVaQBU/6Fsex hO0qR2OAvg1VpwTIR/mJ0lcG1cUm0cT3Co+/XpdPNreouBXHJPbsdxBBTzXqv40= =/RHC -----END PGP SIGNATURE-----
On 01.10.2013 15:18, Dicebot wrote:
On 10/01/2013 12:38 PM, Jan Alexander Steffens wrote:
I assume we will use one git repo per PKGBUILD. The amount of tags should stay manageable.
That slightly complicates creation of new packages though, as server-side support for creating of new repositories is needed instead of just push access. Is this taken into consideration?
Should be easily scriptable.
On Mon, Sep 30, 2013 at 1:36 PM, Florian Pritz <bluewind@xinu.at> wrote:
Hi,
After having read the recent git thread I've come to the conclusion that we should rework dbscripts before we start working on the package backend (although my proposal works better with git because it uses tags).
TLDR: You should care and read it. If you don't I won't care if you later miss features that contradicts my proposal.
Note: The text below is more or less a cleaned up brain dump so it's probably not complete and possibly confusing. If so, please tell me. I'm also open for discussion via irc as long as the important points are later sent to the ML.
I would like to keep the support for directly supplying the commit message: (commitpkg "update to version 1.2.3"). It's useful to have the commit message in the bash history when doing same change to several packages, e.g upstream update, soname rebuild, etc. Will the new dbscripts support split packages with differents arch (it seems to be the case from the pacman example)? For example, most docs packages don't use the 'any' arch because the current dbscripts don't support it. If we could support, it we would reduce server disk space and bandwidth. Eric
On 01.10.2013 15:07, Eric Bélanger wrote:
I would like to keep the support for directly supplying the commit message: (commitpkg "update to version 1.2.3"). It's useful to have the commit message in the bash history when doing same change to several packages, e.g upstream update, soname rebuild, etc.
Good point, noted.
Will the new dbscripts support split packages with differents arch (it seems to be the case from the pacman example)?
I intend to support that, yes.
I've put up a wiki page[1] based upon my original mail and the points raised in this thread so far. It also contains a new idea under the "dbscripts DB" heading, but this is not too relevant right now so I'll put this up for discussion later. [1]: https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite Please keep discussion on the ML and don't edit the page without having talked to me first (ML or IRC).
participants (6)
-
Allan McRae
-
Dicebot
-
Eric Bélanger
-
Florian Pritz
-
Jan Alexander Steffens
-
Massimiliano Torromeo