[arch-dev-public] Rewriting dbscripts (packagers should read this!)
Hi,
After having read the recent git thread I've come to the conclusion that
we should rework dbscripts before we start working on the package
backend (although my proposal works better with git because it uses tags).
TLDR: You should care and read it. If you don't I won't care if you
later miss features that contradicts my proposal.
Note: The text below is more or less a cleaned up brain dump so it's
probably not complete and possibly confusing. If so, please tell me. I'm
also open for discussion via irc as long as the important points are
later sent to the ML.
# Goal
Decouple dbscripts from svn to make it faster and simpler. Moving lots
of packages is awefully slow right now.
svn tracks the PKGBUILDS and tags refering to certain PKGBUILDs.
dbscripts maintains an internal, separate database which tracks in which
repo a given package is (identified by a pkgname and a tag).
# Definitions
- tag: Identifier for a particular pkgbuild + support files;
git tag/svn repos subdir; "$epoch:$pkgver-$pkgrel"
- package: A single .pkg.tar.xz file. -debug and split packages
count as single packages.
# dbscripts DB
This is not important to packagers, it's internal to dbscripts and only
tracks which packages are in a given repo for history purposes.
The db contains files called "$repo/$arch/$pkgname" which contain <
On 01/10/13 03:36, Florian Pritz wrote:
Hi,
After having read the recent git thread I've come to the conclusion that we should rework dbscripts before we start working on the package backend (although my proposal works better with git because it uses tags).
TLDR: You should care and read it. If you don't I won't care if you later miss features that contradicts my proposal.
Two comments: 1) Please use "all" rather than "any" when referring to all architectures. "any" already has a defined use. 2) Does this involve stopping the use of repo-add? I know of at least two projects that aim at replacing it, although none have been proposed on pacman-dev yet. I think it is important for something like repo-add to be tied to pacman, but we could add an option to update an extracted directory (e.g. git repo) rather than a tarball if that would be useful. (Aside: please consider a separate repo for debug packages. The [extra] and [community] databases are already large enough.) Allan
On 01.10.2013 04:29, Allan McRae wrote:
1) Please use "all" rather than "any" when referring to all architectures. "any" already has a defined use.
Will do.
2) Does this involve stopping the use of repo-add?
No, I plan on using repo-add.
(Aside: please consider a separate repo for debug packages. The [extra] and [community] databases are already large enough.)
Noted. Thanks for the review.
On Mon, Sep 30, 2013 at 7:36 PM, Florian Pritz
## commitpkg [-a $arch] (from devtools) - create a tag and upload packages to the server
This will lead to having thousands of tags in the repo. Does anyone have any experience with git repositories with so many tags? I am thinking about possible performance issues since I don't think this is typical git usage.
On Tue, Oct 1, 2013 at 11:25 AM, Massimiliano Torromeo
This will lead to having thousands of tags in the repo. Does anyone have any experience with git repositories with so many tags? I am thinking about possible performance issues since I don't think this is typical git usage.
I assume we will use one git repo per PKGBUILD. The amount of tags should stay manageable. Putting all the packages into one repo would create too much merge/rebase contention. Using branches to separate the packages would mean we'd lose the use of branches to manage development, say adding a new linux stable release in [core] while the next major release cooks in [testing]. That has historically been rather painful in our setup, and making it easier would also help us with staging larger projects like GNOME releases without hindering [extra] maintenance. In any case, git got a lot better lately in dealing with many refs.
On Mon, Sep 30, 2013 at 1:36 PM, Florian Pritz
Hi,
After having read the recent git thread I've come to the conclusion that we should rework dbscripts before we start working on the package backend (although my proposal works better with git because it uses tags).
TLDR: You should care and read it. If you don't I won't care if you later miss features that contradicts my proposal.
Note: The text below is more or less a cleaned up brain dump so it's probably not complete and possibly confusing. If so, please tell me. I'm also open for discussion via irc as long as the important points are later sent to the ML.
I would like to keep the support for directly supplying the commit message: (commitpkg "update to version 1.2.3"). It's useful to have the commit message in the bash history when doing same change to several packages, e.g upstream update, soname rebuild, etc. Will the new dbscripts support split packages with differents arch (it seems to be the case from the pacman example)? For example, most docs packages don't use the 'any' arch because the current dbscripts don't support it. If we could support, it we would reduce server disk space and bandwidth. Eric
On 01.10.2013 15:07, Eric Bélanger wrote:
I would like to keep the support for directly supplying the commit message: (commitpkg "update to version 1.2.3"). It's useful to have the commit message in the bash history when doing same change to several packages, e.g upstream update, soname rebuild, etc.
Good point, noted.
Will the new dbscripts support split packages with differents arch (it seems to be the case from the pacman example)?
I intend to support that, yes.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/01/2013 12:38 PM, Jan Alexander Steffens wrote:
On Tue, Oct 1, 2013 at 11:25 AM, Massimiliano Torromeo
wrote: This will lead to having thousands of tags in the repo. Does anyone have any experience with git repositories with so many tags? I am thinking about possible performance issues since I don't think this is typical git usage.
I assume we will use one git repo per PKGBUILD. The amount of tags should stay manageable.
I really hope this is exactly the case. That slightly complicates creation of new packages though, as server-side support for creating of new repositories is needed instead of just push access. Is this taken into consideration? -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSSsuaAAoJEHYfrWm6BsapCYUIAL3eFdv2g13rx7un4VPRtWon XrRvmNwPnG7mqoicABDQuarJ54BnzwGnYu9PUUncfUPbAs1d5iXzF/6UtsyRTiPV Y1RUMv5IJMjVXZijmnLzv0lsDiwiEE7tUDtV+V+ddYjv7P6q6KHaq9+NxVWQ+9g3 66BQRpeFeFHydgwuNNULBaHATPZ6lozA0VFK7OTM7A9jqTolplRvuI0j6JlN2vZf M6RWlLqnPHTr0ZpIddmxaf0GWTekEGQ5mUBP/5oAiUncPCw7wy5gOVaQBU/6Fsex hO0qR2OAvg1VpwTIR/mJ0lcG1cUm0cT3Co+/XpdPNreouBXHJPbsdxBBTzXqv40= =/RHC -----END PGP SIGNATURE-----
On 01.10.2013 15:18, Dicebot wrote:
On 10/01/2013 12:38 PM, Jan Alexander Steffens wrote:
I assume we will use one git repo per PKGBUILD. The amount of tags should stay manageable.
That slightly complicates creation of new packages though, as server-side support for creating of new repositories is needed instead of just push access. Is this taken into consideration?
Should be easily scriptable.
I've put up a wiki page[1] based upon my original mail and the points raised in this thread so far. It also contains a new idea under the "dbscripts DB" heading, but this is not too relevant right now so I'll put this up for discussion later. [1]: https://wiki.archlinux.org/index.php/User:Bluewind/dbscripts-rewrite Please keep discussion on the ML and don't edit the page without having talked to me first (ML or IRC).
participants (6)
-
Allan McRae
-
Dicebot
-
Eric Bélanger
-
Florian Pritz
-
Jan Alexander Steffens
-
Massimiliano Torromeo