On Thu, Feb 3, 2011 at 6:23 AM, Pierre Schmitz <pierre@archlinux.de> wrote:

On Thu, 3 Feb 2011 05:05:11 -0500, Eric Bélanger wrote:

...

Hi,

Lately, I've started to look into adding support in the dbscripts for splitted packages with mixed arches (i686/x86_64 and any) and/or different pkgver, pkgrel. Depending on my free time and how much time it takes, it could take a while though. I'm just writing to make sure no-one else is doing it so we don't duplicate work.  If anyone wants to help out, let me know.  We'll split the work.

I had put some thoughts into supporting "any" split packages. So far two points need to be looked into: 1) ensure all arches have the same pkgbuild

I suppose we could add a check for this. Diff might be adequate if we ignore the PKGBUILd svn Id tags.

2) check_splitpkgs: get correct arch for svnnames Feel free to send me you ideas or some rough patches for a review.

I got rid of that function. It seemed to be used to make sure split packages all have the same pkgver and pkgrel which is not longer relevant. I suppose it could be readded later on with some of my changes in it. So far, I'm trying to get the correct logic so there's some code duplication that might be moved to a function eventually. more on my ideas below. I could send you my current work or post patches here if other people are interested.

But: I don't like supporting different pkgver or pkgrel for split packages. Imho this is a feature of makepkg that has issues by design: * Source packages with the same file name no longer point to the same packages

I haven't thought about the sourceballs script yet but there might be a way to tell it to update the sourceballs. Also, the reason we have these sourceballs is to comply with licenses. As long as the upstream sources are there, it shouldn't be a major problem if the PKGBUILD inside them is a little bit out-of-date. So it shouldn't be a stopper IMO.

* The repos subdir in svn (might) no longer contains the PKGBUILD that a package was built with

I don't see why or how this would happen. The trunk gets copied to the repos subdir with archrelease.

* Same issue with ABS which is no longer in sync with the actual binary packages

* Parsing the correct pkgver and pkgrel of a sub packages is between hard and nearly impossible Possible and not very hard once you know how to do it. ;) BTW, you need to use the same procedure to get the correct arch for the split

I believe that ABS use svn to gets its data. As I said above, it shouldn't be affected. package. Here's how it's done: #get the default values of variables by sourcing PKGBUILD like below or just do: sourcePKGBUILD pkgname=($(. "PKGBUILD"; echo ${pkgname[@]})) pkgver=$(. "$PKGBUILD"; echo "${pkgver}") pkgrel=$(. "$PKGBUILD"; echo "${pkgrel}") arch=($(. "$PKGBUILD"; echo ${arch[@]})) for _pkgname in "${pkgname[@]}"; do # check if the variables were overridden in the package functions. The awk line gives you the individual package functions for each package. This is what makes it possible. Then all what's left is just gepping and sedding to get the data. unset newpkgver unset newpkgrel unset newarch newpkgver=$(awk "/package_$_pkgname\(/,/^}/" PKGBUILD |grep pkgver= |sed 's/pkgver=//;s/ //g') newpkgrel=$(awk "/package_$_pkgname\(/,/^}/" PKGBUILD |grep pkgrel= |sed 's/pkgrel=//;s/ //g') newarch=($(awk "/package_$_pkgname\(/,/^}/" PKGBUILD |grep arch= |sed "s/arch=//;s/(//;s/)//;s/'//g")) # If variables were overridden in package function, use that value otherwise use the default one if [ -n "$newpkgver" ]; then _pkgver=$newpkgver else _pkgver=$pkgver fi if [ -n "$newpkgrel" ]; then _pkgrel=$newpkgrel else _pkgrel=$pkgrel fi if [ -n "$newarch" ]; then _arch=("${newarch[@]}") else _arch=("${arch[@]}") fi # do stuff echo $_pkgname-$_pkgver-$_pkgrel ${_arch[@]} done I've setup a local repo with svn tree to test this stuff. My test PKGBUILKD produce the following packages: diffarch-1-1-i686.pkg.tar.xz diffarch-1-1-x86_64.pkg.tar.xz diffarch-doc-1-2-any.pkg.tar.xz libdiffarch-1.5-3-i686.pkg.tar.xz libdiffarch-1.5-3-x86_64.pkg.tar.xz As you can see, there quite a mix of different arch, pkgver and pkgrel. With the procedure described above, I was able to upload my test package to my local staging directory and add it in my repo. So it seem that it's going to work.

* We only have one build function; so treating split packages as one unit is best

I don't understang how it's related. I still see them as one unit.

In general this would make things more complicated and introduces quite some points of failure. So I don't think its worth to try to implement this atm. Which packages would benefit from that a lot?

- package that provide docs in a seperate package. Now, if you use a split PKGBUILD, the docs need to be i686/x86_64. If you want the docs in an 'any' arch package, you need to use a seperate PKGBUILD wich is more work for the maintainer. - i18n packages, I've seen it in (open) libreoffice and koffice. Sometime, they don't release new version of some of these i18n package (or release them later) but the old version is still usable. Currently, the package function of these languguage pack is commented out in the PKGBUILD but the package is kept in the repo. So we end up with a broken ABS and the integrity check complains. Support of multi pkgver/pkrel in split package would fix that. - Some split packages provides many packages, think about kdegames which builds 40 packages. If a single game needs to be fixed (missing depends, etc), all the games needs to be packaged, uploaded to repo and downloaded by users. Support of multi pkgver/pkrel in split package will enable us to only build upload the broken one. makepkg can build specific package in a split PKGBUILD. We could add similar support in commitpkg.

There are still some problems with split packages in dbscripts and those are complex enough; so we should be really careful and not make things even worse.

What problems? Are they on the bug tracker?

Greetings,

Pierre

-- Pierre Schmitz, https://users.archlinux.de/~pierre

On Fri, Feb 4, 2011 at 1:49 AM, Pierre Schmitz <pierre@archlinux.de> wrote:

On Thu, 3 Feb 2011 17:33:10 -0500, Eric Bélanger wrote:

...

On Thu, Feb 3, 2011 at 6:23 AM, Pierre Schmitz <pierre@archlinux.de> wrote:

...

2) check_splitpkgs: get correct arch for svnnames Feel free to send me you ideas or some rough patches for a review.

I got rid of that function. It seemed to be used to make sure split packages all have the same pkgver and pkgrel which is not longer relevant. I suppose it could be readded later on with some of my changes in it.  So far, I'm trying to get the correct logic so there's some code duplication that might be moved to a function eventually.

This function should not be removed but improved; e.g. a check if missing if a split package was removed and then automatically remove the split package from the repo. This function ensures that the repo contains the same as abs/svn.

Sure. Once the logic is done, we can add and improve sanity/integrity checks.

...

...

But: I don't like supporting different pkgver or pkgrel for split packages. Imho this is a feature of makepkg that has issues by design: * Source packages with the same file name no longer point to the same packages

I haven't thought about the sourceballs script yet but there might be a way to tell it to update the sourceballs.  Also, the reason we have these sourceballs is to comply with licenses. As long as the upstream sources are there, it shouldn't be a major problem if the PKGBUILD inside them is a little bit out-of-date.  So it shouldn't be a stopper IMO.

...

* The repos subdir in svn (might) no longer contains the PKGBUILD that a package was built with

I don't see why or how this would happen. The trunk gets copied to the repos subdir with archrelease.

We can no longer guarantee that a package in the repo was built with the PKGBUILD in the svn/repos dir or abs. Or in short: you cannot use abs to rebuild the repo.

...

...

* Same issue with ABS which is no longer in sync with the actual binary packages

I believe that ABS use svn to gets its data. As I said above, it shouldn't be affected.

It sure is. What you propose is just being more relaxed on integrity and literally decouple the binary repos from svn.

I think I understand what you mean now. So basically, if I have a split PKGBUILD with multi arch/pkgver/pkgrel and I do a pkgrel bump on the 'any' arch package only, then there will be a mismatch between the PKGBUILD in repos/extra-i686 and the one in repos/extra-any (and similar for extra-x86_64.). Is this the cause of the svn and binary repos decoupling you mentionned and because of that abs won't be able to work? If so, we could make sure that repos/$repo-any is in sync with the $repo-i686 and $repo-x86_64 directory. For example, if I update the 'any' package of a multi-arch split PKGBUILD in extra, the archrelease would run on extra-any and will run on extra-i686 and extra-x86_64 if they exist already. If I update the i686/x86_64 package, then archrelease is ran on i686/x86_64 and on extra-any if it exist already. I believe that would keep the repos and svn coupled. As far as abs goes, for the x86_64 abs tree it would just grab the content of repos/$repo-x86_64. If that directory is missing it would check for repos/$repo-any and add it (so that 'any' arch only PKGBUILD would be in tree). Feel free to let me know if you think it wouldn't work or would over-complicate things. This is not a minor

straight forward change and should be done right. In fact I already had some ideas about this. I could send a more detailed mail if wanted. In short it repos like this: * Use svn just for trunk (or other branches) but no longer try to "tag" what is actually in the repos. Means removing the repos subdir completely.

That would make it difficult to quickly know what PKGBUILD was used to build the packags in the repos as trunk only contains the latest version of the PKGBUILD. With the repos directory, it's easy and fast to know what package version is in extra, whihc one for testing and what are latest chenges in trunk. Of course, we can use svn log or cgit but how dbscripts will be able to do this? Unless we don't check anymore the svn against the packages in the repos, I can't see how it could be done. Wouldn't that decouple the svn from the repos completely?

* Instead switch to a pure package based approach. dbscripts will only know about single packages. (would still need an integrity check to search for removed split packages)

That might simplyfy things. So it might be worthwhile to switch to this proposed setup instead of my above suggestion above to modify the current one. I'm not decided yet.

* Improve makepkg to create source packages with every build. Should be like $pkgbase-$pkgver-$pkgrel-$arch.src.tar.xz

I suppose that would be enabled with a switch. And that also you're talking about "makepkg --source" and not "makepkg --allsource" otherwise we'll have disk space and bandwith issues.

* Upload these source packages together with the binary packages and store them in another repo * If you need the pkgbuild of a package (e.g. for abs) just look for a source package that corresponds to the scheme mentioned above * Note that the db files is the only place where information are stored in which repo a package exists.

Summing up: We really should not try to support different versions of split packages with our current svn repo setup. Instead we should rethink the whole setup first as described above. Then we can easily implement this in a clean way.

As things seem to be more complicated than I initially thought, I've stopped my work on the scripts for the time being until we agree on the best approach. FWIW, I posted my modified scripts here: https://dev.archlinux.org/~eric/reposcripts/ in case someone's interested. If we go with Pierre's proposed setup, then probably the only thing worth to keep is the logic to get each packages version and arch that I posted early in this thread. Eric

Greetings,

Pierre

-- Pierre Schmitz, https://users.archlinux.de/~pierre