[arch-dev-public] Adding multi arch, pkgver and/or pkgrel support in repo scripts

[Eric Bélanger]

On Fri, Feb 4, 2011 at 1:49 AM, Pierre Schmitz <pierre at archlinux.de> wrote:
> On Thu, 3 Feb 2011 17:33:10 -0500, Eric Bélanger wrote:
>> On Thu, Feb 3, 2011 at 6:23 AM, Pierre Schmitz <pierre at archlinux.de> wrote:
>>> 2) check_splitpkgs: get correct arch for svnnames
>>> Feel free to send me you ideas or some rough patches for a review.
>>
>> I got rid of that function. It seemed to be used to make sure split
>> packages all have the same pkgver and pkgrel which is not longer
>> relevant. I suppose it could be readded later on with some of my
>> changes in it.  So far, I'm trying to get the correct logic so there's
>> some code duplication that might be moved to a function eventually.
>
> This function should not be removed but improved; e.g. a check if
> missing if a split package was removed and then automatically remove the
> split package from the repo. This function ensures that the repo
> contains the same as abs/svn.

Sure. Once the logic is done, we can add and improve sanity/integrity checks.

>
>>> But: I don't like supporting different pkgver or pkgrel for split
>>> packages. Imho this is a feature of makepkg that has issues by design:
>>> * Source packages with the same file name no longer point to the same
>>> packages
>>
>> I haven't thought about the sourceballs script yet but there might be
>> a way to tell it to update the sourceballs.  Also, the reason we have
>> these sourceballs is to comply with licenses. As long as the upstream
>> sources are there, it shouldn't be a major problem if the PKGBUILD
>> inside them is a little bit out-of-date.  So it shouldn't be a stopper
>> IMO.
>>
>>> * The repos subdir in svn (might) no longer contains the PKGBUILD that
>>> a package was built with
>>
>> I don't see why or how this would happen. The trunk gets copied to the
>> repos subdir with archrelease.
>
> We can no longer guarantee that a package in the repo was built with
> the PKGBUILD in the svn/repos dir or abs. Or in short: you cannot use
> abs to rebuild the repo.
>
>>> * Same issue with ABS which is no longer in sync with the actual binary
>>> packages
>>
>> I believe that ABS use svn to gets its data. As I said above, it
>> shouldn't be affected.
>
> It sure is. What you propose is just being more relaxed on integrity
> and literally decouple the binary repos from svn.

I think I understand what you mean now.  So basically, if I have a
split PKGBUILD with multi arch/pkgver/pkgrel
and I do a pkgrel bump on the 'any' arch package only, then there will
be a mismatch between the PKGBUILD in repos/extra-i686 and the one in
repos/extra-any (and similar for extra-x86_64.). Is this the cause of
the svn and binary repos decoupling you mentionned and because of that
abs won't be able to work?

If so, we could make sure that repos/$repo-any is in sync with the
$repo-i686 and $repo-x86_64 directory.
For example, if I update the 'any' package of a multi-arch split
PKGBUILD in extra, the archrelease would run on extra-any and will run
on extra-i686 and extra-x86_64 if they exist already. If I update the
i686/x86_64 package, then archrelease is ran on i686/x86_64 and on
extra-any if it exist already.  I believe that would keep the repos
and svn coupled.  As far as abs goes, for the x86_64 abs tree it would
just grab the content of repos/$repo-x86_64.  If that directory is
missing it would check for repos/$repo-any and add it (so that 'any'
arch only PKGBUILD would be in tree).  Feel free to let me know if you
think it wouldn't work or would over-complicate things.

This is not a minor
> straight forward change and should be done right. In fact I already had
> some ideas about this. I could send a more detailed mail if wanted. In
> short it repos like this:
> * Use svn just for trunk (or other branches) but no longer try to "tag"
> what is actually in the repos. Means removing the repos subdir
> completely.

That would make it difficult to quickly know what PKGBUILD was used to
build the packags in the repos as trunk only contains the latest
version of the PKGBUILD.  With the repos directory, it's easy and fast
to know what package version is in extra, whihc one for testing and
what are latest chenges in trunk. Of course, we can use svn log or
cgit but how  dbscripts will be able to do this? Unless we don't check
anymore the svn against the packages in the repos, I can't see how it
could be done. Wouldn't that decouple the svn from the repos
completely?

> * Instead switch to a pure package based approach. dbscripts will only
> know about single packages. (would still need an integrity check to
> search for removed split packages)

That might simplyfy things. So it might be worthwhile to switch to
this proposed setup instead of my above suggestion above to modify the
current one. I'm not decided yet.

> * Improve makepkg to create source packages with every build. Should be
> like $pkgbase-$pkgver-$pkgrel-$arch.src.tar.xz

I suppose that would be enabled with a switch. And that also you're
talking about "makepkg --source" and not "makepkg --allsource"
otherwise we'll have disk space and bandwith issues.

> * Upload these source packages together with the binary packages and
> store them in another repo
> * If you need the pkgbuild of a package (e.g. for abs) just look for a
> source package that corresponds to the scheme mentioned above
> * Note that the db files is the only place where information are stored
> in which repo a package exists.
>
>
> Summing up: We really should not try to support different versions of
> split packages with our current svn repo setup. Instead we should
> rethink the whole setup first as described above. Then we can easily
> implement this in a clean way.
>

As things seem to be more complicated than I initially thought,  I've
stopped my work on the scripts for the time being until we agree on
the best approach. FWIW, I posted my modified scripts here:
https://dev.archlinux.org/~eric/reposcripts/
in case someone's interested. If we go with Pierre's proposed  setup,
then probably the only thing worth to keep is the logic to get each
packages version and arch that I posted early in this thread.

Eric

> Greetings,
>
> Pierre
>
> --
> Pierre Schmitz, https://users.archlinux.de/~pierre
>