On Thu, Jun 11, 2015 at 9:28 PM, Phillip Smith <fukawi2@gmail.com> wrote:

I have already seen your work (thanks by the way, those hooks look great), but unless I'm missing something there, there's nothing to help me understand how to use subtrees etc in this scenario.

I have Googled and read various blogs, maillist posts and GitHub's help page on the subject but I can't related any of it to how I should be doing it with AUR4. The Arch wiki seems rather desolate on the topic at the moment too; that would be the best place for such information I guess.

Use the `./aurpublish` script. It controls everything. Today I added a "speedup" option to save the subtree (it duplicates all those commits though). My goal is that *everything* should be abstracted and just leave you with a simple repo of folders. :) -- Eli Schwartz

On 12 June 2015 at 12:54, Eli Schwartz <eschwartz93@gmail.com> wrote:

Use the `./aurpublish` script. It controls everything. Today I added a "speedup" option to save the subtree (it duplicates all those commits though).

My goal is that *everything* should be abstracted and just leave you with a simple repo of folders. :)

I'm not seeing how that sets up subtrees? The only subtree command in the script is a "push" command.

On 15 June 2015 at 15:25, Eli Schwartz <eschwartz93@gmail.com> wrote:

Like I said, you should be able to pretend it is one vanilla repo, just like many people used to track their AUR3 packages. But .SRCINFO is required for this one, because each folder is *exported* to the AUR4.

Riiiight! Gotcha! That makes a lot more sense -- I was assuming there was something required within the "umbrella" repo to setup/configure the child repos etc. Thanks, I'll go have a play :)

On 16 June 2015 at 14:26, Eli Schwartz <eschwartz93@gmail.com> wrote:
>
> FWIW you can use filter-branch to run mksrcinfo on each folder in your
> current aur-packages repo, then merge in my base branch and continue on as
> before. Might have to resolve a conflict between our .gitignores, (I may be
> biased but I prefer mine :-P ) but other than that it should work fine.

Thanks for your ongoing interest -- I think I've royally screwed it
all though.... Because I've submitted all my packages to AUR4 using
individual (fresh) git repos, now I can't push using `aurpublish`
because there's no common commits.

I tried your 'adopt' option to aurpublish to pull the repo from the
AUR server into my umbrella git repo, but still won't let me push:

 ~/aur-packages $ ./aurpublish stenc
git push using:  aur:/stenc.git master
To aur:/stenc.git
 ! [rejected]        89b88136c6e0b9bf70f0975ebebedd2a05c9f3fa ->
master (non-fast-forward)
error: failed to push some refs to 'aur:/stenc.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

~/aur-packages $ git subtree pull -P stenc aur:/stenc.git master
>From aur:/stenc
 * branch            master     -> FETCH_HEAD
Already up-to-date.


Thanks for you patience with my cluelessness with this!

On 17 June 2015 at 09:00, Joost Bremmer <toost.b@gmail.com> wrote:

Just init a fresh repo, then add your packages and run:

git subtree add -P stenc aur:/stenc.git master

That looks like the command I've been missing this whole time.... Thanks! :) I have 50-odd packages, but a quick bash loop makes it less painful to add them all.

On 6/16/2015 6:00 PM, Joost Bremmer wrote:

Just init a fresh repo, then add your packages and run: git subtree add -P stenc aur:/stenc.git master Note that you'll have to make a bogus first commit (for git rev-parse as I understand it), just an empty README.md will do, you can even remove it after if you don't want it.

FWIW -- I kind of intended that the helper scripts be tracked in the repo, which is why I have a base branch for building off. -- Eli Schwartz

On Thu, Jun 11, 2015 at 12:04 AM, David Kaylor <dpkaylor@gmail.com> wrote:

How about a simpleton who has never pushed, pulled, or committed at all? I maintain only two packages, both very simplistic, and not updated upstream very often. So a git repository of my PKGBUILDS was not something I thought was necessary. As a network admin, SSH is certainly nothing new. Git, not so much. I'm not a developer. My only real exposure has been news of its creation (I remember the first news of Linus's disgust with the Bitkeeper changes), its emergence as THE solution to distributed revision control, and the Bup backup tool (because I regularly backup large VM images and like the block-level data deplucation advantages.) At this point, enough info is available that I know what needs to be done. But it would have been nice if the initial announcements didn't assume that all maintainers, or even most, used Git in their workflow.

IIRC there was some discussion about this originally, and someone suggested that maintainers might even decide to stop maintaining. The suggestion was made that perhaps a tarball upload could be added which would create automatic commits to the git repository. Hopefully, something like that could be added, it would allow people who don't want to deal with the added complexity to continue on as though nothing had ever happened. -- Eli Schwartz

* Joakim Hernberg <jhernberg@alchemy.lu> [2015-06-11 21:03:50 +0200]:

On Thu, 11 Jun 2015 13:36:54 -0500 Eli Schwartz <eschwartz93@gmail.com> wrote:

...

On Thu, Jun 11, 2015 at 12:04 AM, David Kaylor <dpkaylor@gmail.com> wrote:

...

How about a simpleton who has never pushed, pulled, or committed at all?

...

IIRC there was some discussion about this originally, and someone suggested that maintainers might even decide to stop maintaining.

The suggestion was made that perhaps a tarball upload could be added which would create automatic commits to the git repository. Hopefully, something like that could be added, it would allow people who don't want to deal with the added complexity to continue on as though nothing had ever happened.

IMO this would be a good idea.

FWIW, +1 here as well. I love git so I'm excited about all the new possibilites and feel like it's a very good improvement - but I can see how this makes it harder for people who'd prefer not getting into git just to contribute to the AUR. Florian -- http://www.the-compiler.org | me@the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc I love long mails! | http://email.is-not-s.ms/

On Thu, Jun 11, 2015 at 4:01 PM, Lukas Fleischer <lfleischer@archlinux.org> wrote:

I am against adding anything that involves uploading a tarball. I am not averse to including a set of utilities to help with working with to AUR to [community], though (as long as there isn't anything similar to a package manager for the AUR -- please let's not discuss this here).

There could be a simple tool called aur-pkgsubmit that roughly does the following:

1. Check whether the current directory is the top-level directory of a Git repository, otherwise initialize a new repository.

2. Run `mksrcinfo` or something equivalent.

3. `git add` the PKGBUILD, .SRCINFO and other source files.

4. Commit the changes with a predefined commit message that can be edited. If the HEAD commit is not pushed yet, amend the previous commit instead of creating a new one.

5. Setup Git remotes according to the package base name, unless configured already.

6. Push the changes.

Users not knowing Git would be able to just update their PKGBUILDs and run that tool to submit the package, without even knowing that Git is used under the hood.

What do you think?

Regards, Lukas

That still requires packagers to go through several new hoops. And I can see the benefit of allowing tarball uploads for the same reason that the AUR3 included a web interface for adding new packages (rather than having a hard requirement for using a helper). e.g. uploading changes from another computer while traveling. What happens if the package already exists? Say, the packager switches to a new computer. Does the tool also setup an ssh key and add that key to the packager's account? Installing tools to help manage AUR packages is definitely a useful option to have -- but it would be nice if it wasn't *the only way* to do so. I would like to see a method for submitting a tarball, having the aurweb checkout the package's master branch, overwrite the index with the tarball contents, and commit that with a generic message. (e.g. "upgpkg: ${pkgname} ${pkgver}-${pkgrel}"). Aside from the need to actually spend time implementing that, is there any reason not to? -- Eli Schwartz

On Thu, Jun 11, 2015 at 8:44 PM, David Kaylor <dpkaylor@gmail.com> wrote:

I have so few packages to maintain, the burden to me is very low. And I'm just taking it as oppurtunity to finally learn to use Git. But a tool for submitting a tarball and having AUR4 do the conversion is a good one. Might help prrevent a massive pile of orphans after July 8th. Maintainers are going to have to jump through the hoops, though, one way or the other.

Well, sure, git is fun. :) But tarball uploads means maintainers wouldn't have to go through any more hoops than they do already. Primarily, I am wondering what people who refuse to learn git *just* to contribute to the AUR might think. Any local abstraction tool would have to be fairly robust to deal with e.g. authentication, first-time updating a package that already exists, and in combination with transferring to another computer. vs. providing a compatibility layer over the old tarball interface, since aurweb already has everything else. -- Eli Schwartz

On Thu, Jun 11, 2015 at 10:03 PM, Eli Schwartz <eschwartz93@gmail.com> wrote:

On Thu, Jun 11, 2015 at 8:44 PM, David Kaylor <dpkaylor@gmail.com> wrote:

...

I have so few packages to maintain, the burden to me is very low. And I'm just taking it as oppurtunity to finally learn to use Git. But a tool for submitting a tarball and having AUR4 do the conversion is a good one. Might help prrevent a massive pile of orphans after July 8th. Maintainers are going to have to jump through the hoops, though, one way or the other.

Well, sure, git is fun. :)

But tarball uploads means maintainers wouldn't have to go through any more hoops than they do already. Primarily, I am wondering what people who refuse to learn git *just* to contribute to the AUR might think. Any local abstraction tool would have to be fairly robust to deal with e.g. authentication, first-time updating a package that already exists, and in combination with transferring to another computer.

vs. providing a compatibility layer over the old tarball interface, since aurweb already has everything else.

-- Eli Schwartz

I was surprised that the AUR devs didn't include an easy "press a button on the website" or "ssh aur4 setup-repos-seed-with-aur-mirror" way to import a package from the http://pkgbuild.com/git/aur-mirror.git/ git repository... It's unofficial, but it has the daily history. Perhaps an easy solution is to offer such a service on the AUR4 website from a local clone of that aur-mirror git repository to seed the repository, and then also offer a tarball upload ssh command (e.g. "cat example.1.src.tar.gz | ssh aur4 process-tarball", where process-tarball is just a script that untars, commits, and pushes the contents on the server side as-if it was the user)?

On Fri, Jun 12, 2015 at 5:34 AM, Lukas Fleischer <lfleischer@archlinux.org> wrote:

No. It also does not register a new AUR account or setup your Internet connection. It submits packages to the AUR. As I said before, generating and adding the key is a tiny one-time process and automating it doesn't pay off.

Agreed, it's trivial.

* We would have to reintroduce the Archive::Tar library which we had several issues with in the past and were very happy to get rid of.

* You need to be very careful when extracting tarballs. It is quite easy to build ZIP bombs. This is one of the reasons we only extracted the PKGBUILD (and no other files) when the AUR submissions still required uploading source tarballs. What you suggest would require extracting everything, though.

* We would have to create checkouts for the Git repositories that are submitted via tarballs. We put a lot of time into making the storage as space efficient as possible, using a shared object storage with gitnamespaces. We can currently store the ~10000 packages uploaded so far with <40MB disk usage. Creating checkouts would mean that this increases by a factor of ~20. Creating and destroying checkouts on the fly is also quite ugly and something I would like to avoid.

* The AUR web interface is written in PHP, the Git backend is written in Python. We would either have to duplicate all the sanity checks or create some weird interface between the tools that involves reading and converting error messages from the standard file descriptors.

Agreed, sounds like it's non-trivial, and not maintainable.

Also, I do not understand all the fuss about Git. We don't expect people to be a Git expert. There are a lot of tutorials and there are detailed explanations in the Arch wiki. You should be able to submit a package by only copy-pasting snippets from the wiki without even understanding what is going on.

If people really refuse to learn the five most basic Git commands, they better don't maintain any AUR packages. Maintaining packages means that you are willing to understand basic packaging and VCS tools. Tracking down issues with a package often involves using the upstream VCS and nowadays, most projects use Git.

Yep, agreed. Time to learn, and in any case, not that difficult.

It is expected (and intended) that some (hopefully not too many) AUR package maintainers back out due to the new system. It is also intended that a huge number of packages is not imported into the new AUR and anything that automatically transfers packages is undesirable. We only want users to resubmit their stuff if they want to take care of it in the future. Think of it as a huge AUR cleanup.

And finally, you state your position; cull the herd. Again, I don't disagree, but it's usually best to state intentions from the very beginning. So make sure that statement is at the very head of the wiki related to the transition to AUR4. "Basic working knowledge of Git is expected of all AUR package maintainers. Otherwise, sod off."

On Fri, Jun 12, 2015 at 8:22 AM, Lukas Fleischer <lfleischer@archlinux.org> wrote:

"Culling the herd" (using your words, even though I dislike that metaphor) is something that always happens when you introduce a new system. Users need time to get used to it, some of them back out. Just to clarify: We don't want to make the AUR a place for elitists and we are trying to make it as convenient as possible for users to continue contributing. On the other hand, though, we also want to use the chance of having to reimport all packages to improve the overall quality of the AUR (less broken and unmaintained packages, less clutter). We also need to keep an eye on maintainability: I am the only regular aurweb contributor these days.

Regarding your request to state intentions from the very beginning, the first sentence in the AUR 4 section of the Arch wiki is:

Since release 4.0.0, aurweb uses Git repositories for AUR packages which means that the package submission process is a bit different.

In my opinion, that makes it pretty obvious that maintainers need to know or learn Git. If you disagree, feel free to improve that section, it is a wiki!

Regards, Lukas

I just tweaked the language in the wiki a bit. My point is that if one of the objectives is to force more discipline among maintainers going forward, then there is no reason to hide that fact. I suspect it would be more accurate to say you see it as a "fortunate side affect" of modernizing and increasing the maintainable of the AUR, which I think you have achieved with AUR4. And I agree with you on that. As an aside, the english language phrases "culling the herd", "thinning the herd", etc. need not have any negative connotations. Harsh as it may be, modern humans wouldn't exist otherwise. Nor would Arch, and numerous other distros. Or the Linux kernel in general, for that matter.

On Fri, Jun 12, 2015 at 4:34 AM, Lukas Fleischer <lfleischer@archlinux.org> wrote:

On Fri, 12 Jun 2015 at 01:42:41, Eli Schwartz wrote:

...

[...] That still requires packagers to go through several new hoops. [...]

Which ones? Can't think of any apart from generating the key and adding it to the web interface. It literally takes ~1 minute. You need to do the same thing after creating an account on a Git hosting platform like GitHub.

...

And I can see the benefit of allowing tarball uploads for the same reason that the AUR3 included a web interface for adding new packages (rather than having a hard requirement for using a helper). e.g. uploading changes from another computer while traveling.

What happens if the package already exists? Say, the packager switches to a new computer.

What's the issue here? `git clone` will clone the existing repository. Might want to use some options such that it doesn't actually check out HEAD. Or have another command to pull changes before submitting.

If you work on two machines in parallel, you will get a merge conflict or a denied push, sure. But that's an inherent issue. When using the tarball submission process you would overwrite changes from the other machine which is way worse than something telling you that there is something wrong.

...

Does the tool also setup an ssh key and add that key to the packager's account?

No. It also does not register a new AUR account or setup your Internet connection. It submits packages to the AUR. As I said before, generating and adding the key is a tiny one-time process and automating it doesn't pay off.

...

[...] I would like to see a method for submitting a tarball, having the aurweb checkout the package's master branch, overwrite the index with the tarball contents, and commit that with a generic message. (e.g. "upgpkg: ${pkgname} ${pkgver}-${pkgrel}").

Aside from the need to actually spend time implementing that, is there any reason not to? [...]

Just to name a few issues:

* We would have to reintroduce the Archive::Tar library which we had several issues with in the past and were very happy to get rid of.

* You need to be very careful when extracting tarballs. It is quite easy to build ZIP bombs. This is one of the reasons we only extracted the PKGBUILD (and no other files) when the AUR submissions still required uploading source tarballs. What you suggest would require extracting everything, though.

* We would have to create checkouts for the Git repositories that are submitted via tarballs. We put a lot of time into making the storage as space efficient as possible, using a shared object storage with gitnamespaces. We can currently store the ~10000 packages uploaded so far with <40MB disk usage. Creating checkouts would mean that this increases by a factor of ~20. Creating and destroying checkouts on the fly is also quite ugly and something I would like to avoid.

* The AUR web interface is written in PHP, the Git backend is written in Python. We would either have to duplicate all the sanity checks or create some weird interface between the tools that involves reading and converting error messages from the standard file descriptors.

Also, I do not understand all the fuss about Git. We don't expect people to be a Git expert. There are a lot of tutorials and there are detailed explanations in the Arch wiki. You should be able to submit a package by only copy-pasting snippets from the wiki without even understanding what is going on.

If people really refuse to learn the five most basic Git commands, they better don't maintain any AUR packages. Maintaining packages means that you are willing to understand basic packaging and VCS tools. Tracking down issues with a package often involves using the upstream VCS and nowadays, most projects use Git.

It is expected (and intended) that some (hopefully not too many) AUR package maintainers back out due to the new system. It is also intended that a huge number of packages is not imported into the new AUR and anything that automatically transfers packages is undesirable. We only want users to resubmit their stuff if they want to take care of it in the future. Think of it as a huge AUR cleanup.

Regards, Lukas

Thanks for the explanation. :) Sounds like there really are issues beyond simply writing the implementation (which *could've* been a nice-to-have-someday request). I agree people should understand the system, and I understand the AUR is being cleaned up. I suppose it would be too much effort and stress to properly support any sort of bridge. Hopefully most AUR maintainers will switch over. -- Eli Schwartz