[aur-dev] Using git as a backend for the AUR

[Jelle van der Waa]

On 16/03/13 23:10, William Giokas wrote:
> All,
> 
> So in my spare time I was thinking about the AUR and how it could be
> better. Back in January I commented on a bug[1] about integrating the
> AUR and git to have a powerful, robust backend for the AUR. I think that
> the original idea of creating one massive repository was inherently
> flawed for most users, as it requires that the entire repository be
> cloned, and that is no trivial task for people with only moderate
> internet speeds. Were it similar to the way the official repositories
> git setup is, then it would be okay for fetching, but using single
> repositories for each project would make access control much less of a
> nightmare.
> 
> At the moment I have no experience with PHP or messing with the AUR, but
> I have been playing around with git and using it to track packages and
> the like, as well as some minimal experience with access control and
> git.
> 
> Currently, this is kind of the roadmap I see in my head:
> 
> * Set up access controls for the AUR users to access over ssh using
>   keys. While I could see this being somewhat arduous, it shouldn't be
>   terribly hard to automate or set up. Something like gitolite or
>   whatever should make it simpler.
> * Each repository on the server would contain a single package (if
>   someone decides to do a split package on the AUR it would contain the
>   whole set of packages), allowing for multiple users to have push
>   access and maintain the packages.
The AUR currently has ~ 41076 packages, which would bring us to an equal
amount of git repo's. This doesn't seem very efficient to me.

With svn you can put it all in one repo and checkout a single package
without checking out the whole repo, which is how
[core],[extra],[community] work.

I know Exherbo uses Git for their repo's but they seem to split it up
into categories [1].

Git seems to have sparse checkout for this in 1.7, but I don't know how
it works [2]
> * The repositories would be limited to 5M, unless given special
>   permission (certain kernel packages with lots of massive configuration
>   files) which should make enough room for the needed files, while
>   helping to enforce the idea of not putting retrievable sources in the
>   source tarball. I did some quick math using the number of packages on
>   the AUR right now, and if every repo used that 5M limit, it would
>   require a bit over 200G. Granted, I doubt that this limit would be
>   reached for most packages. 
> * With the repository set up, give maintainers a week or so to upload
>   history for their packages (in case they already keep their repos in
>   this set up), and new packages would get put directly into a new repo.
> * Once the week for transition is over, begin moving all of the current
>   packages in the AUR to the git format.
> 
> This basically concludes setting up the git stuff, which I have done on
> a local machine on a very small scale (5 packages) in the past. One
> thing I was thinking of was the use of `makepkg -S` tarballs still being
> available. cryptocrack came up with a good point, that just simply
> copying things over could overwrite the .git directory in the designated
> directory. I don't think that would be too hard to get around, but I'm no
> expert. I wrote a quick and dirty script that extracts and commits to
> the respective repository. Currently, almost no packages use a
> .changelog file, but there could be some minimal parsing done of a
> packages .changelog file to craft a git commit message from a tarball. 
> Obviously there needs to be some kind of security check, but that could be
> done by the frontend using the system we have now for source tarballs.
> 
> At the moment, I am unsure exactly how the AUR parses the PKGBUILDs it
> receives, but I'm sure that this could easily be adapted to the git
> system.
> 
> One major advantage to having the AUR managed this way is that it would
> allow users to check for updates on the AUR without the need for complex
> helpers outside of git. Secondly, it would make mirroring the AUR, or
> just parts of the AUR extremely easy. In case something happens to the
> AUR, people can still get their packages and maintainers can still
> update them very easily using git. It would also still allow people to
> grab tarballs of the current master branch. This way git is not a
> requirement to use the AUR, it just makes things a butt ton easier.
> Also, it would allow users to see the entire source of the package, not
> just the PKGBUILD, similar to how the official repository works, just
> that things would be in distinct git repositories not branches.
> 
> I would really like to see this kind of thing come to fruition, and if
> anyone would be willing to help, then please say so.
> 
> 
> Pardon my scattered thoughts,
> 

Although I like the idea, I really think it needs more thoughts ;). So
thanks for bringing it up.

[1] http://git.exherbo.org/summer/
[2] http://schacon.github.com/git/RelNotes/1.7.0.txt

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 555 bytes
Desc: OpenPGP digital signature
URL: <http://mailman.archlinux.org/pipermail/aur-dev/attachments/20130317/fee4dd00/attachment.asc>