[aur-dev] Using git as a backend for the AUR

[Daniel Wallace]

On Sun, Mar 17, 2013 at 12:43:01PM +0100, Jelle van der Waa wrote:
> On 16/03/13 23:10, William Giokas wrote:
> > All,
> > 
> > So in my spare time I was thinking about the AUR and how it could be
> > better. Back in January I commented on a bug[1] about integrating the
> > AUR and git to have a powerful, robust backend for the AUR. I think that
> > the original idea of creating one massive repository was inherently
> > flawed for most users, as it requires that the entire repository be
> > cloned, and that is no trivial task for people with only moderate
> > internet speeds. Were it similar to the way the official repositories
> > git setup is, then it would be okay for fetching, but using single
> > repositories for each project would make access control much less of a
> > nightmare.
> > 
> > At the moment I have no experience with PHP or messing with the AUR, but
> > I have been playing around with git and using it to track packages and
> > the like, as well as some minimal experience with access control and
> > git.
> > 
> > Currently, this is kind of the roadmap I see in my head:
> > 
> > * Set up access controls for the AUR users to access over ssh using
> >   keys. While I could see this being somewhat arduous, it shouldn't be
> >   terribly hard to automate or set up. Something like gitolite or
> >   whatever should make it simpler.
> > * Each repository on the server would contain a single package (if
> >   someone decides to do a split package on the AUR it would contain the
> >   whole set of packages), allowing for multiple users to have push
> >   access and maintain the packages.
> The AUR currently has ~ 41076 packages, which would bring us to an equal
> amount of git repo's. This doesn't seem very efficient to me.
> 
> With svn you can put it all in one repo and checkout a single package
> without checking out the whole repo, which is how
> [core],[extra],[community] work.
> 
> I know Exherbo uses Git for their repo's but they seem to split it up
> into categories [1].
> 
> Git seems to have sparse checkout for this in 1.7, but I don't know how
> it works [2]

git clone --single-branch git://projects.archlinux.org/svntogit/packages.git -b packages/linux

... then make one for community.git :)

and inside, you can do stuff like...

git fetch origin packages/bash
git checkout -b packages/bash FETCH_HEAD

and checkout the other branches


> > * The repositories would be limited to 5M, unless given special
> >   permission (certain kernel packages with lots of massive configuration
> >   files) which should make enough room for the needed files, while
> >   helping to enforce the idea of not putting retrievable sources in the
> >   source tarball. I did some quick math using the number of packages on
> >   the AUR right now, and if every repo used that 5M limit, it would
> >   require a bit over 200G. Granted, I doubt that this limit would be
> >   reached for most packages. 
> > * With the repository set up, give maintainers a week or so to upload
> >   history for their packages (in case they already keep their repos in
> >   this set up), and new packages would get put directly into a new repo.
> > * Once the week for transition is over, begin moving all of the current
> >   packages in the AUR to the git format.
> > 
> > This basically concludes setting up the git stuff, which I have done on
> > a local machine on a very small scale (5 packages) in the past. One
> > thing I was thinking of was the use of `makepkg -S` tarballs still being
> > available. cryptocrack came up with a good point, that just simply
> > copying things over could overwrite the .git directory in the designated
> > directory. I don't think that would be too hard to get around, but I'm no
> > expert. I wrote a quick and dirty script that extracts and commits to
> > the respective repository. Currently, almost no packages use a
> > .changelog file, but there could be some minimal parsing done of a
> > packages .changelog file to craft a git commit message from a tarball. 
> > Obviously there needs to be some kind of security check, but that could be
> > done by the frontend using the system we have now for source tarballs.
> > 
> > At the moment, I am unsure exactly how the AUR parses the PKGBUILDs it
> > receives, but I'm sure that this could easily be adapted to the git
> > system.
> > 
> > One major advantage to having the AUR managed this way is that it would
> > allow users to check for updates on the AUR without the need for complex
> > helpers outside of git. Secondly, it would make mirroring the AUR, or
> > just parts of the AUR extremely easy. In case something happens to the
> > AUR, people can still get their packages and maintainers can still
> > update them very easily using git. It would also still allow people to
> > grab tarballs of the current master branch. This way git is not a
> > requirement to use the AUR, it just makes things a butt ton easier.
> > Also, it would allow users to see the entire source of the package, not
> > just the PKGBUILD, similar to how the official repository works, just
> > that things would be in distinct git repositories not branches.
> > 
> > I would really like to see this kind of thing come to fruition, and if
> > anyone would be willing to help, then please say so.
> > 
> > 
> > Pardon my scattered thoughts,
> > 
> 
> Although I like the idea, I really think it needs more thoughts ;). So
> thanks for bringing it up.
> 
> [1] http://git.exherbo.org/summer/
> [2] http://schacon.github.com/git/RelNotes/1.7.0.txt
> 



-- 
Daniel Wallace
Archlinux Trusted User (gtmanfred)
Georgia Institute of Technology
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://mailman.archlinux.org/pipermail/aur-dev/attachments/20130317/5802ec4d/attachment.asc>