On Sun, Mar 17, 2013 at 01:44:12PM +0100, Kwpolska wrote:

On 16/03/13 23:10, William Giokas wrote:

...

The repositories would be limited to 5M

Measuring by current size of contents or total size? The latter would be very unpleasant for packages modified often or packages with binary files (eg. dropbox/dropbox.png; binary files in git aren’t using deltas AFAIK).

Binaries (even images) should not be hosted in the AUR. This is simply a waste of space with the plethora of hosting services around. Someone on aur-general mentioned using a 1-2M limit, which actually seems more sane.

On Sun, Mar 17, 2013 at 12:43 PM, Jelle van der Waa wrote:

...

The AUR currently has ~ 41076 packages, which would bring us to an equal amount of git repo's. This doesn't seem very efficient to me.

Well, why do you think it is inefficient? Maybe because of the amount of repos? GitHub hosts (as of now) 5,700,361 repos[0], big and small. That number doesn’t include gists, and there are a lot of them. So, we would be around six million, at the very least.

While I agree with you, even having 41K git repos would take some more thought.

Because of the repo size? Let me disprove you.

Currently, the AUR stores stuff in 41078 tarballs (tar+gz), plus 41078 PKGBUILD files stored without any compression on the server (eg. https://aur.archlinux.org/packages/pk/pkgbuilder/PKGBUILD ). The PKGBUILD and the tarball weigh a total of 1241 bytes[1].

Now, we make a local git repo (git init --bare), and get the aforementioned PKGBUILD into it from a downstream clone (git {add,commit,push}). After removing the sample hooks (unnecessary, although the AUR may make good use of such hooks[2]), we end up with 843 bytes[2]. After an update (pkgver+md5sums) we get 1518 bytes[3]. This is a small price for a complete PKGBUILD modification history.

If somebody wants numbers for Mercurial, it gets 1262 and 1725 bytes[4], respectively. I couldn’t set SVN up easily, so I don’t have any numbers.

[0] https://github.com/search; displayed if you are lucky enough [1] makepkg --source; wc -c PKGBUILD pkgbuilder*.src.tar.gz [2] For example, Heroku uses post-update hooks to trigger a build of the website. GitHub has an awesome Service Hook system, where you can have a bazillion services do stuff after you push (eg. unit tests via Travis CI) [3] wc -c config description HEAD objects/*/* refs/heads/master [4] wc -c 00changelog.i branch dirstate hgrc last-message.txt requires undo.bookmarks undo.branch undo.desc undo.dirstate store/00changelog.i store/00manifest.i store/fncache store/phaseroots store/undo store/undo.phaseroots store/data/* cache/branchheads-served

...

With svn you can put it all in one repo and checkout a single package without checking out the whole repo, which is how [core],[extra],[community] work.

svn is ugly and human-unfriendly. Nobody likes it. And I couldn’t get it to work properly, as mentioned above.

...

Git seems to have sparse checkout for this in 1.7, but I don't know how it works.

http://jasonkarns.com/blog/subdirectory-checkouts-with-git-sparse-checkout/

Thank you, -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF

On Sun, Mar 17, 2013 at 12:43:01PM +0100, Jelle van der Waa wrote:

On 16/03/13 23:10, William Giokas wrote:

...

All,

So in my spare time I was thinking about the AUR and how it could be better. Back in January I commented on a bug[1] about integrating the AUR and git to have a powerful, robust backend for the AUR. I think that the original idea of creating one massive repository was inherently flawed for most users, as it requires that the entire repository be cloned, and that is no trivial task for people with only moderate internet speeds. Were it similar to the way the official repositories git setup is, then it would be okay for fetching, but using single repositories for each project would make access control much less of a nightmare.

At the moment I have no experience with PHP or messing with the AUR, but I have been playing around with git and using it to track packages and the like, as well as some minimal experience with access control and git.

Currently, this is kind of the roadmap I see in my head:

* Set up access controls for the AUR users to access over ssh using keys. While I could see this being somewhat arduous, it shouldn't be terribly hard to automate or set up. Something like gitolite or whatever should make it simpler. * Each repository on the server would contain a single package (if someone decides to do a split package on the AUR it would contain the whole set of packages), allowing for multiple users to have push access and maintain the packages. The AUR currently has ~ 41076 packages, which would bring us to an equal amount of git repo's. This doesn't seem very efficient to me.

With svn you can put it all in one repo and checkout a single package without checking out the whole repo, which is how [core],[extra],[community] work.

I know Exherbo uses Git for their repo's but they seem to split it up into categories [1].

Git seems to have sparse checkout for this in 1.7, but I don't know how it works [2]

git clone --single-branch git://projects.archlinux.org/svntogit/packages.git -b packages/linux ... then make one for community.git :) and inside, you can do stuff like... git fetch origin packages/bash git checkout -b packages/bash FETCH_HEAD and checkout the other branches

...

* The repositories would be limited to 5M, unless given special permission (certain kernel packages with lots of massive configuration files) which should make enough room for the needed files, while helping to enforce the idea of not putting retrievable sources in the source tarball. I did some quick math using the number of packages on the AUR right now, and if every repo used that 5M limit, it would require a bit over 200G. Granted, I doubt that this limit would be reached for most packages. * With the repository set up, give maintainers a week or so to upload history for their packages (in case they already keep their repos in this set up), and new packages would get put directly into a new repo. * Once the week for transition is over, begin moving all of the current packages in the AUR to the git format.

This basically concludes setting up the git stuff, which I have done on a local machine on a very small scale (5 packages) in the past. One thing I was thinking of was the use of `makepkg -S` tarballs still being available. cryptocrack came up with a good point, that just simply copying things over could overwrite the .git directory in the designated directory. I don't think that would be too hard to get around, but I'm no expert. I wrote a quick and dirty script that extracts and commits to the respective repository. Currently, almost no packages use a .changelog file, but there could be some minimal parsing done of a packages .changelog file to craft a git commit message from a tarball. Obviously there needs to be some kind of security check, but that could be done by the frontend using the system we have now for source tarballs.

At the moment, I am unsure exactly how the AUR parses the PKGBUILDs it receives, but I'm sure that this could easily be adapted to the git system.

One major advantage to having the AUR managed this way is that it would allow users to check for updates on the AUR without the need for complex helpers outside of git. Secondly, it would make mirroring the AUR, or just parts of the AUR extremely easy. In case something happens to the AUR, people can still get their packages and maintainers can still update them very easily using git. It would also still allow people to grab tarballs of the current master branch. This way git is not a requirement to use the AUR, it just makes things a butt ton easier. Also, it would allow users to see the entire source of the package, not just the PKGBUILD, similar to how the official repository works, just that things would be in distinct git repositories not branches.

I would really like to see this kind of thing come to fruition, and if anyone would be willing to help, then please say so.

Pardon my scattered thoughts,

Although I like the idea, I really think it needs more thoughts ;). So thanks for bringing it up.

[1] http://git.exherbo.org/summer/ [2] http://schacon.github.com/git/RelNotes/1.7.0.txt

-- Daniel Wallace Archlinux Trusted User (gtmanfred) Georgia Institute of Technology

On Sun, Mar 17, 2013 at 12:33:41PM -0500, William Giokas wrote:

On Sun, Mar 17, 2013 at 01:14:01PM -0400, Daniel Wallace wrote:

...

On Sun, Mar 17, 2013 at 12:43:01PM +0100, Jelle van der Waa wrote:

...

The AUR currently has ~ 41076 packages, which would bring us to an equal amount of git repo's. This doesn't seem very efficient to me.

With svn you can put it all in one repo and checkout a single package without checking out the whole repo, which is how [core],[extra],[community] work.

I know Exherbo uses Git for their repo's but they seem to split it up into categories [1].

Git seems to have sparse checkout for this in 1.7, but I don't know how it works [2]

git clone --single-branch git://projects.archlinux.org/svntogit/packages.git -b packages/linux

... then make one for community.git :)

and inside, you can do stuff like...

git fetch origin packages/bash git checkout -b packages/bash FETCH_HEAD

and checkout the other branches

But this does make it so that either no one has push access, or only a very small subset of people have push access for security reasons. The whole point of the multiple repos is to allow people full control over the whole repo with git, and not have to rely on the `makepkg -S` bits. While great for spectating, it's pretty bad for a bunch of people that can just simply sign up as contributors. Also, you know what would happen if we just gave everyone push access.

Why are people going to be able to actually git push, why not use something like burp, where you run makepkg -S and upload that with a commit message, and the backend takes care of unwrapping and committing it.

Thanks, -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF

-- Daniel Wallace Archlinux Trusted User (gtmanfred) Georgia Institute of Technology

On Sun, Mar 17, 2013 at 1:33 PM, William Giokas <1007380@gmail.com> wrote:

On Sun, Mar 17, 2013 at 01:14:01PM -0400, Daniel Wallace wrote:

...

On Sun, Mar 17, 2013 at 12:43:01PM +0100, Jelle van der Waa wrote:

...

The AUR currently has ~ 41076 packages, which would bring us to an equal amount of git repo's. This doesn't seem very efficient to me.

With svn you can put it all in one repo and checkout a single package without checking out the whole repo, which is how [core],[extra],[community] work.

I know Exherbo uses Git for their repo's but they seem to split it up into categories [1].

Git seems to have sparse checkout for this in 1.7, but I don't know how it works [2]

git clone --single-branch git://projects.archlinux.org/svntogit/packages.git -b packages/linux

... then make one for community.git :)

and inside, you can do stuff like...

git fetch origin packages/bash git checkout -b packages/bash FETCH_HEAD

and checkout the other branches

But this does make it so that either no one has push access, or only a very small subset of people have push access for security reasons. The whole point of the multiple repos is to allow people full control over the whole repo with git, and not have to rely on the `makepkg -S` bits. While great for spectating, it's pretty bad for a bunch of people that can just simply sign up as contributors. Also, you know what would happen if we just gave everyone push access.

I'm fairly certain you can restrict users to only be able to push to certain branches using hooks. So even if there was a single repository users wouldn't be able to push to branches they shouldn't be pushing to. I don't even think full Git push access is the first logical step in implementing version control in the AUR. I think the first logical step would be to have the package submission page have a spot for a changelog entry and then have the backend take care of all the Git business itself rather than having users directly pushing. Then re-assess at a later date and maybe add the ability for direct Git pushing.

Thanks, -- William Giokas | KaiSforza GnuPG Key: 0x73CD09CF Fingerprint: F73F 50EF BBE2 9846 8306 E6B8 6902 06D8 73CD 09CF