Hi all, I discovered this problem back in June with the help of the guys on IRC. The guy who helped me discover the root of the problem went on to discuss with one of the TUs about this problem and the solution for it. IIRC, Lukas was on holiday at the time, so he wasn't pestered with it at the time :)
Three to four months later and nothing seems to have changed, so I thought I'd start a paper trail and bring this to the attention of a few more people.
I committed changes to an already-established AUR package, and attempted to push these commits. Running `git push` on my slow connection just hangs. 'Impatient me' killed it after 5 minutes of hanging. Trying to push from a faster connection, I found it uploaded almost instantly. Setting GIT_TRACE_PACKET=1 when running `git push` showed up a (warning: large) log [1] showing that git is being sent a bunch of refs (?), presumably for packages which aren't even mine.
On a slow connection, this makes updating AUR packages to impossible. It takes a solid 20 minutes for the push to complete for me. I won't maintain AUR packages if this is the way it's going to be. Even though faster connections solve the time problem, it then becomes an issue of the amount of data transferred needlessly.
Why is this happening? IIRC, the IRC folk mentioned something about the underlying structure of the git repository at the AUR end, but I don't know enough about this stuff to say either way.
Log, xzipped for size reasons. Unpacks to ~82 000 lines. [1] https://ptpb.pw/XmET
On Wed, 07 Oct 2015 at 06:40:36, David Phillips wrote:
[...] I committed changes to an already-established AUR package, and attempted to push these commits. Running `git push` on my slow connection just hangs. 'Impatient me' killed it after 5 minutes of hanging. Trying to push from a faster connection, I found it uploaded almost instantly. Setting GIT_TRACE_PACKET=1 when running `git push` showed up a (warning: large) log [1] showing that git is being sent a bunch of refs (?), presumably for packages which aren't even mine.
On a slow connection, this makes updating AUR packages to impossible. It takes a solid 20 minutes for the push to complete for me. I won't maintain AUR packages if this is the way it's going to be. Even though faster connections solve the time problem, it then becomes an issue of the amount of data transferred needlessly.
Why is this happening? IIRC, the IRC folk mentioned something about the underlying structure of the git repository at the AUR end, but I don't know enough about this stuff to say either way.
Exactly. Internally, we use one huge Git repository to store all AUR packages and gitnamespaces(7) to make that single repository look like different repositories to the outside world. One situation where this does not work is ref advertisement. As you noticed, all refs of the internal repository (i.e. the HEAD refs of each package) are advertised. We are working on improving that.
Log, xzipped for size reasons. Unpacks to ~82 000 lines. [1] https://ptpb.pw/XmET
Okay, good to know it's being worked on. I seem to recall the TU who looked at it mentioning that using separate repositories would use up more disc space but avoid this issue?
I may be wrong, but this seems like quite a large/important problem/flaw to me.
I feel like this may have been discussed before—Is there a thread I've missed on the ML which covers this discussion?
Thanks
On Wed, 07 Oct 2015 at 09:41:31, David Phillips wrote:
Okay, good to know it's being worked on. I seem to recall the TU who looked at it mentioning that using separate repositories would use up more disc space but avoid this issue?
Slightly more disc space on the server, yes. One idea was to use $GIT_OBJECT_DIRECTORY, though, which means that we could still use a shared object storage. We would only need to store the refs separately.
The other possibility is to tell Git to not advertise refs outside the currently selected namespace. I am not entirely sure whether that is a good idea and whether it would be accepted upstream, though.
I may be wrong, but this seems like quite a large/important problem/flaw to me.
I feel like this may have been discussed before—Is there a thread I've missed on the ML which covers this discussion?
No, this hasn't been discussed in public yet.
Thanks
On Wed, 07 Oct 2015 at 12:58:27, Lukas Fleischer wrote:
On Wed, 07 Oct 2015 at 09:41:31, David Phillips wrote:
Okay, good to know it's being worked on. I seem to recall the TU who looked at it mentioning that using separate repositories would use up more disc space but avoid this issue?
Slightly more disc space on the server, yes. One idea was to use $GIT_OBJECT_DIRECTORY, though, which means that we could still use a shared object storage. We would only need to store the refs separately.
The other possibility is to tell Git to not advertise refs outside the currently selected namespace. I am not entirely sure whether that is a good idea and whether it would be accepted upstream, though. [...]
We temporarily enabled a undocumented Git setting to suppress these unneeded advertisements. Git 2.7 will probably officially support hiding refs outside the current namespace which we will enable as soon as 2.7 is out and the aurweb hosting server is upgraded.
Regards, Lukas
Looks good this way, I'm very happy you guys were able to find a solution.
Thanks and all the best
aur-general@lists.archlinux.org