On Wed, 18 Aug 2010 14:21:01 +1000 "Allan McRae" <allan@archlinux.org> wrote:

On 18/08/10 12:07, Alexander Duscheleit wrote:

...

Hi,

since I upgraded to pacman 3.4.0 I notice a strange behavior in the way pacman integrates with my local mirror server. First, pacman opens a new ftp connection for every download, which spams the logs a lot. Second, pacman doesn't seem to close those connections either at all, or at least not in a timely fashion. This results in pacman not being able to download more than a hand full of packages before my ftp won't let it connect anymore. The server-log says:

Aug 18 00:42:38 titan proftpd[4666]: titan.huntemann.uni-oldenburg.de - MaxInstances (30) reached, new connection denied

over and over while pacman switches to the next mirror for every download.

The pacman frontend clyde exhibits the same behavior, so I suspect the culprit is somewhere in libalpm. Pacman prior to 3.4.0 did not show this behavior and all downloads worked fine.

This bug has been reported: http://bugs.archlinux.org/task/20371

Great, seems I should remember to actually set the right project in FS :(

However, no-one appears to have investigated whether this was libarchive's fault or libalpm's.

How would I best go about researching this? I have all the components right here, and it's easy enough to trigger, but I have no experience whatsoever in debugging libraries or C code in general. I could try and set up a chroot to bisect pacman / lib{fetch,archive,alpm} but I have no clue how good or bad random pacman versions interact with the rest of the system.

On Wed, 18 Aug 2010 00:06:44 -0700 mike rosset <schizoid29@gmail.com> wrote:

...

How would I best go about researching this? I have all the components right here, and it's easy enough to trigger, but I have no experience whatsoever in debugging libraries or C code in general.

I could try and set up a chroot to bisect pacman / lib{fetch,archive,alpm} but I have no clue how good or bad random pacman versions interact with the rest of the system.

What I did to replicate this was create a chroot ie

sudo mkdir -p archtest/var/lib/pacman

and then test with a large group something like this

sudo pacman -Sy gnome --cachedir $(pwd)/archtest/var/cache/pacman/pkg/ -r ./archtest/

if you need to download again you can just purge chroot cachedir. after that I'm not sure how best to debug libfetch possibly using netstat, lsof and strace?

After playing around i a throwaway-chroot, the problem seems to be libfetch >=2.30. I just modified the PKGBUILD to different versions (without replacing or rebuilding pacman at all). Libfetch 2.26 fetches files without a problem, 2.30+ fails after downloading 5 files while MaxInstances in proftpd is set to 8. If I set MaxInstances to 3, downloading fails outright (also only for 2.30+), so something there seems to consume 3 connections before actually downloading something at all. I'd try to break it down further, but NetBSDs CVS server isn't talking to me at the moment.

On Wed, 18 Aug 2010 00:54:26 -0700 Mike Rosset <schizoid29@gmail.com> wrote:

...

After playing around i a throwaway-chroot, the problem seems to be libfetch >=2.30. I just modified the PKGBUILD to different versions (without replacing or rebuilding pacman at all).

Libfetch 2.26 fetches files without a problem, 2.30+ fails after downloading 5 files while MaxInstances in proftpd is set to 8. If I set MaxInstances to 3, downloading fails outright (also only for 2.30+), so something there seems to consume 3 connections before actually downloading something at all.

I'd try to break it down further, but NetBSDs CVS server isn't talking to me at the moment.

Ok maybe upate the bug with these results, so they have more information.

I've updated the bug-report with the above text as well as my results of a bisection of libfetch. Looking at the patch, the change in libfetch seems sensible and I suspect that something higher up the chain is calling it in a wrong way. As far as I can tell, it is possible, that pacman never called libfetch in The Right Way (TM), but this only got exposed, after libfetch stopped closing connections unconditionally. Let's hope that some of my stumbling around is actually helpfull :)