Roman Kyrylych wrote:

2008/7/31 Dan McGee <dpmcgee@gmail.com>:

...

On Thu, Jul 31, 2008 at 2:01 AM, Eric Belanger <belanger@astro.umontreal.ca> wrote:

...

On Wed, 30 Jul 2008, Dan McGee wrote:

...

Pacman 3.2.0 is in testing for i686. Please signoff. Changes outlined here: http://projects.archlinux.org/?p=pacman.git;a=blob;f=NEWS

When building mythbrowser, I noticed that makepkg now fails when there is an '?' in the sources, e.g. source=("http://www.mythtv.org/modules.php?name=Downloads&d_op=getit&lid=136&foo=/mythplugins-$pkgver.tar.bz2")

Removing the double quotes and escaping the characters doesn't work.

The error message:

==> ERROR: There is no agent set up to handle URLs. Check /etc/makepkg.conf. Aborting...

Not a showstopper as the source can be downloaded maunally in the build dir. I'll try to look into it further tomorrow.

We have a regression here, and it is actually bigger than it looks at first sight. Can you file a bug report for this?

Tracing the code shows that %o and %u are never even used right now, as they don't make it all the way through to the download stage.

Hmm... I didn't look at that code for a long time, dunno why this might happen. :-(

My hacked makepkg shows:

On Thu, Jul 31, 2008 at 2:14 PM, Allan McRae <allan@archlinux.org> wrote:

My hacked makepkg with:

msg "ALLAN!: $netfile" local proto=$(echo $netfile | sed 's|://.*||') msg "ALLAN!: $proto" sleep 5

outputs this: ==> ALLAN!: http://www.mythtv.org/modules.php?name=Downloads&d_op=getit&lid=136&foo=/mythplugins-0.21.tar.bz2 ==> ALLAN!:

So it is that sed line that is failing

Indeed, that is the first problem, netfile should be quoted here. local proto=$(echo "$netfile" | sed 's|://.*||') But once this is fixed, we run into a second problem, with another sed command, which breaks because of all the & in the url. local dlcmd=$(echo "$dlagent" | sed "s|%o|$file.part|" | sed "s|%u|$netfile|" $ netfile="ab&cd&ef"; echo "%u" | sed "s|%u|$netfile|" ab%ucd%uef $ netfile="ab\&cd\&ef"; echo "%u" | sed "s|%u|$netfile|" ab&cd&ef So I don't see how to fix this except manually escaping all & in the url...

On Thu, Jul 31, 2008 at 2:50 PM, Allan McRae <allan@archlinux.org> wrote:

...

Indeed, that is the first problem, netfile should be quoted here. local proto=$(echo "$netfile" | sed 's|://.*||')

And that $netfile should be $url....

Yeah, apparently.

...

But once this is fixed, we run into a second problem, with another sed command, which breaks because of all the & in the url. local dlcmd=$(echo "$dlagent" | sed "s|%o|$file.part|" | sed "s|%u|$netfile|"

$ netfile="ab&cd&ef"; echo "%u" | sed "s|%u|$netfile|" ab%ucd%uef

$ netfile="ab\&cd\&ef"; echo "%u" | sed "s|%u|$netfile|" ab&cd&ef

So I don't see how to fix this except manually escaping all & in the url...

Well, to just make the bug more obscure.

$ netfile="ab&cd&ef"; echo "%u" | sed "s|%u|$netfile|" | sed "s|\%u|\&|g" ab&cd&ef

But that makes problems if there is a %u in the url. But can we do some multistage hackery like that?

I really don't like this. I think we could use bash feature instead. echo ${netfile//foo/bar} But even after converting the two above sed rules with this, I still have other problem. It is driving me crazy. When I print the whole download command myself, and copy/paste it to the shell, it works fine. But when it gets executed in makepkg with : $(get_downloadcmd foo bar), it always fail.

On Sat, Aug 2, 2008 at 2:01 PM, Andreas Radke <a.radke@arcor.de> wrote:

CC'ing my issue here originally posted on arch-dev-public:

I'm getting errors pacman not able to fetch packages from my local mirror with -Syu or packages cannot be found in the db when using -Ss foo. This seems to get fixed with running once -Syy. Whatever the reason is.

But fetching issue stays:

[root@workstation64 /]# LANG=C pacman -Syu :: Synchronizing package databases... testing is up to date core is up to date extra is up to date :: Starting full system upgrade... :: Replace kde-common with extra/kdebase-workspace? [Y/n] resolving dependencies... looking for inter-conflicts...

Remove (1): kde-common-4.1.0-2

Total Removed Size: 0.01 MB

Targets (26): qt-4.4.1-1 phonon-4.2.0-2 raptor-1.4.18-1 redland-1.0.8-1 soprano-2.1.1-1 kdelibs-4.1.0-4 gcc-libs-4.3.1-3 rarian-0.8.0-2 lame-3.98-1 a52dec-0.7.4-4 faad2-2.6.1-1 libmp4v2-1.6-1 faac-1.26-1 xvidcore-1.1.3-1 libid3tag-0.15.1b-2 imlib2-1.4.1-1 x264-20080625-1 ffmpeg-20080715-3 xine-lib-1.1.14-2 kdebase-runtime-4.1.0-2 kdebase-workspace-4.1.0-4 gcc-4.3.1-3 hunspell-1.2.6-1 ttf-dejavu-2.26-1 xulrunner-1.9.0.1-2 zip-3.0-1

Total Download Size: 163.84 MB Total Installed Size: 433.84 MB

Proceed with installation? [Y/n] :: Retrieving packages from core... gcc-4.3.1-3-i686 23.7M 52.1M/s 00:00:00 [#########################################################################################################################] 100% :: Retrieving packages from extra... raptor-1.4.18-1-i686 390.8K 15.0M/s 00:00:00 [#########################################################################################################################] 100% error: failed retrieving file 'redland-1.0.8-1-i686.pkg.tar.gz' from 192.168.0.90 : Command okay redland-1.0.8-1-i686 547.2K 367.6K/s 00:00:01 [#########################################################################################################################] 100% soprano-2.1.1-1-i686 2.2M 22.9M/s 00:00:00 [#########################################################################################################################] 100% error: failed retrieving file 'rarian-0.8.0-2-i686.pkg.tar.gz' from 192.168.0.90 : Command okay rarian-0.8.0-2-i686 163.3K 187.2K/s 00:00:01 [#########################################################################################################################] 100%

[testing] Server = ftp://192.168.0.90/mirror/packages/i686/testing/ Include = /etc/pacman.d/mirrorlist

changing to XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u seems to fix it.

The "Command okay" part is certainly interesting. Looks like libdownload may be returning a non-zero success code? I'm not sure.

On Sun, Aug 3, 2008 at 8:26 PM, Xavier <shiningxc@gmail.com> wrote:

What really bothers me here is that it looks like a libdownload issue, yet it is reported as a pacman 3.2.0 regression. And libdownload has not changed for ages. So is that issue really reproducible without problems with 3.2.0, and when you go back to 3.1.4, everything works perfectly fine again? This is so odd :/ Are you sure it could not have been caused by another change, like a change in the ftp server, or some network settings, or whatever?

So trying to reproduce the issue, I first tried to set up vsftpd. It worked perfectly with ncftp and firefox, but no luck with libdownload :P The only thing it told me is : error: failed retrieving file 'custom.db.tar.gz' from 127.0.0.1 : File unavailable (e.g., file not found, no access) So well, I gave up, and tried with pure-ftpd. And this time. exactly the same error as Andy, "Command okay" So I started inspecting the download code between 3.1.4 and 3.2.0. While the core mostly stayed the same, it has been heavily refactored, so some small details could have changed. I finally found one difference that caught my eyes : /* remove trailing slashes, just to clean up the rest of the code */ for(int i = strlen(u->doc) - 1; u->doc[i] == '/'; --i) u->doc[i] = '\0'; I removed the trailing slash on my Server line in pacman.conf, and it worked perfectly... So I patched pacman to remove an eventual trailing slash, and the patch will follow quickly. But I wonder if libdownload should not be patched as well. "Command ok" seems rather misleading. Returning the following error would be much better if it is possible : [DLERR_URL] Invalid URL

On Mon, Aug 4, 2008 at 1:26 AM, Xavier <shiningxc@gmail.com> wrote:

So I patched pacman to remove an eventual trailing slash, and the patch will follow quickly. But I wonder if libdownload should not be patched as well. "Command ok" seems rather misleading. Returning the following error would be much better if it is possible : [DLERR_URL] Invalid URL

I tried to compare a bit libdownload with current libfetch code, by curiosity. I am a bit confused though now, it looks like the code already handles multiple slashes. But well, I will need to investigate it more. Anyway, while looking at this, I found one difference in libfetch, which was introduced by this patch : http://lists.freebsd.org/pipermail/cvs-all/2007-April/217521.html This looks rather interesting, I always noticed the high latency of ftp access on -Sy operations, and that is why I always use http now. But this patch might solve it.

On Mon, Aug 4, 2008 at 7:34 PM, Henning Garus <henning.garus@googlemail.com> wrote:

If my understanding of the libdownload code is correct "Command okay" occurs, because the server sends a 200 when libdownload expects something different. But I can't say where.

I've been spending some time trying to reproduce this error, but syncing to a local ftp just works, with and without trailing slash.

...

I am a bit confused though now, it looks like the code already handles multiple slashes. But well, I will need to investigate it more.

C is not my strong point, but I think the following lines in _ftp_cwd() should take care of multiple slashes:

while (*beg == '/') ++beg, ++i;

Note that you need to download at least 2 files in a row for this to happen. Doing pacman -Sy <package> is enough because a first connection is made for the -Sy, then a second for downloading the package and it breaks there. If you prefer, I have a simpler test case. I just took libdownload source code (for helping debugging, I set debug=true in the Makefile and then installed that), then I extended the samples/dl.c program to reproduce the issue. I attach it here. Then you just need to run it on one ftp server with two files : ./dl ftp://ftp.archlinux.org/core/os/i686/lastsync ftp://ftp.archlinux.org/core/os/i686//packages.txt But it is probably better to run your own ftp server for debugging / playing purpose. ./dl ftp://localhost/foo ftp://localhost//bar The relevant part of the code is indeed the one you showed, but you have to look around it too: for (beg = file + i; beg < end; beg = file + i + 1) { while (*beg == '/') ++beg, ++i; for (++i; file + i < end && file[i] != '/'; ++i) /* nothing */; if(beg < end) e = _ftp_cmd(conn, "CWD %.*s", file + i - beg, beg); if (e != FTP_FILE_ACTION_OK) { _ftp_seterr(e); return (-1); } } In non trailing case, we have : file : /core/os/i686/packages.txt pwd : /core/os/i686 "end" points to the last / in file "beg" points to the first character which differs between file and pwd, which turns out to be exactly the same / as "end". So here we don't have beg < end (we have beg == end) so the whole code above is skipped and everything works fine. In trailing case : file : /core/os/i686//packages.txt pwd : /core/os/i686 end points to the last / in file, and beg to the first different char, which is the slash just before. So here we have beg < end (beg == end - 1 ), so the above code is run, the beg pointer skips the successive /, so eventually points to the "p" of packages So then we don't have beg < end anymore (beg == end + 1), so the ftp_cmd command is not run. But the final check is still run, and here is where our error appears : if (e != FTP_FILE_ACTION_OK) { _ftp_seterr(e); return (-1); } So as you said, at this point e was set to 200 (FTP_OK), but from a previous part of the code. It looks very weird to check for the e error here, while on the previous line, e was only set conditionally (only if beg < end). And about finding where e was last set (to 200), I am not sure, maybe at the beggining of _ftp_cwd : if ((e = _ftp_cmd(conn, "PWD")) != FTP_WORKING_DIRECTORY || Anyway, this was a very long explanation for a small thing in the end :P Maybe it is easier to just run libdownload in debug mode, and play with dl.c and gdb to get sense of it. I still didn't figure out the whole thing yet, and I have still no idea if there is something to fix and how.

On Mon, Aug 4, 2008 at 1:56 AM, Xavier <shiningxc@gmail.com> wrote:

On Mon, Aug 4, 2008 at 1:26 AM, Xavier <shiningxc@gmail.com> wrote:

...

So I patched pacman to remove an eventual trailing slash, and the patch will follow quickly. But I wonder if libdownload should not be patched as well. "Command ok" seems rather misleading. Returning the following error would be much better if it is possible : [DLERR_URL] Invalid URL

I tried to compare a bit libdownload with current libfetch code, by curiosity. I am a bit confused though now, it looks like the code already handles multiple slashes. But well, I will need to investigate it more.

Anyway, while looking at this, I found one difference in libfetch, which was introduced by this patch : http://lists.freebsd.org/pipermail/cvs-all/2007-April/217521.html

This looks rather interesting, I always noticed the high latency of ftp access on -Sy operations, and that is why I always use http now. But this patch might solve it.

At some point in the near future, I'm going to remove libdownload and maintain it as a patchset against libfetch. I am unclear as to where to pull libfetch from, because dragonflybsd seems to be more active, but has diverged a tad from freebsd, and openbsd is slightly different still.

On Mon, Aug 4, 2008 at 6:52 PM, Xavier <shiningxc@gmail.com> wrote:

On Mon, Aug 4, 2008 at 9:29 PM, Aaron Griffin <aaronmgriffin@gmail.com> wrote:

...

At some point in the near future, I'm going to remove libdownload and maintain it as a patchset against libfetch. I am unclear as to where to pull libfetch from, because dragonflybsd seems to be more active, but has diverged a tad from freebsd, and openbsd is slightly different still.

No matter where you pull it from, I think this is a great idea and would totally support it.

Considering you looked at it more recently, would you use freebsd or dragonfly?

On Mon, Aug 04, 2008 at 02:29:10PM -0500, Aaron Griffin <aaronmgriffin@gmail.com> wrote:

At some point in the near future, I'm going to remove libdownload and maintain it as a patchset against libfetch. I am unclear as to where to pull libfetch from, because dragonflybsd seems to be more active, but has diverged a tad from freebsd, and openbsd is slightly different still.

Great. :) I was always sad about you forked libdownload and you did not do it as a patchset (possibly by having all the patches in upstream after some time).