2006/10/25, Christian Hamar alias krix :

2006. 10. 25, szerda keltezéssel 10.56-kor Aaron Griffin ezt írta:

...

On 10/25/06, Piche, Kevin wrote:

...

Frankly I don't like ftplib - it doesn't work with proxies so I have to use wget. Maybe we should switch to libcurl or something.

On this note, I agree 100% - but libcurl is messy. I would prefer something along the lines of libarchive - maybe libfetch (also from bsd) ?

For points. I agree with that we need to replace ftplib with some "newer" and some active developed project. Because as i see ftplib dead (development)

I dont know libcurl. And never hear from libfetch :) Maybe i see libfetch in these days.

I support this idea.

libcurl is the most featured and thus too heavy for Pacman, IMHO. I don't know if compiling it with some featuress omitted will help much. A list of libraries can be found at http://curl.haxx.se/libcurl/competitors.html libfetch seems to be nice alternative to ftplib. It is small, has simple and stable API, is maintained, supports HTTP and FTP downloads and uploads, HTTP proxies and redirects. Sources are here: http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/ -- Roman Kyrylych (Роман Кирилич)

2006/10/25, Christian Hamar alias krix :

...

libfetch seems to be nice alternative to ftplib. It is small, has simple and stable API, is maintained, supports HTTP and FTP downloads and uploads, HTTP proxies and redirects. Sources are here: http://www.freebsd.org/cgi/cvsweb.cgi/src/lib/libfetch/

Yes as i see. Its great and simple. And it supports file:/// too! Which is not bad. Since as i know pacman now got "own" routin for handling those types.

I support libfetch! :)

Vote for libfetch! libfetch for President! :) (sorry :) )

libfetch is licensed under BSD license with some parts of code under MIT license. As they both are compatible with GPL I think there is no reason why not to use libfetch. ;-) -- Roman Kyrylych (Роман Кирилич)

2006/10/25, Aaron Griffin :

libfetch is a PITA to port... we'll probably have to fork it... as the changes in the pas year or so have been minimal, I don't forsee a problem with that.... I'm gonna try this out

Why it's so PITA? Does libftp use many FreeBSD-specific things? Maybe you should take a look how it's ported by GNU-Darwin project here: http://gnu-darwin.cvs.sourceforge.net/gnu-darwin/4linux/libfetch/ The description of 4linux says "FreeBSD ports and package system as modified for GNU/Linux". I know that this is outdated version, but maybe you can find some hints how to port the latest version here. Yes, libfetch is not changing constantly, so if there will be some patches in mainstream we can port them easy. I wonder if there will be any non-security-related/non-bugfix patches at all, as libfetch API was last changed in 4 years, 10 months ago! -- Roman Kyrylych (Роман Кирилич)

2006/10/26, Aaron Griffin :

Regardless, it's fixed and I integrated it into pacman. It was pretty quick, and let me remove about 500 lines. I also got to change the download callback format (thank god), as using about 50 different 'global' variables was disgusting. Of course, this breaks some things until they use the new format (assuming it's ok).

Aaron, it will be really cool if you make a Linux port of libfetch available not only from Arch CVS. AFAIK many people want something like libfetch under Linux. (I realized this even more while searched over Internet for Linux port of it) -- Roman Kyrylych (Роман Кирилич)

2006/10/26, Aaron Griffin :

...

Aaron, it will be really cool if you make a Linux port of libfetch available not only from Arch CVS. AFAIK many people want something like libfetch under Linux. (I realized this even more while searched over Internet for Linux port of it)

I plan on it, but I will fork it. One of my tabled projects, for a long time, has been a decent "download" library for using external resources. I plan on adding segmented downloading among other things (torrent and metalink support?), but that may be a long ways off. I will provide a patch to allow libfetch to compile under linux, mostly unmolested, but I also plan on forking this for my own ideas 8)

Yes, I meant the fork. :-) Hmm... segmented downloads? Yay! That's really cool 8-D -- Roman Kyrylych (Роман Кирилич)

On Wed, 2006-10-25 at 10:56 -0500, Aaron Griffin wrote:

On 10/25/06, Piche, Kevin wrote:

...

Frankly I don't like ftplib - it doesn't work with proxies so I have to use wget. Maybe we should switch to libcurl or something.

On this note, I agree 100% - but libcurl is messy. I would prefer something along the lines of libarchive - maybe libfetch (also from bsd) ?

Are you sure this is an ftplib issue?

_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

While the SIG11 occurs within ftplib I think it's more of a compiler issue actually after more investigation. There's nothing obviously wrong with the code however. Here's my gdb trace: [23:01 root@puffin pacman]$ gdb --args ./pacman.static -vSw vim --noconfirm GNU gdb 6.5 Copyright (C) 2006 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "x86_64-unknown-linux-gnu"...Using host libthread_db library "/lib/libthread_db.so.1". (gdb) break FtpGet Breakpoint 1 at 0x41e400: file ftplib.c, line 1248. (gdb) run Starting program: /home/kpiche/pacman-devel/src/pacman-lib/src/pacman/pacman.static -vSw vim --noconfirm Root : / DBPath: var/lib/pacman Targets: vim resolving dependencies... done. Targets: vim-7.0.118-1 Total Package Size: 7.6 MB Beginning download... :: Retrieving packages from current... Breakpoint 1, FtpGet ( outputfile=0x7fff3c50e470 "/var/cache/pacman/pkg/vim-7.0.118-1.pkg.tar.gz.part", path=0x7742a0 "vim-7.0.118-1.pkg.tar.gz", mode=73 'I', nControl=0x777810) at ftplib.c:1248 1248 { (gdb) s 1249 return FtpXfer(outputfile, path, nControl, FTPLIB_FILE_READ, mode); (gdb) FtpXfer ( localfile=0x7fff3c50e470 "/var/cache/pacman/pkg/vim-7.0.118-1.pkg.tar.gz.part", path=0x7742a0 "vim-7.0.118-1.pkg.tar.gz", nControl=0x777810, typ=3, mode=73) at ftplib.c:1101 1101 { (gdb) 1108 if (localfile != NULL) (gdb) 1101 { (gdb) 1108 if (localfile != NULL) (gdb) 1110 char ac[4] = "a"; (gdb) 1112 ac[0] = 'r'; (gdb) 1115 local = fopen(localfile, ac); (gdb) 1110 char ac[4] = "a"; (gdb) 1112 ac[0] = 'r'; (gdb) 1114 ac[1] = 'b'; (gdb) 1112 ac[0] = 'r'; (gdb) 1114 ac[1] = 'b'; (gdb) 1115 local = fopen(localfile, ac); (gdb) 1116 if (local == NULL) (gdb) 1115 local = fopen(localfile, ac); (gdb) 1116 if (local == NULL) (gdb) 1125 if (!FtpAccess(path, typ, mode, nControl, &nData)) (gdb) FtpAccess (path=0x7742a0 "vim-7.0.118-1.pkg.tar.gz", typ=3, mode=73, nControl=0x777810, nData=0x7fff3c50d3c8) at ftplib.c:802 802 { (gdb) 805 if ((path == NULL) && (gdb) print nControl $1 = (netbuf *) 0x777810 (gdb) print nControl->idlecb $2 = (FtpCallback) 0x401e70 (gdb) fin Run till exit from #0 FtpAccess (path=0x7742a0 "vim-7.0.118-1.pkg.tar.gz", typ=3, mode=73, nControl=0x777810, nData=0x7fff3c50d3c8) at ftplib.c:805 0x000000000041e298 in FtpXfer ( localfile=0x7fff3c50e470 "/var/cache/pacman/pkg/vim-7.0.118-1.pkg.tar.gz.part", path=0x7742a0 "vim-7.0.118-1.pkg.tar.gz", nControl=0x777810, typ=3, mode=<value optimized out>) at ftplib.c:1125 1125 if (!FtpAccess(path, typ, mode, nControl, &nData)) Value returned is $3 = 1 (gdb) s 1127 dbuf = malloc(FTPLIB_BUFSIZ); (gdb) 1128 if (typ == FTPLIB_FILE_WRITE) (gdb) 1127 dbuf = malloc(FTPLIB_BUFSIZ); (gdb) 1128 if (typ == FTPLIB_FILE_WRITE) (gdb) 1140 while ((l = FtpRead(dbuf, FTPLIB_BUFSIZ, nData))

0) (gdb) FtpRead (buf=0x779d60, max=8192, nData=0x779be0) at ftplib.c:872 872 { (gdb) 874 if (nData->dir != FTPLIB_READ) (gdb) print nData $4 = (netbuf *) 0x779be0 (gdb) l 869 * FtpRead - read from a data connection 870 */ 871 GLOBALDEF int FtpRead(void *buf, int max, netbuf *nData) 872 { 873 int i; 874 if (nData->dir != FTPLIB_READ) 875 return 0; 876 if (nData->buf) 877 i = readline(buf, max, nData); 878 else (gdb) print nData->dir $5 = 1 (gdb) $6 = 1 (gdb) s 872 { (gdb) 874 if (nData->dir != FTPLIB_READ) (gdb) 876 if (nData->buf) (gdb) print nData->buf $7 = 0x0 (gdb) s

So here FtpRead is about to call socket_wait. Note the value of the control structure pointer is 0x779be0. 880 i = socket_wait(nData); (gdb) print nData $8 = (netbuf *) 0x779be0 (gdb) s socket_wait (ctl=0x779be0) at ftplib.c:144 144 { (gdb) 148 if ((ctl->dir == FTPLIB_CONTROL) || (ctl->idlecb == NULL)) (gdb) print ctl->idlecb $9 = (FtpCallback) 0x401e70 (gdb) s 150 if (ctl->dir == FTPLIB_WRITE) (gdb) 154 FD_ZERO(&fd); (gdb) print fd $10 = {__fds_bits = {13, 7838112, 4100, 4518680, 140734205318097, 140734205318097, 140734205318016, 4519571, 747324309664, 365072220369, 566935683113, 8192, 8240, 140734205318096, 140734205322352, 7838112}} (gdb) print *fd Structure has no component named operator*. (gdb) s 157 FD_SET(ctl->handle,&fd); (gdb) print fd $11 = {__fds_bits = {0 }} (gdb) print ctl $12 = (netbuf *) 0x779be0 (gdb) print ctl->handle $13 = 14 (gdb) s 159 rv = select(ctl->handle+1, rfd, wfd, NULL, &tv); ...cut... control ptr is still good. socket_wait (ctl=0x779be0) at ftplib.c:144 144 { (gdb) 148 if ((ctl->dir == FTPLIB_CONTROL) || (ctl->idlecb == NULL)) (gdb) 150 if (ctl->dir == FTPLIB_WRITE) (gdb) 154 FD_ZERO(&fd); (gdb) 157 FD_SET(ctl->handle,&fd); (gdb) 159 rv = select(ctl->handle+1, rfd, wfd, NULL, &tv); (gdb) 157 FD_SET(ctl->handle,&fd); (gdb) 159 rv = select(ctl->handle+1, rfd, wfd, NULL, &tv); (gdb) 157 FD_SET(ctl->handle,&fd); (gdb) 159 rv = select(ctl->handle+1, rfd, wfd, NULL, &tv); (gdb) 157 FD_SET(ctl->handle,&fd); (gdb) 159 rv = select(ctl->handle+1, rfd, wfd, NULL, &tv); (gdb) 158 tv = ctl->idletime; (gdb) 159 rv = select(ctl->handle+1, rfd, wfd, NULL, &tv); (gdb) 160 if (rv == -1) (gdb) 167 else if (rv > 0) (gdb) 173 while ((rv = ctl->idlecb(ctl, ctl->xfered, ctl->idlearg))); (gdb) Now we've called our callback function. ctl is still good. log_progress (ctl=0x779be0, xfered=5792, arg=0x7fff3c51260c) at download.c:56 56 { (gdb) print ctl->handle $31 = 14 (gdb) s 73 if(config->noprogressbar || (pct == 100 && lastpct == 100)) { (gdb) 57 int fsz = *(int*)arg; (gdb) s 58 int pct = ((float)(xfered+offset) / fsz) * 100; (gdb) print fsz $32 = 7968150 (gdb) $33 = 7968150 (gdb) s 73 if(config->noprogressbar || (pct == 100 && lastpct == 100)) { (gdb) 58 int pct = ((float)(xfered+offset) / fsz) * 100; (gdb) 73 if(config->noprogressbar || (pct == 100 && lastpct == 100)) { (gdb) 77 alpm_get_option(PM_OPT_CHOMP, (long *)&chomp); (gdb) alpm_get_option (parm=29 '\035', data=0x7fff3c50d2bc) at alpm.c:153 153 { (gdb) 155 ASSERT(handle != NULL, RET_ERR(PM_ERR_HANDLE_NULL, -1)); (gdb) 153 { (gdb) 155 ASSERT(handle != NULL, RET_ERR(PM_ERR_HANDLE_NULL, -1)); (gdb) 156 ASSERT(data != NULL, RET_ERR(PM_ERR_WRONG_ARGS, -1)); (gdb) 158 return(_alpm_handle_get_option(handle, parm, data)); (gdb) 159 } (gdb) alpm_get_option (parm=29 '\035', data=0x7fff3c50d2bc) at alpm.c:158 158 return(_alpm_handle_get_option(handle, parm, data)); (gdb) _alpm_handle_get_option (handle=0x600560, val=29 '\035', data=0x7fff3c50d2bc) at handle.c:315 315 ASSERT(handle != NULL, RET_ERR(PM_ERR_HANDLE_NULL, -1)); (gdb) 313 { (gdb) 315 ASSERT(handle != NULL, RET_ERR(PM_ERR_HANDLE_NULL, -1)); (gdb) 317 switch(val) { (gdb) 347 case PM_OPT_CHOMP: *data = handle->chomp; break; (gdb) 354 } (gdb) log_progress (ctl=<value optimized out>, xfered=5792, arg=<value optimized out>) at download.c:79 79 gettimeofday(&t1, NULL); (gdb) 80 if(xfered+offset == fsz) { (gdb) 83 timediff = t1.tv_sec-t.tv_sec + (float)(t1.tv_usec-t.tv_usec) / 1000000; (gdb) 94 } else if(timediff > 1) { (gdb) 97 rate = (xfered-xfered1) / (timediff * 1024); (gdb) 99 gettimeofday(&t, NULL); (gdb) 97 rate = (xfered-xfered1) / (timediff * 1024); (gdb) 99 gettimeofday(&t, NULL); (gdb) 97 rate = (xfered-xfered1) / (timediff * 1024); (gdb) 99 gettimeofday(&t, NULL); (gdb) 98 xfered1 = xfered; (gdb) 97 rate = (xfered-xfered1) / (timediff * 1024); (gdb) 99 gettimeofday(&t, NULL); (gdb) 100 eta_s = (fsz-(xfered+offset)) / (rate * 1024); (gdb) 101 eta_h = eta_s / 3600; (gdb) 104 eta_s -= eta_m * 60; (gdb) 101 eta_h = eta_s / 3600; (gdb) 102 eta_s -= eta_h * 3600; (gdb) 103 eta_m = eta_s / 60; (gdb) 104 eta_s -= eta_m * 60; (gdb) 107 if(rate > 1000) { (gdb) 110 printf("%*s %6dK %6.1fK/s %02d:%02d:%02d [", PM_DLFNM_LEN, sync_fnm, ((xfered+offset) / 1024), rate, eta_h, eta_m, eta_s); (gdb) 112 cur = (int)((maxcols-57)*pct/100); (gdb) 113 for(i = 0; i < maxcols-57; i++) { (gdb) 114 if(chomp) { (gdb) 138 (i < cur) ? printf("#") : printf(" "); (gdb) 113 for(i = 0; i < maxcols-57; i++) { ...cut... 113 for(i = 0; i < maxcols-57; i++) { (gdb) 141 printf("] %3d%%\r", pct); (gdb) 142 if(lastpct != 100 && pct == 100) { (gdb) 147 fflush(stdout); (gdb) 145 lastcur = cur; (gdb) 146 lastpct = pct; (gdb) 147 fflush(stdout); (gdb) vim-7.0.118-1 5K 0.0K/s 388:10:38 [ ] 0149 } (gdb) socket_wait (ctl=0x0) at ftplib.c:157 We've printed our progress bar and return to socket_wait but now ctl is NULL for some reason. Then we die on the pointer dereference below. 157 FD_SET(ctl->handle,&fd); (gdb) print ctl $41 = (netbuf *) 0x0 (gdb) print rv $42 = 1 (gdb) s Program received signal SIGSEGV, Segmentation fault. socket_wait (ctl=0x0) at ftplib.c:157 157 FD_SET(ctl->handle,&fd); (gdb) cleanup (signum=11) at pacman.c:183 183 { (gdb) where #0 cleanup (signum=11) at pacman.c:183 #1 <signal handler called> #2 socket_wait (ctl=0x0) at ftplib.c:157 #3 0x000000000041c68b in FtpRead (buf=0x779d60, max=8192, nData=0x5fb860) at ftplib.c:880 #4 0x000000000041e2e5 in FtpXfer ( localfile=0x7fff3c50e470 "/var/cache/pacman/pkg/vim-7.0.118-1.pkg.tar.gz.part", path=<value optimized out>, nControl=0x777810, typ=5792, mode=<value optimized out>) at ftplib.c:1140 #5 0x000000000041b3a3 in _alpm_downloadfiles_forreal ( servers=<value optimized out>, localpath=0x7fff3c5156b0 "/var/cache/pacman/pkg", files=0x7742d0, mtime1=0x0, mtime2=0x0) at server.c:368 #6 0x000000000041bade in _alpm_downloadfiles (servers=0xffffffff, localpath=0x0, files=0x5fb860) at server.c:137 #7 0x00000000004190e3 in _alpm_sync_commit (trans=0x6068d0, db_local=0x606d90, data=0x7fff3c518ad8) at sync.c:820 #8 0x00000000004136cd in _alpm_trans_commit (trans=0x6068d0, data=0x5fb860) at trans.c:217 #9 0x0000000000405ea7 in pacman_sync (targets=<value optimized out>) at sync.c:660 #10 0x0000000000407f4e in main (argc=<value optimized out>, argv=0x7fff3c519e38) at pacman.c:610 (gdb) The wierd thing is that ctl is not modified by log_progress. The only explanation I can think of besides a compiler optimization bug is that log_progress accesses the fsz variable which is just before the ctl variable in main. Moving the variable may fix it or refactoring the code in socket_wait to get different assembler output. k

On Thu, 2006-10-26 at 17:20 -0500, Aaron Griffin wrote:

On 10/26/06, K. Piche wrote:

...

The wierd thing is that ctl is not modified by log_progress. The only explanation I can think of besides a compiler optimization bug is that log_progress accesses the fsz variable which is just before the ctl variable in main. Moving the variable may fix it or refactoring the code in socket_wait to get different assembler output.

This is a close as anything could possibly come to modifying ctl, but in this case it's just being silly and preventing compier warnings, I'd assume.

src/pacman/download.c /* we don't need that parameter */ ctl=NULL;

Just in case the compiler decides to do-something-it-shouldn't, could you try switching this to a: /* we don't need that parameter */ (void)ctl;

That's as close as I can get

_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

I commented the ctl=NULL to no effect. Halfway stepping through log_progress ctl gets optimized away which somehow affects the caller which it obviously shouldn't. I tried playing around with the do {} while loop to hopefully get different code generation to no avail. Since log_progress always returns 1 the loop can become: if (stuff) break; blah ctl->idlecb(blah) } while (1) But ctl still disappears. Compiling with -O1 works as a workaround. k