October 2006 - Pacman-dev - lists.archlinux.org

[pacman-dev] [PATCH] libalpm: new config option - maxtries
by VMiklos 07 Dec '06

07 Dec '06

hi i've split up the patch to 3 parts for easier reviewing: the feature itself: http://darcs.frugalware.org/darcsweb/darcsweb.cgi?r=pacman;a=commit;h=20061… indentation fixup: http://darcs.frugalware.org/darcsweb/darcsweb.cgi?r=pacman;a=commit;h=20061… documentation: http://darcs.frugalware.org/darcsweb/darcsweb.cgi?r=pacman;a=commit;h=20061… udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

5 9

[pacman-dev] problem with the configure.ac change
by VMiklos 01 Nov '06

01 Nov '06

hi - CFLAGS="-g -Wall -Werror -std=c99 -DPACMAN_DEBUG" + CFLAGS="$CFLAGS -g -Wall -Werror -std=c99 -DPACMAN_DEBUG" this change means that in case the user has set CFLAGS, we'll use it. this is very bad during the debug, for example we have -O2 in CFLAGS by default and that means small functions will be optimized out and so on. that "ignore $CFLAGS if debug enabled" thingy was not by accident. if we debug then we should ignore the more or less problematic cflags in addition of you use both -O2 and -Werror, you'll get additional warnings wich will break the build.. udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

2 4

[pacman-dev] cvs unusable on x86_64
by VMiklos 28 Oct '06

28 Oct '06

hi just wanted to let you know that the current cvs segfaults on pacman -S anything. i'm not sure about what is the problem, reverting this diff: http://frugalware.org/~vmiklos/patches/pacman_progress.diff works around the problem. i've tried to fix it but i don't see any problem. it should be some memory corruption since if the build is debug-enabled then it works fine since i'm not the author of the change, it's not my task to fix it - good luck for now i've reverted it in our tree udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

3 7

Re: [pacman-dev] cvs unusable on x86_64
by Piche, Kevin 28 Oct '06

28 Oct '06

I looked at this too. The SIG11 goes away if you compile without -O2. When looking at it with gdb most function arguments seemed to be "optimized out". If you compile without the -O2 it still displays strangely. The SIG11 happens when vim*.part starts downloading. Frankly I don't like ftplib - it doesn't work with proxies so I have to use wget. Maybe we should switch to libcurl or something. k -----Original Message----- From: pacman-dev-bounces(a)archlinux.org [mailto:pacman-dev-bounces@archlinux.org] On Behalf Of Christian Hamar alias krix Sent: Wednesday, October 25, 2006 7:42 AM To: Discussion list for pacman development Subject: Re: [pacman-dev] cvs unusable on x86_64 > still segfaults. i would suggest asking a shell access from an Arch64 > dev, i neither have such a machine phisically. anyway here is the > commandline to reproduce the issue: > > vmiklos@helicon:~/darcs/pacman/src/pacman$ sudo rm -f > /var/cache/pacman/pkg/vim* /tmp/pacman.lck; sudo ./pacman -Sw vim > --noconfirm > resolving dependencies... done. > > Targets: vim-7.0-2 > > Total Package Size: 5.0 MB > > Total Uncompressed Package Size: 16.3 MB > > Beginning download... > > :: Retrieving packages from frugalware-current... > Internal pacman error: Segmentation fault00:00:00 [ > ] 0% > Please submit a full bug report, with the given package if appropriate. Strange. I did some work at this code. I found one thing only which will be x86_64 specific. In pacman.c there is a callback point (as i remember it was in pacman.c :) ) which got some : (long)log_progress() definition. Actually in download.c (and in.h) the log_progress function is int log_progress(...) I tried to change from int -> long in download.c and .h to see that cause the problem or not. No, that wasnt :) (as you know on x86_64 int and long matters, because of size difference) So the problem still exists. I catched up some backtrace with the not debug binary pacman and seems that it segfaults at FtpRead() function. That is strange too. A notice for this, the main progressbar moves to some ~5 or 10% and crashing after that. So it will be some magic buffer overrun somewhere :S Still i will work on it (cause i got x86_64 machine) and maybe see what i can do. Regards Christian Hamar alias krix _______________________________________________ pacman-dev mailing list pacman-dev(a)archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

6 16

[pacman-dev] [PATCH] alpm_parse_config(): add missing sanity checks
by VMiklos 28 Oct '06

28 Oct '06

hi http://darcs.frugalware.org/patches/pacman/20061027195432-e2957-0dcda6bdb2d… i think the patch is obvious, anyway if there are objections, please let me know udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

2 1

[pacman-dev] backends
by Aaron Griffin 28 Oct '06

28 Oct '06

Split. On 10/26/06, Roman Kyrylych <roman.kyrylych(a)gmail.com> wrote: > 2006/10/26, Aaron Griffin <aaronmgriffin(a)gmail.com>: > > On 10/26/06, Roman Kyrylych <roman.kyrylych(a)gmail.com> wrote: > > > [offtopic on] > > > BTW, about complex patches - I have a patch from Martin Devera that > > > adds support for gdbm database backend to pacman 2.9.8. I suggested > > > him to contact VMiklos for help in inclusion it in pacman, but I don't > > > know if he did that. If I adapt it for Pacman 3 API is there a chance > > > it will be included at least in CVS/DARCS? > > > [/offtopic off] > > > > There's no chance that would get included in pacman 2.9.8 - it will die soon. > > > > Converting it to pacman3 is entirely doable and it's already setup to > > do so. You should be able to copy be_files.c to be_gdbm.c and consume > > the interface provided there. > > > > FTR 'be_' means 'backend'. I still need to add compile-time flags for > > selecting a backend, but as we only provide _files at the moment, > > there's no need. > > > > I worked on a 'compressed' backend that required no changes, but read > > directly from the downloaded db files... it caused problems on the > > local db, however, so I never completed it (yet). > > > > As for backend schemes, I think we need to discuss avenues of attack > > here. The files backend is painfully slow on ext filesystems (11 > > seconds to parse [community]). There are many users touting a > > database backend as a good idea (I highly disagree), and if we do > > this, we NEED a generic layer to support any sane database format, not > > just one specific thing. Using a flat-file database is a better > > option, but here people seem to think sqlite is a good idea (it's not) > > - there are many flat-file schemes that can drastically outperform > > sqlite. > > > > There are many many other options, and I think it's worth at least > > _supplying_ multiple backends, even if they're not used. So gdbm is > > cool. > > Maybe we should start separate thread for this? > Personally, I think sqlite will be the best choice because of: > 1) it's small > 2) it's fast > 3) it's embedable Ok, sqlite is "fast and etc etc", right? So if I do [pacman -Ss foo], what does sqlite do? It opens the file (exactly as plaintext searching would), initializes some structures and things like that describing table data, then proceeds to sequentially search (FULL TABLE SCAN) through all rows. It is impossible, even in a good database, to index substrings. Yes, you can index entire strings, that's easy. The search operations check internal substrings, not whole strings. You could make this faster by converting each entry into a suffix tree and storing that somewhere, but that gets irritating and gets into that crappy Computer Science stuf everyone hates. Fact of the matter is, there's two main "slow" operations - searching, and loading the whole DB (in the case of an -Su). Loading the whole database can easilly be sped up by using lazy package evaluation - that is, a simple readdir() or the db directory gives us package name, version, and release - all we need for a upgrade check. The additional data can be read when/if required. Substring searching will never be uber fast, but we can maintain a minor indexed file with all text that's searched... something like: extra/package-name-1.0-1 : This is the description for the package community/package-two-1.1.1-1 : This is the description for this thing extra/another-99-1 : This is the description for some stuff -Ss can check this file, and use the first entry before the colon to construct the path to open (though that may not be needed as all the information is right there). Ok, I'm gonna stop myself before I rant too much. There are cries of "use a database" from many people. Frankly, I don't think that these people have evaluated all the options and are stuck in the "everyone uses databases for their websites" mentality. Take this for instance: The google engineers here in Chicago explained to me something they use to store whatever the hell google stores there: A custom filesystem tooled to their spec. They don't use mysql or oracle or sqlite any of that - the king of searching uses the simplest of all options. > 4) it offers full power of SQL99 (with ACID transactions!!!) Nope. Not at all. It doesn't even support SQL92 - a 14 year old standard. http://www.sqlite.org/omitted.html http://www.sqlite.org/cvstrac/wiki?p=UnsupportedSql They have said publicly they do not plan on adding support. Standards compliance is HUGE in my book.

5 10

[pacman-dev] changes coming soon
by Aaron Griffin 28 Oct '06

28 Oct '06

So I have a bunch of local changes that I should be committing soon. Just wanted to give you all a "heads up". 1) "lazy caching" - default infolevel for pkgcache set to INFRQ_NONE, added infolevel to the get_pkgcache function, to read only the files needed. This improved -Ss speed and -Su speed by *alot* 2) switch to libfetch - mostly internal changes to server.c download_files_forreal 3) pm_dlcb format change (this is the big one). Because the callback was now in my hands (thanks libfetch) and not part of a library, I had free reign on the format. I really don't like the 10-or-so library-level variables for the callback, so I removed those. The new callback signature is: (const char *filename, int xfered, int total) Every single thing needed by the callback can be calculated from the current and total bytes. The callback is also called before download starts with an xfered == 0. This is useful to initialize the start time and things like that. This is still up for interpretation - if someone sees a problem that I do not, let me know. 4) I implemented the "ILoveCandy" progress on the secondary progress bars. I also slightly changed the format from "C*****" to "C o o o o", just for kicks. If any of these changes are problematic, let me know.... it's alot to check in at once, but all this was due to libfetch integration. Thanks, Aaron

3 4

[pacman-dev] [PATCH] small fix in bindings directory
by Roman Kyrylych 27 Oct '06

27 Oct '06

-- Roman Kyrylych (Роман Кирилич)

4 13

[pacman-dev] some differences between Arch's and Frugalware's Pacman branches
by Roman Kyrylych 26 Oct '06

26 Oct '06

Hi! Today I did checkout of CVS and DARCS and found some differences: 1) Why there is "typedef struct __pmlist_t PMList;" in DARCS (libalpm/list.h)? What it gives? IMHO it's inconsistent with pmdb_t, pmpkg_t etc. Isn't this difference completely useless and even bad because of different API? 2) There are differences in progressbar and log implementations. Some code has been moved from libalpm/log.h and libalpm/server.h to libalpm/handle.c in Frugalware's branch. 3) _alpm_db_search, alpm_pkg_load, alpm_pkg_free call _alpm_log in Arch's branch. 4) alpm_cb_log has been moved from log.h to alpm.h in Frugalware's branch. 5) There's no _alpm_db_register callback in Frugalware's branch. 6) There is local time format support in cb_log in Arch's branch. In libalpm\alpm.c lines 1213-1214 (1256-1275 in FwPacman) are different. Does this show the usefulness of _alpm_db_register? Very little differences, some looks a bit strange for me: 1) In pacman/sync.c, lines 164 and 204 - two strange little cosmethic changes. :-/ 2) There's "extern pmhandle_t *handle;" on top of most *.c files in Frugalware's branch in libalpm. 3) alpm_conflict_getinfo abd alpm_db_update have slightly different Doxygen comments. Well, that's all. Sorry if I take your time but I want to learn more about Pacman internals, so it will be nice if someone can explain these differences. ;-) And the last - a bug in Arch's branch: 1) libalpm\alpm.c, lines 470, 724, 753, 797, 976 - "void *" should be "char *" -- Roman Kyrylych (Роман Кирилич)

3 4

[pacman-dev] wrong description of configure switches
by Roman Kyrylych 26 Oct '06

26 Oct '06

Hi! I was trying to build pacman3 few minutes ago and noticed that some configure switches have wrong descriptions: --enable-man2html Build html docs via man2html --disable-po4a Build localized manpages via po4a --disable-doxygen Build API docs via Doxygen --disable-python Build Python bindings --disable-perl Build Perl bindings --disable-java Build Java bindings --enable-debug Enable debugging support Here --disable-* should not be described as "Build something", but "Do not build something" instead. -- Roman Kyrylych (Роман Кирилич)

2 5