- Pacman-dev - lists.archlinux.org

[pacman-dev] perl repo stuff
by Charles Mauch 15 Nov '06

15 Nov '06

Just thought I'd drop a note in here regarding some updates I've made to the perl repo. Today I squashed what I hope is the last bug in the build process. It's been semi-reliable for a while, but often seemed to require babysitting. If it makes it through tonight's build without errors (which i exect), I'm going to turn off the last human-checking mechanism I have in the process ( committing changes via svn to the repo ) to the script. Some notes on convention I've come up with, that maybe others might find useful. Some packages come from CPAN without a good description. Also, I found that including the actual CPAN module name in the description helps when searching. So all package descriptions are prefixed with the 'normal' perl module name. EG: Module::Foo:Bar - A perl module description from upstream I've also modified the wikipage somewhat so that less user-submitted modules are processed. I keep a local list handy which is used to seed the build process with modules that (I think) should be a part of the repo. This means that modules on this private list can't be removed at whim by a user mucking with the wiki. Eventually, it would be nice to have modules on this private list incorporated back into [community] or [extra]. I'm still working on a process to merge both lists so that dependencies from one list don't bleed into the other. But in any case, I'm hopeful that by the end of the week at latest the entire thing should be mostly self-maintaining. -- Take it easy, Charles FSF Apologist, WikiNut, Concrete Analyst, etc. JABBER: cmauch(a)gmail.com GIZMO: charlesmauch AIM: cmauch5 :: Playing "Crash And Carry" by Orbital (Snivilisation)

1 0

[pacman-dev] bug in pacman -Rs
by Roman Kyrylych 12 Nov '06

12 Nov '06

Hi! I was testing http://bugs.archlinux.org/task/5775 and asked Shadowhand to test it in pacman3. Here's his report: http://bugs.archlinux.org/?getfile=884 I think this is very interesting. -- Roman Kyrylych (Роман Кирилич)

1 0

[pacman-dev] issues again about our patch queue
by VMiklos 10 Nov '06

10 Nov '06

hi Aaron (and others), krix just told me that you have problems with our patch queue. may i ask you to: 1) discuss such problems here, not in private mails or on irc 2) mention what is the problem with our patch queue udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

3 10

Re: [pacman-dev] issues again about our patch queue
by Gabriel C 10 Nov '06

10 Nov '06

Christian Hamar alias krix wrote: > On 11/9/06, VMiklos <vmiklos(a)frugalware.org> wrote: > >>> krix just told me that you have problems with our patch queue. may i ask >>> you to: >>> >>> 1) discuss such problems here, not in private mails or on irc >>> >> The problem was brought up in private because I noticed it _as I was >> writing an email to him_. >> >> >>> 2) mention what is the problem with our patch queue >>> >> I'm not sure what you mean by "patch queue". My issue is this from >> darcs.frugalware.org: >> "pacman Frugalware's .tar.bz2 based package manager with depende ..." >> >> Really... I mean, looking around it just appears that you guys are >> running a fork, and not "pacman". >> >> As a corollary, let's look at IceWeasel - it has the Firefox code, >> plus a few additional patches, they regularly sync with upstream, etc >> etc. It sounds alot like what you guys do with "Frugalware's package >> manager". >> Heh , I can tell you we did not fork. Before you post such an example pls go figure the Firefox License and you will notice *you are forced to rename if you don't distribute 'as is'*. So you have to 1) send all changes to the FF devels to be approved ( if they don't and you still want the changes you need rename ) 2) don't do so and follow the license and rename. So this is a bad example. Anyway to _not_ rename a fork is idiotic. BTW do you think a program with some 'cutom patches is a fork ?' Or what is your _real_ problem *now* with the darcs repo ? before you merged all fixes 'it was OK' now isn't anymore ? Every Distro is using pacman 'will' ( and you can't change this ) add custom patches , so do you want all to 'fork' ? You don't think this may be _bad_ for ArchLinux losing all the contributors ? >> I'm not trying to say anything bad by it. It's just customary to make >> a name change when running a fork (I even think the GPL requires it, >> but I may be mistaken). I mean, lets look at it from a user's point >> of view: >> >> "Oh hey, pacman, I used that on Arch! Wait... why doesn't that option >> work? What's going on? Where's the PKGBUILD for that? FrugalBuild, >> huh? foo-1.0-1-i686.fpm, what?". This is the situation with other PM's too ( rpm , apt etc ). Why a Distro should be *forced* to use the same names as ArchLinux ? Or the same build system ? There isn't any reason to do so. So want you want is something different. You want us to 'rename pacman' ( but not fork ) so such ArchLinux -> Frugalware , Frugalware -> ArchLinux merges are still possible , yes ? Regards Gabriel

1 0

[pacman-dev] problem with the configure.ac change
by VMiklos 01 Nov '06

01 Nov '06

hi - CFLAGS="-g -Wall -Werror -std=c99 -DPACMAN_DEBUG" + CFLAGS="$CFLAGS -g -Wall -Werror -std=c99 -DPACMAN_DEBUG" this change means that in case the user has set CFLAGS, we'll use it. this is very bad during the debug, for example we have -O2 in CFLAGS by default and that means small functions will be optimized out and so on. that "ignore $CFLAGS if debug enabled" thingy was not by accident. if we debug then we should ignore the more or less problematic cflags in addition of you use both -O2 and -Werror, you'll get additional warnings wich will break the build.. udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

2 4

[pacman-dev] cvs unusable on x86_64
by VMiklos 28 Oct '06

28 Oct '06

hi just wanted to let you know that the current cvs segfaults on pacman -S anything. i'm not sure about what is the problem, reverting this diff: http://frugalware.org/~vmiklos/patches/pacman_progress.diff works around the problem. i've tried to fix it but i don't see any problem. it should be some memory corruption since if the build is debug-enabled then it works fine since i'm not the author of the change, it's not my task to fix it - good luck for now i've reverted it in our tree udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

3 7

Re: [pacman-dev] cvs unusable on x86_64
by Piche, Kevin 28 Oct '06

28 Oct '06

I looked at this too. The SIG11 goes away if you compile without -O2. When looking at it with gdb most function arguments seemed to be "optimized out". If you compile without the -O2 it still displays strangely. The SIG11 happens when vim*.part starts downloading. Frankly I don't like ftplib - it doesn't work with proxies so I have to use wget. Maybe we should switch to libcurl or something. k -----Original Message----- From: pacman-dev-bounces(a)archlinux.org [mailto:pacman-dev-bounces@archlinux.org] On Behalf Of Christian Hamar alias krix Sent: Wednesday, October 25, 2006 7:42 AM To: Discussion list for pacman development Subject: Re: [pacman-dev] cvs unusable on x86_64 > still segfaults. i would suggest asking a shell access from an Arch64 > dev, i neither have such a machine phisically. anyway here is the > commandline to reproduce the issue: > > vmiklos@helicon:~/darcs/pacman/src/pacman$ sudo rm -f > /var/cache/pacman/pkg/vim* /tmp/pacman.lck; sudo ./pacman -Sw vim > --noconfirm > resolving dependencies... done. > > Targets: vim-7.0-2 > > Total Package Size: 5.0 MB > > Total Uncompressed Package Size: 16.3 MB > > Beginning download... > > :: Retrieving packages from frugalware-current... > Internal pacman error: Segmentation fault00:00:00 [ > ] 0% > Please submit a full bug report, with the given package if appropriate. Strange. I did some work at this code. I found one thing only which will be x86_64 specific. In pacman.c there is a callback point (as i remember it was in pacman.c :) ) which got some : (long)log_progress() definition. Actually in download.c (and in.h) the log_progress function is int log_progress(...) I tried to change from int -> long in download.c and .h to see that cause the problem or not. No, that wasnt :) (as you know on x86_64 int and long matters, because of size difference) So the problem still exists. I catched up some backtrace with the not debug binary pacman and seems that it segfaults at FtpRead() function. That is strange too. A notice for this, the main progressbar moves to some ~5 or 10% and crashing after that. So it will be some magic buffer overrun somewhere :S Still i will work on it (cause i got x86_64 machine) and maybe see what i can do. Regards Christian Hamar alias krix _______________________________________________ pacman-dev mailing list pacman-dev(a)archlinux.org http://www.archlinux.org/mailman/listinfo/pacman-dev

6 16

[pacman-dev] [PATCH] alpm_parse_config(): add missing sanity checks
by VMiklos 28 Oct '06

28 Oct '06

hi http://darcs.frugalware.org/patches/pacman/20061027195432-e2957-0dcda6bdb2d… i think the patch is obvious, anyway if there are objections, please let me know udv / greetings, VMiklos -- Developer of Frugalware Linux, to make things frugal - http://frugalware.org

2 1

[pacman-dev] backends
by Aaron Griffin 28 Oct '06

28 Oct '06

Split. On 10/26/06, Roman Kyrylych <roman.kyrylych(a)gmail.com> wrote: > 2006/10/26, Aaron Griffin <aaronmgriffin(a)gmail.com>: > > On 10/26/06, Roman Kyrylych <roman.kyrylych(a)gmail.com> wrote: > > > [offtopic on] > > > BTW, about complex patches - I have a patch from Martin Devera that > > > adds support for gdbm database backend to pacman 2.9.8. I suggested > > > him to contact VMiklos for help in inclusion it in pacman, but I don't > > > know if he did that. If I adapt it for Pacman 3 API is there a chance > > > it will be included at least in CVS/DARCS? > > > [/offtopic off] > > > > There's no chance that would get included in pacman 2.9.8 - it will die soon. > > > > Converting it to pacman3 is entirely doable and it's already setup to > > do so. You should be able to copy be_files.c to be_gdbm.c and consume > > the interface provided there. > > > > FTR 'be_' means 'backend'. I still need to add compile-time flags for > > selecting a backend, but as we only provide _files at the moment, > > there's no need. > > > > I worked on a 'compressed' backend that required no changes, but read > > directly from the downloaded db files... it caused problems on the > > local db, however, so I never completed it (yet). > > > > As for backend schemes, I think we need to discuss avenues of attack > > here. The files backend is painfully slow on ext filesystems (11 > > seconds to parse [community]). There are many users touting a > > database backend as a good idea (I highly disagree), and if we do > > this, we NEED a generic layer to support any sane database format, not > > just one specific thing. Using a flat-file database is a better > > option, but here people seem to think sqlite is a good idea (it's not) > > - there are many flat-file schemes that can drastically outperform > > sqlite. > > > > There are many many other options, and I think it's worth at least > > _supplying_ multiple backends, even if they're not used. So gdbm is > > cool. > > Maybe we should start separate thread for this? > Personally, I think sqlite will be the best choice because of: > 1) it's small > 2) it's fast > 3) it's embedable Ok, sqlite is "fast and etc etc", right? So if I do [pacman -Ss foo], what does sqlite do? It opens the file (exactly as plaintext searching would), initializes some structures and things like that describing table data, then proceeds to sequentially search (FULL TABLE SCAN) through all rows. It is impossible, even in a good database, to index substrings. Yes, you can index entire strings, that's easy. The search operations check internal substrings, not whole strings. You could make this faster by converting each entry into a suffix tree and storing that somewhere, but that gets irritating and gets into that crappy Computer Science stuf everyone hates. Fact of the matter is, there's two main "slow" operations - searching, and loading the whole DB (in the case of an -Su). Loading the whole database can easilly be sped up by using lazy package evaluation - that is, a simple readdir() or the db directory gives us package name, version, and release - all we need for a upgrade check. The additional data can be read when/if required. Substring searching will never be uber fast, but we can maintain a minor indexed file with all text that's searched... something like: extra/package-name-1.0-1 : This is the description for the package community/package-two-1.1.1-1 : This is the description for this thing extra/another-99-1 : This is the description for some stuff -Ss can check this file, and use the first entry before the colon to construct the path to open (though that may not be needed as all the information is right there). Ok, I'm gonna stop myself before I rant too much. There are cries of "use a database" from many people. Frankly, I don't think that these people have evaluated all the options and are stuck in the "everyone uses databases for their websites" mentality. Take this for instance: The google engineers here in Chicago explained to me something they use to store whatever the hell google stores there: A custom filesystem tooled to their spec. They don't use mysql or oracle or sqlite any of that - the king of searching uses the simplest of all options. > 4) it offers full power of SQL99 (with ACID transactions!!!) Nope. Not at all. It doesn't even support SQL92 - a 14 year old standard. http://www.sqlite.org/omitted.html http://www.sqlite.org/cvstrac/wiki?p=UnsupportedSql They have said publicly they do not plan on adding support. Standards compliance is HUGE in my book.

5 10

[pacman-dev] changes coming soon
by Aaron Griffin 28 Oct '06

28 Oct '06

So I have a bunch of local changes that I should be committing soon. Just wanted to give you all a "heads up". 1) "lazy caching" - default infolevel for pkgcache set to INFRQ_NONE, added infolevel to the get_pkgcache function, to read only the files needed. This improved -Ss speed and -Su speed by *alot* 2) switch to libfetch - mostly internal changes to server.c download_files_forreal 3) pm_dlcb format change (this is the big one). Because the callback was now in my hands (thanks libfetch) and not part of a library, I had free reign on the format. I really don't like the 10-or-so library-level variables for the callback, so I removed those. The new callback signature is: (const char *filename, int xfered, int total) Every single thing needed by the callback can be calculated from the current and total bytes. The callback is also called before download starts with an xfered == 0. This is useful to initialize the start time and things like that. This is still up for interpretation - if someone sees a problem that I do not, let me know. 4) I implemented the "ILoveCandy" progress on the secondary progress bars. I also slightly changed the format from "C*****" to "C o o o o", just for kicks. If any of these changes are problematic, let me know.... it's alot to check in at once, but all this was due to libfetch integration. Thanks, Aaron

3 4