[pacman-dev] My thoughts for development
Since I've been spending quite a bit of time looking at the code lately, I've been making a TODO list of things worth looking into. I thought I'd put it up on the list for input. makepkg issues: 1. We went through and added some of the undocumented options to the makepkg.conf file, but I haven't done a whole lot of testing on this to see if it works correctly. I also don't know if all the checking is correct, and I see Aaron put some x's in front of these checks. Has anyone else? They also aren't documented in the man page if anyone wants to step up to the plate... 2. Remove use of multiple `` or $() calls to the same function. I have a local update of calling $(id -u) once and storing it to a variable for later use, I can publish a patch sometime. 3. I posted a patch for #5223 (source cache) and it has been patched. However, it is now a triple conditional with three different possible locations for package source (current dir, $SRCDEST, and /var/cache/pacman/src). Is the last one necessary since it is highly recommended to not build as root? Should we deprecate this path? Looking even further ahead, should we make fakeroot a requirement for running makepkg? 4. Aaron, you brought up that `` is the older standard and $() should be used. It is a pretty simple find that will allow you to change these for consistency, I've done it on my working copy. 5. Already mentioned, but a modeline in makepkg. 6. Overabundance of options in makepkg- I was trying to find a new letter for a '--repackage' option and we are pretty much out of lowercase letters. At the moment we have two basic types of options: things that can be done in the pkgbuild options line, and things that directly affect the way makepkg works (geninteg, skip integs, location of PKGBUILD). Are the first type truly necessary? It would make makepkg a bit more streamlined if the only options were those that can't be done somewhere else. So i propose the '--nostrip', '--keepdocs', and any other option like this be removed. 7. Integrity checking- currently generating integrity checks will only generate those from the first one in the INTEGRITY_CHECK array. This seems to make sense as long as the array in makepkg.conf is ordered the way we want it to be. When checking, we do get warning messages for other types that may be in the array but are not in the PKGBUILD. I just wanted to confirm this is the behavior you are looking for. Other issues: 1. Move the PKGBUILD man page to section 5 2. How are you guys doing side-by-side builds with the old pacman? At least for me, the current build system seems horribly complicated with shell scripts, configure, automaking and such. Is there a good reason for this? 3. Everyone talks about pacman slowness (even though it isn't really that bad), and looking though the mailing list archives I believe Aaron implemented a form of lazy caching (not loading the descriptions when unnecessary). Anything else on this front? I thought of something and I may be way off track, but what if when generating a repository we make (in addition to the flat file format) a single file with lines consisting of pkgname-pkgver:description? This file could be loaded and searched a lot quicker than flat files, and I think the operation people complain about being slow the most is a search. Wow, long list, but maybe it will provoke other great ideas. Feel free to respond to any or all of it. -Dan
On 12/20/06, Dan McGee <dpmcgee@gmail.com> wrote:
a lot
Wow! That's a lot of text... lemme see here. Before I get into this, IRC or jabber makes discussions like this alot "easier" - do you, perhaps, use either regularly? For anyone else interested: jabber - phrakture@phrakture.net irc - phrakture on #archlinux (I use other protocols as well, but I prefer these two)
1. We went through and added some of the undocumented options to the makepkg.conf file, but I haven't done a whole lot of testing on this to see if it works correctly. I also don't know if all the checking is correct, and I see Aaron put some x's in front of these checks. Has anyone else? They also aren't documented in the man page if anyone wants to step up to the plate...
The "x$foo" bash thing is for safety. I can't remember the exact case exactly, but it has to do with evaluation of strings when comparing (it's one of those things I've just always done)... I'm drawing a blank... As far as testing goes... it's always needed. It's hard, for some reason, to get people testing pacman3. I'm not really sure why. Some people are really good at it (stonecrest, off the top of my head) but they aren't on this list IIRC.
2. Remove use of multiple `` or $() calls to the same function. I have a local update of calling $(id -u) once and storing it to a variable for later use, I can publish a patch sometime. 4. Aaron, you brought up that `` is the older standard and $() should be used. It is a pretty simple find that will allow you to change these for consistency, I've done it on my working copy.
Yeah, I need to do that. /me adds it to my "do this tonight" list
3. I posted a patch for #5223 (source cache) and it has been patched. However, it is now a triple conditional with three different possible locations for package source (current dir, $SRCDEST, and /var/cache/pacman/src). Is the last one necessary since it is highly recommended to not build as root? Should we deprecate this path? Looking even further ahead, should we make fakeroot a requirement for running makepkg?
Hmmm. Building as root _is_ actually used in one or two oddball cases (Judd knows what they are) so explicitly denying them may not work. As for the SRCDEST check... why not default SRCDEST to the pacman cache dir, and just check one single place for it. It makes sense to me.
5. Already mentioned, but a modeline in makepkg.
* Added to the list
6. Overabundance of options in makepkg- I was trying to find a new letter for a '--repackage' option and we are pretty much out of lowercase letters. At the moment we have two basic types of options: things that can be done in the pkgbuild options line, and things that directly affect the way makepkg works (geninteg, skip integs, location of PKGBUILD). Are the first type truly necessary? It would make makepkg a bit more streamlined if the only options were those that can't be done somewhere else. So i propose the '--nostrip', '--keepdocs', and any other option like this be removed.
I couldn't agree more. Perhaps it's time to split makepkg into two different scripts. I don't know if I can see a good dividing line, however. Alternatively, we don't really need command line options for _every_ config setting. Perhaps we can limit these in some way... I'll look into it tonight as well.
7. Integrity checking- currently generating integrity checks will only generate those from the first one in the INTEGRITY_CHECK array. This seems to make sense as long as the array in makepkg.conf is ordered the way we want it to be. When checking, we do get warning messages for other types that may be in the array but are not in the PKGBUILD. I just wanted to confirm this is the behavior you are looking for.
Erm. It should have generated all of them. If it's not looped, then it's an oversight on my part. I actually tested some stuff with md5 and sha1, then moved back to just md5 for building some packages, so I probably missed it.
Other issues: 1. Move the PKGBUILD man page to section 5
5 does make more sense. Good call *added to the list*
2. How are you guys doing side-by-side builds with the old pacman? At least for me, the current build system seems horribly complicated with shell scripts, configure, automaking and such. Is there a good reason for this?
Well, autotools is always a PITA. I'd like to move away from it in the future. I haven't pushed this to this list yet, for a handful of reasons, but here: http://archlinux.org/~aaron/pacman/ That's a pacman3 side-by-side install. However, I noticed an issue (actually travis did) where if you were to upgrade initscripts, it overwrites rc.conf *BUT* it only does it the first time you use this package... it did it once, and now it all seems fine. So it's something goofy and I need to track that down tonight.
3. Everyone talks about pacman slowness (even though it isn't really that bad), and looking though the mailing list archives I believe Aaron implemented a form of lazy caching (not loading the descriptions when unnecessary). Anything else on this front? I thought of something and I may be way off track, but what if when generating a repository we make (in addition to the flat file format) a single file with lines consisting of pkgname-pkgver:description? This file could be loaded and searched a lot quicker than flat files, and I think the operation people complain about being slow the most is a search.
Yeah, well, there's a few things which are slow. In the future, I think the best way to do this is simply to change the backend which pacman is using. The 'files' backend is about at it's limit without hacky stuff surrounding it. I have a few ideas for this. I actually implemented one that never unpacked the archive (db.tar.gz) files, and read them everytime, and it was like 20x faster. The problem, however, is with the 'local' DB. It needs to be written to, so the archive way was actually a detriment there (unpack, add entries, repack). Ideas for backends are always welcome. The interface exists in be_files.c in libalpm. It's fairly straightforward. People seem to prefer a "real database" backend (while at the same time suggesting sqlite which is far from a real database, but I digress), but I think that is overkill. There are a few ideas I had milling around, but haven't fleshed anything out - discussion is always a good idea. Anyway, I'm just responding 1-1 to your ideas. Pacman discussions are always good, so keep it up. And any of you list lurkers who have ideas - feel free to jump in. Every idea has the capability to be ground-breaking, no matter how silly it seems to you. 8)
Hi, On Wed, Dec 20, 2006 at 01:47:10PM -0600, Aaron Griffin wrote:
The "x$foo" bash thing is for safety. I can't remember the exact case exactly, but it has to do with evaluation of strings when comparing (it's one of those things I've just always done)... I'm drawing a blank...
I also was surprised by this: [ "x$NOBUILD" = "1"] will never be true, because it will always evaluate some thing like [ "x..." = "1" ] the safety workaround is only required if you don't double quote (possible empty) variables. For example: bash-3.2$ unset EMPTY bash-3.2$ [ $EMPTY = "" ] && echo empty bash: [: =: unary operator expected yes, $EMPTY evaluates to NOTHING, so one operator is missing bash-3.2$ [ "$EMPTY" = "" ] && echo empty empty "$EMPTY" evaluates to an empty string -> OK bash-3.2$ [ x$EMPTY = x ] && echo empty empty uses a fixed prefix as workaround for empty variables. Jürgen
On 12/20/06, Aaron Griffin <aaronmgriffin@gmail.com> wrote:
On 12/20/06, Dan McGee <dpmcgee@gmail.com> wrote:
a lot
Wow! That's a lot of text... lemme see here. Before I get into this, IRC or jabber makes discussions like this alot "easier" - do you, perhaps, use either regularly?
I'm on IRC every once in a while, but not too likely over the next few weeks here, because I'm at home and not at school. I've never used jabber but have no real reason not to, but like I said, these would be easier in the new year. The only other issue is a common time when people are on. But I definitely have some other things that would be easier to talk about on a real-time basis, so I'll let you know.
3. I posted a patch for #5223 (source cache) and it has been patched. However, it is now a triple conditional with three different possible locations for package source (current dir, $SRCDEST, and /var/cache/pacman/src). Is the last one necessary since it is highly recommended to not build as root? Should we deprecate this path? Looking even further ahead, should we make fakeroot a requirement for running makepkg?
Hmmm. Building as root _is_ actually used in one or two oddball cases (Judd knows what they are) so explicitly denying them may not work. As for the SRCDEST check... why not default SRCDEST to the pacman cache dir, and just check one single place for it. It makes sense to me.
I'll look into this and see if I can't throw another patch together.
7. Integrity checking- currently generating integrity checks will only generate those from the first one in the INTEGRITY_CHECK array. This seems to make sense as long as the array in makepkg.conf is ordered the way we want it to be. When checking, we do get warning messages for other types that may be in the array but are not in the PKGBUILD. I just wanted to confirm this is the behavior you are looking for.
Erm. It should have generated all of them. If it's not looped, then it's an oversight on my part. I actually tested some stuff with md5 and sha1, then moved back to just md5 for building some packages, so I probably missed it.
This happens because of the 'exit 0' line at the end of the 'if [ "$GENINTEG" = "1" ]; then' loop. It exits after running that entire loop...but not the entire INTEGRITY_CHECK loop. The way it is currently structured makes this hard to change besides putting another if after the outer loop checking the $GENINTEG variable and exiting if it is set so we do not continue on and do the building.
2. How are you guys doing side-by-side builds with the old pacman? At least for me, the current build system seems horribly complicated with shell scripts, configure, automaking and such. Is there a good reason for this?
Well, autotools is always a PITA. I'd like to move away from it in the future. I haven't pushed this to this list yet, for a handful of reasons, but here: http://archlinux.org/~aaron/pacman/
That's a pacman3 side-by-side install. However, I noticed an issue (actually travis did) where if you were to upgrade initscripts, it overwrites rc.conf *BUT* it only does it the first time you use this package... it did it once, and now it all seems fine. So it's something goofy and I need to track that down tonight.
Thanks.
2006/12/20, Aaron Griffin <aaronmgriffin@gmail.com>:
3. I posted a patch for #5223 (source cache) and it has been patched. However, it is now a triple conditional with three different possible locations for package source (current dir, $SRCDEST, and /var/cache/pacman/src). Is the last one necessary since it is highly recommended to not build as root? Should we deprecate this path? Looking even further ahead, should we make fakeroot a requirement for running makepkg?
Hmmm. Building as root _is_ actually used in one or two oddball cases (Judd knows what they are) so explicitly denying them may not work. As for the SRCDEST check... why not default SRCDEST to the pacman cache dir, and just check one single place for it. It makes sense to me.
Currently makepkg downloads files with wget into the same dir where PKGBUILD is. So if we gonna use SRCDEST only, it should write it there. But /var/cache/pacman/src is owned by root. :( It's weird, but even when I make .../src dir owned by my user - makepkg doesn't put file there after download. What fakeroot gives instead of chroot?
6. Overabundance of options in makepkg- I was trying to find a new letter for a '--repackage' option and we are pretty much out of lowercase letters. At the moment we have two basic types of options: things that can be done in the pkgbuild options line, and things that directly affect the way makepkg works (geninteg, skip integs, location of PKGBUILD). Are the first type truly necessary? It would make makepkg a bit more streamlined if the only options were those that can't be done somewhere else. So i propose the '--nostrip', '--keepdocs', and any other option like this be removed.
No! Because then user cannot get exactly the same results on his box without knowledge of those --nostrip/--keepdocs etc. options given by package maintainer. Then he will build some package without --nostrip and will wonder why it doesn't work.
I couldn't agree more. Perhaps it's time to split makepkg into two different scripts. I don't know if I can see a good dividing line, however. Alternatively, we don't really need command line options for _every_ config setting. Perhaps we can limit these in some way... I'll look into it tonight as well.
Two scripts? 1) why? 2) what they should do?
Ideas for backends are always welcome. The interface exists in be_files.c in libalpm. It's fairly straightforward. People seem to prefer a "real database" backend (while at the same time suggesting sqlite which is far from a real database, but I digress), but I think that is overkill. There are a few ideas I had milling around, but haven't fleshed anything out - discussion is always a good idea.
There was a patch against 2.9.8 that added gdbm support. I dunno if it was adapted to pacman 3 later. I can resend it to you, or maybe I'll find the bug #. -- Roman Kyrylych (Роман Кирилич)
2006/12/20, Aaron Griffin <aaronmgriffin@gmail.com>:
And any of you list lurkers who have ideas - feel free to jump in. Every idea has the capability to be ground-breaking, no matter how silly it seems to you. 8)
It would be very useful to have an option to rebuild package without rebuilding source. For example I did makepkg without -c, pkg dir is there with all files needed, now I want to make some harmless changes in PKGBUILD, like add license field, I don't wan't to rebuild the whole 150MB game just for this ;) -- Roman Kyrylych (Роман Кирилич)
On 12/20/06, Roman Kyrylych <roman.kyrylych@gmail.com> wrote:
It would be very useful to have an option to rebuild package without rebuilding source.
Yeah that's the --repackage option Dan is apparently working on. So, it's in the works!
It would be very useful to have an option to rebuild package without rebuilding source. Already working on it and have a patch in mind once other things settle down. See FS #2978. It isn't that hard to implement but putting it in makes me want to reexamine the structure of the current code...the previously linear flow of the program is starting to become quite convoluted.
-Dan
7. Integrity checking- currently generating integrity checks will only generate those from the first one in the INTEGRITY_CHECK array. This seems to make sense as long as the array in makepkg.conf is ordered the way we want it to be. When checking, we do get warning messages for other types that may be in the array but are not in the PKGBUILD. I just wanted to confirm this is the behavior you are looking for.
Erm. It should have generated all of them. If it's not looped, then it's an oversight on my part. I actually tested some stuff with md5 and sha1, then moved back to just md5 for building some packages, so I probably missed it.
Patch below fixes this issue. Also addresses a spot where the wrong variable was used, splits up the integrity checks and extracting of sources into two sections for easier readability, and adds a few comments for good measure. --- /home/dmcgee/projects/pacman-lib.orig/scripts/makepkg 2006-12-19 11:40:51.000000000 -0500 +++ makepkg 2006-12-20 20:03:13.000000000 -0500 @@ -590,10 +590,8 @@ fi done - - +#Generate/Check integrity checks if [ "$NOEXTRACT" = "1" ]; then - warning "Skipping source extraction -- using existing src/ tree" warning "Skipping source integrity checks -- using existing src/ tree" else for integ in ${INTEGRITY_CHECK[@]}; do @@ -636,7 +634,6 @@ fi done plain "" - exit 0 #Validate integrity checks else integrity_sums=($(eval echo \${${integrity_name}s[@]})) @@ -662,11 +659,20 @@ exit 1 fi else - warning "Integrity checks ($INTEGRITY_CHECK) are missing or incomplete." + warning "Integrity checks ($integ) are missing or incomplete." fi fi done + if [ "$GENINTEG" = "1" ]; then + exit 0 + fi +fi + +#Extract sources +if [ "$NOEXTRACT" = "1" ]; then + warning "Skipping source extraction -- using existing src/ tree" +else msg "Extracting Sources..." for netfile in "${source[@]}"; do unziphack=0
I implemented much of the changes in this thread: * modelines and indentation * integrity patch * EUID patch * x$foo -> $foo changes when in double quotes Thanks alot guys. brain0 also discovered the issue with the pacman3 snapshot there. It appears that pacman2 does not save the md5sums of NoUpgrade files even if they are in the backup=() array. This causes pacman3 to assume they weren't backed up and overwrite them... I need to figure a way around this one now.
Hi,
2. Remove use of multiple `` or $() calls to the same function. I have a local update of calling $(id -u) once and storing it to a variable for later use, I can publish a patch sometime.
I'm surprised by the usage of back-ticks do get the effective uid, the shell already has an built-in READONLY variable (enclosed). Jürgen
participants (4)
-
Aaron Griffin
-
Dan McGee
-
Jürgen Hötzel
-
Roman Kyrylych