[arch-general] resurrecting srcpac
Hi Archers, in the past weeks there was a little discussion in the forums about the purpose of ABS vs. AUR and when to use which of both tools. Srcpac was mentioned as being the right tool to install customized packages from the already existing PKGBUILDs in the ABS tree. However, srcpac didn't work since abs.conf moved and other minor things that changed (makepkg doesn't like to be used by root etc.). I took to opportunity to fetch the srcpac git repo and get it working again, because I think it makes a good addition to ABS + pacman. Instead of sending a bunch of patches I've setup a public git repo where you can fetch it and give it a testrun if you like (I can send patches too if required): http://dev.chimeric.de/git/srcpac.git I've also changed a couple of things. These are merely suggestions, which are of course disputable ;). I completely dropped the srcpac.conf in favour of another approach. srcpac now looks for config files in /etc/srcpac.d/<pkgname>. You have one config file per package. These files contain only sed expressions (they're just simple sed script files) and have to be named exactly like the corresponding folder (package) in the ABS tree ie. "python-doc", or "most". IMHO this is much more clear (KISS if you like) than having one config file containing stuff like "pythonXdoc=('#sedexpr#' '#sedexpr#')" which gets really messy when you change lots of things. Another possibility would be to keep the modified PKGBUILDs in /var/srcpac/<repo>/<pkgname>/PKBUILD and completely drop sed, though I think the sed approach requires less work. Because makepkg doesn't like to be run as root there were also some implications for building packages. To resolve them I decided to use a new user named srcpac and a dedicated build directory /var/srcpac in which the packages are build instead of building directly in the ABS tree. srcpac will drop privileges for the build process to this user. I also had to introduce a little perl dependency :( (just a oneliner) to be able to sanitize packages names from urls. The bash expression which was originally used didn't work anymore and the suggested fix in FS#8434 worked only partly and not for all packages. However, I failed to get the thing right in sed, thus the perl oneliner. If someone has an idea I'd greatly appreciate it (see get_pkgname()) (examples of packages which fail with the fix suggested in the bug report, because they're versioning scheme is a little different, are feh and most). Instead of using makepkg -i to install the newly build packages on the fly, srcpac uses pacman -U now, this way it's now possible to specify a different target root directory when build packages from source (this was marked as missing feature). From the test I've run up to now everything _seems_ to work as expected, but it would'nt harm if someone else who's interested in this could give it a try :). If you plan to try it you have to do the following things before running it (I can provide a PKGBUILD if needed): - add a new user 'srcpac' - create /var/srcpac and grand permissions for srcpac That's it, what do you think about it? I am open for suggestions in any direction (+thx for reading this rather lenghty mail ;)). Best Regards + TIA for any feedback, Michael PS.: The original srcpac.conf is still in the repo, and I haven't touched the manpage in advance. -- Michael Klier
On Thu, May 22, 2008 at 4:13 PM, Michael Klier <chi@chimeric.de> wrote:
Hi Archers,
in the past weeks there was a little discussion in the forums about the purpose of ABS vs. AUR and when to use which of both tools. Srcpac was mentioned as being the right tool to install customized packages from the already existing PKGBUILDs in the ABS tree. However, srcpac didn't work since abs.conf moved and other minor things that changed (makepkg doesn't like to be used by root etc.).
I took to opportunity to fetch the srcpac git repo and get it working again, because I think it makes a good addition to ABS + pacman.
Instead of sending a bunch of patches I've setup a public git repo where you can fetch it and give it a testrun if you like (I can send patches too if required):
http://dev.chimeric.de/git/srcpac.git
I've also changed a couple of things. These are merely suggestions, which are of course disputable ;).
I completely dropped the srcpac.conf in favour of another approach. srcpac now looks for config files in /etc/srcpac.d/<pkgname>. You have one config file per package. These files contain only sed expressions (they're just simple sed script files) and have to be named exactly like the corresponding folder (package) in the ABS tree ie. "python-doc", or "most". IMHO this is much more clear (KISS if you like) than having one config file containing stuff like "pythonXdoc=('#sedexpr#' '#sedexpr#')" which gets really messy when you change lots of things.
Another possibility would be to keep the modified PKGBUILDs in /var/srcpac/<repo>/<pkgname>/PKBUILD and completely drop sed, though I think the sed approach requires less work.
Because makepkg doesn't like to be run as root there were also some implications for building packages. To resolve them I decided to use a new user named srcpac and a dedicated build directory /var/srcpac in which the packages are build instead of building directly in the ABS tree. srcpac will drop privileges for the build process to this user.
I also had to introduce a little perl dependency :( (just a oneliner) to be able to sanitize packages names from urls. The bash expression which was originally used didn't work anymore and the suggested fix in FS#8434 worked only partly and not for all packages. However, I failed to get the thing right in sed, thus the perl oneliner. If someone has an idea I'd greatly appreciate it (see get_pkgname()) (examples of packages which fail with the fix suggested in the bug report, because they're versioning scheme is a little different, are feh and most).
Instead of using makepkg -i to install the newly build packages on the fly, srcpac uses pacman -U now, this way it's now possible to specify a different target root directory when build packages from source (this was marked as missing feature).
From the test I've run up to now everything _seems_ to work as expected, but it would'nt harm if someone else who's interested in this could give it a try :).
If you plan to try it you have to do the following things before running it (I can provide a PKGBUILD if needed): - add a new user 'srcpac' - create /var/srcpac and grand permissions for srcpac
That's it, what do you think about it? I am open for suggestions in any direction (+thx for reading this rather lenghty mail ;)).
Best Regards + TIA for any feedback, Michael
PS.: The original srcpac.conf is still in the repo, and I haven't touched the manpage in advance.
-- Michael Klier
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFINf4zAuL1i4MI9VERAsscAJ41O7WNrQIKsoDBVOKh5j7B9P4++ACeMNSK XAXVVQ6YiStrCUMBTOBn1A4= =m8CM -----END PGP SIGNATURE-----
Hot! I was hoping someone would do this. I originally wrote srcpac as an example of how such a thing could be done. Then people actually started using it! I will do my best to look at your patches tonight. I'll even put a reminder in my phone. I'll get back to you tomorrow about it. Jason
Jason Chu wrote:
Hot!
I was hoping someone would do this. I originally wrote srcpac as an example of how such a thing could be done. Then people actually started using it!
I will do my best to look at your patches tonight. I'll even put a reminder in my phone. I'll get back to you tomorrow about it.
Thanks for your reply, I am looking forward to read your thoughts on this :). Michael -- Michael Klier
On 5/22/08, Michael Klier <chi@chimeric.de> wrote:
Hi Archers,
<big snip>
oh man. I got excited reading this. I really like the first change you made. Using a directory with package specific sed changes on a per file basis is a kick ass idea.
On Fri, May 23, 2008 at 1:07 AM, eliott <eliott@cactuswax.net> wrote:
On 5/22/08, Michael Klier <chi@chimeric.de> wrote:
Hi Archers,
<big snip>
oh man. I got excited reading this.
I really like the first change you made. Using a directory with package specific sed changes on a per file basis is a kick ass idea.
I agree, this is a smart move. My only real thought is why have anything in /etc/ at all? It would be great if this could all be done non-root for editing any config settings and such. Maybe ~/.srcpac/ or something? It also seems almost overkill to need another user to build packages, but not sure what to think about that. Anyway, great work with this. -Dan
On Fri, May 23, 2008 at 8:55 AM, Dan McGee <dpmcgee@gmail.com> wrote:
On Fri, May 23, 2008 at 1:07 AM, eliott <eliott@cactuswax.net> wrote:
On 5/22/08, Michael Klier <chi@chimeric.de> wrote:
Hi Archers,
<big snip>
oh man. I got excited reading this.
I really like the first change you made. Using a directory with package specific sed changes on a per file basis is a kick ass idea.
I agree, this is a smart move. My only real thought is why have anything in /etc/ at all? It would be great if this could all be done non-root for editing any config settings and such. Maybe ~/.srcpac/ or something?
It also seems almost overkill to need another user to build packages, but not sure what to think about that. Anyway, great work with this.
srcpac is also a wrapper around pacman, so it can install packages - hence the need for root.
Dan McGee wrote:
I agree, this is a smart move. My only real thought is why have anything in /etc/ at all? It would be great if this could all be done non-root for editing any config settings and such. Maybe ~/.srcpac/ or something?
It also seems almost overkill to need another user to build packages, but not sure what to think about that. Anyway, great work with this.
As phrakture pointed out already root is needed for installing packages. Keeping the configs below /etc is disputable though. You could also have them in /var/{?} for example if that fits better. The separate user is needed for the makepkg part of srcpac. Because your invoke srcpac as root/sudo it has to drop privileges for the package build process. Of course one could use an existing user (nobody?) and I said this is merely a suggestion - however, the script has to drop privileges for the build process in a way. Regards, Michael -- Michael Klier
On Sat, May 24, 2008 at 8:12 AM, Michael Klier <chi@chimeric.de> wrote:
Dan McGee wrote:
I agree, this is a smart move. My only real thought is why have anything in /etc/ at all? It would be great if this could all be done non-root for editing any config settings and such. Maybe ~/.srcpac/ or something?
It also seems almost overkill to need another user to build packages, but not sure what to think about that. Anyway, great work with this.
As phrakture pointed out already root is needed for installing packages.
Keeping the configs below /etc is disputable though. You could also have them in /var/{?} for example if that fits better.
The separate user is needed for the makepkg part of srcpac. Because your invoke srcpac as root/sudo it has to drop privileges for the package build process. Of course one could use an existing user (nobody?) and I said this is merely a suggestion - however, the script has to drop privileges for the build process in a way.
Regards, Michael
-- Michael Klier
Alrighty, mostly a couple questions: Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user... I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code? Other than that I like everything else. Jason
On Sat, May 24, 2008 at 12:20 PM, Jason Chu <jason@archlinux.org> wrote:
On Sat, May 24, 2008 at 8:12 AM, Michael Klier <chi@chimeric.de> wrote:
Dan McGee wrote:
I agree, this is a smart move. My only real thought is why have anything in /etc/ at all? It would be great if this could all be done non-root for editing any config settings and such. Maybe ~/.srcpac/ or something?
It also seems almost overkill to need another user to build packages, but not sure what to think about that. Anyway, great work with this.
As phrakture pointed out already root is needed for installing packages.
Keeping the configs below /etc is disputable though. You could also have them in /var/{?} for example if that fits better.
The separate user is needed for the makepkg part of srcpac. Because your invoke srcpac as root/sudo it has to drop privileges for the package build process. Of course one could use an existing user (nobody?) and I said this is merely a suggestion - however, the script has to drop privileges for the build process in a way.
Regards, Michael
-- Michael Klier
Alrighty, mostly a couple questions:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code?
The DB scripts actually do it this way: getpkgname() { local tmp tmp=${1##*/} tmp=${tmp%$PKGEXT} tmp=${tmp%-$CARCH} echo ${tmp%-*-*} }
Aaron Griffin wrote:
Alrighty, mostly a couple questions:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code?
The DB scripts actually do it this way:
getpkgname() { local tmp
tmp=${1##*/} tmp=${tmp%$PKGEXT} tmp=${tmp%-$CARCH} echo ${tmp%-*-*} }
Aaahh ok, there's always a better/correct solution :)! Michael -- Michael Klier
On Sat, 24 May 2008 20:52:23 +0200 Michael Klier <chi@chimeric.de> wrote:
Aaron Griffin wrote:
The DB scripts actually do it this way:
getpkgname() { local tmp
tmp=${1##*/} tmp=${tmp%$PKGEXT} tmp=${tmp%-$CARCH} echo ${tmp%-*-*} }
Aaahh ok, there's always a better/correct solution :)!
Don't forget the 'any' arch!
On Sat, May 24, 2008 at 2:30 PM, Loui <louipc.ist@gmail.com> wrote:
On Sat, 24 May 2008 20:52:23 +0200 Michael Klier <chi@chimeric.de> wrote:
Aaron Griffin wrote:
The DB scripts actually do it this way:
getpkgname() { local tmp
tmp=${1##*/} tmp=${tmp%$PKGEXT} tmp=${tmp%-$CARCH} echo ${tmp%-*-*} }
Aaahh ok, there's always a better/correct solution :)!
Don't forget the 'any' arch!
Very true! The DB scripts don't support the 'any' arch just yet, so the above works for them.
Jason Chu wrote:
Alrighty, mostly a couple questions:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I don't know if that's possible, because you don't know whether srcpac was invoked as root or via sudo. However, maybe a new separate user isn't needed and we could just (ab)use nobody for the makepkg calls.
I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code?
It doesn't work in all cases because not all packages follow the <package-name>-<package-version>-<PKGBUILD-version>-<arch>.pkg.tar.gz naming scheme yet. This has something todo with how srcpac determines which packages are about to be build from source on a system update (-Syu). In this case it fetches the list of package urls to be updated (-Sup) - extracts the the package part without the pkg.tar.gz ending and then removes the version part with the bash expression and then checks if the package was installed from source. There are still lots of packages in extra/community which are missing the <arch> part in the file name and for which the bash expression fails. Im quite sure that what I did in perl could be done with sed, but up to now I failed to get it right in sed :(. The perl oneline however can be replaced with the bash expression once all packages follow the same naming scheme. Michael -- Michael Klier
Michael Klier wrote:
Jason Chu wrote:
Alrighty, mostly a couple questions:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I don't know if that's possible, because you don't know whether srcpac was invoked as root or via sudo. However, maybe a new separate user isn't needed and we could just (ab)use nobody for the makepkg calls.
You could test for SUDO_USER environment variable. If it doesn't exists then it was not called through sudo and USER would give you the real user. Armando
On Sat, 24 May 2008 10:20:37 -0700 "Jason Chu" <jason@archlinux.org> wrote:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code?
Using nobody isn't a bad idea! When it comes to using sudo $HOME isn't changed and $USER becomes $SUDO_USER within the sudo environment. You can then easily find the correct makepkg.conf for user specific build options and such.
Loui wrote:
On Sat, 24 May 2008 10:20:37 -0700 "Jason Chu" <jason@archlinux.org> wrote:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code?
Using nobody isn't a bad idea! When it comes to using sudo $HOME isn't changed and $USER becomes $SUDO_USER within the sudo environment. You can then easily find the correct makepkg.conf for user specific build options and such.
So, I am not 100% sure how to proceed from here. First the bits that seemed to be cleared: - using nobody to build packages when srcpac was called as root is a good idea - the get_pkgname() function should be adjusted to use something along the lines of the DB scripts (including support for the "any" arch) to get rid of the perl stuff Not clear, at least not to me: - undecided whether to keep the package specific configs in /etc/srcpac.d or in /var/srcpac - if called with sudo what would be the benefit to drop privileges to the user who invoked srcpac instead than nobody. AJAIK makpkg doesn't support user specific build options, but the ones set in /etc/makepkg.conf (please correct me if I am wrong on this) I'd be happy to help and add all these changes and provide patches push it to my public repo or whatever you see fit. Michael -- Michael Klier
On Tue, May 27, 2008 at 9:11 AM, Michael Klier <chi@chimeric.de> wrote:
Not clear, at least not to me:
- undecided whether to keep the package specific configs in /etc/srcpac.d or in /var/srcpac
Of the two, I'd suggest /etc/srcpac.d - it's essentially configuration data.
- if called with sudo what would be the benefit to drop privileges to the user who invoked srcpac instead than nobody. AJAIK makpkg doesn't support user specific build options, but the ones set in /etc/makepkg.conf (please correct me if I am wrong on this)
makepkg supports ~/.makepkg.conf as well - additionally, if you use the original user instead of nobody, then permissions on the final package file will be nice and pretty too. :)
On Tue, May 27, 2008 at 6:11 AM, Michael Klier <chi@chimeric.de> wrote:
Loui wrote:
On Sat, 24 May 2008 10:20:37 -0700 "Jason Chu" <jason@archlinux.org> wrote:
Would it be an idea to build as the original users who ran srcpac? I guess if you use sudo, that won't quite work... you'd need something pointing back to that user...
I'm a little confused why ${i-*-*-*} doesn't work in all cases. What are you trying to fix with the perl code?
Using nobody isn't a bad idea! When it comes to using sudo $HOME isn't changed and $USER becomes $SUDO_USER within the sudo environment. You can then easily find the correct makepkg.conf for user specific build options and such.
So, I am not 100% sure how to proceed from here. First the bits that seemed to be cleared:
- using nobody to build packages when srcpac was called as root is a good idea
- the get_pkgname() function should be adjusted to use something along the lines of the DB scripts (including support for the "any" arch) to get rid of the perl stuff
Not clear, at least not to me:
- undecided whether to keep the package specific configs in /etc/srcpac.d or in /var/srcpac
I like srcpac.d. I didn't comment on that one because it was good ;)
- if called with sudo what would be the benefit to drop privileges to the user who invoked srcpac instead than nobody. AJAIK makpkg doesn't support user specific build options, but the ones set in /etc/makepkg.conf (please correct me if I am wrong on this)
There is actually a ~/.makepkg.conf like Travis says. What does it matter about user specific build options?
I'd be happy to help and add all these changes and provide patches push it to my public repo or whatever you see fit.
Yeah, put those in your public repo too and then I'll release a new version of srcpac.
Jason Chu wrote:
Yeah, put those in your public repo too and then I'll release a new version of srcpac.
Ok, almost finished, though one problem remains. Using nobody actually doesn't work because when you su nobody -c <command> the system will enforce a password change. That leaves 3 options: a) we use a dedicated srcpac user in case srcpac was invoked by root or b) make the user configurable in /etc/srcpac.conf or c) invoke makepkg using sudo -u nobody, that however will add sudo as dependency to srcpac. Personally I think c) is the best of them. Other than that I've added the changes, but because of that missing bit the version in my repo is not 100% functional at the moment. What do you think? Michael -- Michael Klier
On Sun, Jun 1, 2008 at 10:14 AM, Michael Klier <chi@chimeric.de> wrote:
Jason Chu wrote:
Yeah, put those in your public repo too and then I'll release a new version of srcpac.
Ok, almost finished, though one problem remains. Using nobody actually doesn't work because when you su nobody -c <command> the system will enforce a password change.
That leaves 3 options: a) we use a dedicated srcpac user in case srcpac was invoked by root or b) make the user configurable in /etc/srcpac.conf or c) invoke makepkg using sudo -u nobody, that however will add sudo as dependency to srcpac. Personally I think c) is the best of them.
Other than that I've added the changes, but because of that missing bit the version in my repo is not 100% functional at the moment.
What do you think?
Michael
I don't mind sudo as a dependency: c) is fine. Jason
Jason Chu wrote:
On Sun, Jun 1, 2008 at 10:14 AM, Michael Klier <chi@chimeric.de> wrote:
Jason Chu wrote:
Yeah, put those in your public repo too and then I'll release a new version of srcpac.
Ok, almost finished, though one problem remains. Using nobody actually doesn't work because when you su nobody -c <command> the system will enforce a password change.
That leaves 3 options: a) we use a dedicated srcpac user in case srcpac was invoked by root or b) make the user configurable in /etc/srcpac.conf or c) invoke makepkg using sudo -u nobody, that however will add sudo as dependency to srcpac. Personally I think c) is the best of them.
Other than that I've added the changes, but because of that missing bit the version in my repo is not 100% functional at the moment.
What do you think?
Michael
I don't mind sudo as a dependency: c) is fine.
OK then, I've applied that as well and updated the man page too. Now sudo is used when invoked by root to drop privileges to nobody and su is used to drop privilegs to the user who called srcpac via sudo (using su here to get the environment right). From the tests I've done so far everything seems to work well (though again, it wouldn't harm if someone maybe checked it too). I also haven't touched the version number ;). Regards Michael -- Michael Klier
On Sun, Jun 1, 2008 at 1:31 PM, Michael Klier <chi@chimeric.de> wrote:
Jason Chu wrote:
On Sun, Jun 1, 2008 at 10:14 AM, Michael Klier <chi@chimeric.de> wrote:
Jason Chu wrote:
Yeah, put those in your public repo too and then I'll release a new version of srcpac.
Ok, almost finished, though one problem remains. Using nobody actually doesn't work because when you su nobody -c <command> the system will enforce a password change.
That leaves 3 options: a) we use a dedicated srcpac user in case srcpac was invoked by root or b) make the user configurable in /etc/srcpac.conf or c) invoke makepkg using sudo -u nobody, that however will add sudo as dependency to srcpac. Personally I think c) is the best of them.
Other than that I've added the changes, but because of that missing bit the version in my repo is not 100% functional at the moment.
What do you think?
Michael
I don't mind sudo as a dependency: c) is fine.
OK then, I've applied that as well and updated the man page too. Now sudo is used when invoked by root to drop privileges to nobody and su is used to drop privilegs to the user who called srcpac via sudo (using su here to get the environment right). From the tests I've done so far everything seems to work well (though again, it wouldn't harm if someone maybe checked it too). I also haven't touched the version number ;).
Regards Michael
-- Michael Klier
I'm happy with these changes. Now the only real question that needs to be answered should this be srcpac 0.6 or srcpac 1.0? Jason
On Wed, 2008-06-04 at 20:38 -0700, Jason Chu wrote:
On Sun, Jun 1, 2008 at 1:31 PM, Michael Klier <chi@chimeric.de> wrote:
Jason Chu wrote:
On Sun, Jun 1, 2008 at 10:14 AM, Michael Klier <chi@chimeric.de> wrote:
Jason Chu wrote:
Yeah, put those in your public repo too and then I'll release a new version of srcpac.
Ok, almost finished, though one problem remains. Using nobody actually doesn't work because when you su nobody -c <command> the system will enforce a password change.
That leaves 3 options: a) we use a dedicated srcpac user in case srcpac was invoked by root or b) make the user configurable in /etc/srcpac.conf or c) invoke makepkg using sudo -u nobody, that however will add sudo as dependency to srcpac. Personally I think c) is the best of them.
Other than that I've added the changes, but because of that missing bit the version in my repo is not 100% functional at the moment.
What do you think?
Michael
I don't mind sudo as a dependency: c) is fine.
OK then, I've applied that as well and updated the man page too. Now sudo is used when invoked by root to drop privileges to nobody and su is used to drop privilegs to the user who called srcpac via sudo (using su here to get the environment right). From the tests I've done so far everything seems to work well (though again, it wouldn't harm if someone maybe checked it too). I also haven't touched the version number ;).
Regards Michael
-- Michael Klier
I'm happy with these changes. Now the only real question that needs to be answered should this be srcpac 0.6 or srcpac 1.0?
Jason
Can you have it run su -c '/path/to/command something' instead of sudo? Not all of us are willing to have to use sudo. If sudo is that necessary, then it is still fine. This is just a small opinion. Thanks.
Jason Chu wrote:
I'm happy with these changes. Now the only real question that needs to be answered should this be srcpac 0.6 or srcpac 1.0?
I'd vote for 0.6 (there might be still room for improvement: ie. localization?). Michael -- Michael Klier
participants (9)
-
Aaron Griffin
-
Armando M. Baratti
-
Dan McGee
-
eliott
-
Hussam Al-Tayeb
-
Jason Chu
-
Loui
-
Michael Klier
-
Travis Willard