Re: [pacman-dev] Status of "Splitting packages with makepkg"

newer
[pacman-dev] [PATCH] Give an error...

Marc - A. Dahlhaus [ Administration | Westermann GmbH ]

13 Nov 2008 13 Nov '08

3:27 p.m.

Hello Allen, the prototype has a little problem in case you give -L option to makepkg as it is now: As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions. Am Donnerstag, den 13.11.2008, 21:34 +1000 schrieb Allan McRae:

...

Jakub Schmidtke wrote:

...
Hi.

I guess this is the right place to ask :) What is the status of this feature? I have read discussions from June and August, and some of the ideas look very interesting. Is there anything going on with this? Is there a way to help (as non dev/TU though)? :)

Jakub Schmidtke

I was taking the lead on getting this into makepkg but have been busy with other things in the last couple of months. About the only thing I have in my git branch is the prototype that was "decided on" (or possibly the least rejected....)

See http://dev.archlinux.org/~allan/gitweb/gitweb.cgi?p=pacman.git;a=commitdiff;...

Allan

_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://archlinux.org/mailman/listinfo/pacman-dev

Show replies by date

Allan McRae

13 Nov 13 Nov

5:05 p.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...

Hello Allen,

the prototype has a little problem in case you give -L option to makepkg as it is now:

As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging. Cheers, Allan

Allan McRae

5:18 p.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Allan McRae wrote:

...

Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...
Hello Allen,

the prototype has a little problem in case you give -L option to makepkg as it is now:

As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine... test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output } echo $output build 2>&1 build 2>&1 | tee buildlog

...

./test.sh FOO BAR BAR

...

cat buildlog BAR

So what exactly are you say is wrong? Allan

Aaron Griffin

5:25 p.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...

Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...
Hello Allen,

the prototype has a little problem in case you give -L option to makepkg as it is now:

As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh FOO BAR BAR

...
cat buildlog BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output } echo $output build 2>&1 | tee buildlog echo $output # end ./test.sh FOO BAR FOO Which, after thinking a little about it, is the way we want this to happen, yes?

Allan McRae

5:33 p.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Aaron Griffin wrote:

...

On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...
Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...
Hello Allen,

the prototype has a little problem in case you give -L option to makepkg as it is now:

As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh

FOO BAR BAR

...
cat buildlog

BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output }

echo $output build 2>&1 | tee buildlog echo $output # end

./test.sh FOO BAR FOO

Which, after thinking a little about it, is the way we want this to happen, yes?

I think, yes it is. From the KDEmod implementation, in the case where we are not logging (and hence no tee) we need to revert the variables to what they were before calling the build function. Anyway, this will all become more apparent as I finish implementing it (with heavy borrowing for all the patches floating around). The implementation can be a bit more complex to accommodate a simpler PKGBUILD as far as I am concerned. Few people look at makepkg, many look at PKGBUILDs. Allan

Aaron Griffin

5:49 p.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

On Thu, Nov 13, 2008 at 11:33 AM, Allan McRae wrote:

...

Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...
Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...
Hello Allen,

the prototype has a little problem in case you give -L option to makepkg as it is now:

As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh

FOO BAR BAR

...
cat buildlog

BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output }

echo $output build 2>&1 | tee buildlog echo $output # end

./test.sh FOO BAR FOO

Which, after thinking a little about it, is the way we want this to happen, yes?

I think, yes it is. From the KDEmod implementation, in the case where we are not logging (and hence no tee) we need to revert the variables to what they were before calling the build function. Anyway, this will all become more apparent as I finish implementing it (with heavy borrowing for all the patches floating around). The implementation can be a bit more complex to accommodate a simpler PKGBUILD as far as I am concerned. Few people look at makepkg, many look at PKGBUILDs.

Hell, you might be able to cheat. LOGFILE=/dev/null # if -L is specified, set LOGFILE to something else and always run: build() 2>&1 | tee $LOGFILE Bam, now variables are reset automatically. Just make sure you comment that nicely 8)

Allan McRae

14 Nov 14 Nov

3:48 a.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Aaron Griffin wrote:

...

On Thu, Nov 13, 2008 at 11:33 AM, Allan McRae wrote:

...
Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...
Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

...
Hello Allen,

the prototype has a little problem in case you give -L option to makepkg as it is now:

As makepkg pipes the output of the build-function into tee, the build-function gets executed in a sub-shell. Any changes of variables in context of main-shell from inside build-function will not work in that case. This leads to that we have to set the values of all split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh

FOO BAR BAR

...
cat buildlog

BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output }

echo $output build 2>&1 | tee buildlog echo $output # end

./test.sh FOO BAR FOO

Which, after thinking a little about it, is the way we want this to happen, yes?

I think, yes it is. From the KDEmod implementation, in the case where we are not logging (and hence no tee) we need to revert the variables to what they were before calling the build function. Anyway, this will all become more apparent as I finish implementing it (with heavy borrowing for all the patches floating around). The implementation can be a bit more complex to accommodate a simpler PKGBUILD as far as I am concerned. Few people look at makepkg, many look at PKGBUILDs.

Hell, you might be able to cheat. LOGFILE=/dev/null # if -L is specified, set LOGFILE to something else and always run: build() 2>&1 | tee $LOGFILE

Bam, now variables are reset automatically. Just make sure you comment that nicely 8)

Very nice idea. That will clean up the whole split package patches nicely. Allan

Jan Mette

11:45 a.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Am Freitag 14 November 2008 04:48:27 schrieb Allan McRae:

...

Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:33 AM, Allan McRae wrote:

...
Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...
Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote: > Hello Allen, > > the prototype has a little problem in case you give -L option to > makepkg > as it is now: > > As makepkg pipes the output of the build-function into tee, the > build-function gets executed in a sub-shell. Any changes of > variables in > context of main-shell from inside build-function will not work in > that case. This leads to that we have to set the values of all > split-package-variables outside of the respective build-functions.

As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh

FOO BAR BAR

...
cat buildlog

BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output }

echo $output build 2>&1 | tee buildlog echo $output # end

./test.sh FOO BAR FOO

Which, after thinking a little about it, is the way we want this to happen, yes?

I think, yes it is. From the KDEmod implementation, in the case where we are not logging (and hence no tee) we need to revert the variables to what they were before calling the build function. Anyway, this will all become more apparent as I finish implementing it (with heavy borrowing for all the patches floating around). The implementation can be a bit more complex to accommodate a simpler PKGBUILD as far as I am concerned. Few people look at makepkg, many look at PKGBUILDs.

Hell, you might be able to cheat. LOGFILE=/dev/null # if -L is specified, set LOGFILE to something else and always run: build() 2>&1 | tee $LOGFILE

Bam, now variables are reset automatically. Just make sure you comment that nicely 8)

Very nice idea. That will clean up the whole split package patches nicely.

Allan

Hi, i just want to give a small update on the KDEmod split patches. I havent updated them since some months, but the last round of hacking on them brought in some nice little changes. The current patches are here: http://www.kdemod.ath.cx/svn/branches/kde41/_buildsystem/patches/ The split patch is: splitpkg/3_makepkg-kdemod-splitpkg.patch I have combined our original approach with the improvements from http://bugs.archlinux.org/task/7982 and fixed all bugs i could find so far. There are also some more changes: - the split stuff has been moved into its own function, which makes the changes a lot more cleaner, imho... - the split packages will be created in subdirs of /pkg, so we dont pollute the builddir so much - repackaging etc works The only drawback i have experienced so far is that the -i switch does not work with split packages, but i guess there is a easy solution... You can also find the patches to create separate debug packages at this place, but these need some more "integration work" and testing. Currently we have 2 makepkg's (makepkg, splitpkg), because i havent had the time yet to write a unified split-debug patch that works with both "normal" and split pkgs. Oh, and i just want to say that i love Allans PKGBUILD prototype. Putting the split packages into the pkgname is an awesome idea :) Greetings Jan

Marc - A. Dahlhaus

11:57 a.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Allan McRae schrieb:

...

Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:33 AM, Allan McRae wrote:

...
Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...
Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

> Hello Allen, > > the prototype has a little problem in case you give -L option to > makepkg > as it is now: > > As makepkg pipes the output of the build-function into tee, the > build-function gets executed in a sub-shell. Any changes of > variables > in > context of main-shell from inside build-function will not work > in that > case. This leads to that we have to set the values of all > split-package-variables outside of the respective build-functions. > > > > As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh

FOO BAR BAR

...
cat buildlog

BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output }

echo $output build 2>&1 | tee buildlog echo $output # end

./test.sh FOO BAR FOO

Which, after thinking a little about it, is the way we want this to happen, yes?

I think, yes it is. From the KDEmod implementation, in the case where we are not logging (and hence no tee) we need to revert the variables to what they were before calling the build function. Anyway, this will all become more apparent as I finish implementing it (with heavy borrowing for all the patches floating around). The implementation can be a bit more complex to accommodate a simpler PKGBUILD as far as I am concerned. Few people look at makepkg, many look at PKGBUILDs.

Hell, you might be able to cheat. LOGFILE=/dev/null # if -L is specified, set LOGFILE to something else and always run: build() 2>&1 | tee $LOGFILE

Bam, now variables are reset automatically. Just make sure you comment that nicely 8)

Very nice idea. That will clean up the whole split package patches nicely.

Allan

_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://archlinux.org/mailman/listinfo/pacman-dev Hello,

the problem is that we need the vars to be the same as inside the build-function as we need them outside of build-function for package creation. The function that creates and compresses the actual package will be executed outside of the scope of the sub-shell so you get allways the same package-vars (the global defined) while the main-shell creates the package for every split-package without knowing about any variable overloading from inside subshell-scope of the build-functions... So i think as long as we stay with this loggin implentation we can scratch the overloading feathure alltogether and have to declare an array inside the scope of the main-shell for each variable...

Marc - A. Dahlhaus

17 Nov 17 Nov

11:21 p.m.

New subject: [pacman-dev] Status of "Splitting packages with makepkg"

Allan McRae schrieb:

...

Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:33 AM, Allan McRae wrote:

...
Aaron Griffin wrote:

...
On Thu, Nov 13, 2008 at 11:18 AM, Allan McRae wrote:

...
Allan McRae wrote:

...
Marc - A. Dahlhaus [ Administration | Westermann GmbH ] wrote:

> Hello Allen, > > the prototype has a little problem in case you give -L option to > makepkg > as it is now: > > As makepkg pipes the output of the build-function into tee, the > build-function gets executed in a sub-shell. Any changes of > variables > in > context of main-shell from inside build-function will not work > in that > case. This leads to that we have to set the values of all > split-package-variables outside of the respective build-functions. > > > > As I said, the implementation is still a bit away... The KDEmod guys override the PKGBUILD variables the same way. I will look into what they for logging.

Testing with this script indicates to me that everything is fine...

test.sh: #! /bin/bash

output="FOO"

build() { output="BAR" echo $output }

echo $output build 2>&1 build 2>&1 | tee buildlog

...
./test.sh

FOO BAR BAR

...
cat buildlog

BAR

So what exactly are you say is wrong?

I think the case he means is this: test.sh: #! /bin/bash output="FOO" build() { output="BAR" echo $output }

echo $output build 2>&1 | tee buildlog echo $output # end

./test.sh FOO BAR FOO

Which, after thinking a little about it, is the way we want this to happen, yes?

I think, yes it is. From the KDEmod implementation, in the case where we are not logging (and hence no tee) we need to revert the variables to what they were before calling the build function. Anyway, this will all become more apparent as I finish implementing it (with heavy borrowing for all the patches floating around). The implementation can be a bit more complex to accommodate a simpler PKGBUILD as far as I am concerned. Few people look at makepkg, many look at PKGBUILDs.

Hell, you might be able to cheat. LOGFILE=/dev/null # if -L is specified, set LOGFILE to something else and always run: build() 2>&1 | tee $LOGFILE

Bam, now variables are reset automatically. Just make sure you comment that nicely 8)

Very nice idea. That will clean up the whole split package patches nicely.

Allan

_______________________________________________ pacman-dev mailing list pacman-dev@archlinux.org http://archlinux.org/mailman/listinfo/pacman-dev

Hello, i played around a bit with the split-package implementation and got some ideas i'd like to share. I think we can actualy fix the case up real easy and provide more usefull information in the logfiles if we use a container function around the actual build and the package creation and pipe it as a whole into tee. An adaption of current makepkg would look like this: ... if [ 1 -lt ${#pkgname[@]} ] then # splitpackage build for package in ${pkgname[@]} do run_build ${package} 2>&1 | tee ${logfile} done else run_build build 2>&1 | tee ${logfile} fi ... run_build() { ... old run_build logic ... if [ 'function' == type -t ${1} ] then if eval ${1} then tidi_install create_package else # build error fi else # undefined error fi } We would get consitent variable overloading in the functions that actualy need it and get the global defined variables back after each package was build but also get all errors that we faced durig the build, tidy and package creation in our logfile... Another thing that could be usefull would be a separate logfile for each splitpackage. What about a special "prepare" function in a splitpackage case where we build everything and use the splitpackage-build functions to just install the content we want for a specific split-package? this could be usefull to e.g. separate development content (headers and static libs) from dynamic librarys... also a option to actualy overload the contents of pkgname from the commandline would be usefull to repackage only a singe split-package and not all split-package inside of PKGBUILD... In case of e.g. php, one could use overloading of pkgname content from the commandline to rebuild only the mysql and mysqli modules if mysql dynamic library changed its version... Marc

5647

Age (days ago)

5651

Last active (days ago)

List overview

Download

9 comments

5 participants

participants (5)

Aaron Griffin
Allan McRae
Jan Mette
Marc - A. Dahlhaus
Marc - A. Dahlhaus [ Administration | Westermann GmbH ]