[arch-general] MAKEFLAGS remodelation
Hello, Some projects (e.g., mpv, in community) are moving from Autotools to other build systems. I think we should create a JOBS variable indicating the number of parallel jobs, leaving MAKEFLAGS for make-specific options. We could also implement SCONSFLAGS, WAFFLAGS,… although that may be a bit premature. The reason behind this is that a given build system might not understand "-j N". (And, also, for SCons projects, the "-Q" flag could be desired.) In the proposed model, each PKGBUILD would appropiately tell the build system how many jobs to do in parallel, or ignore the setting. As an extra question, may I ask why aren't MAKEFLAGS used in official PKGBUILDs? I mean, as in: make ${MAKEFLAGS} Perhaps make automatically fetches options from the environment? Or maybe because package construction is automated in a server and other processes are running in other cores? Thanks, Kalrish
On Sun, 2014-01-05 at 17:55 +0100, Kalrish Bäakjen wrote:
Hello,
Some projects (e.g., mpv, in community) are moving from Autotools to other build systems. I think we should create a JOBS variable indicating the number of parallel jobs, leaving MAKEFLAGS for make-specific options. We could also implement SCONSFLAGS, WAFFLAGS,… although that may be a bit premature.
The reason behind this is that a given build system might not understand "-j N". (And, also, for SCons projects, the "-Q" flag could be desired.)
In the proposed model, each PKGBUILD would appropiately tell the build system how many jobs to do in parallel, or ignore the setting.
As an extra question, may I ask why aren't MAKEFLAGS used in official PKGBUILDs? I mean, as in: make ${MAKEFLAGS} Perhaps make automatically fetches options from the environment? Or maybe because package construction is automated in a server and other processes are running in other cores?
Thanks, Kalrish
Salutations,
Forgive me if I misunderstood, but isn't Makeflags specified in the
MAKEFLAGS array in makepkg.conf (see
https://wiki.archlinux.org/index.php/Makepkg#MAKEFLAGS). They should
be automatically fetched from the environment.
Regarding your idea for supporting different build tools, I think
officially Arch supports the build tools provided in base-devel (which
would be autotools). While there are distcc flags (not in base-devel
group) in makepkg, a change in makepkg.conf would probably reflect a
change in the base-devel group (like adding waf).
Regards,
Mark
--
Mark Lee
Hello, I'm sorry for the confusion. A little research reveals that make automatically recognises the MAKEFLAGS environmental variable. (I didn't know it, and I used to add $MAKEFLAGS to each make invocation.) I agree that a change in makepkg.conf should reflect a change in the base-devel group. However, I think that a separate JOBS variable would be more appropiated, and would also work with other build systems if they eventually get support. $ grep JOBS /etc/makepkg.conf JOBS=4 MAKEFLAGS="-j $JOBS" $ grep make PKGBUILD make # No change here $ grep strange-build-system PKGBUILD strange --parallel=$JOBS Thanks, Kalrish P.S.: at least with community/mpv, it seems that Waf is able to bootstrap itself, so it's not necessary to install it to build the mentioned package.
Hi waf have internal multithreading detection, no need add job options greetings
On 6 January 2014 02:11, SpinFlo
Hi
waf have internal multithreading detection, no need add job options
That may be true, but it does support a job option. [1] I also don't get the point of this topic, since $MAKEFLAGS simply contains a hyphen, a character, and a number: --parellel-jobs=${MAKEFLAGS/-j} In fact, the job option WAF supports has a shorthand -j, making it directly compatible with Autotools. I maintain several WAF-based packages, take a look at them. As for SCONS, I don't recall appending $MAKEFLAGS to any of the builds I have been exposed to, but I do see it being used that way in our repos. So any other real-world cases where the build system makes this difficult? [1] https://code.google.com/p/waf/wiki/EnvironmentVariables -- GPG/PGP ID: C0711BF1
My goal was to make it generic. SCons and Waf support the "-j" option, but that is coincidental. MAKEFLAGS is for make; why tweak it to make it work with other build systems? And, if other build systems support some options (for debugging, for output supression, for whatever), we need a way to tell them. I'm not sure and I can't test it, but I think that your example would be broken if MAKEFLAGS was "-j N" instead of "-jN". Anyway, this was just about names. In practice, and as you point out, you can safely assume that every modern build system accepts "-j". Thanks, Kalrish
On 6 January 2014 03:57, Kalrish Bäakjen
My goal was to make it generic. SCons and Waf support the "-j" option, but that is coincidental. MAKEFLAGS is for make; why tweak it to make it work with other build systems? And, if other build systems support some options (for debugging, for output supression, for whatever), we need a way to tell them.
I'm not sure and I can't test it, but I think that your example would be broken if MAKEFLAGS was "-j N" instead of "-jN".
I gave an obvious example; I didn't intend to cover all cases. My
point was that you could tweak as much as you'd like within the
PKGBUILD, on a case-by-case basis, as and when the need arises (or
rather, increases), until it becomes more than just a few packages.
_JOBS=${MAKEFLAGS// }
--parellel-jobs=${_JOBS/-j}
On 6 January 2014 04:10, SpinFlo
MAKEFLAGS in makepkg.conf is disabled (#) by default. add makeflags variable like "--parellel-jobs=${MAKEFLAGS/-j}" in PKGBUILD only gets errors (or not work).
And our workaround would grow to accommodate that (while still keeping it simple, i.e. no sed magic): _JOBS=${MAKEFLAGS// } _JOBS=${_JOBS/-j} --parellel-jobs=${_JOBS:-1} If this looks nasty, it is. The point is, again, how often do we need to do this, right now? Is it an expected future use case, for sure?
about waf, yes, can use JOBS or --parellel-jobs, but if you test with/without these options, without options waf use all thread available in the system (for example in my system is 8 threads), if set JOBS or --parellel-jobs with all thread available (8) have the same result.
That's good news, then, and your English is just fine.
sorry my english
-- GPG/PGP ID: C0711BF1
hi
MAKEFLAGS in makepkg.conf is disabled (#) by default. add makeflags variable like "--parellel-jobs=${MAKEFLAGS/-j}" in PKGBUILD only gets errors (or not work). i think the best method is set (and enable by default) MAKEFLAGS with nproc value internally in makepkg tools (like MAKEFLAGS="j$(nproc)"). and only set MAKEFLAGS in makepkg.conf if want avoid established value (set less threads, for example) about waf, yes, can use JOBS or --parellel-jobs, but if you test with/without these options, without options waf use all thread available in the system (for example in my system is 8 threads), if set JOBS or --parellel-jobs with all thread available (8) have the same result. sorry my english greetings
I didn't realize what you explain. However, there's no need to use MAKEFLAGS inside the PKGBUILD; it seems 'make' automatically takes its value from the environment if it is present. With respect to the default value, I think it is a sane default to use 'nproc'; PKGBUILDs for applications that can't be built in parallel should explicitly disable it. # I think these would be sane defaults JOBS=$(nproc) MAKEFLAGS="-j ${JOBS}" SCONSFLAGS="-j ${JOBS} -Q" Don't worry about your English. If you had seen me some years ago, you would have thought I was a gorilla.
On Sun, 2014-01-05 at 18:59 +0100, Kalrish Bäakjen wrote:
Hello,
I'm sorry for the confusion. A little research reveals that make automatically recognises the MAKEFLAGS environmental variable. (I didn't know it, and I used to add $MAKEFLAGS to each make invocation.)
I agree that a change in makepkg.conf should reflect a change in the base-devel group. However, I think that a separate JOBS variable would be more appropiated, and would also work with other build systems if they eventually get support.
$ grep JOBS /etc/makepkg.conf JOBS=4 MAKEFLAGS="-j $JOBS" $ grep make PKGBUILD make # No change here $ grep strange-build-system PKGBUILD strange --parallel=$JOBS
Thanks, Kalrish
P.S.: at least with community/mpv, it seems that Waf is able to bootstrap itself, so it's not necessary to install it to build the mentioned package.
Salutations,
For mpv, waf is installed to build the package (it's in the makedepends
array). It's not needed to run the program though.
I think what your recommendation boils down to is adding a jobs string
that specified how many jobs to run independent of the compiler. It
would be difficult to balance the unique flags of different compilers
with a universal flag in makepkg.conf. That may be opening a can of
worms (if compilers use different flags to specify the threading).
Regards,
Mark
--
Mark Lee
participants (4)
-
Kalrish Bäakjen
-
Mark Lee
-
Rashif Ray Rahman
-
SpinFlo