Re: [pacman-dev] LTO and PGO build options
What is the advantage to all this beyond people adding "-flto" to their CFLAGS if they want LTO?
The advantage is not having to do that. Theoretically, the compiler devs expect LTO to always work and want reports when it does not. We should be able to enable LTO for every build, but my experience is that--occasionally--the linker fails. Users so inclined could "set it and forget it" most of the time, and occasionally disable it if a package does not build. I find this more efficient than modifying the PKGBUILD of each package that could benefit from LTO.
All modern builds should handle the rest by themselves.
Should, but don't last time I checked. A number of builds failed for not having {AR,RANLIB,NM}FLAGS set when attempting LTO. I see the complexity as another reason to have this built into makepkg instead of having to do all this tediousness on a per-package basis. The less anyone has to edit PKGBUILDs for generic things like optimization, the better I think.
I'm guessing this is untested...
That's kinda like.. not testing it, yeah? I've been using this for quite some time. Nearly everything compiles successfully with pgo. There were a couple of wrapper leftovers to fix in my initial post, see below. The problem with only me doing the testing is no one will ever beleive my success. I have been successful, believe it--or try it for yourself.
Anyway, I don't think PGO has a place in makepkg itself
Why not, specifically? We have other cflag stuff in there; besides, it should be disabled by default so users could opt-in when they feel ready. Do you know you can PGO just about any package in the repositories? It's still takes two build steps, but this macro saves a lot of tediousness over editing the individual PKGBUILDS. I don't see how it's any more or less worthy than ccache, upx, etc. As for the particular cflags I've used for PGO and LTO, these are *known to work* but may not be the only options, might be overkill, or there could even be a few more minor flags worth setting. I'd appreciate some commentary on this matter, and some tests to prove or disprove which flags are required. "-fwhopr" might be a safter idea than "-flto", to avoid the compiler running out of memory and crashing while compiling large programs. New patchset: In makepkg, "-Wl,-option" must be at the end of LDFLAGS: --- makepkg 2016-02-23 11:52:34.000000000 +0900 +++ makepkg 2016-03-20 16:57:07.260935565 +0900 @@ -866,6 +866,32 @@ export DISTCC_HOSTS fi + + # Use lto if it is requested (check buildenv and PKGBUILD opts) + if check_buildoption "lto" "y" && [[ -f "$(gcc -print-search-dirs | grep install | awk '{print $2 "liblto_plugin.so"}')" ]]; then + CFLAGS+=" -flto=$(getconf _NPROCESSORS_ONLN)" + CXXFLAGS+=" -flto=$(getconf _NPROCESSORS_ONLN)" + LDFLAGS+=" $CFLAGS $CXXFLAGS -Wl,-fuse-linker-plugin" + LTOPLUGIN="$(gcc -print-search-dirs | grep install | awk '{print $2 "liblto_plugin.so"}')" + ARFLAGS+=" --plugin $LTOPLUGIN" + RANLIBFLAGS+=" --plugin $LTOPLUGIN" + NMFLAGS+=" --plugin $LTOPLUGIN" + fi + + # Generate or utilize pgo if it is requested (check buildenv and PKGBUILD opts) + if check_buildoption "pgo" "y"; then + if [ ! -d "$PROFDEST/$pkgbase.gen" ]; then + mkdir "$PROFDEST/$pkgbase.gen" + CFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen" + CXXFLAGS+=" -fprofile-generate -fprofile-dir=$PROFDEST/$pkgbase.gen" + LDFLAGS+=" -Wl,-lgcov" + elif [ ! -d "$PROFDEST/$pkgbase.used" ]; then + mv "$PROFDEST/$pkgbase.gen" "$PROFDEST/$pkgbase.used" + CFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used" + CXXFLAGS+=" -fprofile-correction -fprofile-use -fprofile-dir=$PROFDEST/$pkgbase.used" + fi + fi + run_function_safe "build" } @@ -1865,6 +2028,7 @@ [[ -n ${PKGDEST} ]] && _PKGDEST=$(canonicalize_path ${PKGDEST}) [[ -n ${SRCDEST} ]] && _SRCDEST=$(canonicalize_path ${SRCDEST}) [[ -n ${SRCPKGDEST} ]] && _SRCPKGDEST=$(canonicalize_path ${SRCPKGDEST}) +[[ -n ${PROFDEST} ]] && _PROFDEST=$(canonicalize_path ${PROFDEST}) [[ -n ${LOGDEST} ]] && _LOGDEST=$(canonicalize_path ${LOGDEST}) [[ -n ${BUILDDIR} ]] && _BUILDDIR=$(canonicalize_path ${BUILDDIR}) [[ -n ${PKGEXT} ]] && _PKGEXT=${PKGEXT} @@ -1961,6 +2125,14 @@ IGNOREARCH=1 fi +PROFDEST=${_PROFDEST:-$PROFDEST} +PROFDEST=${PROFDEST:-$startdir} #default to $startdir if undefined +if [[ ! -w $PROFDEST ]] ; then + error "$(gettext "You do not have write permission to store profiles in %s.")" "$PROFDEST" + plain "$(gettext "Aborting...")" + exit 1 +fi + LOGDEST=${_LOGDEST:-$LOGDEST} LOGDEST=${LOGDEST:-$startdir} #default to $startdir if undefined if (( LOGGING )) && [[ ! -w $LOGDEST ]]; then In makepkg.conf, I should have specified lto and pgo as BUILDENV options, not package options: --- makepkg.conf 2016-03-20 17:07:12.708226549 +0900 +++ makepkg.conf 2016-03-20 17:12:09.358079792 +0900 @@ -83,6 +83,8 @@ #-- ccache: Use ccache to cache compilation #-- check: Run the check() function if present in the PKGBUILD #-- sign: Generate PGP signature file +#-- lto: Use link-time optimization to reduce compiled binary size and possibly improve performance +#-- pgo: Generate or use profile guided optimization to reduce compile binary size and improve performance # BUILDENV=(!distcc color !ccache !check !sign) # @@ -142,6 +144,8 @@ #SRCDEST=/home/sources #-- Source packages: specify a fixed directory where all src packages will be placed #SRCPKGDEST=/home/srcpackages +#-- Profile cache: specify a fixed directory where profiles will be stored +#PROFDEST=/home/sources #-- Log files: specify a fixed directory where all log files will be placed #LOGDEST=/home/makepkglogs #-- Packager: name/email of the person or organization building packages I have tested these patches. They work for me and have no ill side effects.
On 20/03/16 19:03, Que Quotion wrote:
What is the advantage to all this beyond people adding "-flto" to their CFLAGS if they want LTO?
The advantage is not having to do that. Theoretically, the compiler devs expect LTO to always work and want reports when it does not. We should be able to enable LTO for every build, but my experience is that--occasionally--the linker fails.
Users so inclined could "set it and forget it" most of the time, and occasionally disable it if a package does not build. I find this more efficient than modifying the PKGBUILD of each package that could benefit from LTO.
Adding -flto to CFLAGS and forgetting also is fine. Remove it when the software is broken with LTO.
All modern builds should handle the rest by themselves.
Should, but don't last time I checked. A number of builds failed for not having {AR,RANLIB,NM}FLAGS set when attempting LTO. I see the complexity as another reason to have this built into makepkg instead of having to do all this tediousness on a per-package basis.
The less anyone has to edit PKGBUILDs for generic things like optimization, the better I think.
Add -flto to CFLAGS, report broken software upstream. I am not adding this to makepkg.
I'm guessing this is untested...
That's kinda like.. not testing it, yeah?
I've been using this for quite some time. Nearly everything compiles successfully with pgo. There were a couple of wrapper leftovers to fix in my initial post, see below. The problem with only me doing the testing is no one will ever beleive my success. I have been successful, believe it--or try it for yourself.
I was talking about the patch you provided being untested. You appear to have understood that as you have fixed it....
Anyway, I don't think PGO has a place in makepkg itself
Why not, specifically? We have other cflag stuff in there; besides, it should be disabled by default so users could opt-in when they feel ready. Do you know you can PGO just about any package in the repositories? It's still takes two build steps, but this macro saves a lot of tediousness over editing the individual PKGBUILDS. I don't see how it's any more or less worthy than ccache, upx, etc.
upx is dropped. optipng is dropped. I am not adding an option to makepkg that does non-deterministic optimisation. A
Adding -flto to CFLAGS and forgetting also is fine. Remove it when the software is broken with LTO.
Add -flto to CFLAGS, report broken software upstream. I am not adding
The only way this makes logical sense is if you mean for *packagers* to do so, but I have never known any to do so. I don't think it's any more appropriate to distribute PKGBUILDs with optimizations enabled by default. Much better to leave that to users; much too troublesome to ask them to edit each PKGBUILD they want to optimize. Doing this in makepkg works better than a wrapper script, and involves less work for both users and packagers--who would also have the opportunity to specify packages that must not use LTO. this to makepkg. That's dissapointing. There's practically no downside to having LTO available in makepkg.>I was talking about the patch you provided being untested. You appear to have understood that as you have fixed it.... Yeah, and you were right--but you came to that conclusion on a whim.
upx is dropped. optipng is dropped.
Oh, I see. Sorry to hear about that. makepkg's automated (opt-in) optimization of packages is one of the features I considered vastly superior to other package creation tools.
I am not adding an option to makepkg that does non-deterministic optimisation.
It always would have been an option for users to choose, not a policy, ie "enable at your own (negligible) risk". I don't think I have the constitution to maintain a makepkg fork for extra optimization features, but its sounding like a good idea.
Oh, you're talking about CFLAGS *in makepkg.conf* as I've pointed out, that won't always work when it should. On Sun, Mar 20, 2016 at 7:16 PM, Que Quotion <quequotion@gmail.com> wrote:
Adding -flto to CFLAGS and forgetting also is fine. Remove it when the software is broken with LTO.
The only way this makes logical sense is if you mean for *packagers* to do so, but I have never known any to do so. I don't think it's any more appropriate to distribute PKGBUILDs with optimizations enabled by default. Much better to leave that to users; much too troublesome to ask them to edit each PKGBUILD they want to optimize. Doing this in makepkg works better than a wrapper script, and involves less work for both users and packagers--who would also have the opportunity to specify packages that must not use LTO.
Add -flto to CFLAGS, report broken software upstream. I am not adding this to makepkg.
That's dissapointing. There's practically no downside to having LTO available in makepkg.>I was talking about the patch you provided being untested. You appear to have understood that as you have fixed it....
Yeah, and you were right--but you came to that conclusion on a whim.
upx is dropped. optipng is dropped.
Oh, I see. Sorry to hear about that. makepkg's automated (opt-in) optimization of packages is one of the features I considered vastly superior to other package creation tools.
I am not adding an option to makepkg that does non-deterministic optimisation.
It always would have been an option for users to choose, not a policy, ie "enable at your own (negligible) risk".
I don't think I have the constitution to maintain a makepkg fork for extra optimization features, but its sounding like a good idea.
On 20/03/16 20:16, Que Quotion wrote:
Adding -flto to CFLAGS and forgetting also is fine. Remove it when the software is broken with LTO.
The only way this makes logical sense is if you mean for *packagers* to do so, but I have never known any to do so. I don't think it's any more appropriate to distribute PKGBUILDs with optimizations enabled by default. Much better to leave that to users; much too troublesome to ask them to edit each PKGBUILD they want to optimize. Doing this in makepkg works better than a wrapper script, and involves less work for both users and packagers--who would also have the opportunity to specify packages that must not use LTO.
Add -flto to CFLAGS, report broken software upstream. I am not adding this to makepkg.
That's dissapointing. There's practically no downside to having LTO available in makepkg.
LTO is available in makepkg. You can add it to your system makepkg.conf CFLAGS, to your ~/.makepkg.conf CFLAGS, or even add it to the CFLAGS in the PKGBUILD itself. We don't need more options. Use the tools already available to you. If any of that breaks, it is not makepkg's place to fix it...
I was talking about the patch you provided being untested. You appear to have understood that as you have fixed it....
Yeah, and you were right--but you came to that conclusion on a whim.
A clearly broken patch means that you have not tested - no whim involved.
upx is dropped. optipng is dropped.
Oh, I see. Sorry to hear about that. makepkg's automated (opt-in) optimization of packages is one of the features I considered vastly superior to other package creation tools.
And it is still available - libmakepkg allows anybody to distribute additions like those to makepkg. These sort of optimizations need to be provided at a distribution (or repo) level, not in makepkg itself.
I am not adding an option to makepkg that does non-deterministic optimisation.
It always would have been an option for users to choose, not a policy, ie "enable at your own (negligible) risk".
I don't think I have the constitution to maintain a makepkg fork for extra optimization features, but its sounding like a good idea.
Create two makepkg.conf files. One with CFLAGS for PGO pass 1 and one with CLFAGS for PGO pass 2. Use the --config switch to point at the one needed for that build. That could even be done in a tiny wrapper. Allan
participants (2)
-
Allan McRae
-
Que Quotion