[arch-general] Problems compiling big projects from AUR
Hey hey, even after my recent hardware upgrade I still notice issues compiling big AUR packages from github, like aws-sdk-cpp-git or mysql. This does not happen when cloning the git repository without AUR and compile using the package's build system. I notice that the AUR compile seems to go quicker. I know that I had the make option -j 4 turned on a while ago, but switched it off again. The thing that happens is for the AUR build process to swap massive amounts of data, grinding the system almost to a halt and finally failing without much of a reason. Nothing typically source code or linkage related. My system has 4GB of RAM and 4GB of swap, with a swapiness of 10. Any ideas? Best wishes, Jeanette -- * Website: http://juliencoder.de - for summer is a state of sound * SoundCloud: https://soundcloud.com/jeanette_c * Youtube: https://www.youtube.com/channel/UCMS4rfGrTwz8W7jhC1Jnv7g * GitHub: https://github.com/jeanette-c * Twitter: https://twitter.com/jeanette_c_s E-mail my heart and say our love will never die... <3 (Britney Spears)
Hi Jeanette,
I notice that the AUR compile seems to go quicker. ... The thing that happens is for the AUR build process to swap massive amounts of data, grinding the system almost to a halt and finally failing without much of a reason. Nothing typically source code or linkage related.
So an AUR build goes quicker, but swaps a lot, and grinds almost to a halt? By quicker, do you mean it makes quicker progress before the swap-grinding occurs?
Any ideas?
Watching `dstat -pcdngsy' would show if many processes are in the run queue, whether it's CPU bound or waiting for I/O, the amount of paging it's doing, and how much swap is used. Is it possible you haven't disabled the `-j', e.g. `MAKEFLAGS' in /etc/makepkg.conf? -- Cheers, Ralph.
Is it possible you haven't disabled the `-j', e.g. `MAKEFLAGS' in /etc/makepkg.conf? ... Hey hey Ralph,
Apr 12 2019, Ralph Corderoy has written: ... thanks, it turned out that the -j option had crept back in. Possibly something during the system update/copy went wrong. Thanks a lot. Thanks everyone, best wishes, Jeanette -- * Website: http://juliencoder.de - for summer is a state of sound * SoundCloud: https://soundcloud.com/jeanette_c * Youtube: https://www.youtube.com/channel/UCMS4rfGrTwz8W7jhC1Jnv7g * GitHub: https://github.com/jeanette-c * Twitter: https://twitter.com/jeanette_c_s We still move to a rhythm just like this We still dream of sharing our first kiss <3 (Britney Spears)
Is it possible you haven't disabled the `-j', e.g. `MAKEFLAGS' in /etc/makepkg.conf? ... Hey hey Ralph,
Apr 12 2019, Ralph Corderoy has written: ... thanks, it turned out that the -j option had crept back in. Possibly something during the system update/copy went wrong. Thanks a lot. Thanks everyone, best wishes, Jeanette ... Jeanette, You have to be careful with the -j option. Some packages are capable of using multiple jobs, but there are a whole lot of packages that can't. So make sure whatever you are compiling supports multiple jobs, otherwise a -j1 option may be the quickest overall.
April 12, 2019 3:07 AM, "Jeanette C. via arch-general" <arch-general@archlinux.org> wrote:
The thing that happens is for the AUR build process to swap massive amounts of data, grinding the system almost to a halt and finally failing without much of a reason. Nothing typically source code or linkage related.
Are you using an AUR helper to run the build, or are you running it manually using makepkg? It seems likely to me that your AUR helper is building somewhere under /tmp, which is of course a tmpfs and thus entirely resident in RAM — and thus causing massive and unnecessary swapping. ~Celti
On Fri, 2019-04-12 at 11:04 +0000, Celti Burroughs wrote:
April 12, 2019 3:07 AM, "Jeanette C. via arch-general" <arch-general@archlinux.org> wrote:
The thing that happens is for the AUR build process to swap massive amounts of data, grinding the system almost to a halt and finally failing without much of a reason. Nothing typically source code or linkage related.
Are you using an AUR helper to run the build, or are you running it manually using makepkg? It seems likely to me that your AUR helper is building somewhere under /tmp, which is of course a tmpfs and thus entirely resident in RAM — and thus causing massive and unnecessary swapping.
I agree on the tmpfs guess. However, when not using a helper, users might anyway tend to do the manual build process in tmpfs. OTOH I sometimes still use a discontinued helper. For small software I just run the helper and for large software I run an alias [1], to build outside tmpfs. When using the manual build process, then depending on the size of the software to build, I also chose between tmpfs and no tmpfs for the build directory. The usage of tmpfs isn't necessarily related to the tools somebody does use to build packages ;). [1] $ grep naourt .bashrc alias naourt='echo "Not AnOther User Repository Tool";rm -rf /{.,}tmp/yaourt-tmp-$(id -un);yaourt --tmp /.tmp'
On 4/12/19 3:33 PM, Ralf Mardorf via arch-general wrote:
On Fri, 2019-04-12 at 11:04 +0000, Celti Burroughs wrote:
April 12, 2019 3:07 AM, "Jeanette C. via arch-general" <arch-general@archlinux.org> wrote:
The thing that happens is for the AUR build process to swap massive amounts of data, grinding the system almost to a halt and finally failing without much of a reason. Nothing typically source code or linkage related.
Are you using an AUR helper to run the build, or are you running it manually using makepkg? It seems likely to me that your AUR helper is building somewhere under /tmp, which is of course a tmpfs and thus entirely resident in RAM — and thus causing massive and unnecessary swapping.
I agree on the tmpfs guess. However, when not using a helper, users might anyway tend to do the manual build process in tmpfs. OTOH I sometimes still use a discontinued helper. For small software I just run the helper and for large software I run an alias [1], to build outside tmpfs.
When using the manual build process, then depending on the size of the software to build, I also chose between tmpfs and no tmpfs for the build directory.
The usage of tmpfs isn't necessarily related to the tools somebody does use to build packages ;).
[1] $ grep naourt .bashrc alias naourt='echo "Not AnOther User Repository Tool";rm -rf /{.,}tmp/yaourt-tmp-$(id -un);yaourt --tmp /.tmp'
yaourt is discontinued. I use Trizen when i really need to use Aur helper. Many options, it's possible to specify other directory than /tmp -- Maderios
On Fri, 2019-04-12 at 15:50 +0200, leoutation@gmx.fr wrote:
On 4/12/19 3:33 PM, Ralf Mardorf via arch-general wrote:
OTOH I sometimes still use a discontinued helper. ^^^^^^^^^^^^ yaourt is discontinued. ^^^^^^^^^^^^ More important:
"Warning: AUR helpers are not supported by Arch Linux. You should become familiar with the manual build process in order to be prepared to troubleshoot problems." - https://wiki.archlinux.org/index.php/AUR_helpers Jeanette didn't mention to use a helper at all.
On 4/12/19 4:23 PM, Ralf Mardorf via arch-general wrote:
On Fri, 2019-04-12 at 15:50 +0200, leoutation@gmx.fr wrote:
On 4/12/19 3:33 PM, Ralf Mardorf via arch-general wrote:
OTOH I sometimes still use a discontinued helper. ^^^^^^^^^^^^ yaourt is discontinued. ^^^^^^^^^^^^ More important:
"Warning: AUR helpers are not supported by Arch Linux. You should become familiar with the manual build process in order to be prepared to troubleshoot problems." - https://wiki.archlinux.org/index.php/AUR_helpers
Jeanette didn't mention to use a helper at all.
Sure. I wrote: "I use Trizen when i really need to use Aur helper." It's usefull to search/test/install packages with a lot of dependencies. Most of time, it's useless. It's like Arch: know what we are doing, otherwise, do not do it... :) -- Maderios
participants (6)
-
Celti Burroughs
-
David C. Rankin
-
Jeanette C.
-
leoutation＠gmx.fr
-
Ralf Mardorf
-
Ralph Corderoy