[arch-general] Can you test machine to set --jobs[=N] from w/i PKGBUILD?
Guys, Is there any way to check the number of cores on a machine to set the number of jobs within a PKGBUILD?: make -j'X' I have some dual-core and some quad-core boxes and the difference between -j2 and -j4 on build time is tremendous. However, for generically creating a source package to be used by someone else, I can't hardcode the jobs number. Anyway to test in pkgbuild or is this just a per box makepkg.conf issue? -- David C. Rankin, J.D.,P.E.
should be the latter. some don't want the system too heavily loaded by building a package, so they may prefer building without parallel enabled. -- 抱歉暂时无法详细说明。这份邮件是使用安装有K-9 Mail的Android移动设备发送的。 "David C. Rankin" <drankinatty@suddenlinkmail.com>写到： Guys, Is there any way to check the number of cores on a machine to set the number of jobs within a PKGBUILD?: make -j'X' I have some dual-core and some quad-core boxes and the difference between -j2 and -j4 on build time is tremendous. However, for generically creating a source package to be used by someone else, I can't hardcode the jobs number. Anyway to test in pkgbuild or is this just a per box makepkg.conf issue? -- David C. Rankin, J.D.,P.E.
On 04/01/2012 12:03 AM, XeCycle wrote:
should be the latter. some don't want the system too heavily loaded by building a package, so they may prefer building without parallel enabled.
Thanks! I guess that's the smart way to do it anyway. -- David C. Rankin, J.D.,P.E.
Makepkg.conf MAKEFLAGS="-jX" Innstall renice and you wont even Notice the building, as gcc's priority is set so low by renice.
Jesse Jaara <jesse.jaara@gmail.com>写到：
Makepkg.conf MAKEFLAGS="-jX"
Innstall renice and you wont even Notice the building, as gcc's priority is set so low by renice.
That doesn't solve everything, though. A notebook user like me may find the CPU gets too hot when building with aggressive parallel options, and renice can't help. -- Carl Lei (XeCycle) Sent with my phone, sorry for inconveniences.
On Sun, Apr 01, 2012 at 04:28:53PM +0800, XeCycle wrote:
Jesse Jaara <jesse.jaara@gmail.com>写到：
Makepkg.conf MAKEFLAGS="-jX"
Innstall renice and you wont even Notice the building, as gcc's priority is set so low by renice.
That doesn't solve everything, though. A notebook user like me may find the CPU gets too hot when building with aggressive parallel options, and renice can't help.
makepkg.conf is imo the way to go, because things are up to the user, how hot he wants his builds to get. I also concur with auguste's opinion that PKGUBILD is not the place to make this kind (rather machine-dependent) setting. thanks for bringing this up btw, I used to add this stuff manually to the custom kernels' PKGBUILDs I use. cheers! mar77i
On Sun, Apr 1, 2012 at 12:59 AM, David C. Rankin <drankinatty@suddenlinkmail.com> wrote:
Guys,
Is there any way to check the number of cores on a machine to set the number of jobs within a PKGBUILD?:
make -j'X'
I have some dual-core and some quad-core boxes and the difference between -j2 and -j4 on build time is tremendous. However, for generically creating a source package to be used by someone else, I can't hardcode the jobs number. Anyway to test in pkgbuild or is this just a per box makepkg.conf issue?
-- David C. Rankin, J.D.,P.E.
i am not sure why are you trying to do it in PKGBUILD. if you have several machines that you want to have different settings, why not set /etc/makepkg.conf or $HOME/.makepkg.conf separately? anyway, i think the following command can easily get the information you need: grep -c processor /proc/cpuinfo
On Sun, 1 Apr 2012 04:45:08 -0400 Auguste Pop <auguste@gmail.com> wrote:
anyway, i think the following command can easily get the information you need: grep -c processor /proc/cpuinfo
This command will make a few 2 Cores processor with some kind of virtual processing technology (like hyper-threading) to be counted as a 4 cores processor. You may use: #cat /proc/cpuinfo | grep "physical id" | sort | uniq | wc -l To get the number of CPU "slots" you have on the machine. And then: #cat /proc/cpuinfo | grep "cpu cores" | uniq To get the number of cores per "slot". Then you just have to multiply both numbers and you know how many "real cpus" you have on the system. Otherwise you may be counting virtual processors, which shouldn't be used for heavy stuff such as compiling. Regards, Vitor
On 01.04.2012 13:05, Vitor Garcia wrote:
On Sun, 1 Apr 2012 04:45:08 -0400 Auguste Pop <auguste@gmail.com> wrote:
anyway, i think the following command can easily get the information you need: grep -c processor /proc/cpuinfo
This command will make a few 2 Cores processor with some kind of virtual processing technology (like hyper-threading) to be counted as a 4 cores processor.
Which is totally fine.
Then you just have to multiply both numbers and you know how many "real cpus" you have on the system. Otherwise you may be counting virtual processors, which shouldn't be used for heavy stuff such as compiling.
Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%. -- Florian Pritz
On Sun, 01 Apr 2012 13:37:27 +0200 Florian Pritz <bluewind@xinu.at> wrote:
Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%.
This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
Often higher cores benefit high I/O applications. If gcc is bottlenecking at reading and writing, sometimes more threads will help. On Apr 1, 2012 7:49 AM, "Vitor Garcia" <vitorlopesgarcia@gmail.com> wrote:
On Sun, 01 Apr 2012 13:37:27 +0200 Florian Pritz <bluewind@xinu.at> wrote:
Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%.
This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
On 04/01/2012 08:37 AM, Calvin Morrison wrote:
Often higher cores benefit high I/O applications. If gcc is bottlenecking at reading and writing, sometimes more threads will help.
How does that help? Sounds backwards to me
On Apr 1, 2012 7:49 AM, "Vitor Garcia"<vitorlopesgarcia@gmail.com> wrote:
On Sun, 01 Apr 2012 13:37:27 +0200 Florian Pritz<bluewind@xinu.at> wrote:
Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%. This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
Because if there are 4 cores doing heavy CPU lifting of 4 threads, when one is finished, you then waste time loading up the next thread because it's reading from the disk. If you have it already done with I/O it will be ready and less time is wasted. Calvin On Apr 1, 2012 9:09 AM, "Baho Utot" <baho-utot@columbus.rr.com> wrote:
On 04/01/2012 08:37 AM, Calvin Morrison wrote:
Often higher cores benefit high I/O applications. If gcc is bottlenecking at reading and writing, sometimes more threads will help.
How does that help?
Sounds backwards to me
On Apr 1, 2012 7:49 AM, "Vitor Garcia"<vitorlopesgarcia@**gmail.com<vitorlopesgarcia@gmail.com>>
wrote:
On Sun, 01 Apr 2012 13:37:27 +0200
Florian Pritz<bluewind@xinu.at> wrote:
Simple tests (building readline because it's small) with -j4 and -j8
on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%.
This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
On Sun, Apr 1, 2012 at 7:49 AM, Vitor Garcia <vitorlopesgarcia@gmail.com> wrote:
On Sun, 01 Apr 2012 13:37:27 +0200 Florian Pritz <bluewind@xinu.at> wrote:
Simple tests (building readline because it's small) with -j4 and -j8 on my i7-920 show that -j8 is around 20% faster than -j4. IIRC wikipedia states that HT core can increase performance by up to 30%.
This is nice to know. I work with mechanichal engineering simulations and we have noted that using more threads then avaiable processors (we have a 2 x 6 cores processors that has HT, so it looks like a 24 cores server) increases the calculation time on the softwares we have. I assumed that the same would apply to any intensive task, and we have even disabled HT on the BIOS. Perhaps I'll enable it again.
from my limited experience with scientific computing, the appropriate number of threads oftens relates not only to the number of cpus/cores, but also to the genuine feature of your application. i would suggest making some smaller-scaled test runs to test the performance and the implementation as well. best regards,
On 04/01/2012 03:45 AM, Auguste Pop wrote:
i am not sure why are you trying to do it in PKGBUILD. if you have several machines that you want to have different settings, why not set /etc/makepkg.conf or $HOME/.makepkg.conf separately?
anyway, i think the following command can easily get the information you need: grep -c processor /proc/cpuinfo
Thanks All, Auguste, I was really just brainstorming and I ran out of answers, so I asked on the list. What started this was just the question: "What can I do (if anything) to optimize the --jobs setting for trinity builds for anyone who decided to try building it?" I didn't even know how much difference the make --jobs=X could make on build time. It turns out to make a LOT of difference with quad core processors. A quick example is Amarok. With --jobs=0 on a phenom 9850 w/8G, the build takes 10 minutes. With --jobs=4, it builds in just over 2 minutes. That is a factor of five on rebuilds - literally cutting hours off the build time of trinity. I always wondered why anyone would care how many cores a processor has, because for desktop use -- I can't tell much difference at all on how a desktop behaves. However -- the usefulness of the additional cores was instantly clear when building large packages. So I was wanting to tweak the: make || return 1 line in the pkgbuild files to optimize build times for whoever was building (even if they hadn't made friends with makepkg.conf yet) Then thinking further about it, the Arch setup using makepkg.conf just made that much more sense. If a user is building that large of projects, then they should be at a point to learn to optimize makepkg themselves. That and the additional build instability (small, but it makes a difference with some code) of parallel builds by forcing --jobs=X in the pkgbuild, looked like a bad idea all the way around. However, I do thank you for the grep -c processor /proc/cpuinfo tip. -- David C. Rankin, J.D.,P.E.
su, 2012-04-01 kello 14:44 -0500, David C. Rankin kirjoitti:
make || return 1
Those '|| return 1' statements havent been used in long time anymore, yoy can just remove all them :D
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/01/2012 05:47 PM, Jesse Juhani Jaara wrote:
su, 2012-04-01 kello 14:44 -0500, David C. Rankin kirjoitti:
make || return 1
Those '|| return 1' statements havent been used in long time anymore, yoy can just remove all them :D
Thanks! I also did some testing on old and new boxes to find a makepkg.conf test that is reliable. I hit on the following: CPUCORES=$(grep -c "^processor" /proc/cpuinfo) if test $CPUCORES -gt 1; then MAKEFLAGS="-j${CPUCORES}" fi The beginning of line (^) is important to guard against incorrect results like: 16:01 nemesis:~> ( CPUCORES=$(grep -c "processor" /proc/cpuinfo); echo "CPUCORES: $CPUCORES"; ) CPUCORES: 2 16:01 nemesis:~> cat /proc/cpuinfo processor : 0 vendor_id : AuthenticAMD cpu family : 6 model : 4 model name : AMD Athlon(tm) processor stepping : 2 cpu MHz : 800.054 cache size : 256 KB fdiv_bug : no hlt_bug : no f00f_bug : no coma_bug : no fpu : yes fpu_exception : yes cpuid level : 1 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow up bogomips : 1602.51 clflush size : 32 Notice the extra 'processor' in 'model name'. - -- David C. Rankin, J.D.,P.E. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk94xLAACgkQZMpuZ8CyrciRDACfbW3UR/fcKLgye6G8Da2/XA7L efwAnRUDOD3QSFe5zp9IPFKi1Hu0Jttt =VL8G -----END PGP SIGNATURE-----
On 1 April 2012 10:59, David C. Rankin <drankinatty@suddenlinkmail.com>wrote:
Guys,
Is there any way to check the number of cores on a machine to set the number of jobs within a PKGBUILD?:
make -j'X'
I have some dual-core and some quad-core boxes and the difference between -j2 and -j4 on build time is tremendous. However, for generically creating a source package to be used by someone else, I can't hardcode the jobs number. Anyway to test in pkgbuild or is this just a per box makepkg.conf issue?
I had this for a one-fit Arch UFD: CPUCORES=$(grep processor /proc/cpuinfo | wc -l) [ $CPUCORES -gt 1 ] && MAKEFLAGS="-j$(($CPUCORES + 1))" But that's in makepkg.conf :) You shouldn't enforce this from within a PKGBUILD. -- GPG/PGP ID: C0711BF1
participants (11)
-
Auguste Pop
-
Baho Utot
-
Calvin Morrison
-
David C. Rankin
-
Florian Pritz
-
Jesse Jaara
-
Jesse Juhani Jaara
-
Martti Kühne
-
Rashif Ray Rahman
-
Vitor Garcia
-
XeCycle