[arch-proaudio] JACK 1: extreme CPU usage (since update from 5.0 kernel)
Hey hey everyone, JACK has long had a habit of grabbing a lot of CPU time, even in idel mode, getting to 100% (one of four cores) and raising the CPU from 800MHz to 3GHz. But it usually took some time. Restarting the JACK daemon resolved the issue for some time. Since the update from the 5.0.x kernel to first a 5.1(?) (last Sunday) and now 5.2.5 has incrased the issue. JACK quickly starts grabbing lots of CPU, heating up the system (and waisting energy). This gets to a point where I can't change terminals. Only when I stop JACK, the system will react to an already pressed alt+fN command, as if it was stored in a bugffer, but couldn't be reacted upon. The keystrokes are read immediately, as verified by showkey, the problem is JACK as verified, by always getting a prompt from tty1 after shutting it down. I have set the kernel commandline parameter audit=0 . Another suggestion recommended trying something with spectre mitigation. There was mention of a few kernel parameters and possible security risks, which went over my head. Can someone please help me through this? Hints, tips, maybe similar experiences and good solutions? I have run: realTimeConfigQuickScan which aggreed with everything except that my kernel isn't real lowlatency. It's the standard pacman kernel. And it didn't like the schedutil governor, which I change to performance, when I'm recording. More info? CPU: AMD Athlon(tm) II X4 640 Processor (/proc/cpuinfo) quadcore with up to 3GHz 4GB of RAM (mostly no issues at all) no GUI Soundcard: twl M-Audio Delta 1010LT sync'ed by SP/DIF and bundled into one device through ALSA through multi device type. Any help is appreciated! Thanks you and best wishes, Jeanette -- * Website: http://juliencoder.de - for summer is a state of sound * SoundCloud: https://soundcloud.com/jeanette_c * Youtube: https://www.youtube.com/channel/UCMS4rfGrTwz8W7jhC1Jnv7g * GitHub: https://github.com/jeanette-c * Twitter: https://twitter.com/jeanette_c_s There's a girl in the mirror I wonder who she is Sometimes I think I know her Sometimes I really wish I did <3 (Britney Spears)
Hi Jeanette, On 2019-08-05 19:19:34 (+0200), Jeanette C. wrote:
JACK has long had a habit of grabbing a lot of CPU time, even in idel mode, getting to 100% (one of four cores) and raising the CPU from 800MHz to 3GHz. How do you start jack? At what samplerate/buffersize? Is it the same with jack2?
You can also check what's going on in iotop. Not sure how easy that is to use for you, but at least it's a CLI tool (and has a man page!).
I have set the kernel commandline parameter audit=0 . It's not necessarily *that* performance degrading, but YMMV.
Another suggestion recommended trying something with spectre mitigation. Please don't attempt that, unless you won't be connected to the internet. ;-)
Can someone please help me through this? Hints, tips, maybe similar experiences and good solutions? I've also had some degrading performance issues with my Lenovo W540 starting with linux > 5.0. Guess it's a mixture of Intel mitigations and ucode updates. Yay :-/
You can try with linux-rt and/or linux-rt-lts. They're now pre-compiled in an unofficial repository [1]. As I build/signed these, you won't have to import a key or anything, but merely add the additional entry to /etc/pacman.conf [2].
I have run: realTimeConfigQuickScan which aggreed with everything except that my kernel isn't real lowlatency. It's the standard pacman kernel. And it didn't like the schedutil governor, which I change to performance, when I'm recording. Do you use irqbalance? This could help in distributing the (hardware) Interrupt Request (IRQ) handling on your system over your various cores/threads.
More info? CPU: AMD Athlon(tm) II X4 640 Processor (/proc/cpuinfo) quadcore with up to 3GHz 4GB of RAM (mostly no issues at all) That's not a superlot of RAM, but you also don't have any hungry GUIs :-) Do you use amd-ucode? Which bootloader do you use?
Soundcard: twl M-Audio Delta 1010LT sync'ed by SP/DIF and bundled into one device through ALSA through multi device type. So, you're saying, that you're using two soundcards? That could in theory be a problem (at least to performance).
As you're not using a realtime kernel currently, I assume, you're also not using something to tune your IRQs (e.g. rtirq). However, that could be quite interesting for boosting the performance. It requires you to figure out, whether you have IRQs, that are shared: ``` cat /proc/interrupts |grep ", " ``` and whether those are somehow related to the hardware you would like to use (e.g. your builtin soundcard). Those devices (names are in the last column), can be boosted to a higher RTPRIO using rtirq. The wiki articles about most of this are still lacking big time IMHO... if I find time in the coming weeks, I'll try to improve them. I hope this got you some ideas where to look and what to try nonetheless. Best, David [1] https://wiki.archlinux.org/index.php/Unofficial_user_repositories#dvzrv [2] https://wiki.archlinux.org/index.php/Pacman#Repositories_and_mirrors -- https://sleepmap.de
Hi David, thanks for your great and fast reply. I'd like to go through it, maybe we could delve into it together a bit more? Aug 5 2019, David Runge has written: ...
How do you start jack? My own systemd service as: jackd -R --timeout 4500 -d alsa -C cdelta -P pdelta -r 48000 -n 3 -p 256 -z shaped Is it the same with jack2? Never tried, since it's not been recommended for a long time and used to be heavily integrated with dbus. ... Trying iotop, have to figure out a few settings for better readabililty with my braille display. Could be quite useful.
You can try with linux-rt and/or linux-rt-lts. Do they also support schedutil, mostly my system is quite idle. :) Thanks for the package! ... Regarding irqs: no shared irq, the soundcards are at 18 and 19. No rtirq not irqbalance. Would it help with that situation? ... That's not a superlot of RAM, but you also don't have any hungry GUIs :-) Definitely not! :) top ... KiB Mem : 3780140 total, 587016 free, 399196 used, 2793928 buff/cache KiB Swap: 4194300 total, 4167676 free, 26624 used. 3072468 avail Mem ... Do you use amd-ucode? Which bootloader do you use? grub2-pc (no uEFI) and no amd-ucore yet. If threatened with your life for an estimated impact (ballpark figure), what would you say for the amd-ucode? ... So, you're saying, that you're using two soundcards? That could in theory be a problem (at least to performance). Yes, two soundcards, sync'ed amongst each other by SP/DIF, the first one clocked/sync'ed from external SP/DIF. Can't do anything about it, too many synths. :) Not that that's a reason: but it has worked for years, starting with 2.6.3x kernels and was easy enough to setup on my first Arch install in 2015 or so. ... Once again, thanks for your pragmatic and concise answer. I am sorry to take up more time. I'm really a little out of my usual water here. Haven't felt a need in a long time to tune my system on that level. This is good in general and bad for the case in hand. :(
Best wishes, Jeanette -- * Website: http://juliencoder.de - for summer is a state of sound * SoundCloud: https://soundcloud.com/jeanette_c * Youtube: https://www.youtube.com/channel/UCMS4rfGrTwz8W7jhC1Jnv7g * GitHub: https://github.com/jeanette-c * Twitter: https://twitter.com/jeanette_c_s There's a girl in the mirror I wonder who she is Sometimes I think I know her Sometimes I really wish I did <3 (Britney Spears)
On 2019-08-05 22:20:10 (+0200), Jeanette C. wrote:
Hi David, thanks for your great and fast reply. I'd like to go through it, maybe we could delve into it together a bit more? Aug 5 2019, David Runge has written: ...
How do you start jack? My own systemd service as: jackd -R --timeout 4500 -d alsa -C cdelta -P pdelta -r 48000 -n 3 -p 256 -z shaped Hm, that sounds reasonable. Note, `-R` is the default and usually starts jackd at RTPRIO 10. On a realtime kernel you can set the maximum RTPRIO higher than the lower halves (higher than 50) with the `-P` flag. Check `man 1 jackd` for more info.
Is it the same with jack2? Never tried, since it's not been recommended for a long time and used to be heavily integrated with dbus. Would be interesting to find out, whether you have the same issues there as with jack(1). The jack2 package also offers jackd, which should just work the same with your service file (jack2 is a API compatible replacement). Fun fact: The next jack2 release will offer a systemd user service out of the box. Also: Unless you have a bunch of GUI tools starting it via dbus activation, the jack2 package defaults to launching jackd!
Trying iotop, have to figure out a few settings for better readabililty with my braille display. Could be quite useful. It's sometimes interesting to figure out, what's clogging the system.
You can try with linux-rt and/or linux-rt-lts. Do they also support schedutil, mostly my system is quite idle. :) Yes.
Regarding irqs: no shared irq, the soundcards are at 18 and 19. No rtirq not irqbalance. Would it help with that situation? I guess irqbalance could improve your situation.
That's not a superlot of RAM, but you also don't have any hungry GUIs :-) Definitely not! :) top ... KiB Mem : 3780140 total, 587016 free, 399196 used, 2793928 buff/cache KiB Swap: 4194300 total, 4167676 free, 26624 used. 3072468 avail Mem ... Do you use amd-ucode? Which bootloader do you use? grub2-pc (no uEFI) and no amd-ucore yet. If threatened with your life for an estimated impact (ballpark figure), what would you say for the amd-ucode? The amd-ucode package only provides the ucode updates from AMD (before you boot the kernel image). They are basically firmware enhancements for bugs in the hardware and are generally recommended. With grub you won't have to do much to use it (after installing it), but to update your configuration [1].
Closing, you do mean the package grub when you write grub2-pc, yes?
So, you're saying, that you're using two soundcards? That could in theory be a problem (at least to performance). Yes, two soundcards, sync'ed amongst each other by SP/DIF, the first one clocked/sync'ed from external SP/DIF. Can't do anything about it, too many synths. :) Not that that's a reason: but it has worked for years, starting with 2.6.3x kernels and was easy enough to setup on my first Arch install in 2015 or so. Hm, yes, I guess this shouldn't be a problem. However, ALSA with several soundcards apparently tends to be unstable for some (with different soundcards) due to slightly differing clocks and such.
Haven't felt a need in a long time to tune my system on that level. This is good in general and bad for the case in hand. :( No worries. Usually, there's really not much needed, besides a realtime kernel and starting jackd with a reasonably high RTPRIO. I currently use this systemd user unit [2] and these environment configs [3] (depending on hardware).
Best, David [1] https://wiki.archlinux.org/index.php/Microcode#GRUB [2] https://git.sleepmap.de/dave/dotfiles.git/tree/.config/systemd/user/jack@.se... [3] https://git.sleepmap.de/dave/dotfiles.git/tree/.config/jack -- https://sleepmap.de
On a realtime kernel you can set the maximum RTPRIO higher than the lower halves (higher than 50) with the `-P` flag. Check `man 1 jackd` for more info. To extend this: The systemd user unit needs to allow for it to happen (see LimitRTPRIO, LimitRTTIME and LimitMEMLOCK in `man systemd.exec`) and your user must be allowed to actually do this as well (e.g. by using
I realized, that I wrote some inconsistent stuff in regards to IRQ tuning last night, which I'd like to clarify. On 2019-08-06 00:12:18 (+0200), David Runge wrote: the realtime-privileges package - which is Arch specific! - and adding your user to the 'realtime' group).
No worries. Usually, there's really not much needed, besides a realtime kernel and starting jackd with a reasonably high RTPRIO. I currently use this systemd user unit [2] and these environment configs [3] (depending on hardware). Note: If your audio hardware IRQs have a lower RTPRIO, than jackd (e.g. your audio card runs at 50, but you start jackd at 80), this *will* lead to problems! Ordering is important when tuning IRQs!
Again: The Arch Wiki really lacks info on all of this. Best, David -- https://sleepmap.de
Hello David, I have now installed and booted a realtime kernel, thanks for your repo, and installed and enabled irqbalance. There's one question however, to which I can't find a satisfactory answer. Aug 6 2019, David Runge has written: ...
Note: If your audio hardware IRQs have a lower RTPRIO, than jackd (e.g. your audio card runs at 50, but you start jackd at 80), this *will* lead to problems! Ordering is important when tuning IRQs! ... How do I check my soundcards realtime priority. iotop gives a kind of RTprio as something like be/4, the manual is unclear on that classification. Well, they don't explain it. :)
Best wishes, Jeanette -- * Website: http://juliencoder.de - for summer is a state of sound * SoundCloud: https://soundcloud.com/jeanette_c * Youtube: https://www.youtube.com/channel/UCMS4rfGrTwz8W7jhC1Jnv7g * GitHub: https://github.com/jeanette-c * Twitter: https://twitter.com/jeanette_c_s Make all the clouds disappear Put all your fears to rest <3 (Britney Spears)
On 2019-08-06 13:51:42 (+0200), Jeanette C. wrote:
How do I check my soundcards realtime priority. iotop gives a kind of RTprio as something like be/4, the manual is unclear on that classification. Well, they don't explain it. :) If you install and setup rtirq (that's quite straight forward) with a realtime kernel, its status output will show you just that: `rtirq status`
Else, if you require more output, you can try something like: ``` ps -eo pid,class,rtprio,ni,pri,pcpu,stat,comm --sort -rtprio ``` (That's actually what's being called within rtirq as part of the status call.) Note, that rtirq doesn't make much sense (or just won't have the same effect), when not using a realtime kernel (or the 'threadirqs' kernel parameter [1] - which is the default on linux-rt{,-lts}, but can be activated for the vanilla kernel as well - for that matter). While you can set very specific IRQs (based on the names in the most right column in `/proc/interrupts`), rtirq also allows for more general approaches (e.g. 'usb' for all USB devices, or 'snd' for all PCI based internal audio devices), when setting the priorities in its configuration file. Best, David P.S.: Closing, it of course also makes sense to monitor your jackd service closely, by attaching to its journal (e.g. `journalctl --user -f -u myjackd.service`) and potentially momentarily increasing its verbosity (for debugging, if you think there's still something wrong). [1] https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html -- https://sleepmap.de
participants (2)
-
David Runge
-
Jeanette C.