[arch-general] System Freeze on sleep
Hello All, After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough to be irritating. I have to reboot the system to recover. I am running kernel 5.12.7-arch1-1 The issue was not seen with kernel 5.11. I am running arch on btrfs and I suspect it is a btrfs issue somehow. When I check dmesg (journactl --dmesg --boot -1) I see that it always ends with the following two lines when the issue is seen. May 29 07:12:40 matrix kernel: PM: suspend entry (deep) May 29 07:12:40 matrix kernel: Filesystems sync: 0.025 seconds I wanted to know how do I attempt to debug this issue? Regards, Prateek
On 5/29/21 2:23 PM, Prateek Sadhukhan via arch-general wrote:
Hello All,
After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough
Hi It can be many things for sure so it might be helpful for you to narrow it a bit; things to consider include: CPU - AMD vs Intel Graphics - intel, amd, nvidia etc. Any 3rd party drivers (nvidia e.g.) - look for 'tainted' in logs. google can be quite helpful too, as can reading the git logs from kernel or bugs on freedesktop.org if you think it might be graphics related. If you do have any 3rd party drivers, you can try not using them to see if makes a difference. Quite a few reports of problems with both suspend or coming out of suspend (I'm equating 'sleep' with suspend as distinct from hibernate here). Please report back what you learn and good luck. regards, gene
When I check dmesg (journactl --dmesg --boot -1)
/var/log/errors.log usually provide useful information. -- Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma brièveté.
On 29/05/21 11:53 pm, Prateek Sadhukhan via arch-general wrote:
Hello All,
After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough to be irritating. I have to reboot the system to recover.
I am running kernel 5.12.7-arch1-1
See and report here. https://bugs.archlinux.org/task/70773 Only problem here is that for many who reported the bug, it is resolved in kernel 5.12.7 but for you it still remains. Regards, Amish
On 5/29/21 8:23 PM, Prateek Sadhukhan via arch-general wrote:
I am running arch on btrfs and I suspect it is a btrfs issue somehow. When I check dmesg (journactl --dmesg --boot -1) I see that it always ends with the following two lines when the issue is seen.
May 29 07:12:40 matrix kernel: PM: suspend entry (deep) May 29 07:12:40 matrix kernel: Filesystems sync: 0.025 seconds
This doesn't necessarily mean that the error is file system-related. The latest message is "File systems sync" because, well, that's when the file system synced, and therefore the log got written to disk. The error might have occurred at a completely different step afterwards, but isn't visible on the next reboot, because the file systems weren't synced again, so it got never written to disk. Keep that in mind when diagnosing this issue, it might be something completely unrelated to file systems. Regards, LuKaRo
Hey Thanks LuKaRo, I will keep this in mind. I was so far trying to search btrfs issues since Arch wiki says it is unstable and all. Unfortunately with this now I have no idea where to start looking :-). Regards, Prateek LuKaRo via arch-general <arch-general@lists.archlinux.org> writes:
On 5/29/21 8:23 PM, Prateek Sadhukhan via arch-general wrote:
I am running arch on btrfs and I suspect it is a btrfs issue somehow. When I check dmesg (journactl --dmesg --boot -1) I see that it always ends with the following two lines when the issue is seen.
May 29 07:12:40 matrix kernel: PM: suspend entry (deep) May 29 07:12:40 matrix kernel: Filesystems sync: 0.025 seconds
This doesn't necessarily mean that the error is file system-related. The latest message is "File systems sync" because, well, that's when the file system synced, and therefore the log got written to disk. The error might have occurred at a completely different step afterwards, but isn't visible on the next reboot, because the file systems weren't synced again, so it got never written to disk. Keep that in mind when diagnosing this issue, it might be something completely unrelated to file systems.
Regards,
LuKaRo
Yes, I had found this and similar ones. And yes 5.12 I had no such issues. Cheers, Prateek Amish via arch-general <arch-general@lists.archlinux.org> writes:
On 29/05/21 11:53 pm, Prateek Sadhukhan via arch-general wrote:
Hello All,
After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough to be irritating. I have to reboot the system to recover.
I am running kernel 5.12.7-arch1-1
See and report here.
https://bugs.archlinux.org/task/70773
Only problem here is that for many who reported the bug, it is resolved in kernel 5.12.7 but for you it still remains.
Regards,
Amish
CPU is intel. 2 GPUs - Intel and AMD. No third party drivers that I installed explicitly. Laptop is a Dell Vostro. I get grief from the network driver sometimes and I have to re-load the ath10k_pci driver/module manually sometimes. I checked Google first thing. Could not find anything relevant. Cheers, Prateek Genes Lists via arch-general <arch-general@lists.archlinux.org> writes:
On 5/29/21 2:23 PM, Prateek Sadhukhan via arch-general wrote:
Hello All, After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough
Hi
It can be many things for sure so it might be helpful for you to narrow it a bit; things to consider include:
CPU - AMD vs Intel Graphics - intel, amd, nvidia etc. Any 3rd party drivers (nvidia e.g.) - look for 'tainted' in logs.
google can be quite helpful too, as can reading the git logs from kernel or bugs on freedesktop.org if you think it might be graphics related.
If you do have any 3rd party drivers, you can try not using them to see if makes a difference.
Quite a few reports of problems with both suspend or coming out of suspend (I'm equating 'sleep' with suspend as distinct from hibernate here).
Please report back what you learn and good luck.
regards,
gene
I could not find this file. I have to enable this log?
syslog-ng creates this file amongst others. But it may not be able to write anything anywhere in a frozen system during wakeup.
On 5/30/21 12:53 AM, Prateek Sadhukhan via arch-general wrote:
CPU is intel. 2 GPUs - Intel and AMD. No third party drivers that I installed explicitly.
Hi - If were me and using AMD graphics I'd try blacklisting the AMD one and see if it helps.
On 5/30/21 12:53 AM, Prateek Sadhukhan via arch-general wrote:
...No third party drivers that I installed explicitly.
To confirm you're not using 3rd party drivers you can run: # journalctl -lb |grep taint Look for something like: kernel: xxx_driver: loading out-of-tree module taints kernel
I recently had a similar issue. As it was breaking my workflow for day-to-day use I investigated the issue. If the computer is unresposive while resuming it's very likely to be a kernel panic. If there is no display output at all, either the graphics driver itself causes the panic, or some kernel module before that. The only way to debug these kinds of kernel panics was to use a serial port and a second computer to get a tty. As most computers have a COM-header on the mainboard, it's fairly easy to get things connected. I purchased a COM-RS232 PCIe Shield and found a Serial to USB adapter at home. I then set the following kernel cmdline no_console_suspend console=tty0 console=ttyS0,115200 no_console_suspend was needed in my case, but it might not be needed in all cases, I am sure it's not recommended to always have that enabled, but as we are debugging here we should be fine. The two console= arguments set the console (kernel log, systemd log and panics) to both tty0 (the default) as well as ttyS0 at a baud of 115200 (Check if ttyS0 is your serial port while booted). Now you can get your USB-Serial adapter, connect it with your computer's COM port and start a serial console like screen, minicom or picocom. Now you should see the kernel log while booting. You can even have getty on there to login via serial. (I don't know right now if it is autostarted or if you need to start it manually with systemctl start getty@ttyS0.service) Now you can get to debugging: Reproduce your issue and see what you get there. P.S. There are other ways to see the console without working graphics. There is USB-3.0 debugging (couldn't get that to work myself) and netconsole (didn't work for my issue, maybe it wasn't initialised before the panic) P.S.P.S. I planned to write this process up somewhere, I guess having it here is a start. I may create a Wiki article about this in the future. P.S.P.S.P.S. My issue was an actual kernel bug, which I fixed and got merged with recent 5.11 and 5.12 releases. See https://git.kernel.org/pub/scm/linux/kernel/git/ torvalds/linux.git/commit/?id=7df4ceb60fa9a3c5160cfd5b696657291934a2c9
Hello All,
After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough to be irritating. I have to reboot the system to recover.
I am running kernel 5.12.7-arch1-1
The issue was not seen with kernel 5.11.
I am running arch on btrfs and I suspect it is a btrfs issue somehow. When I check dmesg (journactl --dmesg --boot -1) I see that it always ends with the following two lines when the issue is seen.
May 29 07:12:40 matrix kernel: PM: suspend entry (deep) May 29 07:12:40 matrix kernel: Filesystems sync: 0.025 seconds
I wanted to know how do I attempt to debug this issue?
Regards, Prateek
-- Sefa Eyeoglu https://scrumplex.net
This is interesting. I will store this in my notes too. The issue is I am working on a laptop and so dont have a serial port. But I think I have found the problem. I am soaking it for now, I will respond to the thread if my hunch works. Thanks, Prateek Sefa Eyeoglu via arch-general <arch-general@lists.archlinux.org> writes:
I recently had a similar issue.
As it was breaking my workflow for day-to-day use I investigated the issue.
If the computer is unresposive while resuming it's very likely to be a kernel panic.
If there is no display output at all, either the graphics driver itself causes the panic, or some kernel module before that.
The only way to debug these kinds of kernel panics was to use a serial port and a second computer to get a tty.
As most computers have a COM-header on the mainboard, it's fairly easy to get things connected. I purchased a COM-RS232 PCIe Shield and found a Serial to USB adapter at home.
I then set the following kernel cmdline
no_console_suspend console=tty0 console=ttyS0,115200
no_console_suspend was needed in my case, but it might not be needed in all cases, I am sure it's not recommended to always have that enabled, but as we are debugging here we should be fine.
The two console= arguments set the console (kernel log, systemd log and panics) to both tty0 (the default) as well as ttyS0 at a baud of 115200 (Check if ttyS0 is your serial port while booted).
Now you can get your USB-Serial adapter, connect it with your computer's COM port and start a serial console like screen, minicom or picocom.
Now you should see the kernel log while booting. You can even have getty on there to login via serial. (I don't know right now if it is autostarted or if you need to start it manually with systemctl start getty@ttyS0.service)
Now you can get to debugging: Reproduce your issue and see what you get there.
P.S. There are other ways to see the console without working graphics. There is USB-3.0 debugging (couldn't get that to work myself) and netconsole (didn't work for my issue, maybe it wasn't initialised before the panic)
P.S.P.S. I planned to write this process up somewhere, I guess having it here is a start. I may create a Wiki article about this in the future.
P.S.P.S.P.S. My issue was an actual kernel bug, which I fixed and got merged with recent 5.11 and 5.12 releases. See https://git.kernel.org/pub/scm/linux/kernel/git/ torvalds/linux.git/commit/?id=7df4ceb60fa9a3c5160cfd5b696657291934a2c9
Hello All,
After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough to be irritating. I have to reboot the system to recover.
I am running kernel 5.12.7-arch1-1
The issue was not seen with kernel 5.11.
I am running arch on btrfs and I suspect it is a btrfs issue somehow. When I check dmesg (journactl --dmesg --boot -1) I see that it always ends with the following two lines when the issue is seen.
May 29 07:12:40 matrix kernel: PM: suspend entry (deep) May 29 07:12:40 matrix kernel: Filesystems sync: 0.025 seconds
I wanted to know how do I attempt to debug this issue?
Regards, Prateek
Thank you for all the ideas. The issue is now solved. This is somehow caused by the ath10k_pci wireless driver. On my Dell Vostro laptop this is a known issue (Arch Wiki), that after resume wireless connection fails and I have to reload the module. In my case looks like it was causing resume from sleep to fail altogether. I added a systemd hook to remove this module when going to sleep and re-adding it when resuming. This seems to hold good so far. Regards, Prateek Prateek Sadhukhan via arch-general <arch-general@lists.archlinux.org> writes:
Hello All,
After upgrading to the new kernel I see that my system freezes when I try to resume from sleep. This happens intermittently but frequent enough to be irritating. I have to reboot the system to recover.
I am running kernel 5.12.7-arch1-1
The issue was not seen with kernel 5.11.
I am running arch on btrfs and I suspect it is a btrfs issue somehow. When I check dmesg (journactl --dmesg --boot -1) I see that it always ends with the following two lines when the issue is seen.
May 29 07:12:40 matrix kernel: PM: suspend entry (deep) May 29 07:12:40 matrix kernel: Filesystems sync: 0.025 seconds
I wanted to know how do I attempt to debug this issue?
Regards, Prateek
participants (6)
-
Amish
-
Genes Lists
-
LuKaRo
-
Prateek Sadhukhan
-
Sefa Eyeoglu
-
SET