Hello,
I want to have a sub-30 second boot time, so as to make it possible for me to power down my computer and power back up at a moment's notice. Right now, I have a (according to my watch) ~35 second boot time. I used systemd-analyze, and discovered that NetworkManager.service is running for 12 seconds of that time! Also, polkit.service runs quite a while after everything else, but that is less important.
How do I optimize NetworkManager.service to shorten boot times, especially as I don't need a network first-thing?
On Wed, Apr 30, 2014 at 12:06 AM, Toyam Cox csupercomputergeek@gmail.com wrote:
Hello,
I want to have a sub-30 second boot time, so as to make it possible for me to power down my computer and power back up at a moment's notice. Right now, I have a (according to my watch) ~35 second boot time. I used systemd-analyze, and discovered that NetworkManager.service is running for 12 seconds of that time! Also, polkit.service runs quite a while after everything else, but that is less important.
How do I optimize NetworkManager.service to shorten boot times, especially as I don't need a network first-thing?
--
- Toyam
Are you using a display manager? If not (getty), add a snippet for getty@.service to use Type=simple instead of Type=idle (making it start ASAP) and add "quiet" to the kernel command line so boot messages are suppressed. This will allow you to log in with getty while the system is still booting.
On Tue, Apr 29, 2014 at 6:17 PM, Jan Alexander Steffens < jan.steffens@gmail.com> wrote:
On Wed, Apr 30, 2014 at 12:06 AM, Toyam Cox csupercomputergeek@gmail.com wrote:
Hello,
I want to have a sub-30 second boot time, so as to make it possible for
me
to power down my computer and power back up at a moment's notice. Right now, I have a (according to my watch) ~35 second boot time. I used systemd-analyze, and discovered that NetworkManager.service is running
for
12 seconds of that time! Also, polkit.service runs quite a while after everything else, but that is less important.
How do I optimize NetworkManager.service to shorten boot times,
especially
as I don't need a network first-thing?
--
- Toyam
Are you using a display manager? If not (getty), add a snippet for getty@.service to use Type=simple instead of Type=idle (making it start ASAP) and add "quiet" to the kernel command line so boot messages are suppressed. This will allow you to log in with getty while the system is still booting.
I'm using LXDM, my attempt to be lightweight with a GUI. Would Type=simple still work with LXDM? I notice it starts relatively soon in the boot process, meaning I'd need to figure out getting polkit to start even sooner, but is it an option?
On Tue, Apr 29, 2014 at 06:28:42PM -0400, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 6:17 PM, Jan Alexander Steffens < jan.steffens@gmail.com> wrote:
On Wed, Apr 30, 2014 at 12:06 AM, Toyam Cox csupercomputergeek@gmail.com wrote:
Hello,
I want to have a sub-30 second boot time, so as to make it possible for
me
to power down my computer and power back up at a moment's notice. Right now, I have a (according to my watch) ~35 second boot time. I used systemd-analyze, and discovered that NetworkManager.service is running
for
12 seconds of that time! Also, polkit.service runs quite a while after everything else, but that is less important.
How do I optimize NetworkManager.service to shorten boot times,
especially
as I don't need a network first-thing?
--
- Toyam
Are you using a display manager? If not (getty), add a snippet for getty@.service to use Type=simple instead of Type=idle (making it start ASAP) and add "quiet" to the kernel command line so boot messages are suppressed. This will allow you to log in with getty while the system is still booting.
I'm using LXDM, my attempt to be lightweight with a GUI. Would Type=simple still work with LXDM? I notice it starts relatively soon in the boot process, meaning I'd need to figure out getting polkit to start even sooner, but is it an option?
--
- Toyam
LXDM is separate from getty. Getty runs on a TTY and prompts the user for a username. LXDM is a display manager, it runs on the X server and also prompts the user for a username and password. I doubt changing settings for one will affect the other. (Even though getty should by default run on all 6 initially available TTYs. Correct me if I'm wrong.)
If you really want to speed up boot times you might want to consider cutting down on the Display Manager as it is not essential to using a DE/WM with X server. But in the end, because LXDM is supposedly so lightweight, and I presume that you count starting X and running a DE/WM a part of the boot process, you might not get any speed advantage.
In the end, it really is a matter of what you consider a "booted" machine. I can get from halted and powered off to getty in under or around 20 seconds, a few seconds to log in, a few more for X server and under a second for i3wm (Yes it's lightning speed.) and I can be "booted" into a graphical environment in around 30 seconds.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
Am 30.04.2014 00:06, schrieb Toyam Cox:
NetworkManager.service is running for 12 seconds
Can you use static ip address in your network? The dhcp client did eat a lot of time here, too. 9 sec boot here without cryptsetup and static ip. Server needs 20 sec without ssd, 15 sec for dhcpcd, mysql and php-fpm
On Tue, Apr 29, 2014 at 7:20 PM, Simon Brand simon.brand@postadigitale.dewrote:
Am 30.04.2014 00:06, schrieb Toyam Cox:
NetworkManager.service is running for 12 seconds
Can you use static ip address in your network? The dhcp client did eat a lot of time here, too. 9 sec boot here without cryptsetup and static ip. Server needs 20 sec without ssd, 15 sec for dhcpcd, mysql and php-fpm
I do not believe that would help, because often I start up in areas without a network. Perhaps there is a way to get Network Manager to start after the boot is completed, or at least not be a boot dependency?
On 29/04/14 07:34 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 7:20 PM, Simon Brand simon.brand@postadigitale.dewrote:
Am 30.04.2014 00:06, schrieb Toyam Cox:
NetworkManager.service is running for 12 seconds
Can you use static ip address in your network? The dhcp client did eat a lot of time here, too. 9 sec boot here without cryptsetup and static ip. Server needs 20 sec without ssd, 15 sec for dhcpcd, mysql and php-fpm
I do not believe that would help, because often I start up in areas without a network. Perhaps there is a way to get Network Manager to start after the boot is completed, or at least not be a boot dependency?
NetworkManager works fine with roaming and can be configured to use a static IP on some networks but not others. I don't see what you have to gain by removing it from the regular boot process... just make sure you're not letting stuff block on it.
This is with NetworkManager enabled on a wireless network with a Samsung 840 EVO (it varies from ~2-3s for kernel + userspace):
Startup finished in 3.070s (firmware) + 60ms (loader) + 1.655s (kernel) + 676ms (userspace)
160ms NetworkManager.service
AFAIK it doesn't count the time needed to connect over DHCP... it's often not connected by the time I have a browser and a few terminals open in i3 since it takes 10 seconds.
Not that boot time should matter to anyone, since kernel upgrades aren't every day and there's not much reason to reboot otherwise :P.
On Tue, Apr 29, 2014 at 7:47 PM, Daniel Micay danielmicay@gmail.com wrote:
On 29/04/14 07:34 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 7:20 PM, Simon Brand simon.brand@postadigitale.dewrote:
Am 30.04.2014 00:06, schrieb Toyam Cox:
NetworkManager.service is running for 12 seconds
Can you use static ip address in your network? The dhcp client did eat a lot of time here, too. 9 sec boot here without cryptsetup and static ip. Server needs 20 sec without ssd, 15 sec for dhcpcd, mysql and php-fpm
I do not believe that would help, because often I start up in areas
without
a network. Perhaps there is a way to get Network Manager to start after
the
boot is completed, or at least not be a boot dependency?
NetworkManager works fine with roaming and can be configured to use a static IP on some networks but not others. I don't see what you have to gain by removing it from the regular boot process... just make sure you're not letting stuff block on it.
This is with NetworkManager enabled on a wireless network with a Samsung 840 EVO (it varies from ~2-3s for kernel + userspace):
Startup finished in 3.070s (firmware) + 60ms (loader) + 1.655s (kernel)
- 676ms (userspace)
160ms NetworkManager.service
AFAIK it doesn't count the time needed to connect over DHCP... it's often not connected by the time I have a browser and a few terminals open in i3 since it takes 10 seconds.
Not that boot time should matter to anyone, since kernel upgrades aren't every day and there's not much reason to reboot otherwise :P.
So something seems to be wrong here. Startup finished in 4.637s (firmware) + 131ms (loader) + 2.790s (kernel) + 20.066s (userspace) = 27.626s
12s NetworkManager.service
What sort of things should I check for? Is there an /etc config file I can play with?
On 29/04/14 07:55 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 7:47 PM, Daniel Micay danielmicay@gmail.com wrote:
On 29/04/14 07:34 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 7:20 PM, Simon Brand simon.brand@postadigitale.dewrote:
Am 30.04.2014 00:06, schrieb Toyam Cox:
NetworkManager.service is running for 12 seconds
Can you use static ip address in your network? The dhcp client did eat a lot of time here, too. 9 sec boot here without cryptsetup and static ip. Server needs 20 sec without ssd, 15 sec for dhcpcd, mysql and php-fpm
I do not believe that would help, because often I start up in areas
without
a network. Perhaps there is a way to get Network Manager to start after
the
boot is completed, or at least not be a boot dependency?
NetworkManager works fine with roaming and can be configured to use a static IP on some networks but not others. I don't see what you have to gain by removing it from the regular boot process... just make sure you're not letting stuff block on it.
This is with NetworkManager enabled on a wireless network with a Samsung 840 EVO (it varies from ~2-3s for kernel + userspace):
Startup finished in 3.070s (firmware) + 60ms (loader) + 1.655s (kernel)
- 676ms (userspace)
160ms NetworkManager.service
AFAIK it doesn't count the time needed to connect over DHCP... it's often not connected by the time I have a browser and a few terminals open in i3 since it takes 10 seconds.
Not that boot time should matter to anyone, since kernel upgrades aren't every day and there's not much reason to reboot otherwise :P.
So something seems to be wrong here. Startup finished in 4.637s (firmware) + 131ms (loader) + 2.790s (kernel) + 20.066s (userspace) = 27.626s
12s NetworkManager.service
What sort of things should I check for? Is there an /etc config file I can play with?
Use an efistub loader like gummiboot if you're not already, use lz4 compression for the kernel, disable staggered spin-up, use a single unsplit root partition and avoid remounting it, etc. I'm sure these things are all on the wiki somewhere, because I remember writing some of it.
There's not really any magic to speed up starting a large number of services, if that's what you're doing. All I have enabled is chrony/pdnsd/NetworkManager and they don't block the boot process. I just use agetty to start up i3.
I assume you've got something on the critical path depending on NetworkManager like a Type=idle service.
On Tue, Apr 29, 2014 at 8:09 PM, Daniel Micay danielmicay@gmail.com wrote:
On 29/04/14 07:55 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 7:47 PM, Daniel Micay danielmicay@gmail.com
wrote:
On 29/04/14 07:34 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 7:20 PM, Simon Brand simon.brand@postadigitale.dewrote:
Am 30.04.2014 00:06, schrieb Toyam Cox:
NetworkManager.service is running for 12 seconds
Can you use static ip address in your network? The dhcp client did eat a lot of time here, too. 9 sec boot here without cryptsetup and static ip. Server needs 20 sec without ssd, 15 sec for dhcpcd, mysql and php-fpm
I do not believe that would help, because often I start up in areas
without
a network. Perhaps there is a way to get Network Manager to start after
the
boot is completed, or at least not be a boot dependency?
NetworkManager works fine with roaming and can be configured to use a static IP on some networks but not others. I don't see what you have to gain by removing it from the regular boot process... just make sure you're not letting stuff block on it.
This is with NetworkManager enabled on a wireless network with a Samsung 840 EVO (it varies from ~2-3s for kernel + userspace):
Startup finished in 3.070s (firmware) + 60ms (loader) + 1.655s (kernel)
- 676ms (userspace)
160ms NetworkManager.service
AFAIK it doesn't count the time needed to connect over DHCP... it's often not connected by the time I have a browser and a few terminals open in i3 since it takes 10 seconds.
Not that boot time should matter to anyone, since kernel upgrades aren't every day and there's not much reason to reboot otherwise :P.
So something seems to be wrong here. Startup finished in 4.637s (firmware) + 131ms (loader) + 2.790s (kernel)
20.066s (userspace) = 27.626s
12s NetworkManager.service
What sort of things should I check for? Is there an /etc config file I
can
play with?
Use an efistub loader like gummiboot if you're not already, use lz4 compression for the kernel, disable staggered spin-up, use a single unsplit root partition and avoid remounting it, etc. I'm sure these things are all on the wiki somewhere, because I remember writing some of it.
There's not really any magic to speed up starting a large number of services, if that's what you're doing. All I have enabled is chrony/pdnsd/NetworkManager and they don't block the boot process. I just use agetty to start up i3.
I assume you've got something on the critical path depending on NetworkManager like a Type=idle service.
I read the wiki page today, actually. I have 1 root partition, and another for /usr and another for /var. / is /sda2, near the beginning of the physical disk. How do I check what is depending on what? It seems it's only the multi-boot.target that waits for it...
On 29/04/14 09:00 PM, Toyam Cox wrote:
On Tue, Apr 29, 2014 at 8:09 PM, Daniel Micay danielmicay@gmail.com wrote:
I read the wiki page today, actually. I have 1 root partition, and another for /usr and another for /var. / is /sda2, near the beginning of the physical disk.
That's going to take more time to deal with than a single partition, and you're probably doing a pointless remount of / via fstab.
In my opinion, there's little advantage to splitting it up into pieces now that we have the protected hardlink option enabled. If it's about space usage, then quotas seem saner.
How do I check what is depending on what? It seems it's only the multi-boot.target that waits for it...
Use `systemctl list-dependencies` on whatever service you're using to log in (display manager, agetty).
On 30 April 2014 00:06:43 CEST, Toyam Cox csupercomputergeek@gmail.com wrote:
Hello,
I want to have a sub-30 second boot time, so as to make it possible for me to power down my computer and power back up at a moment's notice. Right now, I have a (according to my watch) ~35 second boot time. I used systemd-analyze, and discovered that NetworkManager.service is running for 12 seconds of that time! Also, polkit.service runs quite a while after everything else, but that is less important.
How do I optimize NetworkManager.service to shorten boot times, especially as I don't need a network first-thing?
--
- Toyam
Just a wild guess: https://bbs.archlinux.org/viewtopic.php?id=180099
On Wed, Apr 30, 2014 at 3:22 AM, Jakub Klinkovský j.l.k@gmx.com wrote:
On 30 April 2014 00:06:43 CEST, Toyam Cox csupercomputergeek@gmail.com wrote:
Hello,
I want to have a sub-30 second boot time, so as to make it possible for me to power down my computer and power back up at a moment's notice. Right now, I have a (according to my watch) ~35 second boot time. I used systemd-analyze, and discovered that NetworkManager.service is running for 12 seconds of that time! Also, polkit.service runs quite a while after everything else, but that is less important.
How do I optimize NetworkManager.service to shorten boot times, especially as I don't need a network first-thing?
--
- Toyam
Just a wild guess: https://bbs.archlinux.org/viewtopic.php?id=180099
Startup finished in 4.215s (firmware) + 176ms (loader) + 2.939s (kernel) + 11.166s (userspace) = 18.497s
Thank you very much! NetworkManager takes a much more reasonable 1.4 seconds now. It would not have occured to me to have the journal checked. I also enabled readahead, and now LXDE starts before I'm finished booting, a very nice feature.
On Wed, Apr 30, 2014 at 10:56 AM, Toyam Cox csupercomputergeek@gmail.comwrote:
Startup finished in 4.215s (firmware) + 176ms (loader) + 2.939s (kernel) + 11.166s (userspace) = 18.497s
Thank you very much! NetworkManager takes a much more reasonable 1.4 seconds now. It would not have occured to me to have the journal checked. I also enabled readahead, and now LXDE starts before I'm finished booting, a very nice feature.
Just a comment about boot times. The overall boot performance will depend not only on optimising an individual setup, but also is dependent on the hardware as well as which boot manager is being used. So an older laptop with a hard drive, using BIOS boot and optimised will still see much longer boot time than say a new laptop running a fast i7 processor, with an ssd, using UEFI and also optimised. Certainly I have an old laptop that takes around 35 seconds to boot to the login prompt from when the boot manager takes over after POST using BIOS legacy boot, but a similarly set up and optimised new Haswell i7 laptop, with msata ssd using refind for UEFI boot takes about 7 seconds to reach the KDM login prompt. Of course for a specific system it may be possible to shave some seconds off the boot time, but it will also depend on which server daemons need to be started as well. So adding dovecot, an MTA, and maybe a DHCP server all add to the time taken to complete the boot process.
So comparisons of absolute boot times from different machines are difficult to interpret.
On Wednesday 30 Apr 2014 11:08:14 Mike Cloaked wrote:
Just a comment about boot times. The overall boot performance will depend not only on optimising an individual setup, but also is dependent on the hardware as well as which boot manager is being used. So an older laptop with a hard drive, using BIOS boot and optimised will still see much longer boot time than say a new laptop running a fast i7 processor, with an ssd, using UEFI and also optimised. Certainly I have an old laptop that takes around 35 seconds to boot to the login prompt from when the boot manager takes over after POST using BIOS legacy boot, but a similarly set up and optimised new Haswell i7 laptop, with msata ssd using refind for UEFI boot takes about 7 seconds to reach the KDM login prompt. Of course for a specific system it may be possible to shave some seconds off the boot time, but it will also depend on which server daemons need to be started as well. So adding dovecot, an MTA, and maybe a DHCP server all add to the time taken to complete the boot process.
So comparisons of absolute boot times from different machines are difficult to interpret.
Since we're on the topic, does anyone have a clue how I can find out why systemd hangs for ages when I shut down or reboot? The display server is shut down, I'm placed in a TTY with a "shutting-down" message, but then it looks like it's waiting for something that never happens, and then I think I see something flash past about a watchdog timeout before it proceeds. If I could get rid of that hanging step, it would save me waiting 60 seconds or however long each time I reboot (which is infrequently enough that it's only been a mild annoyance so far).
What's the correct way to diagnose this? I don't think systemd-analyze can handle shutdown. Could this be an initrd thing?
Paul
On April 30, 2014 1:28:52 PM GMT+03:00, Paul Gideon Dann pdgiddie@gmail.com wrote:
On Wednesday 30 Apr 2014 11:08:14 Mike Cloaked wrote:
Just a comment about boot times. The overall boot performance will
depend
not only on optimising an individual setup, but also is dependent on
the
hardware as well as which boot manager is being used. So an older
laptop
with a hard drive, using BIOS boot and optimised will still see much
longer
boot time than say a new laptop running a fast i7 processor, with an
ssd,
using UEFI and also optimised. Certainly I have an old laptop that
takes
around 35 seconds to boot to the login prompt from when the boot
manager
takes over after POST using BIOS legacy boot, but a similarly set up
and
optimised new Haswell i7 laptop, with msata ssd using refind for UEFI
boot
takes about 7 seconds to reach the KDM login prompt. Of course for a specific system it may be possible to shave some seconds off the boot
time,
but it will also depend on which server daemons need to be started as
well.
So adding dovecot, an MTA, and maybe a DHCP server all add to the
time
taken to complete the boot process.
So comparisons of absolute boot times from different machines are
difficult
to interpret.
Since we're on the topic, does anyone have a clue how I can find out why systemd hangs for ages when I shut down or reboot? The display server is shut down, I'm placed in a TTY with a "shutting-down" message, but then it looks like it's waiting for something that never happens, and then I think I see something flash past about a watchdog timeout before it proceeds. If I could get rid of that hanging step, it would save me waiting 60 seconds or however long each time I reboot (which is infrequently enough that it's only been a mild annoyance so far).
What's the correct way to diagnose this? I don't think systemd-analyze can handle shutdown. Could this be an initrd thing?
Paul
I see the same thing on my box. The system shuts down, I get a message to that effect printed, yet the hardware keeps running for more than 3m. Eventually, I always cold-shutdown the computer when it reaches that state. Any ideas why this is? Gesh
On 30-04-2014 11:28, Paul Gideon Dann wrote:
Since we're on the topic, does anyone have a clue how I can find out why systemd hangs for ages when I shut down or reboot? The display server is shut down, I'm placed in a TTY with a "shutting-down" message, but then it looks like it's waiting for something that never happens, and then I think I see something flash past about a watchdog timeout before it proceeds. If I could get rid of that hanging step, it would save me waiting 60 seconds or however long each time I reboot (which is infrequently enough that it's only been a mild annoyance so far).
What's the correct way to diagnose this? I don't think systemd-analyze can handle shutdown. Could this be an initrd thing?
Check the output of journalctl and look for lines with timeout (use grep -i timeout).
I've experienced this before (90 seconds timeout) and I found out that some systemd service related to the user session was not terminating, and systemd waits for it until the timeout elapses.
This happened to me when I had the rootfs in a f2fs partition, ext4 and btrfs don't seem to trigger this problem. The workaround I used at the time was to extend the unit file by explicitly adding a shorter timeout 10~15s.
On Wednesday 30 Apr 2014 11:46:03 Mauro Santos wrote:
Check the output of journalctl and look for lines with timeout (use grep -i timeout).
I've experienced this before (90 seconds timeout) and I found out that some systemd service related to the user session was not terminating, and systemd waits for it until the timeout elapses.
This happened to me when I had the rootfs in a f2fs partition, ext4 and btrfs don't seem to trigger this problem. The workaround I used at the time was to extend the unit file by explicitly adding a shorter timeout 10~15s.
Thanks, this got me on the right track. It was indeed nothing specifically to do with shutdown, but just a custom unit I'd written for fcgiwrap. Looks like there was some bad interaction between systemd trying to kill all the process tree and the master process trying to keep the workers alive. The unit was timing out (90 seconds, as you said). Fixed by adding:
PostExec=/usr/bin/kill $MAINPID
Reboot seems to be fine now. This was one of those things I assumed would be messy until I actually took the time to look into it :p That'll teach me to test custom units a little more carefully!
Thanks, Paul
On Wed, Apr 30, 2014 at 12:28 PM, Paul Gideon Dann pdgiddie@gmail.com wrote:
Since we're on the topic, does anyone have a clue how I can find out why systemd hangs for ages when I shut down or reboot? The display server is shut down, I'm placed in a TTY with a "shutting-down" message, but then it looks like it's waiting for something that never happens, and then I think I see something flash past about a watchdog timeout before it proceeds. If I could get rid of that hanging step, it would save me waiting 60 seconds or however long each time I reboot (which is infrequently enough that it's only been a mild annoyance so far).
What's the correct way to diagnose this? I don't think systemd-analyze can handle shutdown. Could this be an initrd thing?
Although the original problem has already been solved, I'll post my trick to debug this kind of issues with systemd:
* Before doing the thing that causes the problem run as root `systemctl start debug-shell`. If the problem is in the boot sequence, then run `systemctl enable debug-shell` and then restart. * That unit will open a root shell in TTY9, so beware of the security issues if you let that enabled. This shell will be enabled from the early boot until the final shutdown. * When the system hangs, switch to TTY9, with Alt+F9 or Ctrl+Alt+F9 and get access to the debug shell. There you can run `systemctl list-jobs` or `journalctl -b` to see what is wrong with your system.
HTH - Rodrigo
On Wednesday 30 Apr 2014 23:26:48 Rodrigo Rivas wrote:
Although the original problem has already been solved, I'll post my trick to debug this kind of issues with systemd:
- Before doing the thing that causes the problem run as root
`systemctl start debug-shell`. If the problem is in the boot sequence, then run `systemctl enable debug-shell` and then restart.
- That unit will open a root shell in TTY9, so beware of the security
issues if you let that enabled. This shell will be enabled from the early boot until the final shutdown.
- When the system hangs, switch to TTY9, with Alt+F9 or Ctrl+Alt+F9
and get access to the debug shell. There you can run `systemctl list-jobs` or `journalctl -b` to see what is wrong with your system.
Very useful; thank you. I wonder if you might write that up on the Wiki page for systemd?
Paul
arch-general@lists.archlinux.org