Hey all,
I got a very strange behavior with tmux and systemd on my server and maybe someone knows a way around it.
The way I start tmux is as follows: Setting TMUX_TMPDIR to my user's XDG_RUNTIME_DIR and starting tmux in a systemd user service [1]. I do the same for other things, such as weechat [2]. They both use the same tmux.conf [3]. Lingering is activated for my user, so the enabled user services will start with the system and stay running, even when it's disconnected... in theory (and this used to work very well, without a problem for years).
With some changes in systemd however, this seems to fail now, and I can't seem to find the reason why. Tmux just exits with "server lost" once I do a detach. At first I could only observe this behavior when I had a shared ssh session to the server (so I thought), that I then quit, or when I attached and detached the tmux session quickly. Now I'm getting this nearly every time I detach from the tmux session!
I tried several things by now, and it doesn't seem to make any difference: - using legacy cgroups [4], because of maybe being affected by this [5] - switching to single connections instead of shared connections for ssh (lowered the amount of random tmux quits!) - using hardened kernel (just to try another one) - logging a tmux session, without getting any output, that would explain, why this happens - crossing fingers (actually helps, because the bug seems completely random ;/ ) - using a system scope unit (tmux@username) [6], getting the same results - downgrading tmux to >=2.3 (no difference) - downgrading systemd to <=233-7 (seems to happen less frequently)
For some awkward reason, it seems to have worked again for a short time, when Arch first switched to systemd 233 a few days back [6], but ever since it moved to systemd-stable [7], the problem is back (and even worse than before). Me thinking that might only be due to a "less frequent occurence" though.
I'm just wondering, how to solve this. Maybe someone has a great idea to share? It's getting beyond annoying by now and I can't seem to find out, if it's tmux, systemd or some kernel problem... or all of them together.
David
P.S.: Doesn't seem to happen with screen. I don't want to adapt everything to screen now though ;)
[1] https://git.sleepmap.de/software/uenv.git/tree/user/tmux.service [2] https://git.sleepmap.de/software/uenv.git/tree/user/weechat.service [3] https://git.sleepmap.de/config/dotfiles.git/tree/.tmux.conf [4] https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.legacy... [5] https://github.com/systemd/systemd/issues/3388 [6] https://wiki.archlinux.org/index.php/Tmux#Autostart_with_systemd [7] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/syst... [8] https://github.com/systemd/systemd-stable
Not sure if related, but try setting KillUserProcesses=no in /etc/systemd/logind.conf
https://askubuntu.com/questions/802189/how-to-run-tmux-screen-with-systemd-2...
Hey Andre Vitor,
On 2017-07-07 10:53:32 (-0300), André Vitor de Lima Matos wrote:
Not sure if related, but try setting KillUserProcesses=no in /etc/systemd/logind.conf
https://askubuntu.com/questions/802189/how-to-run-tmux-screen-with-systemd-2...
That is an Arch default setting (commented variables in the systemd config files reflect compile-time defaults).
Check with: $ loginctl show-session|grep Kill
Also: The issue at hand is not a logind problem (at least not a login/logout issue), as far as I can see, as it happens on detach from tmux (but only sometimes). That one often times disconnects from the machine after a tmux detach doesn't help to figure out why it happens unfortunately :>
Thanks anyhow!
Best, David
Hey again,
On 2017-07-07 15:24:39 (+0200), David Runge wrote:
I got a very strange behavior with tmux and systemd on my server and maybe someone knows a way around it.
The way I start tmux is as follows: Setting TMUX_TMPDIR to my user's XDG_RUNTIME_DIR and starting tmux in a systemd user service [1]. I do the same for other things, such as weechat [2]. They both use the same tmux.conf [3]. Lingering is activated for my user, so the enabled user services will start with the system and stay running, even when it's disconnected... in theory (and this used to work very well, without a problem for years).
With some changes in systemd however, this seems to fail now, and I can't seem to find the reason why. Tmux just exits with "server lost" once I do a detach. At first I could only observe this behavior when I had a shared ssh session to the server (so I thought), that I then quit, or when I attached and detached the tmux session quickly. Now I'm getting this nearly every time I detach from the tmux session!
I tried several things by now, and it doesn't seem to make any difference:
- using legacy cgroups [4], because of maybe being affected by this [5]
- switching to single connections instead of shared connections for ssh (lowered the amount of random tmux quits!)
- using hardened kernel (just to try another one)
- logging a tmux session, without getting any output, that would explain, why this happens
- crossing fingers (actually helps, because the bug seems completely random ;/ )
- using a system scope unit (tmux@username) [6], getting the same results
- downgrading tmux to >=2.3 (no difference)
- downgrading systemd to <=233-7 (seems to happen less frequently)
After a week of testing: tmux - in above mentioned setup - works much more reliable with systemd 233-7.
Maybe any of the package maintainers can comment on it?
For some awkward reason, it seems to have worked again for a short time, when Arch first switched to systemd 233 a few days back [6], but ever since it moved to systemd-stable [7], the problem is back (and even worse than before). Me thinking that might only be due to a "less frequent occurence" though.
So, in conclusion, the issue is most likely related to some changes applied to the 233 branch in systemd-stable.
[1] https://git.sleepmap.de/software/uenv.git/tree/user/tmux.service [2] https://git.sleepmap.de/software/uenv.git/tree/user/weechat.service [3] https://git.sleepmap.de/config/dotfiles.git/tree/.tmux.conf [4] https://www.freedesktop.org/software/systemd/man/systemd.html#systemd.legacy... [5] https://github.com/systemd/systemd/issues/3388 [6] https://wiki.archlinux.org/index.php/Tmux#Autostart_with_systemd [7] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/syst... [8] https://github.com/systemd/systemd-stable
Best, David
Hey all,
as follow up on this (probably not so interesting thread for too many), I have tested systemd 234 successfully. The weechat instance within tmux (or tmux itself) still ends sporadically, but not the main tmux user session. So, in conclusion, whatever has been broken in systemd 233, was probably fixed again in 234.
Best, David
as follow up on this (probably not so interesting thread for too many), I have tested systemd 234 successfully. The weechat instance within tmux (or tmux itself) still ends sporadically, but not the main tmux user session. So, in conclusion, whatever has been broken in systemd 233, was probably fixed again in 234.
Hey David and all, I'm in 234.11-8 and I still have this issue. What did you end up with during your tests? I do have KillUserProcesses=no and lingering for my user (it was working for ages without issues before this summer for me too), and made other various tests among the ones you mention, but it seems like you have the right combination.
Kind regards,
Hi Ismael,
On 2017-09-25 10:16:26 (+0200), Ismael Bouya wrote:
as follow up on this (probably not so interesting thread for too many), I have tested systemd 234 successfully. The weechat instance within tmux (or tmux itself) still ends sporadically, but not the main tmux user session. So, in conclusion, whatever has been broken in systemd 233, was probably fixed again in 234.
Hey David and all, I'm in 234.11-8 and I still have this issue. What did you end up with during your tests? I do have KillUserProcesses=no and lingering for my user (it was working for ages without issues before this summer for me too), and made other various tests among the ones you mention, but it seems like you have the right combination.
Well, I'm sad to tell you, that I'm experiencing the issues again with 234.11-9, but didn't with 234.11-8 or 234.11-6.
This is getting pretty random, untracable and annoying... :( Especially, as there is no "real change"[1] between the last pkgrels...
Best, David
[1] https://git.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/syst...
Well, I'm sad to tell you, that I'm experiencing the issues again with 234.11-9, but didn't with 234.11-8 or 234.11-6.
This is getting pretty random, untracable and annoying... :( Especially, as there is no "real change"[1] between the last pkgrels...
Thanks. I searched a little longer and found that: https://github.com/tmux/tmux/issues/1001
So we can now cross fingers and wait until 2.6...
In the meantime, if you keep at least one session open all the time, the problem is much less frequent (In my case, I have a mosh session open all the time from my phone to the server, which kind of helps). Still annoying, but less frequent.
Kind regards,
On 09/25/2017 09:33 AM, Ismael Bouya wrote:
Well, I'm sad to tell you, that I'm experiencing the issues again with 234.11-9, but didn't with 234.11-8 or 234.11-6.
This is getting pretty random, untracable and annoying... :( Especially, as there is no "real change"[1] between the last pkgrels...
Thanks. I searched a little longer and found that: https://github.com/tmux/tmux/issues/1001
So we can now cross fingers and wait until 2.6...
In the meantime, if you keep at least one session open all the time, the problem is much less frequent (In my case, I have a mosh session open all the time from my phone to the server, which kind of helps). Still annoying, but less frequent.
Bugreport opened on our tracker, so this fix may get backported. ;)
https://bugs.archlinux.org/task/55746
(Mon, Sep 25, 2017 at 03:28:07PM +0000) Eli Schwartz :
Bugreport opened on our tracker, so this fix may get backported. ;)
I didn’t know this was "possible", thanks for reporting it :)
On 09/25/2017 11:36 AM, Ismael Bouya wrote:
(Mon, Sep 25, 2017 at 03:28:07PM +0000) Eli Schwartz :
Bugreport opened on our tracker, so this fix may get backported. ;)
I didn’t know this was "possible", thanks for reporting it :)
What... I didn't do anything, eworm opened that. :p I do of course watch all new bugs (it's my job!) and therefore find out about these things.
Yes, it is possible to request a backport for bugs if the fix is upstream, though the maintainer may not always agree that the "something" in question merits doing a backport especially if a release is expected soon.
arch-general@lists.archlinux.org