[arch-general] NFS server broke after reboot - *urgent* need help
I have a problem where nfs failed. 2 units fail to start rpc-statd.service nfs-server.service. ------------------------------------- rpc-statd: systemctl status rpc-statd rpc.statd[736]: Version 1.3.2 starting rpc.statd[736]: Flags: TI-RPC rpc.statd[736]: Running as root. chown /var/lib/nfs to choose different user rpc.statd[736]: failed to create RPC listeners, exiting systemd[1]: rpc-statd.service: control process exited, code=exited status=1 systemd[1]: Failed to start NFS status monitor for NFSv2/3 locking.. systemd[1]: Unit rpc-statd.service entered failed state. systemd[1]: rpc-statd.service failed. ------------------------------------- I am unable to start this by hand either - continues to fail same way. I had seen this once a month or so back - but was able to start it by hand after machine was up. ------------------------------------- nfs-server: systemctl-status nfs-server ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2015-01-31 16:05:32 EST; 5min ago Process: 743 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE) Process: 741 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 743 (code=exited, status=1/FAILURE) rpc.nfsd[743]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused) rpc.nfsd[743]: rpc.nfsd: unable to set any sockets for nfsd systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE systemd[1]: Failed to start NFS server and services. systemd[1]: Unit nfs-server.service entered failed state. systemd[1]: nfs-server.service failed. ------------------------------------- Machine is fully updated from testing repo. I did try kernel 3.19.rc6 but it does not help. nfs-utils 1.3.2-1 linux 3.18.5-1 rpcbind 0.2.2-1 Really appreciate some help - this is my main nfs server and I can no longer get it working ... Thanks for any advice. gene
On 01/31/2015 10:14 PM, Genes Lists wrote:
I have a problem where nfs failed. 2 units fail to start
rpc-statd.service nfs-server.service.
------------------------------------- rpc-statd: systemctl status rpc-statd
rpc.statd[736]: Version 1.3.2 starting rpc.statd[736]: Flags: TI-RPC rpc.statd[736]: Running as root. chown /var/lib/nfs to choose different user rpc.statd[736]: failed to create RPC listeners, exiting systemd[1]: rpc-statd.service: control process exited, code=exited status=1 systemd[1]: Failed to start NFS status monitor for NFSv2/3 locking.. systemd[1]: Unit rpc-statd.service entered failed state. systemd[1]: rpc-statd.service failed.
------------------------------------- I am unable to start this by hand either - continues to fail same way. I had seen this once a month or so back - but was able to start it by hand after machine was up.
------------------------------------- nfs-server:
systemctl-status nfs-server
● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2015-01-31 16:05:32 EST; 5min ago Process: 743 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=1/FAILURE) Process: 741 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 743 (code=exited, status=1/FAILURE)
rpc.nfsd[743]: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused) rpc.nfsd[743]: rpc.nfsd: unable to set any sockets for nfsd systemd[1]: nfs-server.service: main process exited, code=exited, status=1/FAILURE systemd[1]: Failed to start NFS server and services. systemd[1]: Unit nfs-server.service entered failed state. systemd[1]: nfs-server.service failed.
-------------------------------------
Machine is fully updated from testing repo. I did try kernel 3.19.rc6 but it does not help.
nfs-utils 1.3.2-1 linux 3.18.5-1 rpcbind 0.2.2-1
Really appreciate some help - this is my main nfs server and I can no longer get it working ...
I'm unlikely to be able to help, but it might be a good idea to try journalctl -u nfs-server journalctl -u rpc.statd etc. just to see if there's some output you've missed.
On 01/31/2015 04:28 PM, Bardur Arantsson wrote:
On 01/31/2015 10:14 PM, Genes Lists wrote:
This was original thread: https://lists.archlinux.org/pipermail/arch-general/2014-June/036617.html So I tried the same commands - specifically: systemctl restart proc-fs-nfsd.mount systemctl restart rpcbind systemctl restart nfs-mountd.service systemctl restart rpc-statd.service systemctl restart nfs-idmapd.service systemctl restart rpc-svcgssd.service systemctl restart rpc-statd-notify.service systemctl restart nfs-mountd systemctl restart rpc-gssd.service rpc-svcgssd.service rpc-statd still does not start but now .. it failes with rpc-statd.service start operation timed out. Terminating. After 5 mins i tried nfs-server which now starts ... rpc-statd is still not running. And the server is once again serving NFS. This is a real mess ... eeeek!! At last NFS server is up now ... but yikes I need help so this starts by itself again. thanks!
On 31/01/15 21:32, Genes Lists wrote:
On 01/31/2015 04:28 PM, Bardur Arantsson wrote:
On 01/31/2015 10:14 PM, Genes Lists wrote:
This was original thread: https://lists.archlinux.org/pipermail/arch-general/2014-June/036617.html
So I tried the same commands - specifically:
systemctl restart proc-fs-nfsd.mount
systemctl restart rpcbind systemctl restart nfs-mountd.service systemctl restart rpc-statd.service systemctl restart nfs-idmapd.service systemctl restart rpc-svcgssd.service systemctl restart rpc-statd-notify.service systemctl restart nfs-mountd systemctl restart rpc-gssd.service rpc-svcgssd.service
rpc-statd still does not start but now .. it failes with rpc-statd.service start operation timed out. Terminating.
After 5 mins i tried nfs-server which now starts ... rpc-statd is still not running.
And the server is once again serving NFS.
This is a real mess ... eeeek!!
At last NFS server is up now ... but yikes I need help so this starts by itself again.
thanks!
I've been having a few rather strange problems recently with services requiring an upped network interface starting before the interface is up, this is using systemd-networkd-wait-online.service. In any case, it is possible you are having a similar problem, are you sure the interfaces nfs-server is set to listen on are up? -- Tomasz Kramkowski E-Mail: tk@the-tk.com PGP: 6FCE87503AAF42AB3BF4 94FE40B037BA0A5B8680
On Sat, Jan 31, 2015 at 9:50 PM, Tomasz Kramkowski <tk@the-tk.com> wrote:
I've been having a few rather strange problems recently with services requiring an upped network interface starting before the interface is up, this is using systemd-networkd-wait-online.service.
In any case, it is possible you are having a similar problem, are you sure the interfaces nfs-server is set to listen on are up?
This is now reported at https://bugs.archlinux.org/task/43647 so hopefully some enlightenment will appear there to help too. -- mike c
On 31/01/15 22:02, Mike Cloaked wrote:
On Sat, Jan 31, 2015 at 9:50 PM, Tomasz Kramkowski <tk@the-tk.com> wrote:
I've been having a few rather strange problems recently with services requiring an upped network interface starting before the interface is up, this is using systemd-networkd-wait-online.service.
In any case, it is possible you are having a similar problem, are you sure the interfaces nfs-server is set to listen on are up?
This is now reported at https://bugs.archlinux.org/task/43647 so hopefully some enlightenment will appear there to help too.
Could it be assumed then that this problem happened after an update and is happening to multiple people? I'm getting worried about updating now, very few things rely on my nfs server but it's still relatively important it stays running in case I do need to use it. I guess I'll just wait to see what comes of this bug report. -- Tomasz Kramkowski E-Mail: tk@the-tk.com PGP: 6FCE87503AAF42AB3BF4 94FE40B037BA0A5B8680
On 01/31/2015 04:50 PM, Tomasz Kramkowski wrote:
In any case, it is possible you are having a similar problem, are you sure the interfaces nfs-server is set to listen on are up?
Yes server is up - i administer via ssh remotely :)
On 01/31/2015 10:32 PM, Genes Lists wrote:
On 01/31/2015 04:28 PM, Bardur Arantsson wrote:
On 01/31/2015 10:14 PM, Genes Lists wrote:
This was original thread: https://lists.archlinux.org/pipermail/arch-general/2014-June/036617.html
First, in case it's relevant: DON'T PANIC! That's the worst you can possibly do in a situation like this. Secondly, I mentioned "journalctl -u", not "systemctl". There may be relevant stuff in there that isn't mentioned by "systemctl status" (due to cutoff). Good luck,
On 01/31/2015 05:09 PM, Bardur Arantsson wrote:
First, in case it's relevant: DON'T PANIC!
Ha ha good advice ... :)
Secondly, I mentioned "journalctl -u", not "systemctl". There may be relevant stuff in there that isn't mentioned by "systemctl status" (due to cutoff).
yes I did look thru the logs - and posted one relevant one (nfsd) in the bug report. The rest are already covered by the systemctl status one and are not additive. Thanks for suggestion.
Am 31.01.2015 um 23:19 schrieb Genes Lists:
On 01/31/2015 05:09 PM, Bardur Arantsson wrote:
First, in case it's relevant: DON'T PANIC!
Ha ha good advice ... :)
Secondly, I mentioned "journalctl -u", not "systemctl". There may be relevant stuff in there that isn't mentioned by "systemctl status" (due to cutoff).
yes I did look thru the logs - and posted one relevant one (nfsd) in the bug report. The rest are already covered by the systemctl status one and are not additive.
Thanks for suggestion. Could it be, that all you suffer from issues are using NFS3 and not NFS4.x My simple setup just works fine with NFS4 without any hiccups.
greetings tpowa -- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa@archlinux.org
Only use V4 here. I believe the references to older versions come from the rpc-statd systemd unit file. I'm away from computers at the moment but will double check. Also I believe it's nfs-server unit file that asks for rpc-statd. It's been working fine since system was installed in December 2013 other than the occasional quirk such as mentioned in June 2014 bug report. I forgot to mention that the new nfs-utils came in after the first reboot to pick up 3.18.5. However installing that did not by itself fix anything either. Thank you for your help. On February 1, 2015 4:13:40 AM EST, Tobias Powalowski <tobias.powalowski@googlemail.com> wrote:
Am 31.01.2015 um 23:19 schrieb Genes Lists:
On 01/31/2015 05:09 PM, Bardur Arantsson wrote:
First, in case it's relevant: DON'T PANIC!
Ha ha good advice ... :)
Secondly, I mentioned "journalctl -u", not "systemctl". There may be relevant stuff in there that isn't mentioned by "systemctl status"
(due
to cutoff).
yes I did look thru the logs - and posted one relevant one (nfsd) in the bug report. The rest are already covered by the systemctl status one and are not additive.
Thanks for suggestion. Could it be, that all you suffer from issues are using NFS3 and not NFS4.x My simple setup just works fine with NFS4 without any hiccups.
greetings tpowa
-- Tobias Powalowski Archlinux Developer & Package Maintainer (tpowa) http://www.archlinux.org tpowa@archlinux.org
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Sorry for top post. Shouldn't reply from phone. Gene -- Sent from my Android device. Please excuse my brevity.
On Sun, Feb 1, 2015 at 2:25 PM, Gene Cohler <lists@sapience.com> wrote:
Only use V4 here. I believe the references to older versions come from the rpc-statd systemd unit file. I'm away from computers at the moment but will double check. Also I believe it's nfs-server unit file that asks for rpc-statd.
It's been working fine since system was installed in December 2013 other than the occasional quirk such as mentioned in June 2014 bug report.
I forgot to mention that the new nfs-utils came in after the first reboot to pick up 3.18.5. However installing that did not by itself fix anything either.
Thank you for your help.
Is this Fedora bug related? https://bugzilla.redhat.com/show_bug.cgi?id=1183992 -- mike c
participants (6)
-
Bardur Arantsson
-
Gene Cohler
-
Genes Lists
-
Mike Cloaked
-
Tobias Powalowski
-
Tomasz Kramkowski