'showmount -e nfs-server' fails with: root@impala:/root(43)# qfile -v /usr/sbin/showmount net-fs/nfs-utils-1.3.3 (/usr/sbin/showmount) root@impala:/root(44)# showmount -e caiman clnt_create: RPC: Program not registered At the nfs-server caiman everything seems to be OK. root@caiman:/root(4)# journalctl -u nfs-server | tail Sep 26 20:38:18 caiman systemd[1]: Started NFS server and services. Oct 05 10:11:48 caiman systemd[1]: Stopping NFS server and services... Oct 05 10:11:48 caiman systemd[1]: Stopped NFS server and services. -- Reboot -- Oct 05 10:13:26 caiman systemd[1]: Starting NFS server and services... Oct 05 10:13:26 caiman systemd[1]: Started NFS server and services. Oct 25 10:47:44 caiman systemd[1]: Stopped NFS server and services. -- Reboot -- Oct 25 10:49:56 caiman systemd[1]: Starting NFS server and services... Oct 25 10:49:57 caiman systemd[1]: Started NFS server and services. root@caiman:/root(5)# systemctl status nfs-server ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled; vendor preset: enabled) Active: active (exited) since Sun 2015-10-25 10:49:57 CET; 2h 22min ago Process: 2173 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS) Process: 2162 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 2173 (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs-server.service Oct 25 10:49:56 caiman systemd[1]: Starting NFS server and services... Oct 25 10:49:57 caiman systemd[1]: Started NFS server and services. root@caiman:/root(6)# systemctl status rpcbind ● rpcbind.service - RPC Bind Loaded: loaded (/usr/lib64/systemd/system/rpcbind.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2015-10-25 10:49:56 CET; 2h 23min ago Process: 2119 ExecStart=/sbin/rpcbind (code=exited, status=0/SUCCESS) Main PID: 2127 (rpcbind) CGroup: /system.slice/rpcbind.service └─2127 /sbin/rpcbind Oct 25 10:49:55 caiman systemd[1]: Starting RPC Bind... Oct 25 10:49:56 caiman systemd[1]: Started RPC Bind. root@caiman:/root(7)# Mounting of shares with autofs fails as well: root@lynx:/root(4)# df | grep caiman root@lynx:/root(5)# systemctl status autofs ● autofs.service - Automounts filesystems on demand Loaded: loaded (/usr/lib64/systemd/system/autofs.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2015-10-25 13:05:19 CET; 13min ago Process: 15361 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS) Main PID: 15363 (automount) CGroup: /system.slice/autofs.service └─15363 /usr/sbin/automount --pid-file /run/autofs.pid Oct 25 13:05:19 lynx systemd[1]: Starting Automounts filesystems on demand... Oct 25 13:05:19 lynx systemd[1]: Started Automounts filesystems on demand. Oct 25 13:05:37 lynx automount[15363]: get_exports: lookup(hosts): exports lookup failed for caiman Oct 25 13:05:37 lynx automount[15363]: key "caiman" not found in map source(s). root@lynx:/root(6)# systemctl restart autofs root@lynx:/root(7)# systemctl status autofs ● autofs.service - Automounts filesystems on demand Loaded: loaded (/usr/lib64/systemd/system/autofs.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2015-10-25 13:18:42 CET; 2s ago Process: 15898 ExecStart=/usr/sbin/automount $OPTIONS --pid-file /run/autofs.pid (code=exited, status=0/SUCCESS) Main PID: 15900 (automount) CGroup: /system.slice/autofs.service └─15900 /usr/sbin/automount --pid-file /run/autofs.pid Oct 25 13:18:42 lynx systemd[1]: Starting Automounts filesystems on demand... Oct 25 13:18:42 lynx systemd[1]: Started Automounts filesystems on demand. root@lynx:/root(8)# ll /net/caiman/usr /bin/ls: cannot access /net/caiman/usr: No such file or directory root@lynx:/root(9)# ll /net/caiman/home_caiman /bin/ls: cannot access /net/caiman/home_caiman: No such file or directory If I try to mount these shares manually, it works: root@lynx:/root(10)# mount -t nfs caiman:/usr /mnt/test1 root@lynx:/root(11)# mount -t nfs caiman:/home_caiman /mnt/test2 root@lynx:/root(12)# df | grep caiman caiman:/usr 136110080 39296000 89877504 31% /mnt/test1 caiman:/home_caiman 264093696 191240192 59415552 77% /mnt/test2 root@lynx:/root(13)# ll /mnt/test2/ total 8 drwxrwxr-x 5 root mseifert 63 Nov 30 2014 ftp/ drwx------ 2 root root 4096 Jan 21 2011 lost+found/ drwxr-xr-x 66 rose rose 4096 Oct 25 11:55 rose/
Until now I was running on caiman my own network.service: rose@caiman:/home/rose(1)$ cat /etc/systemd/system/network.service [Unit] Description=network bridge for virtualbox After=network.target [Service] Type=oneshot RemainAfterExit=yes EnvironmentFile=/etc/conf.d/network_systemd ExecStart=/sbin/brctl addbr ${bridge_name} #ExecStart=/bin/echo "user=${user} tap_name=${tap_name} interface=${interface} bridge_name=${bridge_name}" ExecStart=/usr/bin/tunctl -b -u ${user} -t ${tap_name} ExecStart=/bin/ifconfig ${bridge_name} ${address} netmask ${netmask} up ExecStart=/bin/ifconfig ${interface} up ExecStart=/bin/ifconfig ${tap_name} up 0.0.0.0 promisc ExecStart=/sbin/brctl addif ${bridge_name} ${tap_name} ${interface} ExecStart=/sbin/sysctl net.ipv4.conf.${bridge_name}.forwarding=1 ExecStart=/sbin/iptables -t nat -A POSTROUTING -o ${interface} -j MASQUERADE ExecStart=/bin/ip route add default via ${gateway} ExecStop=/bin/ip route del default via ${gateway} ExecStop=/sbin/iptables -t nat -F ExecStop=/sbin/iptables -F ExecStop=/sbin/sysctl net.ipv4.conf.${bridge_name}.forwarding=0 ExecStop=/bin/ifconfig ${tap_name} down ExecStop=/sbin/brctl delif ${bridge_name} ${tap_name} ExecStop=/usr/bin/tunctl -d ${tap_name} ExecStop=/bin/ifconfig ${bridge_name} down ExecStop=/bin/ifconfig ${interface} down ExecStop=/sbin/brctl delbr ${bridge_name} [Install] WantedBy=multi-user.target After switching to systemd-networkd.service the issue disappeared.
The problem appears again, even with running systemd-networkd at the nfs-server: ● systemd-networkd.service - Network Service Loaded: loaded (/usr/lib64/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled) Active: inactive (dead) since Wed 2015-11-04 13:44:17 CET; 1h 21min ago Docs: man:systemd-networkd.service(8) Process: 2151 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=0/SUCCESS) Main PID: 2151 (code=exited, status=0/SUCCESS) Status: "Shutting down..." Nov 04 13:43:29 caiman systemd[1]: Starting Network Service... Nov 04 13:43:29 caiman systemd-networkd[2151]: br0: netdev ready Nov 04 13:43:29 caiman systemd-networkd[2151]: Enumeration completed Nov 04 13:43:29 caiman systemd[1]: Started Network Service. Nov 04 13:43:29 caiman systemd-networkd[2151]: eth0: Configured Nov 04 13:43:29 caiman systemd-networkd[2151]: br0: Configured Nov 04 13:43:32 caiman systemd-networkd[2151]: eth0: Gained carrier Nov 04 13:43:32 caiman systemd-networkd[2151]: br0: Gained carrier root@caiman:/root(4)# systemctl status nfs-server ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2015-11-04 13:43:30 CET; 1h 22min ago Process: 2182 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS) Process: 2179 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 2182 (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs-server.service Nov 04 13:43:30 caiman systemd[1]: Starting NFS server and services... Nov 04 13:43:30 caiman systemd[1]: Started NFS server and services. root@caiman:/root(10)# journalctl -u nfs-server | tail -- Reboot -- Oct 29 11:09:25 caiman systemd[1]: Starting NFS server and services... Oct 29 11:09:25 caiman systemd[1]: Started NFS server and services. -- Reboot -- Oct 31 09:56:50 caiman systemd[1]: Starting NFS server and services... Oct 31 09:56:51 caiman systemd[1]: Started NFS server and services. Nov 04 13:41:53 caiman systemd[1]: Stopping NFS server and services... -- Reboot -- Nov 04 13:43:30 caiman systemd[1]: Starting NFS server and services... Nov 04 13:43:30 caiman systemd[1]: Started NFS server and services. root@caiman:/root(11)# root@lynx:/root(20)# showmount -e caiman clnt_create: RPC: Program not registered
I found, that in contrast to other systems, where 'showmount -e host' works, 'rpcinfo -p' does not show mountd for caiman. root@caiman:/root(15)# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100024 1 udp 36272 status 100024 1 tcp 40108 status 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 3 tcp 2049 nfs_acl 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 3 udp 2049 nfs_acl 100021 1 udp 35707 nlockmgr 100021 3 udp 35707 nlockmgr 100021 4 udp 35707 nlockmgr 100021 1 tcp 40021 nlockmgr 100021 3 tcp 40021 nlockmgr 100021 4 tcp 40021 nlockmgr root@impala:/root(25)# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100005 1 udp 42363 mountd 100005 1 tcp 50680 mountd 100005 2 udp 49991 mountd 100005 2 tcp 58666 mountd 100005 3 udp 48567 mountd 100005 3 tcp 49865 mountd 100024 1 udp 40208 status 100024 1 tcp 46194 status 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 3 tcp 2049 nfs_acl 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 3 udp 2049 nfs_acl 100021 1 udp 46489 nlockmgr 100021 3 udp 46489 nlockmgr 100021 4 udp 46489 nlockmgr 100021 1 tcp 36341 nlockmgr 100021 3 tcp 36341 nlockmgr 100021 4 tcp 36341 nlockmgr I have the same links in /etc/systemd/system/multi-user.target.wants/ on caiman and impala: root@caiman:/root(24)# ls /etc/systemd/system/multi-user.target.wants/ autofs.service nfs-server.service sshd.service clamd.service ntpd.service systemd-networkd.service cups.path remote-fs.target vixie-cron.service gpm.service rpcbind.service root@impala:/root(30)# ls /etc/systemd/system/multi-user.target.wants autofs.service nfs-server.service sshd.service clamd.service ntpd.service systemd-networkd.service cups.path remote-fs.target vixie-cron.service gpm.service rpcbind.service 'ps -ef' shows a running rpc.mountd for both systems: root@caiman:/root(25)# ps -ef | grep mount root 2177 1 0 Nov04 ? 00:00:00 /usr/sbin/rpc.mountd root 2198 1 0 Nov04 ? 00:00:04 /usr/sbin/automount --p root@impala:/root(31)# ps -ef | grep mount root 19656 1 0 Nov04 ? 00:00:05 /usr/sbin/automount --p root 19856 1 0 Nov04 ? 00:00:00 /usr/sbin/rpc.mountd Is there any idea, what I still could check?
'systemctl status rpcbind.service' and 'systemctl status nfs-server.service' do not show any error: root@caiman:/root(33)# systemctl status rpcbind.service ● rpcbind.service - RPC Bind Loaded: loaded (/usr/lib64/systemd/system/rpcbind.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2015-11-04 13:43:30 CET; 1 day 20h ago Process: 2170 ExecStart=/sbin/rpcbind (code=exited, status=0/SUCCESS) Main PID: 2171 (rpcbind) CGroup: /system.slice/rpcbind.service └─2171 /sbin/rpcbind Nov 04 13:43:29 caiman systemd[1]: Starting RPC Bind... Nov 04 13:43:30 caiman systemd[1]: Started RPC Bind. root@caiman:/root(34)# systemctl status nfs-server.service ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib64/systemd/system/nfs-server.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2015-11-04 13:43:30 CET; 1 day 20h ago Process: 2182 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS) Process: 2179 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 2182 (code=exited, status=0/SUCCESS) CGroup: /system.slice/nfs-server.service Nov 04 13:43:30 caiman systemd[1]: Starting NFS server and services... Nov 04 13:43:30 caiman systemd[1]: Started NFS server and services. root@caiman:/root(35) Also 'journalctl -b | grep -U3 rpc' did not show any significant problem: root@caiman:/root(43)# journalctl -b | grep -U3 rpc Nov 04 13:43:28 caiman systemd-tmpfiles[2129]: "/home" already exists and is not a directory. Nov 04 13:43:28 caiman systemd[1]: Started Create Volatile Files and Directories. Nov 04 13:43:28 caiman systemd[1]: Starting Update UTMP about System Boot/Shutdown... Nov 04 13:43:28 caiman systemd[1]: var-lib-nfs-rpc_pipefs.mount: Directory /var/lib/nfs/rpc_pipefs to mount over is not empty, mounting anyway. Nov 04 13:43:28 caiman systemd[1]: Mounting RPC Pipe File System... Nov 04 13:43:28 caiman systemd[1]: Mounted RPC Pipe File System. Nov 04 13:43:28 caiman systemd[1]: Starting NFSv4 ID-name mapping service... -- Nov 04 13:43:29 caiman systemd-networkd[2151]: br0: Configured Nov 04 13:43:29 caiman systemd[1]: Starting Generic ClamAV scanner daemon... Nov 04 13:43:29 caiman systemd[1]: Starting OpenSSH server daemon... Nov 04 13:43:29 caiman rpc.mountd[2167]: mountd: No V2 or V3 listeners created! Nov 04 13:43:29 caiman systemd[1]: Starting RPC Bind... Nov 04 13:43:29 caiman systemd[1]: Reached target Network is Online. Nov 04 13:43:30 caiman systemd[1]: Starting Automounts filesystems on demand... -- Nov 04 13:43:30 caiman systemd[1]: Reached target RPC Port Mapper. Nov 04 13:43:30 caiman systemd[1]: Starting NFS status monitor for NFSv2/3 locking.... Nov 04 13:43:30 caiman systemd[1]: Started OpenSSH server daemon. Nov 04 13:43:30 caiman rpc.mountd[2177]: Version 1.3.3 starting Nov 04 13:43:30 caiman systemd[1]: Started NFS Mount Daemon. Nov 04 13:43:30 caiman rpc.statd[2178]: Version 1.3.3 starting Nov 04 13:43:30 caiman rpc.statd[2178]: Flags: TI-RPC Nov 04 13:43:30 caiman systemd[1]: Started NFS status monitor for NFSv2/3 locking.. Nov 04 13:43:30 caiman systemd[1]: Starting NFS server and services... Nov 04 13:43:30 caiman sshd[2176]: Server listening on 0.0.0.0 port 22. But now the issue disappeard after restarting rpcbind.service: root@caiman:/root(5)# systemctl restart rpcbind.service root@caiman:/root(6)# rpcinfo -p program vers proto port service 100000 4 tcp 111 portmapper 100000 3 tcp 111 portmapper 100000 2 tcp 111 portmapper 100000 4 udp 111 portmapper 100000 3 udp 111 portmapper 100000 2 udp 111 portmapper 100005 1 udp 57159 mountd 100005 1 tcp 39604 mountd 100005 2 udp 56301 mountd 100005 2 tcp 60691 mountd 100005 3 udp 56926 mountd 100005 3 tcp 36935 mountd 100003 3 tcp 2049 nfs 100003 4 tcp 2049 nfs 100227 3 tcp 2049 nfs_acl 100003 3 udp 2049 nfs 100003 4 udp 2049 nfs 100227 3 udp 2049 nfs_acl 100021 1 udp 36986 nlockmgr 100021 3 udp 36986 nlockmgr 100021 4 udp 36986 nlockmgr 100021 1 tcp 44482 nlockmgr 100021 3 tcp 44482 nlockmgr 100021 4 tcp 44482 nlockmgr
The issue happens again. After reboot 'showmount -e server' fails with "clnt_create: RPC: Program not registered" 'systemctl status rpcbind.service' and 'systemctl status nfs-server.service' do not show any error on the server. After 'systemctl restart rpcbind.service' the issue disappeares.
The same error happens with an other server. After reboot 'rpcinfo -p' does not show mountd on the server. 'systemctl restart rcpbind' restarts mountd on the server and showmount and autofs works again on the client side.
Could you try to add rpcbind.target or rpcbind.service in the "After" list of your nfs-mountd.service unit ? just : # systemctl edit nfs-mountd.service type : After=rpcbind.target save and reboot.
