After installing systemd-231 the network is not running. 'systemctl status systemd-networkd' says: ● systemd-networkd.service - Network Service Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled) Active: failed (Result: start-limit-hit) since Sun 2016-07-31 02:41:19 CEST; 49min ago Docs: man:systemd-networkd.service(8) Process: 4069 ExecStart=/usr/lib/systemd/systemd-networkd (code=exited, status=226/NAMESPACE) Main PID: 4069 (code=exited, status=226/NAMESPACE) Jul 31 02:41:19 condor systemd[1]: Failed to start Network Service. Jul 31 02:41:19 condor systemd[1]: systemd-networkd.service: Unit entered failed state. Jul 31 02:41:19 condor systemd[1]: systemd-networkd.service: Failed with result 'exit-code'. Jul 31 02:41:19 condor systemd[1]: systemd-networkd.service: Service has no hold-off time, scheduling restart. Jul 31 02:41:19 condor systemd[1]: Stopped Network Service. Jul 31 02:41:19 condor systemd[1]: systemd-networkd.service: Start request repeated too quickly. Jul 31 02:41:19 condor systemd[1]: Failed to start Network Service. Jul 31 02:41:19 condor systemd[1]: systemd-networkd.service: Unit entered failed state. Jul 31 02:41:19 condor systemd[1]: systemd-networkd.service: Failed with result 'start-limit-hit'. NAMESPACES are configured for this kernel: rose@condor:/home/rose(4)$ zgrep NAMESPACE /proc/config.gz CONFIG_NAMESPACES=y rose@condor:/home/rose(5)$ uname -a Linux condor 4.7.0-gentoo #1 SMP Fri Jul 29 18:44:20 CEST 2016 x86_64 Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz GenuineIntel GNU/Linux After downgrading to systemd-230-r2 the network works again. rose@condor:/home/rose(6)$ emerge -pvuND systemd These are the packages that would be merged, in order: Calculating dependencies... done! [ebuild R ] virtual/perl-Test-Simple-1.1.14-r2::gentoo 0 KiB [ebuild U ] sys-apps/systemd-231:0/2::gentoo [230-r2:0/2::gentoo] USE="acl curl doc gcrypt http kmod lz4 lzma pam policykit seccomp ssl -apparmor -audit -cryptsetup -elfutils -gnuefi -idn -importd -nat -qrcode (-selinux) -sysv-utils {-test} -vanilla -xkb (-kdbus%*)" ABI_X86="32 (64) (-x32)" 4279 KiB Total: 2 packages (1 upgrade, 1 reinstall), Size of downloads: 4279 KiB WARNING: One or more updates/rebuilds have been skipped due to a dependency conflict: virtual/perl-Test-Simple:0 (virtual/perl-Test-Simple-1.1.14_p522:0/0::gentoo, ebuild scheduled for merge) conflicts with ~virtual/perl-Test-Simple-1.1.14 required by (dev-perl/Test-Tester-0.114.0:0/0::gentoo, installed) ^ ^^^^^^
Two new settings were added to /usr/lib/systemd/system/systemd-networkd.service: MemoryDenyWriteExecute=yes SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount @obsolete @raw-io Can you try removing these settings one at a time to see if either makes a difference?
(In reply to Mike Gilbert from comment #1) > Two new settings were added to > /usr/lib/systemd/system/systemd-networkd.service: > > MemoryDenyWriteExecute=yes > SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module @mount > @obsolete @raw-io > > Can you try removing these settings one at a time to see if either makes a > difference? Removing of these lines does not change anything. I have seen two further points, which are maybe connected with this issue: - 'ethtool eth0' says "Link detected: no" - 'journalctl -xb' complains about Too many levels of symbolic links. /etc/systemd/network is a link to /etc/systemd/network.ANYWHRE I have serveral /etc/systemd/network.*. If I change the network, I change this link. I am using this for many month without any problems. Furthermore /etc/systemd/network/50-static_eth0.network is a link as well, it points to /etc/systemd/network/50-static_eth0.network_TECNETAIR. This works at least with <systemd-231.
Even if I replace the last link by its target, it does not improve anything.
Created attachment 442036 [details] Output of 'journalctl -xb' with systemd-231
I mask >=systemd-231 at least on this system.
I cannot reproduce this error. It sounds like you have a rather unorthodox configuration, and I cannot guess at what other oddities you may have on your system. Please feel free to reopen this if you can identify the cause of the problem.
OK, I remove the links. /etc/systemd/network is an ordinary directory containing only the file 50-static_eth0.network: root@condor:/etc/systemd/network(18)# ll -d /etc/systemd/network drwxr-xr-x 2 root root 4096 Jul 31 07:22 /etc/systemd/network/ root@condor:/etc/systemd/network(19)# ll /etc/systemd/network/ total 4 -rw-r--r-- 1 root root 130 Jul 31 05:43 50-static_eth0.network root@condor:/etc/systemd/network(20)# cat /etc/systemd/network/50-static_eth0.network [Match] Name=eth0 [Network] DNS=192.168.3.1 DNS=192.168.1.1 DNS=141.89.64.1 Address=192.168.3.76/24 [Route] Gateway=192.168.3.1 /etc/resolv.conf is still a link managed by systemd-resolved. root@condor:/etc/systemd/network(21)# ll /etc/resolv.conf lrwxrwxrwx 1 root root 32 Jan 9 2002 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf root@condor:/etc/systemd/network(22)# cat /etc/resolv.conf # This file is managed by systemd-resolved(8). Do not edit. # # Third party programs must not access this file directly, but # only through the symlink at /etc/resolv.conf. To manage # resolv.conf(5) in a different way, replace the symlink by a # static file or a different symlink. nameserver 192.168.3.1 nameserver 192.168.1.1 nameserver 141.89.64.1 If I install systemd-231, the network service fails with status 226/NAMESPACE. The relevant error information from journalctl seems to be: ... Jul 31 07:56:36 condor systemd[1]: Starting Network Service... -- Subject: Unit systemd-networkd.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-networkd.service has begun starting up. Jul 31 07:56:36 condor systemd[1]: Network Manager is not active. Jul 31 07:56:36 condor systemd[1]: Dependency failed for Network Manager Wait Online. -- Subject: Unit NetworkManager-wait-online.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit NetworkManager-wait-online.service has failed. -- -- The result is dependency. Jul 31 07:56:36 condor systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start failed with result 'dependency'. Jul 31 07:56:36 condor systemd[2157]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Too many levels of symbolic links -- Subject: Process /usr/lib/systemd/systemd-networkd could not be executed Is it really necessary for systemd-231 to run the NetworkManager? If it is anyhow possible to avoid to run the NetworkManager, I would like live without the NetworkManager.
(In reply to Juergen Rose from comment #7) > Jul 31 07:56:36 condor systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start failed with result 'dependency'. This means NetworkManager-wait-online.service is enabled, but NetworkManager.service is disabled. To resolve the error, simply disable NetworkManager-wait-online.service. > Jul 31 07:56:36 condor systemd[2157]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Too many levels of symbolic links I see this is still happening for you. systemd-networkd.service sets ProtectSystem=full and ProtectHome=yes. This means it will try to do the following in a new namespace: Remount /home, /run/user, /root as inaccessible. Remount /usr, /boot, /etc as read-only. Have you created any symlinks which might affect the above paths?
Another quick test would be to play with those ProtectHome and ProtectSystem settings to see if they make any difference. Also, make sure to run systemctl daemon-reload after editing any unit file.
(In reply to Mike Gilbert from comment #8) > (In reply to Juergen Rose from comment #7) > > > Jul 31 07:56:36 condor systemd[1]: NetworkManager-wait-online.service: Job NetworkManager-wait-online.service/start failed with result 'dependency'. > > This means NetworkManager-wait-online.service is enabled, but > NetworkManager.service is disabled. To resolve the error, simply disable > NetworkManager-wait-online.service. > > > Jul 31 07:56:36 condor systemd[2157]: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Too many levels of symbolic links > > I see this is still happening for you. > > systemd-networkd.service sets ProtectSystem=full and ProtectHome=yes. This > means it will try to do the following in a new namespace: > > Remount /home, /run/user, /root as inaccessible. > Remount /usr, /boot, /etc as read-only. > > Have you created any symlinks which might affect the above paths? Thank you Mike for your hints. First of all I want to say, that I see this issues all systems, which I rebooted after the installation of systemd-231. The first advice to disable NetworkManager-wait-online.service helps a little bit. 'ethtool eth0' detects a link. And I get per DHCP an IP address for eth0. This is not the configuration which I get with systemd-230-r2, where I get a bridged configuration with a static IP address for br0. But at least the network is not completely dead. With respect to /home your assumption are again right. /home is a link to /home_LOCALHOST, where LOCALHOST is the current hostname. So I have usually eg. on 'condor' something like /home -> /home_condor /home_lynx -> /net/lynx/home_lynx /home_cheetah -> /net/cheetah/home_cheetah update respectively locate works then for the whole group. 'journalctl -xb' still shows 32 times: systemd-networkd.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-networkd: Too many levels of symbolic links systemd-networkd.service: Main process exited, code=exited, status=226/NAMESPACE Is there a configuration file where ProtectSystem and ProtectHome can be set and where maybe /home can be replaced by /home_LOCALHOST for remounting inaccessible or have I to edit /usr/lib/systemd/system/systemd-networkd.service? If I set /usr/lib/systemd/system/systemd-networkd.service ProtectHome=no my bridged network configuration works again. But I still get errors assigning VLANs to br0 and eth0 (I did not have the intention to assign VLANs to these interfaces) Jul 31 19:27:37 lynx systemd-networkd[2261]: br0: IPv6 enabled for interface: Success Jul 31 19:27:37 lynx systemd-networkd[2261]: br0: Could not append VLANs: Operation not permitted Jul 31 19:27:37 lynx systemd-networkd[2261]: br0: Failed to assign VLANs to bridge port: Operation not permitted Jul 31 19:27:37 lynx systemd-networkd[2261]: br0: Could not set bridge vlan: Operation not permitted Jul 31 19:27:37 lynx kernel: IPv6: ADDRCONF(NETDEV_UP): br0: link is not ready Jul 31 19:27:37 lynx kernel: br0: port 1(eth0) entered blocking state and I get errors connected with systemd-hostnamed.service: Jul 31 19:27:37 lynx dbus[2256]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' Jul 31 19:27:37 lynx bluetoothd[2275]: Bluetooth management interface 1.12 initialized Jul 31 19:27:37 lynx bluetoothd[2275]: Failed to obtain handles for "Service Changed" characteristic Jul 31 19:27:37 lynx systemd[1]: Starting Hostname Service... -- Subject: Unit systemd-hostnamed.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-hostnamed.service has begun starting up. Jul 31 19:27:37 lynx systemd-networkd[2261]: eth0: Could not append VLANs: Operation not permitted Jul 31 19:27:37 lynx systemd-networkd[2261]: eth0: Failed to assign VLANs to bridge port: Operation not permitted Jul 31 19:27:37 lynx systemd-networkd[2261]: eth0: Could not set bridge vlan: Operation not permitted Jul 31 19:27:37 lynx systemd[2288]: systemd-hostnamed.service: Failed at step NAMESPACE spawning /usr/lib/systemd/systemd-hostnamed: Too many levels of sy -- Subject: Process /usr/lib/systemd/systemd-hostnamed could not be executed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- The process /usr/lib/systemd/systemd-hostnamed could not be executed and failed. -- -- The error number returned by this process is 40. Jul 31 19:27:37 lynx systemd[1]: systemd-hostnamed.service: Main process exited, code=exited, status=226/NAMESPACE Jul 31 19:27:37 lynx systemd[1]: Failed to start Hostname Service. -- Subject: Unit systemd-hostnamed.service has failed
(In reply to Juergen Rose from comment #10) > Is there a configuration file where ProtectSystem and ProtectHome can be set > and where maybe /home can be replaced by /home_LOCALHOST for remounting > inaccessible or have I to edit > /usr/lib/systemd/system/systemd-networkd.service? There is no global config file, but you can disable it for each service using "drop-in" files -- see systemd.unit(5). This will prevent your changes from being overwritten on upgrades/reinstalls. For example: mkdir -p /etc/systemd/system/systemd-networkd.service.d cat >> /etc/systemd/system/systemd-networkd.service.d/noprotect.conf <<EOF [Service] ProtectHome=no ProtectSystem=no EOF > Jul 31 19:27:37 lynx systemd-networkd[2261]: br0: Could not append VLANs: > Operation not permitted This might indicate an error in CapabilityBoundingSet or SystemCallFilter. Again, you can try commenting-out these settings in the unit file to test that. > Jul 31 19:27:37 lynx systemd[2288]: systemd-hostnamed.service: Failed at > step NAMESPACE spawning /usr/lib/systemd/systemd-hostnamed: Too many levels > of symlinks systemd-hostnamed.service also has ProtectHome enabled, so you are running into the same error with a different service. Given that this all worked before systemd-231, it can't hurt to open a bug report upstream. Just make sure to fully explain your unique system setup, and the troubleshooting steps you have taken thus far.
I believe this commit may be the cause of your symlink problem: https://github.com/systemd/systemd/commit/c4b41707462a74eb7008e8d12a0b4d0a0c09bff4#diff-f315b72505d5f0a92ec2f4d068d4f916R301 It makes it an error to attempt to make a symlink inaccessible. If this is attempted, ELOOP is returned. The English description of ELOOP is "Too many levels of symbolic links". That essentially means that any service with ProtectHome enabled will break if /home is a symlink.
Looks like someone else reported this upstream.
(In reply to Mike Gilbert from comment #13) > Looks like someone else reported this upstream. Can confirm this problem, also with /home a symlink. Downgrading for now. Are we to understand that the config in comment #11 will continue to be required for systemd-231 in these circumstances or that this behaviour will be reverted upstream?
(In reply to Adrian.Bassett from comment #14) I have no idea what action (if any) upstream will eventually take.
(In reply to Mike Gilbert from comment #15) > (In reply to Adrian.Bassett from comment #14) > > I have no idea what action (if any) upstream will eventually take. Granted. In order to upgrade to systemd-231 I have made /home a bind mount (rather than a symlink) and this appears to work OK.
(In reply to Adrian.Bassett from comment #16) > (In reply to Mike Gilbert from comment #15) > > (In reply to Adrian.Bassett from comment #14) > > > > I have no idea what action (if any) upstream will eventually take. > > Granted. > > In order to upgrade to systemd-231 I have made /home a bind mount (rather > than a symlink) and this appears to work OK. 'bind mount' of /home is a good idea, I tried now also. I have now in /etc/fstab: root@lynx:/root(1)# grep home /etc/fstab /dev/mapper/vg-home /home_lynx ext4 noatime 1 2 /home_lynx /home ext4 bind 1 2 /dev/mapper/vg-ftp /home_lynx/ftp xfs noauto,noatime 1 2 and I set again in /usr/lib/systemd/system/systemd-networkd.service ProtectHome=yes The most systemd-networkd related errors disappeared. But there is at least one warning regarding to fstab and /home_lynx: root@lynx:/root(6)# journalctl -xb | grep -U2 systemd-fstab Aug 05 13:54:22 lynx systemd[1]: Detected architecture x86-64. Aug 05 13:54:22 lynx systemd[1]: Set hostname to <lynx>. Aug 05 13:54:22 lynx systemd-fstab-generator[1765]: Checking was requested for "/home_lynx", but it is not a device. Aug 05 13:54:22 lynx systemd[1]: Reached target Host and Network Name Lookups. Aug 05 13:54:22 lynx systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point. @Adrian.Bassett, how did you add the 'bind mount' to /etc/fstab? @Mike Gilbert, I still have the VLAN issues: root@lynx:/root(7)# journalctl -xb | grep -U2 VLAN -- The start-up result is done. Aug 05 13:54:42 lynx systemd-networkd[2306]: br0: IPv6 enabled for interface: Success Aug 05 13:54:42 lynx systemd-networkd[2306]: br0: Could not append VLANs: Operation not permitted Aug 05 13:54:42 lynx systemd-networkd[2306]: br0: Failed to assign VLANs to bridge port: Operation not permitted Aug 05 13:54:42 lynx systemd-networkd[2306]: br0: Could not set bridge vlan: Operation not permitted Aug 05 13:54:42 lynx systemd-networkd[2306]: eth0: IPv6 disabled for interface: Success Aug 05 13:54:42 lynx systemd-networkd[2306]: eth0: Could not append VLANs: Operation not permitted Aug 05 13:54:42 lynx systemd-networkd[2306]: eth0: Failed to assign VLANs to bridge port: Operation not permitted Aug 05 13:54:42 lynx systemd-networkd[2306]: eth0: Could not set bridge vlan: Operation not permitted Aug 05 13:54:43 lynx NetworkManager[2302]: <info> [1470398083.0292] NetworkManager (version 1.2.4) is starting... Do you have any idea how to handle?
> @Adrian.Bassett, how did you add the 'bind mount' to /etc/fstab? Just one line with four elements: /mnt/<rest-of-path>/home /home none bind Obviously I had to make the /home directory mount-point for the mount to succeed.
(In reply to Adrian.Bassett from comment #18) > > @Adrian.Bassett, how did you add the 'bind mount' to /etc/fstab? > > Just one line with four elements: > > /mnt/<rest-of-path>/home /home none bind > > Obviously I had to make the /home directory mount-point for the mount to > succeed. Compared with my line you have none instead of ext4. I will try with none too.
> Compared with my line you have none instead of ext4. I will try with none > too. From mount(8): Bind mounts Since Linux 2.4.0 it is possible to remount part of the file hierarchy somewhere else. The call is: mount --bind olddir newdir or by using this fstab entry: /olddir /newdir none bind
(In reply to Juergen Rose from comment #19) You should also set the last two fields (fs_freq and fs_passno) to 0 for bind mounts. It makes no sense to run dump or fsck on a bind mounted directory.
(In reply to Juergen Rose from comment #17) No, I have no idea on your VLAN issue. Please seek support upstream.
(In reply to Mike Gilbert from comment #22) > (In reply to Juergen Rose from comment #17) > > No, I have no idea on your VLAN issue. Please seek support upstream. It looks like this has been reported/fixed upstream. https://github.com/systemd/systemd/issues/3876
Should be fixed int systemd-232.
