When starting a new container openrc-0.9.8.4 first brings down the lxc preconfigured (from lxc.conf) network interface before booting. This occurs even though: (A) openrc has been explicitly informed of lxc boot in /etc/rc.conf (B) /etc/conf.d/net contains only config_eth0="null" This results in a loss of network connectivity to the guest. Reproducible: Always Steps to Reproduce: 1. Build a guest with type 'veth' (as generated by 'lxc-gentoo') 2. Configure the guest.conf with 'veth' type, and an IP address 3. 'lxc-start -n guest -f guest.conf' to start the container 4. Type 'ifconfig -a' to review the missing interface configuration 5. Shift+PageUp back to the first lines of startup to see OpenRC removing it! Actual Results: eth0 has no configuration. Startup shows it explicitly being removed by OpenRC. Expected Results: eth0 retains configuration.
A temporary workaround (Really shouldn't be necessary! This is a bug that should be fixed!) is to add 'net.eth0' to RC_NOSTOP in /etc/rc.conf Apparently some kind of runlevel change early in the init process within this type of LXC container triggers the random seed, network filesystems and the network interface to be downed.
Created attachment 325008 [details] screenshot of failure
I need rc.log from the container showing the error. If you do not know how to get rc.log, please set rc_logger="YES" in /etc/rc.conf in the container. Then reboot the container and attach /var/log/rc.log from the container to this bug report.
Getting rc.log with the failure case is going to be a major mission as the nodes are diskless. I suppose I could set up a temporary read-write NFS share. Isn't rc.log output just the same as a screenshot? What information are you looking for?
Created attachment 325076 [details] requested rc.log
lxc.cap.drop = audit_control audit_write dac_read_search fowner fsetid ipc_owner kill linux_immutable mac_admin mac_override mknod setfcap sys_admin sys_boot sys_module sys_pacct sys_ptrace sys_rawio sys_resource sys_time sys_tty_config syslog
Note that lxc-gentoo has implemented the rc_nodrop workaround.
Diego, can you give me input on this bug as well? Thanks, William
From rc.log, I do see in sysinit where eth0 is being brought down as part of sysinit. My only question is, why is an lxc guest messing around with interfaces on the host? I'm not sure if this is an lxc issue or an OpenRC issue. I also see that dmesg is being run in the container and shouldn't be, so I will fix that.
any virtual ethernet interface is by default called eth0 in lxc. You can change the name though. And you can assign one interface of a multi-interface host to the container to actually use the hardware interface (I do that often enough). You can also set the interface up and set up addressing and everything before starting the container, although when I do that I usually get rid of the net_admin capability.
William: It's definitely an OpenRC issue. It happens inside the LXC container. All that the LXC container sees are interfaces of its own ("the guest"). It can't see the external ("host") ones. A reliable workaround is to set 'rc_nostop'. I am just not sure what the trigger condition is within OpenRC that causes it to want to down the network interfaces on system start. Indeed, given that the host set them up already for the guest, I wonder how it discovers they are running at all. It appears that the trigger for OpenRC to shut them down is something to do with some capability or other being disabled which causes an error somewhere in the early boot sequence (during 'sysinit' runlevel). That's as far as I can figure without delving in to code which unfortunately I don't have time for this week. If I get time in a few days I can try to narrow down which capability causes the issue.
(In reply to comment #11) > William: It's definitely an OpenRC issue. It happens inside the LXC > container. All that the LXC container sees are interfaces of its own ("the > guest"). It can't see the external ("host") ones. > > A reliable workaround is to set 'rc_nostop'. OpenRC stops all services that are started, but not listed in the runlevel and not in rc_nostop when you enter the runlevel. > I am just not sure what the trigger condition is within OpenRC that causes > it to want to down the network interfaces on system start. Indeed, given > that the host set them up already for the guest, I wonder how it discovers > they are running at all. > It appears that the trigger for OpenRC to shut them down is something to do > with some capability or other being disabled which causes an error somewhere > in the early boot sequence (during 'sysinit' runlevel). That's as far as I > can figure without delving in to code which unfortunately I don't have time > for this week. Do the host and guest share the /run directory? If they do, you may be getting an issue if /run/openrc is the same for both. > If I get time in a few days I can try to narrow down which capability causes > the issue.
(In reply to comment #12) > > William: It's definitely an OpenRC issue. It happens inside the LXC > > container. All that the LXC container sees are interfaces of its own ("the > > guest"). It can't see the external ("host") ones. > > > > A reliable workaround is to set 'rc_nostop'. > > OpenRC stops all services that are started, but not listed in the runlevel > and not in rc_nostop when you enter the runlevel. OK, so *assuming* that the 'up' detection for net.* services is automatic from the interface status, then it seems like this is the root cause of the bug. A possible solution might then be to somehow possible to tell OpenRC that: "When initializing 'sysinit' when starting under LXC, consider all net.* services to be down until we have actually started their OpenRC init scripts". This means that OpenRC will not down the interface and lose addressing, solving the reported issue. It also allows LXC users to either set 'null' (keep the preconfigured address) or some alternate config on any of the interfaces, just as in any other environment. Once OpenRC has run a net.* script to bring an interface 'up' (even if that is a null configuration), then standard start/stop behaviour should apply. > Do the host and guest share the /run directory? If they do, you may be > getting an issue if /run/openrc is the same for both. No. The /run is mounted as tmpfs in the guest. > > If I get time in a few days I can try to narrow down which capability causes > > the issue. I'm happy to do this if you rule out the above hypothesis.
(In reply to comment #13) > (In reply to comment #12) > > OpenRC stops all services that are started, but not listed in the runlevel > > and not in rc_nostop when you enter the runlevel. > > OK, so *assuming* that the 'up' detection for net.* services is automatic > from the interface status, then it seems like this is the root cause of the > bug. We don't check the interface status to determine if the net.* services are active; they are checked just like any other service in OpenRC. The reason they are being stopped is they are not listed in the sysinit runlevel in the guest. The rc_nostop setting works because it tells OpenRC not to stop those specific services regardless of whether they are in a runlevel. Try this: 1) remove the nostop workaround. 2) add the following to /etc/conf.d/net in the guest: rc_keyword="-stop" Does that make the interfaces stay up?
The problem is associated with dropping CAP_DAC_OVERRIDE within a container. CAP_DAC_OVERRIDE is described in capabilities(7) as: 'Bypass file read, write, and execute permission checks. (DAC is an abbreviation of "discretionary access control".)' It turns out I'd removed dac_override from that container (and lxc-gentoo) due to some other issue (see end notes) and so the list posted to this thread previously reflected incorrectly the issue occurring without that being dropped. I tested just now on a brand new container, saw the issue didn't appear, re-added dac_override to lxc.cap.drop and the issue re-occurred immediately. Either the previously listed rc_nostop workaround OR adding the line that you suggested to /etc/conf.d/net resolves the problem and allows normal boot without CAP_DAC_OVERRIDE. IMHO the ideal solution would be for OpenRC to handle elegantly the error seems to be encountering when CAP_DAC_OVERRIDE is enabled and to maintain a normal boot sequence. (End notes: However, IIRC from recent testing other problems with OpenRC service management persist without CAP_DAC_OVERRIDE. IIRC it feels like it basically makes root a normal user that doesn't have rights to kill other users' processes, so any attempt to stop or restart fail weirdly. Perhaps start-stop-daemon could be hacked to enable a control channel from root that bypasses these service management issues? Anyway, a that's tangent for another bug. I will check in to it and post separately.)
(In reply to comment #15) > The problem is associated with dropping CAP_DAC_OVERRIDE within a container. > > CAP_DAC_OVERRIDE is described in capabilities(7) as: 'Bypass file read, > write, and execute permission checks. (DAC is an abbreviation of > "discretionary access control".)' > > It turns out I'd removed dac_override from that container (and lxc-gentoo) > due to some other issue (see end notes) and so the list posted to this > thread previously reflected incorrectly the issue occurring without that > being dropped. > > I tested just now on a brand new container, saw the issue didn't appear, > re-added dac_override to lxc.cap.drop and the issue re-occurred immediately. > > Either the previously listed rc_nostop workaround OR adding the line that > you suggested to /etc/conf.d/net resolves the problem and allows normal boot > without CAP_DAC_OVERRIDE. > > IMHO the ideal solution would be for OpenRC to handle elegantly the error > seems to be encountering when CAP_DAC_OVERRIDE is enabled and to maintain a > normal boot sequence. Ok, I can agree with this. I'm not quite sure how to find what this error is. Do you have any suggestions for how to track it down?
How do you normally debug OpenRC behaviour issues when rc.log contains insufficient detail?
Other options... - Read code and add many 'einfo' debugging lines. - inotify() from host, if it's filesystem-related. - strace or kernel profiling tools
(In reply to comment #9) > I also see that dmesg is being run in the container and shouldn't be, so I > will fix that. This part is fixed in commit 6770a6b.
