Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 750410 - app-emulation/lxd LXD startup failing on non-systemd environments
Summary: app-emulation/lxd LXD startup failing on non-systemd environments
Status: CONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Joonas Niilola
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-10-20 11:09 UTC by Nico Baggus
Modified: 2021-05-02 20:12 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nico Baggus 2020-10-20 11:09:08 UTC
Suggest the following addition to the lxd startup stript.

--- /etc/init.d/lxd     2020-10-18 22:01:00.857748741 +0200
+++ /etc/init.d/._cfg0000_lxd   2020-10-20 12:58:31.757567010 +0200
@@ -17,6 +17,7 @@
        ebegin "Starting lxd service"
 
        modprobe -f loop > /dev/null 2>&1
+       [ -d /sys/fs/cgroup/systemd ] || ( mkdir -p /sys/fs/cgroup/systemd ; mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd )
 
        # fix permissions on /var/lib/lxd and make sure it exists
        install -d /var/lib/lxd --group lxd --owner root --mode 0775


This addendum follows the gentoo wiki page on lxd.

Reproducible: Always

Steps to Reproduce:
1.install lxd on a system without systemd.
2.
3.
Actual Results:  
non starting containers

Expected Results:  
Starting containers
Comment 1 Joonas Niilola gentoo-dev 2020-10-20 18:51:23 UTC
I'm running lxd without systemd happily. Do you have all the required kernel options satisfied? What does 'lxc-checkconfig' say?
Comment 2 Nico Baggus 2020-10-20 19:23:47 UTC
lxc != lxd.....

I am using the lxd environment (which confusingly does use the lxc command to manage it, not the lxc-* ).

My guest / container would not start.

/etc/init.d/lxd[5947]: Call to flock failed: Resource temporarily unavailable
/etc/init.d/lxd[5947]: ERROR: lxd stopped by something else


/var/log/lxd/VM/lxc.log:
lxc VM 20201020182546.105 ERROR    utils - utils.c:lxc_rm_rf:1759 - No such file or directory - Failed to open dir "/sys/fs/cgroup/openrc//lxc.payload.VM-6"
lxc VM 20201020182546.106 WARN     cgfsng - cgroups/cgfsng.c:cgroup_tree_remove:965 - Failed to destroy "/sys/fs/cgroup/openrc//lxc.payload.VM-6"

Where the file /sys/fs/cgroup/openrc//lxc.payload.VM-6 does exist


I based it on https://wiki.gentoo.org/wiki/LXD     Section 5.2:

Running systemd based containers on OpenRC hosts
To support systemd for e.g. ubuntu containers the host must be modified:

Create the system cgroup directory and mount the cgroup there:

root #mkdir -p /sys/fs/cgroup/systemd
root #mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd

And indeed i need to run an ubuntu 18.04 for this VM instance
And don't want to do this manual after each boot...
(Mosts guest are based on systemd alas, so it makes send to make this available in the startup script)
Comment 3 Nico Baggus 2020-10-20 19:25:23 UTC
Anyway here follows:
$ lxc-checkconfig
LXC version 4.0.4
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /lib/modules/5.4.48-gentoo-x86_64/build/.config
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points: 
/sys/fs/cgroup/openrc
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu
/sys/fs/cgroup/cpuacct
/sys/fs/cgroup/blkio
/sys/fs/cgroup/memory
/sys/fs/cgroup/devices
/sys/fs/cgroup/freezer
/sys/fs/cgroup/net_cls
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/net_prio
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/pids
/sys/fs/cgroup/rdma
/sys/fs/cgroup/systemd

Cgroup v2 mount points: 
/sys/fs/cgroup/unified

Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_NF_NAT_IPV4: missing
CONFIG_NF_NAT_IPV6: missing
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, loaded

--- Checkpoint/Restore ---
checkpoint restore: missing
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
Comment 4 Nico Baggus 2020-10-20 19:26:59 UTC
anyway after adding that one line the container did start.
Comment 5 Joonas Niilola gentoo-dev 2020-10-21 04:48:22 UTC
(In reply to Nico Baggus from comment #2)
> lxc != lxd.....
> 
> I am using the lxd environment (which confusingly does use the lxc command
> to manage it, not the lxc-* ).

Heh, yes, but the kernel config requirements are rather similar and lxc-checkconfig rules out the most common issues.

> 
> My guest / container would not start.
> 
> /etc/init.d/lxd[5947]: Call to flock failed: Resource temporarily unavailable
> /etc/init.d/lxd[5947]: ERROR: lxd stopped by something else
> 
> 
> /var/log/lxd/VM/lxc.log:
> lxc VM 20201020182546.105 ERROR    utils - utils.c:lxc_rm_rf:1759 - No such
> file or directory - Failed to open dir
> "/sys/fs/cgroup/openrc//lxc.payload.VM-6"
> lxc VM 20201020182546.106 WARN     cgfsng -
> cgroups/cgfsng.c:cgroup_tree_remove:965 - Failed to destroy
> "/sys/fs/cgroup/openrc//lxc.payload.VM-6"
> 
> Where the file /sys/fs/cgroup/openrc//lxc.payload.VM-6 does exist
> 
> 
> I based it on https://wiki.gentoo.org/wiki/LXD     Section 5.2:
> 
> Running systemd based containers on OpenRC hosts
> To support systemd for e.g. ubuntu containers the host must be modified:
> 
> Create the system cgroup directory and mount the cgroup there:
> 
> root #mkdir -p /sys/fs/cgroup/systemd
> root #mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
> 
> And indeed i need to run an ubuntu 18.04 for this VM instance
> And don't want to do this manual after each boot...
> (Mosts guest are based on systemd alas, so it makes send to make this
> available in the startup script)

Now I understand the problem, indeed I've never tried running systemd-container on openrc-host. I'll try this out, thanks!
Comment 6 Larry the Git Cow gentoo-dev 2020-10-21 06:04:23 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=4b4cbd6f7c78abe9d831c8425b2a4ebdbba298ca

commit 4b4cbd6f7c78abe9d831c8425b2a4ebdbba298ca
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2020-10-21 06:03:53 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2020-10-21 06:03:53 +0000

    app-emulation/lxd: fix init.d to allow systemd cont on openrc host
    
    Closes: https://bugs.gentoo.org/750410
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 app-emulation/lxd/files/lxd-4.0.0.initd                     | 3 +++
 app-emulation/lxd/{lxd-4.0.3.ebuild => lxd-4.0.3-r1.ebuild} | 2 +-
 2 files changed, 4 insertions(+), 1 deletion(-)
Comment 7 darthm 2020-10-24 20:12:59 UTC
Hey,

sorry for opening this up again.
But this small one liner is opening up another problem.

A non-systemd container (gentoo) on a non-systemd host os (gentoo) with application using cgroups
is searching for the folder /sys/fs/cgroup/systemd which is not existing 
inside the container BUT 

cat /proc/1/cgroup:

14:name=systemd:/
13:pids:/
12:hugetlb:/
11:net_prio:/
10:perf_event:/
9:net_cls:/
8:freezer:/
7:devices:/
6:memory:/
5:blkio:/
4:cpuacct:/
3:cpu:/
2:cpuset:/
1:name=openrc:/
0::/

is mentioning the systemd folder.

I noticed the problem with docker inside of the lxd container which gave me this error "cgroups: cannot find cgroup mount destination: unknown".

I used lxd-4.01 before and everything worked fine. 
With lxd-4.0.3-r1 I get the problem.

So maybe the solution for container with systemd is not that ideal for the rest and
a better solution is needed?

I mean, I read I can use this fix also inside the container but I don't know the implications, 
because I don't understand the technically behind this (?)namespace(?).

Greetings
Comment 8 Nico Baggus 2020-10-24 21:48:58 UTC
That prohibits the running of systemd requiring lxd guests together with non-systemd lxd guests.... 
I hope someone with more systemd experience can shine a light here...
I'l look and try if there is another solution.
Comment 9 Joonas Niilola gentoo-dev 2020-10-25 15:01:02 UTC
(In reply to darthm from comment #7)
> Hey,
> 
> sorry for opening this up again.
> But this small one liner is opening up another problem.
> 

Don't be, if it's broken, it needs to be fixed.

1: If you manually do 
   mkdir -p /sys/fs/cgroup/systemd ; mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
inside the container, does it fix the problem?
2: If you use the previous file, 
   https://gitweb.gentoo.org/repo/gentoo.git/plain/app-emulation/lxd/files/lxd-4.0.0.initd?id=a7d7c673797b5ff17c5a28b9d1a131d000cac3d7
does it work alright?

Not sure whether the "systemd fix" should be reverted or not when weighing gains. Might be other way to solve this.
Comment 10 darthm 2020-10-26 00:34:17 UTC
Hey,

if you use the old init.d script, sure it will work.

But if you started the service before, you have to restart the system first,
because just unmounting /sys/fs/cgroup/systemd won't remove the entry from /proc/1/cgroup.
And I don't know how to remove the entry.

Like I said, I'm just using cgroups now, because I got into kubernetes/docker stuff.
But I don't understand the technical implications by using and modifying it.
Maybe it's not a big deal at all because it's not important in which namespace (is this entry a namespace?) 
the programs are running.

Btw. this systemd-fix is not only affecting the containers inside of lxd, it's affecting the WHOLE host os system too, 
because all services and application which are started after the lxd service
will use the /sys/fs/cgroup/systemd path because of the name=systemd entry.

At least it breaks all non-systemd, cgroup using containers because a docker inside of lxd will break the whole container.
It tries to start the docker containers till it is OOM and then it will kill/stop the lxd container.
And I guess docker is only one example of it.

In my eyes this fix needs to be reverted or someone with good cgroup knowledge has to explain the implications
of this name entry and that it is not a problem or even introducing a systemd dependency into a non-systemd system.

Greetings
Comment 11 Sam James archtester gentoo-dev Security 2020-10-26 00:37:07 UTC
(I think juippis was just trying to check if that even is a workaround to understand the issue fully.)
Comment 12 Larry the Git Cow gentoo-dev 2020-10-26 06:30:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1e6334d771a7f7b7b38c8221a5649f2e81c5ce8a

commit 1e6334d771a7f7b7b38c8221a5649f2e81c5ce8a
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2020-10-26 06:25:22 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2020-10-26 06:29:59 +0000

    app-emulation/lxd: revert openrc -> systemd-cgroups fix
    
     - it now breaks openrc -> openrc-cgroups instead. A better solution needs to
       be found. Please see
       https://wiki.gentoo.org/wiki/LXD#Running_systemd_based_containers_on_OpenRC_hosts
       for a workaround, that you can add to your own init.d file if needed.
    
    Bug: https://bugs.gentoo.org/750410
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 app-emulation/lxd/files/lxd-4.0.0.initd                        | 3 ---
 app-emulation/lxd/{lxd-4.0.3-r1.ebuild => lxd-4.0.3-r2.ebuild} | 0
 app-emulation/lxd/{lxd-4.0.4.ebuild => lxd-4.0.4-r1.ebuild}    | 0
 3 files changed, 3 deletions(-)
Comment 13 Nico Baggus 2020-10-26 09:04:14 UTC
I did some testing.

devuan (sysv-init) doesn't seem to have issues...  I didn't verify a lot of programs.

launing a gentoo container + emerge docker inside it:
Any openrc script fill mention a ulimit failure:
/lib/rc/sh/openrc-run.sh: line 258: ulimit: open files: cannot modify limit: Operation not permitted

openrc by it's own fails:
* Configuring kernel parameters ...
sysctl: permission denied on key "fs.protected_symlinks"
sysctl: permission denied on key "fs.protected_hardlinks"
 * Unable to configure some kernel parameters                                                                                                                                                                        [ !! ]
 * ERROR: sysctl failed to start
(caused because the "root" inside the lxd != root on the host.)


The docker fails with:
time="2020-10-26T07:53:03.525663749Z" level=error msg="Failed to built-in GetDriver graph overlay /var/lib/docker"
time="2020-10-26T07:53:03.525681112Z" level=error msg="Failed to built-in GetDriver graph devicemapper /var/lib/docker"
time="2020-10-26T07:53:03.956940242Z" level=warning msg="Your kernel does not support cgroup blkio weight"
time="2020-10-26T07:53:03.957005179Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
time="2020-10-26T07:53:03.958153537Z" level=info msg="Loading containers: start."
time="2020-10-26T07:53:04.666254685Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
time="2020-10-26T07:53:05.020395691Z" level=info msg="Loading containers: done."
time="2020-10-26T07:53:05.051482457Z" level=info msg="Docker daemon" commit=4484c46d9d graphdriver(s)=vfs version=19.03.13
time="2020-10-26T07:53:05.051612900Z" level=info msg="Daemon has completed initialization"
time="2020-10-26T07:53:05.185888042Z" level=info msg="API listen on /var/run/docker.sock"


With the mount command issued INSIDE an lxd container:
 [ -d /sys/fs/cgroup/systemd ] || ( mkdir -p /sys/fs/cgroup/systemd ; mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd )
docker does start.


Now i restarted the lxd without after umounting the /sys/fs/cgroup/systemd
All the guest containers do start.  (the systemd directory still does exist).
(also the oes requiring systemd..)

/etc/init.d/docker start
/lib/rc/sh/openrc-run.sh: line 258: ulimit: open files: cannot modify limit: Operation not permitted
 * Starting docker ...                                                                                                                                                                                               [ ok ]
blabla2 ~ # docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES


So to just create the directory on the host seems to be sufficient for anything HAVING systemd, no need to mount on the host.
Guests without system don't care as there is no mount.

I suggest the following is tested as well:

@@ -17,6 +17,7 @@
        ebegin "Starting lxd service"
 
        modprobe -f loop > /dev/null 2>&1
+       [ -d /sys/fs/cgroup/systemd ] || mkdir -p /sys/fs/cgroup/systemd 
 
        # fix permissions on /var/lib/lxd and make sure it exists
        install -d /var/lib/lxd --group lxd --owner root --mode 0775
Comment 14 Nico Baggus 2020-10-26 09:17:51 UTC
Another consideration:

running docker within lxd means running nested cgroup handling.
I am not sure what issues that may cause.
Comment 15 Nico Baggus 2020-10-26 11:10:04 UTC
I later saw the systemd mount is only forgotten after a host reboot.
And indeed that is the case.

So only creating the directory is not a solution.

Without it an ubuntu 18.04 guest looks like this (20 minutes after starting..):
 9310 ?        Ss     0:00 [lxc monitor] /var/lib/lxd/containers code
 9318 ?        Ss     0:00  \_ /sbin/init
(/sbin/init is linked to lib/systemd/systemd)

so an systemd guest is a nonstarter.
It should run a Collabora service for a nextcloud environtment.


Next suggestion:

Put a conditional line in init.d/lxd
and create a conf.d/lxd   that allows to start it (or not).
Comment 16 Nico Baggus 2020-10-26 11:17:06 UTC
Adding the line by hand means ADDING at every update.

or at least allow all init.d/ files to be handled with dispatch-conf / etc-update and manual prevent updating.
Comment 17 darthm 2020-10-26 11:44:42 UTC
As I said with mounting the cgroup folder, you are not only affecting the containers but the host system itself. 
Any docker (just an example for an application which is using cgroups) running outside of lxd (on the host) 
would use the system path instead of the openrc path and I don't know which implications that has. 
Keep that in mind.

But I could live with a conditional clause inside the init script 
which checks for settings in the conf.d file.
The setting should be off by default and commented with a hint that everybody who activates it and has problems with cgroup, 
has to disable it first, for filling any bug (something like that, I guess you get the point).

By the way, for docker inside of lxd you have to configure the guest for special permissions, but that's off topic I guess.

Thanks for caring and greetings
Comment 18 Nico Baggus 2020-10-26 14:01:09 UTC
IMHO systemd is a turd... however polished, a turd.
Problem is Collabora server can only run with systemd enabled environments.

I havent noticed ill side effects on the host yet. (N=1 observation).
Comment 19 Aisha Tammy 2021-01-30 01:25:17 UTC
This issue has been addressed in the wiki - https://wiki.gentoo.org/wiki/LXD#Running_systemd_based_containers_on_OpenRC_hosts

Please confirm this fix works and close if appropriate.
Comment 20 Nico Baggus 2021-02-23 18:10:15 UTC
I tried the solution and the LXC clients could not start.
I'll try to verify where the issue might be, i need to check if it is with cgroupsv2.
Comment 21 Joonas Niilola gentoo-dev 2021-05-01 05:34:49 UTC
Is this still a problem? OpenRC has been updated multiple times and even lxd's init.d file received, quite a many, updates since.
Comment 22 Nico Baggus 2021-05-01 11:48:35 UTC
It now seems to work.


With the most recent update: devices are now not renamed to "eth0" but keep their veth.... random name causing the containers to NOT fetch their addresses).
Comment 23 Joonas Niilola gentoo-dev 2021-05-01 16:14:42 UTC
(In reply to Nico Baggus from comment #22)
> It now seems to work.
> 

Glad to hear it!

> 
> With the most recent update: devices are now not renamed to "eth0" but keep
> their veth.... random name causing the containers to NOT fetch their
> addresses).

AFAIK this change didn't land on LTS branch, that is present in the Gentoo repo, but somewhere later. 

I tested with lxd-4.0.6 and have no problems here creating new containers with network. 

Maybe it's also a profile thing for new setups... hmm...
Comment 24 Nico Baggus 2021-05-02 15:41:41 UTC
profile:

config: {}
description: Default LXD profile
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/instances/nc-beta

nc-beta is such a debian based system that needs the name to be hardcoded in config files.
Comment 25 Nico Baggus 2021-05-02 20:12:59 UTC
Alas,

If setup with
rc_cgroup_mode="unified"
rc_cgroup_controllers="systemd"

The eth0 device are not setup in the container systems.
they are still mentioned with random vethXXXXXX names.

Centos containers can cope (CentOS detects devices by MAC address)
en will receive their DHCP assigned addresses.

Debian Based containers are missing eth0: and therefore have no concept of what to do. (no match on device name).
Due to unpredictable nature of veth devicenames no fix here.

So I reverted back to:
rc_cgroup_mode="hybrid"

And mounting my /sys/fs/cgroup.. bingo eth0 devices... 

The problem might verify well be in lxcfs:
in /var/lib/lxcfs now shows:
drwxr-xr-x 2 root root 0 May  2 22:07 cgroup 
dr-xr-xr-x 2 root root 0 May  2 22:07 proc
dr-xr-xr-x 2 root root 0 May  2 22:07 sys


While rc_cgroup_mode=unified then  cgroup shows as:
????????? ? ? ? ? ??? ?? ??:?? cgroup 



rc_cgroup_mode=hybrid has also been tried without my private mount.. resulting in no starting containers.