Bug 795126 – =app-emulation/lxc-4.0.9-r1 breaks containers startup

• Product: Gentoo Linux
• Status: RESOLVED WORKSFORME

[Description Nikolay Kichukov 2021-06-09 20:27:24 UTC]

Upgrading to =app-emulation/lxc-4.0.9-r1 broke existing containers startup. It also prevented new containers to be created, they would also fail to start. Detailed logs will be attached, but in short the failure is:

'conf.c:lxc_setup_rootfs_prepare_root:3452 - Failed to setup rootfs for'

This has been observed to fail with amd64 gentoo and debian containers.

Without any other changes, just going back to the previous version: app-emulation/lxc-4.0.6 allows the containers to start again. 

This seems like a regression.

Gentoo kernel: 5.12.9 on amd64 arch

Reproducible: Always

Steps to Reproduce:
1. Compile app-emulation/lxc-4.0.9-r1
2. Try to start existing container with:
lxc-start -n <container name>
or try to create a new one and then start with:
lxc-create -t download -n test -- -d debian -r bullseye -a amd64 --keyserver hkp://p80.pool.sks-keyservers.net:80

Actual Results:  
Containers fail to start

Expected Results:  
Containers start successfully

[Comment 1 Nikolay Kichukov 2021-06-09 20:30:05 UTC]

Created attachment 715002 [details]
lxc start log, debian container

[Comment 2 Nikolay Kichukov 2021-06-09 20:30:21 UTC]

Created attachment 715005 [details]
lxc start log, gentoo container

[Comment 3 Joonas Niilola 2021-06-10 09:51:14 UTC]

Well it seems to work for me...

Are you perhaps using openrc, and if you are, what are your rc_cgroup_mode, rc_cgroup_controllers (or rc_controller_cgroups) and related rc_cgroup_settings?
Some recent openrc update seems to have broken cgroups mode.

[Comment 4 Nikolay Kichukov 2021-06-11 12:41:26 UTC]

Hello Sam,
This is openRC init system, indeed. There have been no modification on those settings in rc.conf, so the system uses the package default settings:

$ grep -e "rc_cgroup_mode\|rc_cgroup_controllers\|rc_controller_cgroups\|rc_cgroup_settings" /etc/rc.conf 
#rc_cgroup_mode="hybrid"
#rc_cgroup_controllers=""
#rc_cgroup_settings="
#rc_cgroup_settings=""
#rc_controller_cgroups="YES"

[Comment 5 Nikolay Kichukov 2021-06-11 14:56:19 UTC]

and the version of openrc is: =sys-apps/openrc-0.42.1-r1

[Comment 6 Nikolay Kichukov 2021-06-14 13:37:42 UTC]

From 'strace -f':

[pid  7509] openat2(24, ".lxc_keep", {flags=O_RDONLY|O_CREAT|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC, mode=0600, resolve=RESOLVE_NO_XDEV|RESOLVE_NO_MAGICLINKS|RESOLVE_NO_SYMLINKS|RESOLVE_BENEATH}, 24) = 25

I could identify that it has 'RESOLVE_NO_SYMLINKS', and my /var/lib/lxc directory is a symlink to the containers directory on a different filesystem.

Replacing the symlink with a bind mount resolved the problem and now the containers startup with =app-emulation/lxc-4.0.9-r1.

This can be closed now. Thanks for looking into it.

[Comment 7 Joonas Niilola 2021-06-15 05:31:18 UTC]

Thanks for explaining the solution, I should still look into openrc-cgroups issues but it's definitely something outside my area of expertise.

[Comment 8 geever0 2022-09-29 21:04:43 UTC]

Hi! I ran into the same problem and I believe I found the proper solution!

You need to have /etc/lxc/lxc.conf ( lxc.system.conf (5) ) with the setting:
    lxc.lxcpath = <your lxc root>

^ This will make lxc find the containers in other directory than /var/lib/lxc

And need also container configs ( lxc.container.conf (5) ) specify:
    lxc.rootfs.mount = <temporary rootfs mountpoint>

^ This is not to look for /var/lib/lxc/rootfs directory


The error logs of LXC should be a lot better. Thanks for the strace debugging idea, it helped me a lot!