When we are close any container, on host machine in dmesg, we are can see output: [ 4682.775566] lxc-start[20988]: segfault at 7f110000000f ip 00007f103be30681 sp 00007ffc368c8d70 error 4 in libur ing.so.2.1.0[7f103be30000+2000] [ 4682.775583] Code: ff 48 8b 54 24 78 64 48 2b 14 25 28 00 00 00 75 08 48 81 c4 88 00 00 00 c3 e8 0b fa ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 53 <48> 8b 47 18 48 89 fb 48 8b 7f 38 8b 30 48 c1 e6 06 e8 99 fa ff ff [ 4682.822008] device XXX left promiscuous mode [ 4682.857420] XFS (dm-10): Unmounting Filesystem Error in liburing rice BEFORE unmounting file system. Data loss possible. Solution: do not enable "io-uring" flag in make.conf. Second, app-emulation/qemu do not affected this ussue. Required for io-uring, if run as user. Root is not affected. $ cat /etc/security/limits.conf username_which_start_qemu soft memlock 64000 username_which_start_qemu hard memlock 640000 With best wishes - YD.
Problem found. If I manually do # sync , no matter host or container - no errors, no segfault. This is no gentoo-related bug. lxc-developers no write sync buffers code. But BIG WARNING!!! Until fixed, manually issue # sync && poweroff
Reference: https://github.com/lxc/lxc/blob/master/src/lxc/mainloop.c Buffer size is 512. When container is shutdown, nessesary "commit transaction", force store buffer data to disk. qemu also use this library(liburing). In qemu, buffer size defined as const 128. Need more code investigation. Regards - YD.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f3e5c4ad2fbd47d7e766cf0705612af4038755c2 commit f3e5c4ad2fbd47d7e766cf0705612af4038755c2 Author: Joonas Niilola <juippis@gentoo.org> AuthorDate: 2021-10-28 04:44:16 +0000 Commit: Joonas Niilola <juippis@gentoo.org> CommitDate: 2021-10-28 04:44:16 +0000 profiles/base: package.use.mask app-emulation/lxc[io-uring] - new feature since 4.0.11, buggy with possible data loss! Bug: https://bugs.gentoo.org/820545 Signed-off-by: Joonas Niilola <juippis@gentoo.org> profiles/base/package.use.mask | 4 ++++ 1 file changed, 4 insertions(+)
I imagine the upstream issue was made by you. Thank you!
(In reply to Joonas Niilola from comment #4) > I imagine the upstream issue was made by you. Thank you! Thanks!))) When compile lxc, lxc autodetect liburing via pkgconfig mechanism. Not related to USE flags. We are need do some more by hand. 0. Stop all containers and VM's. 1. emerge -C liburing 2. emerge -DNu world This steps must recompile lxc and qemu without liburing. If all completed correctly, we are must see empty list at # lsof -n |grep uring when running lxc or qemu again. If some people need liburing, installation again is possible AFTER compilation lxc && qemu. With best wishes - YD.
(In reply to Yuriy Dmitriev from comment #5) > (In reply to Joonas Niilola from comment #4) > > I imagine the upstream issue was made by you. Thank you! > > When compile lxc, lxc autodetect liburing via pkgconfig mechanism. Not > related to USE flags. > That shouldn't happen, https://github.com/lxc/lxc/blob/48e079bf318982ae7d5684feeb7358870fa71c10/configure.ac#L680 I also couldn't reproduce this via USE="-io-uring" while liburing was installed. Can you please show the output of following for more info: # cat /var/db/pkg/app-emulation/lxc-4.0.11/NEEDED | grep -i uring Then again I do see how having +io-uring, launching a container, and then trying to get rid of it (while containers are running) could be an issue. (As you showed with lsof)
(In reply to Joonas Niilola from comment #6) > (In reply to Yuriy Dmitriev from comment #5) > > (In reply to Joonas Niilola from comment #4) > > > I imagine the upstream issue was made by you. Thank you! > > > > When compile lxc, lxc autodetect liburing via pkgconfig mechanism. Not > > related to USE flags. > > > > That shouldn't happen, > https://github.com/lxc/lxc/blob/48e079bf318982ae7d5684feeb7358870fa71c10/ > configure.ac#L680 > > I also couldn't reproduce this via USE="-io-uring" while liburing was > installed. Can you please show the output of following for more info: > > # cat /var/db/pkg/app-emulation/lxc-4.0.11/NEEDED | grep -i uring > > Then again I do see how having +io-uring, launching a container, and then > trying to get rid of it (while containers are running) could be an issue. > (As you showed with lsof) Gentoo's ebuild lxc dont have any USE-related to io-uring flags. QEMU have. In fact, if liburing installed BEFORE lxc, lcx compiled automatic with io-uring support. $ equery uses lxc [ Legend : U - final flag setting for installation] [ : I - package is installed with flag ] [ Colors : set, unset ] * Found these USE flags for app-emulation/lxc-4.0.11: U I - - apparmor : Enable AppArmor support + + caps : Use Linux capabilities library to control privilege - - doc : Add extra documentation (API, Javadoc, etc). It is recommended to enable per package instead of globally - - man : Build and install man pages + + pam : Add support for PAM (Pluggable Authentication Modules) - DANGEROUS to arbitrarily flip + + seccomp : Enable seccomp (secure computing mode) to perform system call filtering at runtime to increase security of programs + + ssl : Add support for SSL/TLS connections (Secure Socket Layer / Transport Layer Security) + + tools : Build and install additional command line tools - - verify-sig : Verify upstream signatures on distfiles
Correct, Line 680: AC_ARG_ENABLE([liburing], [AS_HELP_STRING([--enable-liburing], [enable liburing support [default=auto]])], [enable_liburing=$enableval], [enable_liburing=auto]) As we are can see, ebuild use auto detection, as I say above.
(In reply to Yuriy Dmitriev from comment #7) > > Gentoo's ebuild lxc dont have any USE-related to io-uring flags. QEMU have. > > In fact, if liburing installed BEFORE lxc, lcx compiled automatic with > io-uring support. It should have, and 4.0.11 always did: https://gitweb.gentoo.org/repo/gentoo.git/tree/app-emulation/lxc/lxc-4.0.11.ebuild#n122 [binary R ] app-emulation/lxc-4.0.11-1::gentoo USE="caps pam seccomp* ssl* tools* -apparmor* -doc* (-io-uring) -man (-selinux) -verify-sig" 0 KiB (it's use.masked now, so *can't* be enabled) Also I did try to compile lxc-4.0.11 with liburing installed, and -io-uring USE flag set. It didn't at least link to any library/binary. That's why I asked if you could show your # cat /var/db/pkg/app-emulation/lxc-4.0.11/NEEDED | grep -i uring for more information. Are you using an ebuild for lxc outside ::gentoo? If the $(use_enable io-uring liburing) isn't specified in the ebuild, then it'll resort to automatically detecting the liburing dep, but we had that option always in 4.0.11.
For example, if lxc and qemu compiled with liburing, running software seems like: Important: run lsof as # # lsof -n |grep uring qemu-syst 2678 triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2679 qemu-syst triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2679 qemu-syst triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2682 qemu-syst triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2682 qemu-syst triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2684 qemu-syst triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2684 qemu-syst triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2685 qemu-syst triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2685 qemu-syst triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2687 qemu-syst triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2687 qemu-syst triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2696 SPICE\x20 triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2696 SPICE\x20 triod 13u a_inode 0,14 0 16771 [io_uring] qemu-syst 2678 2699 iou-wrk-2 triod mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 qemu-syst 2678 2699 iou-wrk-2 triod 13u a_inode 0,14 0 16771 [io_uring] lxc-start 2736 root mem REG 0,14 16771 anon_inode:[io_uring] (stat: No such file or directory) lxc-start 2736 root mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 lxc-start 2736 root 3u a_inode 0,14 0 16771 [io_uring] lxc-start 2736 root 4u a_inode 0,14 0 16771 [io_uring] lxc-start 3321 root mem REG 0,14 16771 anon_inode:[io_uring] (stat: No such file or directory) lxc-start 3321 root mem REG 259,2 22504 134229553 /usr/lib64/liburing.so.2.1.0 lxc-start 3321 root 3u a_inode 0,14 0 16771 [io_uring] lxc-start 3321 root 4u a_inode 0,14 0 16771 [io_uring] s
(In reply to Joonas Niilola from comment #9) > (In reply to Yuriy Dmitriev from comment #7) > > > > Gentoo's ebuild lxc dont have any USE-related to io-uring flags. QEMU have. > > > > In fact, if liburing installed BEFORE lxc, lcx compiled automatic with > > io-uring support. > > It should have, and 4.0.11 always did: > https://gitweb.gentoo.org/repo/gentoo.git/tree/app-emulation/lxc/lxc-4.0.11. > ebuild#n122 > [binary R ] app-emulation/lxc-4.0.11-1::gentoo USE="caps pam seccomp* > ssl* tools* -apparmor* -doc* (-io-uring) -man (-selinux) -verify-sig" 0 KiB > (it's use.masked now, so *can't* be enabled) > > Also I did try to compile lxc-4.0.11 with liburing installed, and -io-uring > USE flag set. It didn't at least link to any library/binary. That's why I > asked if you could show your > # cat /var/db/pkg/app-emulation/lxc-4.0.11/NEEDED | grep -i uring > for more information. > > Are you using an ebuild for lxc outside ::gentoo? If the > $(use_enable io-uring liburing) > isn't specified in the ebuild, then it'll resort to automatically detecting > the liburing dep, but we had that option always in 4.0.11. Please, few hours. I move on road.
Fully agree. My portage tree was some out of sync. ebuild work as expected, thanks alot!!!
Folks, I have good news!!! Upstream developers make patch. https://github.com/brauner/lxc/commit/a585382b972c25ee8489147d94918d001ef439a7 I make overlay, download main branch and some correct ebuild for reenable liburing. I tested some time, open and close few containers(test equipment). No errors, all work like charm!!! We are can reenable io-uring flag in lxc for testing. With best wishes - Yuriy.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c8329548cefb98b251ac2dc88fd0214240dcebff commit c8329548cefb98b251ac2dc88fd0214240dcebff Author: Joonas Niilola <juippis@gentoo.org> AuthorDate: 2021-10-29 07:12:50 +0000 Commit: Joonas Niilola <juippis@gentoo.org> CommitDate: 2021-10-29 07:14:27 +0000 profiles/base: remove app-emulation/lxc[io-uring] mask - fixed by upstream, and in lxc-4.0.11_p1. Closes: https://bugs.gentoo.org/820545 Signed-off-by: Joonas Niilola <juippis@gentoo.org> profiles/base/package.use.mask | 4 ---- 1 file changed, 4 deletions(-) Additionally, it has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=190499d396a2eac96f71a02e4b9e65d1d169d0e7 commit 190499d396a2eac96f71a02e4b9e65d1d169d0e7 Author: Joonas Niilola <juippis@gentoo.org> AuthorDate: 2021-10-29 07:11:32 +0000 Commit: Joonas Niilola <juippis@gentoo.org> CommitDate: 2021-10-29 07:14:27 +0000 app-emulation/lxc: add io-uring patches for 4.0.11 Bug: https://bugs.gentoo.org/820545 Signed-off-by: Joonas Niilola <juippis@gentoo.org> .../lxc/files/lxc-4.0.11_p1-liburing-sync1.patch | 29 ++++ .../lxc/files/lxc-4.0.11_p1-liburing-sync2.patch | 28 +++ app-emulation/lxc/lxc-4.0.11_p1.ebuild | 188 +++++++++++++++++++++ 3 files changed, 245 insertions(+)
