Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 752480 - app-emulation/lxd-4.0.4 lxd-agent needs to be a statically compiled executable
Summary: app-emulation/lxd-4.0.4 lxd-agent needs to be a statically compiled executable
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal with 1 vote (vote)
Assignee: Joonas Niilola
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-11-03 09:19 UTC by Joe Kowalski
Modified: 2021-05-02 15:40 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joe Kowalski 2020-11-03 09:19:52 UTC
The lxd-agent executable in the lxd package is used to connect lxd virtual machine shell to the host. On an Ubuntu lxd host the agent binary is injected into the vm via virtio-fs and shows up as a static binary. The gentoo binary that gets compiled is dynamicly linked and ends up not working because it can't find its libraries within the vm:
VM within a gentoo host:
root@testvm:/run/lxd_config/9p# ldd lxd-agent
./lxd-agent: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./lxd-agent)
	linux-vdso.so.1 (0x00007ffce55c4000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fbd77822000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbd777ff000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbd777f9000)
	liblxc.so.1 => not found
	libacl.so.1 => /lib/x86_64-linux-gnu/libacl.so.1 (0x00007fbd777ee000)
	libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007fbd776c5000)
	libraft.so.0 => not found
	libdqlite.so.0 => not found
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fbd776a8000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fbd774b6000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fbd7782f000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fbd77367000)

Ubuntu host:
root@testvm:~# ldd /run/lxd_config/9p/lxd-agent
	not a dynamic executable


Reproducible: Always

Steps to Reproduce:
1. Setup a lxd host on gentoo (several different kernel modules and qemu use flags needed, point the lxd daemon to the edk2-ovmf files). 
2. Launch a lxd virtual machine.
Actual Results:  
You can't connect to a shell in the vm with lxc exec containername bash.

Expected Results:  
A shell in the vm opens up

$ sudo emerge --info
Password: 
Portage 3.0.8 (python 3.7.9-final-0, default/linux/amd64/17.0/desktop, gcc-9.2.0, glibc-2.32-r2, 5.4.74 x86_64)
=================================================================
System uname: Linux-5.4.74-x86_64-AMD_Ryzen_9_3900X_12-Core_Processor-with-gentoo-2.7
KiB Mem:    65792636 total,  49818060 free
KiB Swap:   33554428 total,  33554428 free
Timestamp of repository gentoo: Mon, 02 Nov 2020 22:15:01 +0000
Head commit of repository gentoo: 313352e4f5371689b4bccc25ca27718ecf273201
Timestamp of repository jorgicio: Sun, 01 Nov 2020 03:05:20 +0000
Head commit of repository jorgicio: 6ca2fa23e77c8690153ec5300aca667a045d6762

Head commit of repository steam-overlay: 3413c387e5026226d29b04f97c0846f536364ba9

sh bash 5.0_p18
ld GNU ld (Gentoo 2.32 p2) 2.32.0
distcc 3.3.3 x86_64-pc-linux-gnu [disabled]
app-shells/bash:          5.0_p18::gentoo
dev-java/java-config:     2.3.1::gentoo
dev-lang/perl:            5.30.3::gentoo
dev-lang/python:          2.7.18-r4::gentoo, 3.6.12::gentoo, 3.7.9::gentoo, 3.8.5::gentoo
dev-util/cmake:           3.17.4-r1::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.7::gentoo
sys-apps/sandbox:         2.18::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r5::gentoo
sys-devel/automake:       1.11.6-r3::gentoo, 1.15.1-r2::gentoo, 1.16.1-r1::gentoo
sys-devel/binutils:       2.32-r1::gentoo, 2.33.1-r1::gentoo, 2.34-r2::gentoo
sys-devel/gcc:            9.2.0-r2::gentoo, 9.3.0-r1::gentoo
sys-devel/gcc-config:     2.3.2::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers)
sys-libs/glibc:           2.32-r2::gentoo
Repositories:

gentoo
    location: /var/portage/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.us.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-metamanifest: yes
    sync-rsync-extra-opts: 
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-max-age: 24

jorgicio
    location: /var/db/repos/jorgicio
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/jorgicio.git
    masters: gentoo

x-portage
    location: /usr/local/portage
    masters: gentoo
    priority: 0

steam-overlay
    location: /var/lib/layman/steam
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/steam-overlay.git
    masters: gentoo
    priority: 50

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="@FREE"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-march=znver2 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-march=znver2 -O2 -pipe"
DISTDIR="/var/portage/distfiles/"
ENV_UNSET="CARGO_HOME DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN GOPATH PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe"
GENTOO_MIRRORS="http://gentoo.osuosl.org/"
LANG="en_US.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j24"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X a52 aac acl acpi alsa amd64 berkdb bluetooth branding bzip2 cairo cdda cdr cli crypt cups dbus dri dts dvd dvdr emboss encode exif ffmpeg flac fortran gdbm gif gpm gtk gudev gui iconv icu ipv6 jpeg lcms libglvnd libnotify libtirpc mad mng mp3 mp4 mpeg multilib ncurses nls nptl ogg opengl openmp pam pango pcre pdf png policykit ppds pulseaudio qt5 readline sdl seccomp spell split-usr ssl startup-notification svg systemd tcpd tiff truetype udev udisks unicode upower usb vdpau vorbis vulkan wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="aes avx avx2 f16c fma3 mmx mmxext pclmul popcnt sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="evdev" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" LUA_SINGLE_TARGET="lua5-1" LUA_TARGETS="lua5-1" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2 php7-3 php7-4" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_7" PYTHON_TARGETS="python2_7 python3_7" RUBY_TARGETS="ruby25 ruby26" USERLAND="GNU" VIDEO_CARDS="amdgpu radeonsi" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Joonas Niilola gentoo-dev 2020-11-03 13:11:15 UTC
Does this work with 4.0.3-r2? I could introduce a 'static' USE to restore previous behaviour.
Comment 2 Joe Kowalski 2020-11-22 23:16:10 UTC
(In reply to Joonas Niilola from comment #1)
> Does this work with 4.0.3-r2? I could introduce a 'static' USE to restore
> previous behaviour.

Sorry for the delay - it took me a bit to get a non-production box I could downgrade to the 4.0.3-r2 ebuild on to test this (lxd does not like downgrades). But it looks like even the older ebuild still has lxd-agent as a dynamicly linked binary:

[~]>equery l lxd
 * Searching for lxd ...
[IP-] [  ] app-emulation/lxd-4.0.3-r2:0
[~]>ldd /usr/bin/lxd-agent 
	linux-vdso.so.1 (0x00007ffcc0b80000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00007fe8d93c5000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe8d93a5000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007fe8d93a0000)
	liblxc.so.1 => /usr/lib64/liblxc.so.1 (0x00007fe8d92d7000)
	libacl.so.1 => /lib64/libacl.so.1 (0x00007fe8d92cc000)
	libsqlite3.so.0 => /usr/lib/lxd/libsqlite3.so.0 (0x00007fe8d91ef000)
	libraft.so.0 => /usr/lib/lxd/libraft.so.0 (0x00007fe8d91cb000)
	libco.so.0 => /usr/lib/lxd/libco.so.0 (0x00007fe8d91c6000)
	libdqlite.so.0 => /usr/lib/lxd/libdqlite.so.0 (0x00007fe8d91a4000)
	libgcc_s.so.1 => /usr/lib/gcc/x86_64-pc-linux-gnu/9.3.0/libgcc_s.so.1 (0x00007fe8d918a000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fe8d8fce000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fe8d9418000)
	libcrypto.so.1.1 => /usr/lib64/libcrypto.so.1.1 (0x00007fe8d8d11000)
	libseccomp.so.2 => /usr/lib64/libseccomp.so.2 (0x00007fe8d8cc2000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007fe8d8cb7000)
	libz.so.1 => /lib64/libz.so.1 (0x00007fe8d8c9d000)
	libuv.so.1 => /usr/lib64/libuv.so.1 (0x00007fe8d8c6a000)

Not a staticly linked binary that lxd seems to build by default as the main makefile in git seems to set:

lxd-agent:
	CGO_ENABLED=0 go install -v -tags agent,netgo ./lxd-agent
	@echo "LXD agent built successfully"

The rest of the binaries could probably still be dynamically linked to the system libs, but I don't see a way to make the agent work without it being static since it needs to run in a wide variety of vm's.
Comment 3 Joe Kowalski 2020-11-23 00:27:16 UTC
Manually compiling lxd from scratch and replacing the static lxd-agent binary that produces does make a working lxd agent in a lxd vm.
Comment 4 Joonas Niilola gentoo-dev 2020-11-23 07:37:42 UTC
Did you try compiling from their Github master, or 4.0.4 release tarball? Because I'm trying with 4.0.4 and am running into multiple issues with static-linking. I saw their upstream issue, 
  https://github.com/lxc/lxd/issues/7187
which seemed to have landed on 4.2 release. Can you get it to work on 4.0.4 and if yes, how? 

I've tried
	agent? (
		app-emulation/lxc[static-libs]
		dev-db/sqlite[static-libs]
		dev-libs/dqlite[static-libs]
		dev-libs/libuv[static-libs]
		dev-libs/raft[static-libs]
		sys-devel/gettext[static-libs]
		sys-libs/libcap[static-libs]
		sys-libs/libseccomp[static-libs]
		sys-libs/zlib[static-libs]
		virtual/acl[static-libs]

	if use agent; then
		go build -v -tags "agent" -ldflags '-extldflags "-static -lm -ldl -lz -lpthread -lz -lraft -ldqlite -luv -lseccomp -lcap"' -o . ./lxd-agent/... || die
		go build -v -tags "netgo" -ldflags '-extldflags "-static -lm -ldl -lz -lpthread -lz -lraft -ldqlite -luv -lseccomp -lcap"' -o . ./lxd-p2c/... || die
	fi

(and it doesn't even work if I try to build it manually outside portage)
Comment 5 Larry the Git Cow gentoo-dev 2021-05-01 05:38:54 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=5d05a7c3f79a987247f5e43260a299eccab75c06

commit 5d05a7c3f79a987247f5e43260a299eccab75c06
Author:     Joonas Niilola <juippis@gentoo.org>
AuthorDate: 2021-05-01 05:37:09 +0000
Commit:     Joonas Niilola <juippis@gentoo.org>
CommitDate: 2021-05-01 05:38:47 +0000

    app-emulation/lxd: bump to 4.0.6
    
     - build lxd-agent statically, #752480
    
    Bug: https://bugs.gentoo.org/752480
    Signed-off-by: Joonas Niilola <juippis@gentoo.org>

 app-emulation/lxd/Manifest         |   2 +
 app-emulation/lxd/lxd-4.0.6.ebuild | 156 +++++++++++++++++++++++++++++++++++++
 2 files changed, 158 insertions(+)
Comment 6 Joonas Niilola gentoo-dev 2021-05-01 05:39:54 UTC
Attempted a fix, at least it builds for me now. But I have no means to test whether it works or not, and would appreciate feedback.
Comment 7 Michael Meier 2021-05-02 15:40:30 UTC
Not the original reporter, but I hit a similar problem when trying out LXD first with 4.0.5.
Regardless of the VM image I tried, all returned "failed to connect to lxd-agent". 

I'm not able to confirm that the dynamically linked executable was the problem, as I wasn't able to log into the VM images at all (no root PW set, no SSH installed in the standard images).

But I just found this bug, updated to 4.0.6 via portage, and now I'm able to connect with "lxc exec my-vm bash" successfully.