Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 496612

Summary: >=sys-fs/cryptsetup-1.6.0[gcrypt,static]: libgcrypt with threads enabled is not compatible with static linking
Product: Gentoo Linux Reporter: Marko D <markodevelop>
Component: [OLD] Core systemAssignee: Gentoo's Team for Core System packages <base-system>
Status: CONFIRMED ---    
Severity: normal CC: alonbl, amade, anton.kochkov, antu456, bitlord0xff, crypto+disabled, freddie, gentoo, gentoobugs, guillaume.ramelet, help, info, kripton, mark, pageexec, rei4dan, Storklerk, ViliusSutkus89
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=520450
Whiteboard: Maintainer note: REQUIRED_USE="static? ( !gcrypt )" in ebuild to prevent this bug.
Package list:
Runtime testing required: ---
Bug Depends on: 521236    
Bug Blocks:    
Attachments: emerge --info output
build.log

Description Marko D 2013-12-31 18:52:07 UTC
Building latest cryptsetup

[ebuild     U  ] sys-fs/cryptsetup-1.6.3 [1.6.2] USE="gcrypt nls python static -kernel -nettle -openssl -reencrypt -static-libs -udev -urandom" PYTHON_SINGLE_TARGET="python2_7 -python2_6" PYTHON_TARGETS="python2_6 python2_7"

yields the following error:

libtool: link: x86_64-pc-linux-gnu-gcc -Wall -march=native -O2 -pipe -Wl,-O1 -Wl,--sort-common -Wl,-z -Wl,now -Wl,--hash-style=gnu -o .libs/veritysetup veritysetup-utils_crypt.
o veritysetup-utils_loop.o veritysetup-utils_tools.o veritysetup-veritysetup.o  -Wl,--as-needed ../lib/.libs/libcryptsetup.so -L/lib64 -luuid -ldevmapper /usr/lib64/libgcrypt.s
o -lgpg-error -lpopt
/usr/lib64/libgcrypt.a(libgcrypt_la-ath.o): In function `_gcry_ath_mutex_lock':
(.text+0x284): undefined reference to `pthread_mutex_lock'
/usr/lib64/libgcrypt.a(libgcrypt_la-ath.o): In function `_gcry_ath_mutex_unlock':
(.text+0x2d4): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pool_create':
(.text+0x108): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pool_create':
(.text+0x127): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pool_destroy':
(.text+0x18a): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pool_destroy':
(.text+0x1a2): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pools_check_leaks':
(.text+0xa6e): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pools_check_leaks':
(.text+0xa8a): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/../../../../lib64/libdevmapper.a(pool.o): In function `dm_pools_check_leaks':
(.text+0xbb7): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__register_frame_info_bases':
(.text+0x170c): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__register_frame_info_table_bases':
(.text+0x17cb): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__deregister_frame_info_bases':
(.text+0x186e): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__deregister_frame_info_bases':
(.text+0x18f6): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `_Unwind_Find_FDE':
(.text+0x19c6): undefined reference to `pthread_mutex_lock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `_Unwind_Find_FDE':
(.text+0x1a16): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__register_frame_info_bases':
(.text+0x1733): undefined reference to `pthread_mutex_unlock'
/usr/lib/gcc/x86_64-pc-linux-gnu/4.7.3/libgcc_eh.a(unwind-dw2-fde-dip.o): In function `__register_frame_info_table_bases':
(.text+0x17ea): undefined reference to `pthread_mutex_unlock'
collect2: error: ld returned 1 exit status
Makefile:632: recipe for target 'veritysetup.static' failed
make[2]: *** [veritysetup.static] Error 1

emerge   --info attached

Reproducible: Always
Comment 1 Marko D 2013-12-31 18:52:32 UTC
Created attachment 366656 [details]
emerge --info output
Comment 2 Jeroen Roovers (RETIRED) gentoo-dev 2014-01-01 15:41:19 UTC
Please attach the entire build log to this bug report.
Comment 3 Marko D 2014-01-01 18:27:02 UTC
Created attachment 366704 [details]
build.log

Here it is
Comment 4 SpanKY gentoo-dev 2014-01-02 15:52:51 UTC
relies heavily on the fact that libgcrypt doesn't provide a pkg-config file, so querying for its linkage info is hard (impossible?)

maybe we just add REQUIRED_USE="static? ( !gcrypt )"
Comment 5 Samuli Suominen (RETIRED) gentoo-dev 2014-01-03 16:32:14 UTC
(In reply to SpanKY from comment #4)
> relies heavily on the fact that libgcrypt doesn't provide a pkg-config file,
> so querying for its linkage info is hard (impossible?)

there is libgcrypt-config, but I don't see libgcrypt.so linked against -lpthread on my system
looks like a direct duplicate of bug 491806 instead, broken .pc file in lvm2

*** This bug has been marked as a duplicate of bug 491806 ***
Comment 6 SpanKY gentoo-dev 2014-01-03 17:22:52 UTC
(In reply to Samuli Suominen from comment #5)

look at the first error in the list:

/usr/lib64/libgcrypt.a(libgcrypt_la-ath.o): In function `_gcry_ath_mutex_lock':
(.text+0x284): undefined reference to `pthread_mutex_lock'

that's the libgcrypt library using pthread symbols

note that simply not being linked against libpthread doesn't mean a library can't support pthreads.  the C library provides dummy symbols for a lot of pthread_* funcs so that a library can transparently support threads, but only if the application itself has linked against it.
Comment 7 Doug Goldstein (RETIRED) gentoo-dev 2014-01-04 05:00:20 UTC
(In reply to SpanKY from comment #6)
> (In reply to Samuli Suominen from comment #5)
> 
> look at the first error in the list:
> 
> /usr/lib64/libgcrypt.a(libgcrypt_la-ath.o): In function
> `_gcry_ath_mutex_lock':
> (.text+0x284): undefined reference to `pthread_mutex_lock'
> 
> that's the libgcrypt library using pthread symbols
> 
> note that simply not being linked against libpthread doesn't mean a library
> can't support pthreads.  the C library provides dummy symbols for a lot of
> pthread_* funcs so that a library can transparently support threads, but
> only if the application itself has linked against it.

What if we tried -Wl,--whole-archive -pthread -Wl,--no-whole-archive? Pretty common on Linux boxes that anything involving pthreads + static linking has fits without that. I know that dev-util/pkgconfig respects the whole archive wrapping now (I believe it was fixed in 0.26 or 0.27 and we have 0.28 stable) but not sure about dev-util/pkgconf.
Comment 8 SpanKY gentoo-dev 2014-01-04 13:47:22 UTC
(In reply to Doug Goldstein from comment #7)

pretty sure we don't want to do that.  just make sure -lpthread shows up after the libs that need it (gcrypt in this case).
Comment 9 Artemii 2014-01-19 21:38:08 UTC
It compiles with "-Wl,--whole-archive -pthread -Wl,--no-whole-archive". Cryptsetup linked without pthreads library with other linking optimization flags, even if I pasted "-lpthread" in Makefiles.
Comment 10 Alon Bar-Lev (RETIRED) gentoo-dev 2014-02-15 19:35:10 UTC
At libcrypt-1.6 side we have:
---
/* Define if the POSIX multithreading library can be used. */
#define USE_POSIX_THREADS 1

/* Define if references to the POSIX multithreading library should be made
   weak. */
#define USE_POSIX_THREADS_WEAK 1

#if USE_POSIX_THREADS_WEAK
# include <pthread.h>
#endif

/* On an ELF system it is easy to use pthreads using weak references.
   Take care not to test the address of a weak referenced function we
   actually use; some GCC versions have a bug were &foo != NULL is
   always evaluated to true in PIC mode.  USING_PTHREAD_AS_DEFAULT is
   used by ath_install to detect the default usage of pthread.  */
#if USE_POSIX_THREADS_WEAK
# pragma weak pthread_cancel
# pragma weak pthread_mutex_init
# pragma weak pthread_mutex_lock
# pragma weak pthread_mutex_unlock
# pragma weak pthread_mutex_destroy
#endif
---

As much as I hate these magics, I do not think it is capable of working in static mode.

Maybe add --disable-threads at libgcrypt to be able to be linked as static?
Comment 11 Jeroen Roovers (RETIRED) gentoo-dev 2014-03-30 22:33:45 UTC
*** Bug 506242 has been marked as a duplicate of this bug. ***
Comment 12 Christian Schmidt 2014-03-31 06:56:44 UTC
My personal solution for static linking (yes, it fails horribly, causing segfaults, see #506242):

sed -i 's/#define USE_POSIX_THREADS_WEAK 1//' ${S}_build/config.h
sed -i 's/libs="-lgcrypt"/libs="-lgcrypt -lpthread"/' ${S}_build/src/libgcrypt-config

in the end of libgcrypt's src_configure section, after configure has run. This will turn the weak reference into a hard reference, and tell all programs that use libgcrypt-config to link against pthreads as well.

WorksForMeâ„¢
Comment 13 Jeroen Roovers (RETIRED) gentoo-dev 2014-07-02 12:02:49 UTC
*** Bug 516058 has been marked as a duplicate of this bug. ***
Comment 14 Samuli Suominen (RETIRED) gentoo-dev 2014-07-03 20:19:09 UTC
*** Bug 516224 has been marked as a duplicate of this bug. ***
Comment 15 Felix Tiede 2014-08-21 04:02:05 UTC
What about users who need cryptsetup statically linked binary for their initrd to open an encrypted root fs?
Comment 16 Samuli Suominen (RETIRED) gentoo-dev 2014-08-21 04:09:25 UTC
(In reply to Felix Tiede from comment #15)
> What about users who need cryptsetup statically linked binary for their
> initrd to open an encrypted root fs?

Users who use initrd specifically *don't* need static cryptsetup or lvm2, they should simply copy the dynamic libraries cryptsetup or lvm2 needs to the initrd, genkernel for example has copied the libraries it needs for quite a while now

USE="static" is a specific hack for users who *don't* use initrd/initramfs and who use separate /usr, and by the way, it has been decided such an setup
is not really viable, which is why you don't see people actively working on the bug, which is why USE="static" is now in package.use.stable.mask
Comment 17 parafin 2014-08-21 20:34:27 UTC
My workaround for both this bug and #516224:
darkhorse ~ # cat /etc/portage/env/pthread.conf 
LDFLAGS="${LDFLAGS} -pthread"
darkhorse ~ # grep cryptsetup /etc/portage/package.env/local 
sys-fs/cryptsetup pthread.conf
Comment 18 parafin 2014-08-22 10:59:17 UTC
Hmm, my bad, this workaround makes cryptsetup compile, but it segfaults when trying to use luksOpen command:(
Comment 19 Samuli Suominen (RETIRED) gentoo-dev 2014-08-22 11:36:25 UTC
(In reply to parafin from comment #18)
> Hmm, my bad, this workaround makes cryptsetup compile, but it segfaults when
> trying to use luksOpen command:(

Yep, that's why I didn't commit the compile fixes for static + pthread because the end result was just crashing here as well

That's why USE="static" is now in /usr/portage/profiles/base/package.use.stable.mask, and I'm considering making it a package.use.mask instead, so it'll be non-option also for ~arch users

The current situation is like...

- Running `sed` on configure of udev (systemd) to allow --enable-static flag to work again, despite udev upstream explicitely declaring static linking unsupported
This all to get libudev.a for USE="static" of lvm2, cryptsetup
- Carrying dozen patches to make the static build work, none of them have been accepted by upstream, and more patching is required

So, I think it might be time to drop USE=static for cryptsetup and lvm2
Comment 20 parafin 2014-08-22 11:59:28 UTC
It crashes only with gcrypt backend, tested with nettle USE flag - works fine (workaround is still needed because of bug #516224). I would prefer static USE flag to remain, even if it will block some others (gcrypt, udev) - it is much easier to build custom initrd using static binaries.
Comment 21 Torsten Kaiser 2014-08-23 17:00:47 UTC
The segfaults really seems specific to the gcrypt, because the openssl backend is working fine for me (See: https://bugs.gentoo.org/show_bug.cgi?id=516224#c10 )

So would some USEDEP like 'static? (!gcrypt)' acceptable instead of masking 'static'?

I am using cryptsetup in a custom initrd and I am rather happy to just to put busybox, a static mdadm and a static cryptsetup into it. ldd shows 12 libraries for the nonstatic build and I would like to stay with one file instead of 13. (And I'm not sure what else would be needed for the dynamic linker. Do I need to create a /etc/ld.so.conf for the initrd?)

Wrt to udev: http://code.google.com/p/cryptsetup/source/browse/lib/libdevmapper.c#67
That seems to be the only place in cryptsetup that has anything to do with the udev option from configure.
http://code.google.com/p/cryptsetup/issues/detail?id=167#c3 seems to agree with that: Udev is not really needed for cryptsetup, only as an indirect dependency via libdevmapper / lvm2.

As udev is an optional dependency of lvm2, why not add a USEDEP like static? (!udev). For the case of an simple initrd the missing udev support would not really matter. (My initrd does not contain anything from udev...)
Maybe add a comment to that dependency pointing to the upstreams decision to drop static support from libudev and their bugzilla, so anyone who hits it can complain there.
Comment 22 Samuli Suominen (RETIRED) gentoo-dev 2014-08-24 05:52:34 UTC
(In reply to Torsten Kaiser from comment #21)
> The segfaults really seems specific to the gcrypt, because the openssl
> backend is working fine for me (See:
> https://bugs.gentoo.org/show_bug.cgi?id=516224#c10 )
> 
> So would some USEDEP like 'static? (!gcrypt)' acceptable instead of masking
> 'static'?

and the link to problems with openssl backend is here, bug 516224

and even if both gcrypt and openssl are excluded, it's not enough, because there is also bug 520450

and once all these 3 are solved, i'm sure more will crop up as well
Comment 23 Torsten Kaiser 2014-08-24 06:30:59 UTC
(In reply to Samuli Suominen from comment #22)
> [3 current issues with USE=static]

1. Issue: This bug wrt gcrypt backend. I dont know the cause of it or how to fix it, thats why I suggested disabling this backend instead of completly killig USE=static

2. Issue: bug 516224 wrt openssl backend: I filed that bug and as written believe that is easy to fix. It looks to me that what is causing this bug is not a general problem with static cryptsetup, but that the known and working fix for lvm2 not declaring its dependencys correctly (missing "Libs.private: -L${libdir} @PTHREAD_LIBS@" line in its pkconfig file) was accidently dropped when regeneration the lvm2 patch that fixed this.

Why was this hunk dropped? The CVS commit says "refresh 2 patches so that they apply again (no real changes in them)" which very strongly suggest that the removal was *not* intended.

3. Issue: bug 520450: lvm2-2.02.109[static] fails. I never hit that bug. Probably because only [static-libs] are needed to build the static cryptsetup that I needed / requested.
Yes, it might make sense to drop USE=static from lvm2, if #520450 can't be solved. It might even be nessesary to drop it, if a USEDEP of static?(!udev) gets intoduced, if too much of lvm2 really need udev.
But I'm not seeing this blocking cryptsetup[static]...

> i'm sure more will crop up as well
... after rebuilding lvm2-2.02.109 with the fixed patch, cryptsetup[static] works fine for me, Not only mounting /home during the normal boot phase, but also in my custom initrd for mounting /.

... on my stable amd64 system the upgrade to lvm2-2.02.109 also worked fine, when it was marked stable. Its only that there cryptsetup is now a dynamic executable, because its 'static' has been masked.
Comment 24 Samuli Suominen (RETIRED) gentoo-dev 2014-08-27 08:47:56 UTC
For cryptsetup, REQUIRED_USE="static? ( !gcrypt )" is in place now, and bug 521236 has been opened to get libgcrypt[-threads] for static linking

For lvm2, 2.02.109-r1 to ~arch was revision bumped with REQUIRED_USE="static? ( !udev )" because of >=util-linux-2.25 -- no idea yet what to do with this one, this might be the end of USE="static" for lvm2 :-/

Bottom line is, USE="static" masks are removed now, *for now*
Comment 25 Alexander Hof 2014-09-19 08:44:34 UTC
"cryptsetup static gcrypt" builds and works fine with sys-fs/lvm2-2.02.103, which is the only stable package for it now and why I simply masked newer versions. So forcing "static? ( !gcrypt )" (even for the stable versions of cryptsetup) really looks inappropiate to me.
Comment 26 Alexander Hof 2014-11-12 20:28:41 UTC
So, no one feels the need to comment this? I now added cryptsetup to my overlay with changed ebuilds. Still compiling and working fine with +gcrypt +static in combination with sys-fs/lvm2-2.02.103, which is a stable release inside the official tree.
Comment 27 Branko Grubic 2014-11-13 15:11:17 UTC
(In reply to Alexander Hof from comment #26)
> So, no one feels the need to comment this? I now added cryptsetup to my
> overlay with changed ebuilds. Still compiling and working fine with +gcrypt
> +static in combination with sys-fs/lvm2-2.02.103, which is a stable release
> inside the official tree.

I also have copy of ebuild in local overlay, and using 'static gcrypt', works fine for me on two systems.
sys-fs/cryptsetup-1.6.5 (gcrypt nls python_single_target_python2_7 python_targets_python2_7 python_targets_python3_3 static udev)
sys-fs/lvm2-2.02.109 (readline static static-libs thin) 
I did have some issues with 'libgcrypt' so I masked it and using (runtime issues I think related to bug #497654 )
dev-libs/libgcrypt-1.5.4-r1 (abi_x86_64 static-libs)
Comment 28 unlord 2020-07-20 05:26:43 UTC
I am running cryptsetup-2.3.2 with USE="gcrypt static" with the following modification in a local overlay:

--- /var/db/repos/gentoo/sys-fs/cryptsetup/cryptsetup-2.3.2.ebuild	2020-05-20 16:09:05.000000000 -0000
+++ cryptsetup-2.3.2.ebuild	2020-07-19 16:32:28.180027126 -0000
@@ -18,8 +18,7 @@
 # and it's missing ripemd160 support so it can't provide full backward compatibility
 IUSE="${CRYPTO_BACKENDS} +argon2 libressl luks1_default nls pwquality reencrypt static static-libs +udev urandom"
 REQUIRED_USE="^^ ( ${CRYPTO_BACKENDS//+/} )
-	libressl? ( openssl )
-	static? ( !gcrypt )" #496612
+	libressl? ( openssl )"
 
 LIB_DEPEND="
 	dev-libs/json-c:=[static-libs(+)]

Can we remove this restriction from upstream?