Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 713940 - =app-arch/zstd-1.4.4-r3: USE=threads underlinks against pthread
Summary: =app-arch/zstd-1.4.4-r3: USE=threads underlinks against pthread
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Patrick Lauer
URL: https://github.com/facebook/zstd/issu...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-03-22 12:35 UTC by Sergei Trofimovich (RETIRED)
Modified: 2021-10-07 06:28 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
app-arch:zstd-1.4.4-r3:20200322-123156.log (app-arch:zstd-1.4.4-r3:20200322-123156.log,52.60 KB, text/x-log)
2020-03-22 12:35 UTC, Sergei Trofimovich (RETIRED)
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Sergei Trofimovich (RETIRED) gentoo-dev 2020-03-22 12:35:40 UTC
Created attachment 624186 [details]
app-arch:zstd-1.4.4-r3:20200322-123156.log

Simplest reproducer is to add LDFLAGS=-Wl,--no-undefined:

$ LANG=C LDFLAGS=-Wl,--no-undefined USE=-static-libs ebuild zstd-1.4.4-r3.ebuild clean install


compress/zstd_compress.c: In function 'ZSTD_loadZstdDictionary':
compress/zstd_compress.c:2799:64: warning: dereferencing type-punned pointer might break strict-aliasing rules [-Wstrict-aliasing]
 2799 |         size_t const hufHeaderSize = HUF_readCTable((HUF_CElt*)bs->entropy.huf.CTable, &maxSymbolValue, dictPtr, dictEnd-dictPtr);
      |                                                                ^~
compress/zstd_compress_literals.c: In function 'ZSTD_compressLiterals':
compress/zstd_compress_literals.c:106:28: warning: dereferencing type-punned pointer might break strict-aliasing rules [-Wstrict-aliasing]
  106 |                 (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2) :
      |                            ^~~~~~~
compress/zstd_compress_literals.c:110:28: warning: dereferencing type-punned pointer might break strict-aliasing rules [-Wstrict-aliasing]
  110 |                 (HUF_CElt*)nextHuf->CTable, &repeat, preferRepeat, bmi2);
      |                            ^~~~~~~
/usr/lib/gcc/x86_64-pc-linux-gnu/10.0.1/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/portage/app-arch/zstd-1.4.4-r3/temp/cclQCvFm.o: in function `POOL_free.part.0':
pool.c:(.text+0x17f): undefined reference to `pthread_join'
/usr/lib/gcc/x86_64-pc-linux-gnu/10.0.1/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/portage/app-arch/zstd-1.4.4-r3/temp/cclQCvFm.o: in function `POOL_create_advanced':
pool.c:(.text+0x30c): undefined reference to `pthread_create'
/usr/lib/gcc/x86_64-pc-linux-gnu/10.0.1/../../../../x86_64-pc-linux-gnu/bin/ld: /tmp/portage/app-arch/zstd-1.4.4-r3/temp/cclQCvFm.o: in function `POOL_resize':
pool.c:(.text+0x4c5): undefined reference to `pthread_create'
collect2: error: ld returned 1 exit status
make: *** [Makefile:172: libzstd.so.1.4.4] Error 1
make: Leaving directory '/tmp/portage/app-arch/zstd-1.4.4-r3/work/zstd-1.4.4-abi_x86_32.x86/lib'
 * ERROR: app-arch/zstd-1.4.4-r3::gentoo failed (compile phase):
 *   emake failed
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2020-03-22 14:08:05 UTC
https://github.com/facebook/zstd/blob/dev/lib/README.md#multithreading-support says:

"""
Multithreading support

Multithreading is disabled by default when building with make. Enabling multithreading requires 2 conditions :

    set build macro ZSTD_MULTITHREAD (-DZSTD_MULTITHREAD for gcc)
    for POSIX systems : compile with pthread (-pthread compilation flag for gcc)

Both conditions are automatically applied when invoking make lib-mt target.

When linking a POSIX program with a multithreaded version of libzstd, note that it's necessary to invoke the -pthread flag during link stage.

Multithreading capabilities are exposed via the advanced API defined in lib/zstd.h.
"""

But I don't see -pthread being added anywhere for 'lib-mt': https://github.com/facebook/zstd/blob/dev/lib/Makefile
Comment 2 Jan Ziak (atomsymbol) 2020-03-22 14:23:48 UTC
Emerging zstd does not print any QA message about the missing symbols.

In my opinion, this indicates a deeper problem and the deeper problem is that "ldd -r" isn't being run on installed shared libraries and executables as a regular part of emerging all Gentoo packages.

$ ldd -r /usr/lib64/libzstd.so
	linux-vdso.so.1 (0x00007ffc467f6000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fee76cd5000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fee76fe4000)
undefined symbol: pthread_join	(/usr/lib64/libzstd.so)
undefined symbol: pthread_create	(/usr/lib64/libzstd.so)
Comment 3 Sergei Trofimovich (RETIRED) gentoo-dev 2020-03-22 14:25:02 UTC
Filed upstream issue as: https://github.com/facebook/zstd/issues/2045
Comment 4 Sergei Trofimovich (RETIRED) gentoo-dev 2020-03-22 14:33:17 UTC
(In reply to Jan Ziak (http://atom-symbol.net) from comment #2)
> Emerging zstd does not print any QA message about the missing symbols.

Some shared libraries are underlinked on purpose: samba (and other) plugins that use symbols from loaded exceutable exported wih `-rdynamic`. You can try to set LDFLAGS=-Wl,--no-undefined in make.conf and see how far you can get rebuilding world.

I expect `glibc` to be the first fault (unless our aggressive filtering does not filter it out), opengl related plugin loaders the next, samba, pidgin and the like are next.

> In my opinion, this indicates a deeper problem and the deeper problem is
> that "ldd -r" isn't being run on installed shared libraries and executables
> as a regular part of emerging all Gentoo packages.
> 
> $ ldd -r /usr/lib64/libzstd.so
> 	linux-vdso.so.1 (0x00007ffc467f6000)
> 	libc.so.6 => /lib64/libc.so.6 (0x00007fee76cd5000)
> 	/lib64/ld-linux-x86-64.so.2 (0x00007fee76fe4000)
> undefined symbol: pthread_join	(/usr/lib64/libzstd.so)
> undefined symbol: pthread_create	(/usr/lib64/libzstd.so)

I don't think we can sort it in this zstd bug. I suggest filing a separate bug against portage or start a thread in gentoo-dev.
Comment 5 Martin Väth 2020-03-22 14:58:33 UTC
(In reply to Sergei Trofimovich from comment #4)
> You can try to set LDFLAGS=-Wl,--no-undefined in make.conf and see how
> far you can get rebuilding world.

I have this since ages. There are not many packages (on my systems) which break; all are marked by +Wl,--no-undefined in https://github.com/vaeth/portage-env-mv/blob/master/package.cflags/checks

> I expect `glibc` to be the first fault (unless our aggressive filtering does
> not filter it out)

Probably; I never tried to override any filtering inside of the ebuilds.

(Not replying any further here since this is unrelated to the actual bug.)
Comment 6 Larry the Git Cow gentoo-dev 2020-03-22 15:34:28 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=42bc10c0190c408533447d5ab0a980095c4574b2

commit 42bc10c0190c408533447d5ab0a980095c4574b2
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-03-22 15:33:54 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-03-22 15:34:22 +0000

    app-arch/zstd: fix underlinking against pthreads, bug #713940
    
    buscher noticed late gcc-10 link failure which exposed an
    underlinked libzstd.so when built with multithreaded support.
    
    The change adds an ebuild workaround to compile/link against
    -pthread on platforms where this options exist. Real fix is
    left to upstream.
    
    Reported-by: buscher on #gentoo-toolchain
    Closes: https://bugs.gentoo.org/713940
    Package-Manager: Portage-2.3.94, Repoman-2.3.21
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 app-arch/zstd/zstd-1.4.4-r4.ebuild | 74 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 74 insertions(+)
Comment 7 Larry the Git Cow gentoo-dev 2021-10-07 06:28:09 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=33aed1171970621f1a29506c71b9cf24aca9e137

commit 33aed1171970621f1a29506c71b9cf24aca9e137
Author:     Mike Frysinger <vapier@chromium.org>
AuthorDate: 2021-10-07 06:26:53 +0000
Commit:     Mike Frysinger <vapier@gentoo.org>
CommitDate: 2021-10-07 06:28:01 +0000

    app-arch/zstd: drop obsolete workaround
    
    This release fixed the problem so don't need this workaround.
    
    Bug: https://bugs.gentoo.org/713940
    Signed-off-by: Mike Frysinger <vapier@gentoo.org>

 app-arch/zstd/zstd-1.5.0.ebuild | 7 +------
 1 file changed, 1 insertion(+), 6 deletions(-)