Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 737996 - sys-libs/glibc-2.32 installation hangs on ppc (-mcpu=7450)
Summary: sys-libs/glibc-2.32 installation hangs on ppc (-mcpu=7450)
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: PPC Linux
: Normal normal (vote)
Assignee: Gentoo Toolchain Maintainers
URL: https://sourceware.org/PR26522
Whiteboard:
Keywords:
Depends on:
Blocks: glibc-2.32
  Show dependency tree
 
Reported: 2020-08-19 10:31 UTC by Daniel Gurney
Modified: 2020-09-15 22:15 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Build log (build.log.gz,382.40 KB, application/gzip)
2020-08-19 10:32 UTC, Daniel Gurney
Details
emerge info (info.txt,5.16 KB, text/plain)
2020-08-19 10:32 UTC, Daniel Gurney
Details
gcc -mcpu=native -Q -O2 --help=target output (target_native.txt,6.64 KB, text/plain)
2020-08-21 08:17 UTC, Daniel Gurney
Details
gcc -mcpu=native -Q -O2 --help=param output (param_native.txt,10.02 KB, text/plain)
2020-08-21 08:18 UTC, Daniel Gurney
Details
0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch (0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch,2.56 KB, patch)
2020-08-22 14:14 UTC, Sergei Trofimovich (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Gurney 2020-08-19 10:31:42 UTC
Glibc appears to build fine, but when it comes time to install, the glibc sanity check hangs.

When I look at htop, I see a ld-2.32.so process using no resources, so it got as far as trying to do the sanity check. I left it for several minutes, and nothing changed. There is no delay with 2.31-r6, so it's unfortunately not me being impatient.

I was able to reproduce this problem with a local build, and a binary package cross-compiled on another system, both using the same gcc version.

The computer is a PowerBook G4.
Comment 1 Daniel Gurney 2020-08-19 10:32:07 UTC
Created attachment 655466 [details]
Build log
Comment 2 Daniel Gurney 2020-08-19 10:32:19 UTC
Created attachment 655468 [details]
emerge info
Comment 3 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-19 21:12:35 UTC
Can you paste process tree and maybe extract a backtrace of a hung process?
Comment 4 Daniel Gurney 2020-08-20 17:39:40 UTC
pstree output: 
sudo(30954)---emerge(30956)---emerge(31145)---ebuild.sh(31194)--ebuild.sh(31200)---ld-2.32.so(31216)

backtrace:
#0  0x6fe03c60 in __lll_lock_wait_private (futex=futex@entry=0x6ff4aea8 <__exit_funcs_lock>)
    at ./lowlevellock.c:35
#1  0x6fdb8e68 in __internal_atexit (func=0xb7be4a74 <_dl_fini>, arg=arg@entry=0x0, d=d@entry=0x0, 
    listp=0x6ff4adec <__exit_funcs>) at cxa_atexit.c:43
#2  0x6fdb8ef0 in __GI___cxa_atexit (func=<optimized out>, arg=arg@entry=0x0, d=d@entry=0x0) at cxa_atexit.c:70
#3  0x6fd9cba4 in generic_start_main (main=0x6ffc1310, argc=1, argv=0xbfc5da80, auxvec=0xbfc5de34, 
    init=0x6ffcc090, rtld_fini=<optimized out>, stack_end=<optimized out>, fini=<optimized out>)
    at ../csu/libc-start.c:240
#4  0x6fd9cdc8 in __libc_start_main (argc=<optimized out>, argv=<optimized out>, ev=<optimized out>, 
    auxvec=<optimized out>, rtld_fini=<optimized out>, stinfo=<optimized out>, stack_on_entry=<optimized out>)
    at ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:98
#5  0x00000000 in ?? ()
Comment 5 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-21 06:42:55 UTC
Please post your expansion of -mcpu=native.

You can use something similar to -march-native expander: https://wiki.gentoo.org/wiki/Gcc-ICE-reporting-guide#Expand_-march.3Dnative.2C_exact_gcc_version_and_other_system-specific_options
Comment 6 Daniel Gurney 2020-08-21 08:17:43 UTC
Created attachment 655870 [details]
gcc -mcpu=native -Q -O2 --help=target output
Comment 7 Daniel Gurney 2020-08-21 08:18:11 UTC
Created attachment 655872 [details]
gcc -mcpu=native -Q -O2 --help=param output
Comment 8 Daniel Gurney 2020-08-21 08:19:22 UTC
Raw outputs since diff didn't show any differences between 7450 and native.
Comment 9 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-21 17:11:29 UTC
(In reply to Daniel Gurney from comment #8)
> Raw outputs since diff didn't show any differences between 7450 and native.

No diff means your -mcpu=native is equal to -mcpu=7450.

Looks like I reproduced it in qemu-user as (default CFLAGS=-O2 works as expected):

$ CFLAGS="-mcpu=7450 -O2" ebuild glibc-2.32.ebuild clean install
$ cd $WORKDIR
$ qemu-ppc -cpu 7450 ./ld-2.32.so --library-path . /usr/powerpc-unknown-linux-gnu/bin/date
Comment 10 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-21 22:51:09 UTC
At least for qemu the hangup happens because __exit_funcs_lock used by cxa_atexit.c is initialized with garbage instead of zeros.

'__exit_funcs_lock' is in .bss section (I think) and should be zero-initialized.

Comparing libc.so.6 build with '-O2' and '-O2 -mcpu=7450' I noticed that '-mcpu=7450' got an '.iplt' sectin. AFAIU this is PLT for internal indirect functions. It contains 2 entries. Not sure which functions are registered there.

Maybe it's not related, or maybe that's what breaks bss initialization.
Comment 11 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-21 23:22:04 UTC
> It contains 2 entries. Not sure which functions are registered there.

Those are:
- __strchrnul_ifunc
- __memrchr_ifunc

Checking it any of them is called before IREL is initialized for libc.so.6.
Comment 12 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-22 07:59:27 UTC
Poked a bit more at it. iplt is unrelated, it's just a  side-effect of more CPU-spefic optimizations applied.

The real problem is broken memset():

What works:

in -O2 mode sysdeps/powerpc/powerpc32/rtld-memset.c (aka string/memset.c) is used: https://sourceware.org/git/?p=glibc.git;a=blob;f=string/memset.c;h=9c213e82dc90f3eb426bc0b6c97ec5312b9d5c46;hb=HEAD

What does not work:

in -O2 -mcpu=7450 mode ./sysdeps/powerpc/powerpc32/power4/multiarch/rtld-memset.S (aka sysdeps/powerpc/powerpc32/power4/memset.S) is used:  https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/powerpc/powerpc32/power4/memset.S;h=f4ada9afc991bdbddedc99815bc17209b64cae6e;hb=HEAD

When it attempts to initialize .sbss / .iplt /.bss (all should be zeroed out) it fails to do so. __exit_funcs_lock is in .sbss.

Looking at where assembly code breaks now.
Comment 13 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-22 12:39:21 UTC
https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/powerpc/powerpc32/rtld-memset.c;h=f3ed8ad1e74b6bbf9af8764b37c140179767744e;hb=HEAD even says why memset.c is preferred:

"""
   1 /* PPCA2 has a different cache-line size than the usual 128 bytes.  To avoid
   2    using code that assumes cache-line size to be 128 bytes (with dcbz
   3    instructions) we use the generic code instead.  */
   4 #include <string/memset.c>
"""

It should say 'cache-block' though as memset.S uses 'dcbz' instruction, not dclz.

qemu sets block-size to 64 bytes. and thus memsets only 64 of 128 bytes at each iteration leaving uninitialized holes in large chunks.
Comment 14 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-22 14:04:18 UTC
Filed upstream bug as https://sourceware.org/PR26522.
Comment 15 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-22 14:14:21 UTC
Created attachment 656176 [details, diff]
0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch

If you feel brave you can try the simple patch that switches to C implementation for rtld: 0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch

It seems to work for small tests for me.
Comment 16 Daniel Gurney 2020-08-22 15:27:01 UTC
(In reply to Sergei Trofimovich from comment #15)
> Created attachment 656176 [details, diff] [details, diff]
> 0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch
> 
> If you feel brave you can try the simple patch that switches to C
> implementation for rtld:
> 0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch
> 
> It seems to work for small tests for me.

Well, let's just say my bravery didn't pay off at all. System went out with a bunch of segfaults, and now panics on boot due to systemd instantly dying.
Comment 17 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-22 20:00:58 UTC
(In reply to Daniel Gurney from comment #16)
> (In reply to Sergei Trofimovich from comment #15)
> > Created attachment 656176 [details, diff] [details, diff] [details, diff]
> > 0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch
> > 
> > If you feel brave you can try the simple patch that switches to C
> > implementation for rtld:
> > 0001-powerpc-use-C-implementation-of-memset-in-rtld-BZ-26.patch
> > 
> > It seems to work for small tests for me.
> 
> Well, let's just say my bravery didn't pay off at all. System went out with
> a bunch of segfaults, and now panics on boot due to systemd instantly dying.

Ah, that's unfortunate. Probably more arch-specific implementations have the same problems in libc.so.6 and also need a switch back to C version. Looks like multiarch is in a very bad shape on ppc32.

Once you recover can you show what the following program prints for you?

"""
#include <stdio.h>
#include <sys/auxv.h>

int main()
{
    printf("getauxval (AT_DCACHEBSIZE) = %u\n", (unsigned)getauxval (AT_DCACHEBSIZE));
}
"""
Comment 18 Daniel Gurney 2020-08-22 20:08:38 UTC
(In reply to Sergei Trofimovich from comment #17)
> Once you recover can you show what the following program prints for you?

It reports 32.
Comment 19 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-23 09:55:02 UTC
sysdeps/powerpc/powerpc32/power7/memset.S used by sysdeps/powerpc/powerpc32/power4/multiarch/memset-power7.S also assumes dcbz to cover 128 bytes at a time. This breaks memset() for normal (non-rtld) case.
Comment 20 Larry the Git Cow gentoo-dev 2020-08-23 10:19:20 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3e3d3f661fa3df6911db31e7496951aa23497389

commit 3e3d3f661fa3df6911db31e7496951aa23497389
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-08-23 10:19:03 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-08-23 10:19:16 +0000

    sys-libs/glibc: disable multiarch on ppc32
    
    By default glibc does not enable multiarch on
    powerpc-unknown-linux-gnu targets.
    
    Bug triggers only when CFLAGS= enables power4
    or above (like, CFLAGS=-mcpu=7450).
    
    multiarch does not work on ppc with cache-block not equal to
    128 bytes and breaks memset implementations. There 'dcbz'
    instruction is used for optimization to zero out cache-block
    with zeros at a time. memset implementation assumes
    cache-block=128.
    
    G4 hardware and qemu-ppc both have cache-block=32.
    
    This means memset() zero-initializes only 32 of each 128-byte
    block including .sbss and .bss sections. This breaks glibc's
    locks and many other things.
    
    The change disables multiarch for pps32 in Gentoo until
    glibc is fixed upstream.
    
    Reported-by: Daniel Gurney
    Bug: https://sourceware.org/PR26522
    Closes:https://bugs.gentoo.org/737996
    
    Package-Manager: Portage-3.0.4, Repoman-3.0.1
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-libs/glibc/glibc-2.31-r6.ebuild | 5 +++++
 sys-libs/glibc/glibc-2.32.ebuild    | 5 +++++
 sys-libs/glibc/glibc-9999.ebuild    | 5 +++++
 3 files changed, 15 insertions(+)
Comment 21 Sergei Trofimovich (RETIRED) gentoo-dev 2020-08-23 10:29:37 UTC
Having though a bit more about it disabling multiarch is not enough. We also need to disable static platform detection on glibc side and filter out -mcpu= options.
Comment 22 Larry the Git Cow gentoo-dev 2020-08-23 10:47:18 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=81b4106ebd12af6e3fc6ec488c6dd8d07a2344d3

commit 81b4106ebd12af6e3fc6ec488c6dd8d07a2344d3
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-08-23 10:45:19 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-08-23 10:47:09 +0000

    sys-libs/glibc: revert "disable multiarch on ppc32"
    
    Disabling multiarch is not enough as static -mcpu=
    selection will still attempt to close best fit
    implementation (as opposed to refault C-based
    implementation). And these still don't work.
    
    This reverts commit 3e3d3f661fa3df6911db31e7496951aa23497389.
    
    Bug: https://bugs.gentoo.org/737996
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-libs/glibc/glibc-2.31-r6.ebuild | 5 -----
 sys-libs/glibc/glibc-2.32.ebuild    | 5 -----
 sys-libs/glibc/glibc-9999.ebuild    | 5 -----
 3 files changed, 15 deletions(-)
Comment 23 Larry the Git Cow gentoo-dev 2020-08-23 11:33:05 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8cf656db6d13ce7e97966bb31c0a068731f8530e

commit 8cf656db6d13ce7e97966bb31c0a068731f8530e
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-08-23 11:32:05 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-08-23 11:33:01 +0000

    sys-libs/glibc: disable submachine detection on ppc32
    
    By default glibc does not enable multiarch on
    powerpc-unknown-linux-gnu targets.
    
    Bug triggers only when CFLAGS value enables power4
    or above (like, CFLAGS=-mcpu=7450).
    
    multiarch does not work on ppc with cache-block not equal to
    128 bytes and breaks memset implementations. There 'dcbz'
    instruction is used for optimization to zero out cache-block
    with zeros at a time. memset implementation assumes
    cache-block=128.
    
    G4 hardware and qemu-ppc both have cache-block=32.
    
    This means memset() zero-initializes only 32 of each 128-byte
    block including .sbss and .bss sections. This breaks glibc's
    locks and many other things.
    
    The change disables submachine detection for ppc32 in Gentoo
    until glibc is fixed upstream.
    
    Reported-by: Daniel Gurney
    Bug: https://sourceware.org/PR26522
    Closes:https://bugs.gentoo.org/737996
    
    Package-Manager: Portage-3.0.4, Repoman-3.0.1
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-libs/glibc/glibc-2.31-r6.ebuild | 8 ++++++++
 sys-libs/glibc/glibc-2.32.ebuild    | 8 ++++++++
 sys-libs/glibc/glibc-9999.ebuild    | 8 ++++++++
 3 files changed, 24 insertions(+)
Comment 24 Larry the Git Cow gentoo-dev 2020-09-15 22:15:28 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d1cd3ed5e4670b1c71b4519256ffd0f9b73642df

commit d1cd3ed5e4670b1c71b4519256ffd0f9b73642df
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-09-15 22:14:46 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-09-15 22:15:24 +0000

    sys-libs/glibc: extend -mcpu= filter from ppc to ppc+ppc64
    
    glibc shares CPU detection code for powerpc/powerpc64.
    gcc-10 will advertise G5 as power7:
    
        $ echo 'int main(){}' |  powerpc64-unknown-linux-gnu-gcc \
            -mcpu=970 -S -x c - -o - | fgrep machine
          .machine power7
    
    which is too high for multiarch.
    
    Bug: https://sourceware.org/PR26522
    Bug: https://bugs.gentoo.org/737996
    Package-Manager: Portage-3.0.7, Repoman-3.0.1
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 sys-libs/glibc/glibc-2.31-r6.ebuild | 2 +-
 sys-libs/glibc/glibc-2.31-r7.ebuild | 2 +-
 sys-libs/glibc/glibc-2.32-r1.ebuild | 2 +-
 sys-libs/glibc/glibc-2.32.ebuild    | 2 +-
 sys-libs/glibc/glibc-9999.ebuild    | 2 +-
 5 files changed, 5 insertions(+), 5 deletions(-)