I'm getting a segfault during the check_config() portion of the openrc init script on ppc64. I'm having some issues producing a useful backtrace, though. I'm building with -g -gdb -O0 and I have the feature nostrip when I build it, but I'm still not getting symbols: Program received signal SIGSEGV, Segmentation fault. 0x00003fffb7fda014 in ?? () from /lib64/ld64.so.1 (gdb) bt #0 0x00003fffb7fda014 in ?? () from /lib64/ld64.so.1 #1 0x00003fffb7922e30 in ?? () from /usr/lib64/libivykis.so.0 #2 0x00003fffb7fd3598 in ?? () from /lib64/ld64.so.1 #3 0x00003fffb7fd36ec in ?? () from /lib64/ld64.so.1 #4 0x00003fffb7fc1e3c in ?? () from /lib64/ld64.so.1 The lack of symbols is probably something wrong on my part, but a nudge in the right direction would be appreciated. I do wonder if this segfault has anything to do with the use after free fix that went into 3.19.1.
Please post your `emerge --info' output in a comment.
(In reply to Jeroen Roovers from comment #1) > Please post your `emerge --info' output in a comment. adam@g5box ~ $ emerge --info Portage 2.3.55 (python 3.6.6-final-0, default/linux/powerpc/ppc64/17.0/64bit-userland/desktop, gcc-8.2.0, glibc-2.28-r5, 4.10.0-gentoo ppc64) ================================================================= System uname: Linux-4.10.0-gentoo-ppc64-PPC970MP,_altivec_supported-with-gentoo-2.6 KiB Mem: 8469560 total, 6995520 free KiB Swap: 1048572 total, 1048572 free Timestamp of repository gentoo: Fri, 11 Jan 2019 23:00:01 +0000 Head commit of repository gentoo: d2d39f9ac079509a6e29098825f1e5e20ffc2db1 sh bash 4.4_p23 ld GNU ld (Gentoo 2.31.1 p5) 2.31.1 app-shells/bash: 4.4_p23::gentoo dev-lang/perl: 5.26.2::gentoo dev-lang/python: 2.7.15::gentoo, 3.6.6::gentoo dev-util/cmake: 3.13.2::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.40.3::gentoo sys-apps/sandbox: 2.15::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.13.4-r1::gentoo, 1.14.1-r1::gentoo, 1.15-r2::gentoo, 1.16.1-r1::gentoo sys-devel/binutils: 2.26.1::gentoo, 2.27::gentoo, 2.31.1-r3::gentoo sys-devel/gcc: 7.4.0::gentoo, 8.2.0-r6::gentoo sys-devel/gcc-config: 2.0::gentoo sys-devel/libtool: 2.4.6-r5::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 4.20::gentoo (virtual/os-headers) sys-libs/glibc: 2.28-r5::gentoo Repositories: gentoo location: /usr/portage sync-type: rsync sync-uri: rsync://rsync.gentoo.org/gentoo-portage priority: -1000 sync-rsync-extra-opts: sync-rsync-verify-max-age: 24 sync-rsync-verify-jobs: 1 sync-rsync-verify-metamanifest: yes localoverlay location: /usr/local/portage masters: gentoo haskell location: /var/lib/layman/haskell masters: gentoo priority: 50 ACCEPT_KEYWORDS="ppc64 ~ppc64" ACCEPT_LICENSE="* -@EULA" CBUILD="powerpc64-unknown-linux-gnu" CFLAGS="-O2 -pipe -mcpu=G5 -mabi=altivec -maltivec -mpowerpc-gpopt -mpowerpc-gfxopt" CHOST="powerpc64-unknown-linux-gnu" CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt" CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo" CXXFLAGS="-O2 -pipe -mcpu=G5 -mabi=altivec -maltivec -mpowerpc-gpopt -mpowerpc-gfxopt" DISTDIR="/usr/portage/distfiles" ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR" FCFLAGS="-O2 -pipe" FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr" FFLAGS="-O2 -pipe" GENTOO_MIRRORS="http://lug.mtu.edu/gentoo/ http://mirror.leaseweb.com/gentoo/ http://gentoo.osuosl.org/" LANG="en_US.utf8" LDFLAGS="-Wl,-O1 -Wl,--as-needed" LINGUAS="en en_US" MAKEOPTS="-j5" PKGDIR="/usr/portage/packages" PORTAGE_CONFIGROOT="/" PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git" PORTAGE_TMPDIR="/var/tmp" USE="X a52 aac acl alsa altivec berkdb big-endian bluetooth branding bzip2 cairo cdda cdr cli consolekit crypt cscope cups cxx dbus dri dts dvd dvdr emboss encode exif fam flac fortran gdbm gif glamor gpm gtk ibm iconv ipv6 jpeg lcms ldap libnotify mad mng mp3 mp4 mpeg ncurses nls nptl offensive ogg opengl openmp pam pango pcre pdf png policykit ppc64 ppds qt5 readline sdl seccomp spell ssl startup-notification svg tcpd threads tiff truetype udev udisks unicode upower usb vim-syntax vorbis wxwidgets x264 xattr xcb xml xv xvid zlib" ABI_PPC="64" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-1" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24" USERLAND="GNU" VIDEO_CARDS="nouveau" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account" Unset: CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, INSTALL_MASK, LC_ALL, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS This bug looks to have existed for a while now, all versions after 3.7.3 in portage do it, and I cannot build 3.7.3 again as it seems to be broken against newer openssl headers. Let me know how to build debug symbols with this and I'll see if I can produce a more useful backtrace.
So there is no version in portage that works for you?
(In reply to Tomáš Mózes from comment #3) > So there is no version in portage that works for you? Correct. I could possibly downgrade openSSL to a version that contains the headers which 3.7.3 will compile against (or patch 3.7.3 to leverage the newer interface), but I'd be happier if I could figure out which memory access is causing the segfault. 3.7.3 was working prior to me updating the world.
Have you also tried rebuilding ivykis with the debug options? Just guessing from your first comment. Maybe it's something with dev-libs/ivykis and not syslog-ng itself.
(In reply to Tomáš Mózes from comment #5) > Have you also tried rebuilding ivykis with the debug options? Just guessing > from your first comment. Maybe it's something with dev-libs/ivykis and not > syslog-ng itself. Ahh, hadn't realized that's a separate library in portage. Will do so as soon as I get some time.
FYI: (gdb) bt #0 0x00003fffb7fda014 in ?? () from /lib64/ld64.so.1 #1 0x00003fffb792d3c8 in pthr_atfork (prepare=@0x3fffb794b500: 0x3fffb792db30 <iv_signal_prepare>, parent=@0x3fffb794b518: 0x3fffb792dbd0 <iv_signal_parent>, child=@0x3fffb794b530: 0x3fffb792dc70 <iv_signal_child>) at pthr.h:76 #2 0x00003fffb792dd78 in iv_signal_init () at iv_signal.c:114 #3 0x00003fffb7fd3598 in ?? () from /lib64/ld64.so.1 #4 0x00003fffb7fd36ec in ?? () from /lib64/ld64.so.1 #5 0x00003fffb7fc1e3c in ?? () from /lib64/ld64.so.1 Investigating a little further: adam@g5box /tmp/ivykis-0.42.3/src $ gdb syslog-ng GNU gdb (Gentoo 8.2.1 p1) 8.2.1 Copyright (C) 2018 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "powerpc64-unknown-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from syslog-ng...done. (gdb) set directories . (gdb) r -s -f /etc/syslog-ng/syslog-ng.conf Starting program: /usr/sbin/syslog-ng -s -f /etc/syslog-ng/syslog-ng.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x00003fffb7fda014 in ?? () from /lib64/ld64.so.1 (gdb) up #1 0x00003fffb792d3c8 in pthr_atfork (prepare=@0x3fffb794b500: 0x3fffb792db30 <iv_signal_prepare>, parent=@0x3fffb794b518: 0x3fffb792dbd0 <iv_signal_parent>, child=@0x3fffb794b530: 0x3fffb792dc70 <iv_signal_child>) at pthr.h:76 76 return pthread_atfork(prepare, parent, child); (gdb) prepare Undefined command: "prepare". Try "help". (gdb) p prepare $1 = (void (*)(void)) @0x3fffb794b500: 0x3fffb792db30 <iv_signal_prepare> (gdb) p parent $2 = (void (*)(void)) @0x3fffb794b518: 0x3fffb792dbd0 <iv_signal_parent> (gdb) p child $3 = (void (*)(void)) @0x3fffb794b530: 0x3fffb792dc70 <iv_signal_child> All valid looking function pointers. Perhaps the function being called is the issue? Does this need to be explicitly linked to pthreads in order to have a function pthreads API? adam@g5box /tmp/ivykis-0.42.3/src $ ldd /usr/lib64/libivykis.so.0 linux-vdso64.so.1 (0x00003fff819a0000) libc.so.6 => /lib64/libc.so.6 (0x00003fff81747000) /lib64/ld64.so.1 (0x0000000049d5f000) The x86 binary I have here isn't linked, and explicitly linking it didn't help. Perhaps pthreads has an issue on ppc64 right now? I'm calling other things that leverage the pthreads API indirectly and haven't had any issues (I use TBB for an library I build and test on here). Tiny test case says there's nothing inherently wrong with that functionality from pthreads: #include <pthread.h> #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/wait.h> #include <unistd.h> void prepFork() { printf("Prepping the fork\n"); } void parentCleanup() { printf("Cleaning up stuff the fork did from the parent\n"); } void childCleanup() { printf("Cleaning up the process from the child's perspective\n"); } int main(void) { int retVal = pthread_atfork(prepFork, parentCleanup, childCleanup); pid_t childPid = fork(); int status; waitpid(childPid, &status, 0); return retVal; } adam@g5box /tmp $ ./test.out Prepping the fork Cleaning up stuff the fork did from the parent Cleaning up the process from the child's perspective adam@g5box /tmp $ echo $? 0 Chasing through the code a bit more, I noticed it calls pthread's spinlock quite a bit without an explicit unlock. I don't quite know if this is the case with the spinlock, but in my experience on other architectures, behavior is pretty undefined if you don't call unlock on pthread mutexes and rwlocks. Perhaps this is the real root of the issue?
Hmm, perhaps the fact that this is being called from a shared library that we have an issue... https://sourceware.org/bugzilla/show_bug.cgi?id=13502
Hmm, I think I've stumbled onto this same issue: https://github.com/buytenh/ivykis/issues/15
(In reply to Adam Stylinski from comment #9) > Hmm, I think I've stumbled onto this same issue: > > https://github.com/buytenh/ivykis/issues/15 So it would appear a newer version of ivykis would fix this, but the crux of the issue appears that the linker, when looking at the static archive for pthread, is noticing that it's marked as a weak symbol, so the function becomes a null reference. Basically glibc changed which static archive this was being placed in, as can be seen in this pull request: https://github.com/buytenh/ivykis/pull/16/commits/a844abca43349739c36a4cdb7ea6f3bce560bd7f
Created attachment 561168 [details, diff] Patch to ebuild to apply pthread patch for autoconf
Created attachment 561170 [details, diff] Patch to properly statically link in pthread_atfork()
Those patches will also search the old static archives if glibc is older, so that should work for everybody.
Thanks Adam for your detailed analysis and searching. Have you tested locally that with https://patch-diff.githubusercontent.com/raw/buytenh/ivykis/pull/16.patch it works for you correctly?
(In reply to Tomáš Mózes from comment #14) > Thanks Adam for your detailed analysis and searching. Have you tested > locally that with > https://patch-diff.githubusercontent.com/raw/buytenh/ivykis/pull/16.patch it > works for you correctly? Yes, it works now without issue.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=94497c9737e91347b528b31d5c4a60e649b8a45e commit 94497c9737e91347b528b31d5c4a60e649b8a45e Author: Tomas Mozes <hydrapolic@gmail.com> AuthorDate: 2019-01-15 06:33:52 +0000 Commit: Thomas Deutschmann <whissi@gentoo.org> CommitDate: 2019-01-19 03:21:45 +0000 dev-libs/ivykis: fix building with glibc-2.28 Closes: https://bugs.gentoo.org/675338 Package-Manager: Portage-2.3.55, Repoman-2.3.12 Signed-off-by: Tomáš Mózes <hydrapolic@gmail.com> Closes: https://github.com/gentoo/gentoo/pull/10836 Signed-off-by: Thomas Deutschmann <whissi@gentoo.org> .../files/ivykis-fix-segfault-glibc-2.28.patch | 29 ++++++++++++++++++++++ dev-libs/ivykis/ivykis-0.42.3-r1.ebuild | 12 +++++++++ 2 files changed, 41 insertions(+)
Looks like you committed the patch for ivykis. Marking as resolved.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=56b2616368f175312573acc9ad847930999ab293 commit 56b2616368f175312573acc9ad847930999ab293 Author: Tomas Mozes <hydrapolic@gmail.com> AuthorDate: 2019-05-17 10:17:43 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2019-05-21 08:54:49 +0000 dev-libs/ivykis: bump to 0.42.4 Bug: https://bugs.gentoo.org/675338 Package-Manager: Portage-2.3.66, Repoman-2.3.12 Signed-off-by: Tomáš Mózes <hydrapolic@gmail.com> Closes: https://github.com/gentoo/gentoo/pull/12028 Signed-off-by: Michał Górny <mgorny@gentoo.org> dev-libs/ivykis/Manifest | 1 + dev-libs/ivykis/ivykis-0.42.4.ebuild | 31 +++++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+)
