Summary: | net-vpn/strongswan: General Protection Fault when started, crashes in sys-apps/systemd (fill_iovec_sprintf->malloc_sizeof_safe) | ||
---|---|---|---|
Product: | Gentoo Linux | Reporter: | Steve Moyes <gen2bugs> |
Component: | Current packages | Assignee: | Dennis Eisele <kernlpanic> |
Status: | UNCONFIRMED --- | ||
Severity: | critical | CC: | esigra, proxy-maint, rndxelement, systemd |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | AMD64 | ||
OS: | Linux | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 915000 |
Description
Steve Moyes
2023-10-31 07:21:39 UTC
Could you try get a backtrace via coredumpctl? Thanks. (May need to follow https://wiki.gentoo.org/wiki/Debugging#Per-package to get more useful output.0 >!../../usr/portage/profiles/default/linux/amd64/17.1/no-multilib/systemd/merged-usr
By the way, you should fix your profile, as your repo now seems to be in /var/db/repos/gentoo.
(In reply to Sam James from comment #1) > Could you try get a backtrace via coredumpctl? Thanks. > > (May need to follow https://wiki.gentoo.org/wiki/Debugging#Per-package to > get more useful output.0 # coredumpctl gdb PID: 2461753 (charon-systemd) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Tue 2023-10-31 18:16:23 GMT (5min ago) Command Line: /usr/sbin/charon-systemd Executable: /usr/bin/charon-systemd Control Group: /user.slice/user-1001.slice/user@1004.service/app.slice/app-org.gnome.Terminal.slice/vte-spawn-8e0bdda3-8a5f-45a7-9e71-8cd0780d2915.scope Unit: user@1001.service User Unit: vte-spawn-8e0bdda3-8a5f-45a7-9e71-8cd0780d2915.scope Slice: user-1001.slice Owner UID: 1001 (me) Boot ID: c05309a31de341309ab0e80026a58827 Machine ID: 0d72cbc666e91c5fd93656c956a23fb0 Hostname: test.example.com Storage: /var/lib/systemd/coredump/core.charon-systemd.0.c05309a31de341309ab0e80026a58827.2461753.1698776183000000.zst (present) Size on Disk: 113.5K Message: Process 2461753 (charon-systemd) of user 0 dumped core. GNU gdb (Gentoo 13.2 vanilla) 13.2 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/charon-systemd... (No debugging symbols found in /usr/bin/charon-systemd) [New LWP 2461753] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/charon-systemd'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f3247e25ac9 in malloc_usable_size () from /usr/lib64/libc.so.6 In gdb, run "bt"? (In reply to Sam James from comment #4) > In gdb, run "bt"? Reading symbols from /usr/sbin/charon-systemd... (No debugging symbols found in /usr/sbin/charon-systemd) (gdb) run Starting program: /usr/bin/charon-systemd [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib64/libthread_db.so.1". 00[DMN] Starting charon-systemd IKE daemon (strongSwan 5.9.11, Linux 6.5.9-gentoo, x86_64) Program received signal SIGSEGV, Segmentation fault. 0x00007ffff7c4bac9 in malloc_usable_size () from /usr/lib64/libc.so.6 You should be able to run "bt" once it segfaults to get an untruncated backtrace (In reply to Sam James from comment #6) > You should be able to run "bt" once it segfaults to get an untruncated > backtrace Sorry I pasted the wrong text.. that would make sense :facepalm: #0 0x00007f3247e25ac9 in malloc_usable_size () from /usr/lib64/libc.so.6 #1 0x00007f3247fee565 in ?? () from /usr/lib64/libsystemd.so.0 #2 0x00007f3247fccd99 in ?? () from /usr/lib64/libsystemd.so.0 #3 0x00007f3247fcd889 in sd_journal_send () from /usr/lib64/libsystemd.so.0 #4 0x0000563f35d5dbf7 in ?? () #5 0x00007f3248095033 in _cb_vlog_cb (entry=<optimized out>, args=<optimized out>) at bus/bus.c:358 #6 0x00007f324814008e in invoke_function (this=<optimized out>, fn=0x7f3248094f80 <_cb_vlog_cb>) at collections/linked_list.c:443 #7 0x00007f32480953aa in vlog (this=0x563f36a5b1d0, group=<optimized out>, level=LEVEL_CTRL, format=0x563f35d5e248 "Starting charon-systemd IKE daemon (strongSwan 5.9.11, %s %s, %s)", args=args@entry=0x7ffc94d1ed10) at bus/bus.c:427 #8 0x00007f32480954aa in log_ (this=<optimized out>, group=<optimized out>, level=<optimized out>, format=<optimized out>) at bus/bus.c:440 #9 0x0000563f35d5d44a in ?? () #10 0x00007f3247db478a in ?? () from /usr/lib64/libc.so.6 #11 0x00007f3247db4845 in __libc_start_main () from /usr/lib64/libc.so.6 #12 0x0000563f35d5d7e1 in ?? () No worries! Could you build systemd + glibc with debug symbols too like you did for strongswan then get a new backtrace? That place it's dying is quite suspicious... (In reply to Sam James from comment #8) > No worries! > > Could you build systemd + glibc with debug symbols too like you did for > strongswan then get a new backtrace? > > That place it's dying is quite suspicious... I don't get anything. It just comes back with "no stack"? (In reply to Steve Moyes from comment #9) > (In reply to Sam James from comment #8) > > No worries! > > > > Could you build systemd + glibc with debug symbols too like you did for > > strongswan then get a new backtrace? > > > > That place it's dying is quite suspicious... > > I don't get anything. It just comes back with "no stack"? Ignore my last comment :) # coredumpctl gdb PID: 3317096 (charon-systemd) UID: 0 (root) GID: 0 (root) Signal: 11 (SEGV) Timestamp: Wed 2023-11-01 06:09:11 GMT (14s ago) Command Line: /usr/sbin/charon-systemd Executable: /usr/bin/charon-systemd Control Group: /system.slice/strongswan.service Unit: strongswan.service Slice: system.slice Boot ID: c05309a31de341309ab0e80026a58827 Machine ID: 0d72cbc666e91c5fd93656c956a23fb0 Hostname: mail.cdstealer.com Storage: /var/lib/systemd/coredump/core.charon-systemd.0.c05309a31de341309ab0e80026a58827.3317096.1698818951000000.zst (present) Size on Disk: 111.8K Message: Process 3317096 (charon-systemd) of user 0 dumped core. GNU gdb (Gentoo 13.2 vanilla) 13.2 Copyright (C) 2023 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-pc-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <https://bugs.gentoo.org/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/bin/charon-systemd... (No debugging symbols found in /usr/bin/charon-systemd) [New LWP 3317096] [Thread debugging using libthread_db enabled] Using host libthread_db library "/usr/lib64/libthread_db.so.1". Core was generated by `/usr/sbin/charon-systemd'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f541f825ac9 in malloc_usable_size () from /usr/lib64/libc.so.6 (gdb) bt #0 0x00007f541f825ac9 in malloc_usable_size () from /usr/lib64/libc.so.6 #1 0x00007f541f9ee565 in malloc_sizeof_safe (xp=0x7ffc5b7c3c88) at ../systemd-stable-254.5/src/basic/alloc-util.h:204 #2 greedy_realloc (p=p@entry=0x7ffc5b7c3c88, need=need@entry=2, size=size@entry=16) at ../systemd-stable-254.5/src/basic/alloc-util.c:56 #3 0x00007f541f9ccd99 in fill_iovec_sprintf (format=<optimized out>, ap=ap@entry=0x7ffc5b7c3f10, extra=extra@entry=0, ret_iov=ret_iov@entry=0x7ffc5b7c3f00, ret_n_iov=ret_n_iov@entry=0x7ffc5b7c3f08) at ../systemd-stable-254.5/src/libsystemd/sd-journal/journal-send.c:189 #4 0x00007f541f9cd889 in sd_journal_send (format=<optimized out>) at ../systemd-stable-254.5/src/libsystemd/sd-journal/journal-send.c:210 #5 0x000055905f78ebf7 in ?? () #6 0x00007f541fa95033 in _cb_vlog_cb (entry=<optimized out>, args=<optimized out>) at bus/bus.c:358 #7 0x00007f541fb4008e in invoke_function (this=<optimized out>, fn=0x7f541fa94f80 <_cb_vlog_cb>) at collections/linked_list.c:443 #8 0x00007f541fa953aa in vlog (this=0x5590614fa1d0, group=<optimized out>, level=LEVEL_CTRL, format=0x55905f78f248 "Starting charon-systemd IKE daemon (strongSwan 5.9.11, %s %s, %s)", args=args@entry=0x7ffc5b7c5800) at bus/bus.c:427 #9 0x00007f541fa954aa in log_ (this=<optimized out>, group=<optimized out>, level=<optimized out>, format=<optimized out>) at bus/bus.c:440 #10 0x000055905f78e44a in ?? () #11 0x00007f541f7b478a in ?? () from /usr/lib64/libc.so.6 #12 0x00007f541f7b4845 in __libc_start_main () from /usr/lib64/libc.so.6 #13 0x000055905f78e7e1 in ?? () Thanks. Did you definitely build glibc with debug symbols + installsource too? (In reply to Sam James from comment #11) > Thanks. Did you definitely build glibc with debug symbols + installsource > too? Yup, followed the process as instructed. (In reply to Steve Moyes from comment #12) > (In reply to Sam James from comment #11) > > Thanks. Did you definitely build glibc with debug symbols + installsource > > too? > > Yup, followed the process as instructed. Sorry, back home now. # cat /etc/portage/package.env/glibc sys-libs/glibc debugsyms installsources # cat /etc/portage/package.env/systemd sys-apps/systemd debugsyms # cat /etc/portage/package.env/strongswan net-vpn/strongswan debugsyms # cat /etc/portage/env/debugsyms CFLAGS="${CFLAGS} -ggdb3" CXXFLAGS="${CXXFLAGS} -ggdb3" # nostrip is disabled here because it negates splitdebug FEATURES="${FEATURES} splitdebug compressdebug -nostrip" It dying in malloc_sizeof_safe is suspicious given some of the shenanigans in the past there... It's possible strongswan is just passing an invalid format string or similar. The message is pretty new, added in https://github.com/strongswan/strongswan/commit/d12e6c9e551587099e81bcb32b2df9abc1040216. FWIW, if I just build and start strongswan with 0 configuration, I do see that log message w/ no crash. (In reply to Sam James from comment #16) > The message is pretty new, added in > https://github.com/strongswan/strongswan/commit/ > d12e6c9e551587099e81bcb32b2df9abc1040216. > > FWIW, if I just build and start strongswan with 0 configuration, I do see > that log message w/ no crash. yeah. I found a VM that I hadn't updated in a few months, installed strongswan and worked just fine. I guess it could be possible that my glib libraries are too new? One or two of those packages are blocked from downgrading. I've even done an emerge -qe @system && emerge -qe @world. I'll keep digging. Hi, I found this command which produced this output. # ipsec start --attach-gdb Starting strongSwan 5.9.11 IPsec [starter]... can't execv(/usr/bin/gdb,...): No such file or directory 24 bytes total, 1 allocations, 24 bytes average: dumping 7 stack frame addresses: /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__tsearch+0x14a) [0x7ff521310e2a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff5212496ac] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee9c1f0] -> ??:0 /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee99335] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 16 bytes total, 1 allocations, 16 bytes average: dumping 6 stack frame addresses: /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52124963f] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee9c1f0] -> ??:0 /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee99335] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 512 bytes total, 1 allocations, 512 bytes average: dumping 6 stack frame addresses: /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff521249557] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee9c1f0] -> ??:0 /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee99335] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 56 bytes total, 1 allocations, 56 bytes average: dumping 5 stack frame addresses: /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee9a912] -> ??:0 /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee98bd2] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 24 bytes total, 1 allocations, 24 bytes average: dumping 6 stack frame addresses: /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff5212859d8] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__asprintf_chk+0x9f) [0x7ff52131ee8f] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee98778] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 16 bytes total, 1 allocations, 16 bytes average: dumping 6 stack frame addresses: /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff5212859d8] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__asprintf_chk+0x9f) [0x7ff52131ee8f] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee98745] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 26 bytes total, 1 allocations, 26 bytes average: dumping 6 stack frame addresses: /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff5212859d8] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__asprintf_chk+0x9f) [0x7ff52131ee8f] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee98712] -> ??:0 /usr/lib64/libc.so.6 @ 0x7ff52120b000 [0x7ff52122ec8a] -> ??:? /usr/lib64/libc.so.6 @ 0x7ff52120b000 (__libc_start_main+0x85) [0x7ff52122ed45] -> ??:? /usr/libexec/ipsec/starter @ 0x56407ee94000 [0x56407ee993f1] -> ??:0 7 leaks detected, 674 bytes, 3 suppressed by whitelist charon has died -- restart scheduled (5sec) oh interesting.. I booted the VM and the exact same thing is happening after upgrading to gcc 13. I'll downgrade to 12 and see what happens. So I tracked down a glibc-2.37.r3 ebuild (last working version) and downgraded. No change, so I rolled back to the last working version of systemd (systemd-253.6) and boom, strongswan starts again. updated glibc back to version 2.37-r7 and strongswan still works. :) Thanks for you time and patience. Can reproduce this with --enable-leak-detective (via USE=debug), but runs okay without it even with latest glibc and systemd. I had this issue for another package also that hooks their own realloc and malloc. |