Bug 725334 – net-misc/networkmanager-1.22.10-r11: segfault when connecting to openvswitch database

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Stijn Tintel 2020-05-25 13:32:18 UTC]

After upgrading my system over the weekend, NetworkManager is segfaulting. This happens when connecting to the openvswitch database.

An additional problem is that NetworkManager seems to look for the openvswitch database socket in /var/lib/run/openvswitch/db.sock. This can be worked around with symlinks, and I have been using NetworkManager 1.22.10 for about a month without issues, until my upgrade over the weekend.

I initially installed NM 1.22.10-r10 on 23/04:
2020-04-23T14:27:21 >>> net-misc/networkmanager-1.22.10-r10: 1 minute, 22 seconds

On 6/05 NM 1.22.10-r11 was installed:
2020-05-06T13:26:39 >>> net-misc/networkmanager-1.22.10-r11: 1 minute, 7 seconds

I've updated openvswitch to 2.13.0 on 23/04:
2020-04-23T23:27:53 >>> net-misc/openvswitch-2.13.0: 1 minute, 39 seconds

So from 23/04 until rebooting my machine earlier today, this combination was working without issue.

As the segfault happens in json-c, and json-c was upgraded this weekend, I tried downgrading dev-libs/json-c-0.14-r3 to dev-libs/json-c-0.13.1-r1. Afterwards I rebuilt NetworkManager. Unfortunately this does not help.

Reproducible: Always

Steps to Reproduce:
1. USE=ovs emerge networkmanager
2. emerge --config openvswitch
3. start ovsdb-server
4. mkdir /var/lib/run/openvswitch/
5. cd /var/lib/run/openvswitch/
6. ln -s /var/run/openvswitch/db.sock
7. start NetworkManager
Actual Results:  
NetworkManager segfaults

Expected Results:  
NetworkManager should not segfault

GNU gdb (Gentoo 9.1 vanilla) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://bugs.gentoo.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/NetworkManager...
Reading symbols from /usr/lib/debug//usr/sbin/NetworkManager.debug...
[New LWP 365000]
[New LWP 365002]
[New LWP 365001]
[New LWP 365003]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `/usr/sbin/NetworkManager --no-daemon'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f70a0691533 in json_object_iter_next (iter=iter@entry=0x56433cdf0470) at /var/tmp/portage/dev-libs/json-c-0.14-r3/work/json-c-0.14/json_object_iterator.c:100
100     /var/tmp/portage/dev-libs/json-c-0.14-r3/work/json-c-0.14/json_object_iterator.c: No such file or directory.
[Current thread is 1 (Thread 0x7f70a054efc0 (LWP 365000))]
(gdb) thread apply all bt

Thread 4 (Thread 0x7f709f0ef700 (LWP 365003)):
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00007f70a183b2f2 in g_cond_wait_until (cond=cond@entry=0x56433ccc2418, mutex=mutex@entry=0x56433ccc2410, end_time=end_time@entry=6592140801) at ../glib-2.62.6/glib/gthread-posix.c:1470
#2  0x00007f70a17c1501 in g_async_queue_pop_intern_unlocked (queue=0x56433ccc2410, wait=wait@entry=1, end_time=6592140801) at ../glib-2.62.6/glib/gasyncqueue.c:422
#3  0x00007f70a17c1b6d in g_async_queue_timeout_pop_unlocked (queue=<optimized out>, timeout=timeout@entry=500000) at ../glib-2.62.6/glib/gasyncqueue.c:574
#4  0x00007f70a1819472 in g_thread_pool_wait_for_new_task (pool=<optimized out>) at ../glib-2.62.6/glib/gthreadpool.c:263
#5  g_thread_pool_thread_proxy (data=<optimized out>) at ../glib-2.62.6/glib/gthreadpool.c:297
#6  0x00007f70a1818c3d in g_thread_proxy (data=0x56433ccc1b20) at ../glib-2.62.6/glib/gthread.c:805
#7  0x00007f70a13f7e97 in start_thread (arg=<optimized out>) at pthread_create.c:479
#8  0x00007f70a132f6ff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7f70a00f1700 (LWP 365001)):
#0  0x00007f70a13250ff in __GI___poll (fds=0x56433ccbdaf0, nfds=2, timeout=4400) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f70a17efe4e in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x56433ccbdaf0, timeout=<optimized out>, context=0x56433ccc0690) at ../glib-2.62.6/glib/gmain.c:4253
#2  g_main_context_iterate (context=context@entry=0x56433ccc0690, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib-2.62.6/glib/gmain.c:3949
#3  0x00007f70a17eff6f in g_main_context_iteration (context=0x56433ccc0690, may_block=may_block@entry=1) at ../glib-2.62.6/glib/gmain.c:4015
#4  0x00007f70a17effc1 in glib_worker_main (data=<optimized out>) at ../glib-2.62.6/glib/gmain.c:5895
#5  0x00007f70a1818c3d in g_thread_proxy (data=0x56433ccc1800) at ../glib-2.62.6/glib/gthread.c:805
#6  0x00007f70a13f7e97 in start_thread (arg=<optimized out>) at pthread_create.c:479
#7  0x00007f70a132f6ff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7f709f8f0700 (LWP 365002)):
#0  0x00007f70a13250ff in __GI___poll (fds=0x56433ccd74c0, nfds=2, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007f70a17efe4e in g_main_context_poll (priority=<optimized out>, n_fds=2, fds=0x56433ccd74c0, timeout=<optimized out>, context=0x56433ccd45e0) at ../glib-2.62.6/glib/gmain.c:4253
#2  g_main_context_iterate (context=0x56433ccd45e0, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../glib-2.62.6/glib/gmain.c:3949
#3  0x00007f70a17f01c3 in g_main_loop_run (loop=0x56433ccd46d0) at ../glib-2.62.6/glib/gmain.c:4148
#4  0x00007f70a1a28b76 in gdbus_shared_thread_func (user_data=0x56433ccd45b0) at ../glib-2.62.6/gio/gdbusprivate.c:279
#5  0x00007f70a1818c3d in g_thread_proxy (data=0x56433ccc18a0) at ../glib-2.62.6/glib/gthread.c:805
#6  0x00007f70a13f7e97 in start_thread (arg=<optimized out>) at pthread_create.c:479
#7  0x00007f70a132f6ff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7f70a054efc0 (LWP 365000)):
#0  0x00007f70a0691533 in json_object_iter_next (iter=iter@entry=0x56433cdf0470) at /var/tmp/portage/dev-libs/json-c-0.14-r3/work/json-c-0.14/json_object_iterator.c:100
#1  0x00007f709e8d0bd6 in do_dump (json=0x56433cdf0470, flags=0, depth=0, parents=0x7ffcabcb7850, dump=0x7f709e8d0550 <dump_to_strbuffer>, data=0x7ffcabcb78c0) at /var/tmp/portage/dev-libs/jansson-2.11/work/jansson-2.11/src/dump.c:400
#2  0x00007f709e8d128c in json_dump_callback (json=json@entry=0x56433cdf0470, callback=callback@entry=0x7f709e8d0550 <dump_to_strbuffer>, data=data@entry=0x7ffcabcb78c0, flags=flags@entry=0) at /var/tmp/portage/dev-libs/jansson-2.11/work/jansson-2.11/src/dump.c:500
#3  0x00007f709e8d1318 in json_dumps (json=json@entry=0x56433cdf0470, flags=flags@entry=0) at /var/tmp/portage/dev-libs/jansson-2.11/work/jansson-2.11/src/dump.c:443
#4  0x00007f709e8e79cb in ovsdb_next_command (self=self@entry=0x56433cd265f0 [NMOvsdb]) at ../NetworkManager-1.22.10/src/devices/ovs/nm-ovsdb.c:827
#5  0x00007f709e8e7e67 in _client_connect_cb (source_object=<optimized out>, res=<optimized out>, user_data=0x56433cd265f0) at ../NetworkManager-1.22.10/src/devices/ovs/nm-ovsdb.c:1433
#6  0x00007f70a19c5153 in g_task_return_now (task=0x7f7090003e40 [GTask]) at ../glib-2.62.6/gio/gtask.c:1212
#7  0x00007f70a19c5afd in g_task_return (task=0x7f7090003e40 [GTask], type=<optimized out>) at ../glib-2.62.6/gio/gtask.c:1281
#8  0x00007f70a19c611c in g_task_return (type=G_TASK_RETURN_SUCCESS, task=<optimized out>) at ../glib-2.62.6/gio/gtask.c:1684
#9  g_task_return_pointer (task=<optimized out>, result=<optimized out>, result_destroy=<optimized out>) at ../glib-2.62.6/gio/gtask.c:1689
#10 0x0000000000000000 in  ()

[Comment 1 Stijn Tintel 2020-05-25 15:09:08 UTC]

Thanks to thaller and bengal for helping me figure this out.

Looks like the problem is caused by util-linux. It was upgraded from 2.33.2 to 2.35.1-r2 during my upgrade this weekend. The latter seems to link against libcryptsetup, which in turn links to libjson-c.

NM uses jansson instead of json-c, but both libraries have a symbol clash and neither use symbol versioning. So NM now tries to use json-c instead of jansson to talk to ovsdb-server, but the libraries are not compatible and hence the crash.

As I have cryptsetup as a global USE flag, the new version of util-linux was built with support for this.

From the 2.35 release notes:

mount(8) and libmount now provides built-in dm-verity support if linked with
libcryptsetup. This new feature is EXPERIMENTAL and disabled by default; use
--with-cryptsetup to enable.

So I am going to disable this for now. Maybe the NetworkManager ebuild can be changed to depend on util-linux[-cryptsetup]?

[Comment 2 Stijn Tintel 2020-05-25 15:58:43 UTC]

Reported upstream: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/issues/453

[Comment 3 Matt Turner 2021-08-22 19:29:30 UTC]

Looks like this is now fixed.