Bug 705574 – >=net-libs/grpc-1.25 - ServerBuilder::BuildAndStart hangs

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Yuriy Dmitriev 2020-01-16 16:36:31 UTC]

Please mask versions from [1.25 to 1.26] inclusive, which is now in portage.
Main branch few hours ago fixed.

There from comment user  rustyx commented on 18 Nov 2019.


There are some ODR violations in libgrpc++ (master branch); for example client_channel.cc is included in both libgrpc.so and libgrpc++.so.
That leads to the same grpc_core::TraceFlag grpc_client_channel_call_trace being constructed twice (notice this=0x7fffff7a07f0 in both traces below), creating a loop in the trace flag chain.

Global objects are merged during dynamic loading, but not their initializers, it's a well-known phenomenon (examples: 1, 2, 3, 4).

#1  0x00007fffff6e4e24 in grpc_core::TraceFlagList::Add (flag=0x7fffff7a07f0 <grpc_core::grpc_client_channel_call_trace>) at src/core/lib/debug/trace.cc:80
#2  0x00007fffff6e4f01 in grpc_core::TraceFlag::TraceFlag (this=0x7fffff7a07f0 <grpc_core::grpc_client_channel_call_trace>, default_enabled=false, name=0x7fffff288a06 "client_channel_call") at src/core/lib/debug/trace.cc:98
#3  0x00007fffff0816bc in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at src/core/ext/filters/client_channel/client_channel.cc:100
#4  0x00007fffff0816ed in _GLOBAL__sub_I_client_channel.cc(void) () at src/core/ext/filters/client_channel/client_channel.cc:4046
#5  0x00007fffff7cf37a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffffffedde8, env=env@entry=0x7ffffffeddf8) at dl-init.c:72
#6  0x00007fffff7cf476 in call_init (env=0x7ffffffeddf8, argv=0x7ffffffedde8, argc=1, l=<optimized out>) at dl-init.c:30
#7  _dl_init (main_map=0x7fffff7e9190, argc=1, argv=0x7ffffffedde8, env=0x7ffffffeddf8) at dl-init.c:119
#8  0x00007fffff7c10ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#9  0x0000000000000001 in ?? ()
#10 0x00007ffffffedfff in ?? ()
#11 0x0000000000000000 in ?? ()

(gdb) info sym 0x00007fffff0816ed
_GLOBAL__sub_I_client_channel.cc + 19 in section .text of /usr/local/lib/libgrpc.so.9

#1  0x00007fffff6e4e24 in grpc_core::TraceFlagList::Add (flag=0x7fffff7a07f0 <grpc_core::grpc_client_channel_call_trace>) at src/core/lib/debug/trace.cc:80
#2  0x00007fffff6e4f01 in grpc_core::TraceFlag::TraceFlag (this=0x7fffff7a07f0 <grpc_core::grpc_client_channel_call_trace>, default_enabled=false, name=0x7fffff6f5006 "client_channel_call") at src/core/lib/debug/trace.cc:98
#3  0x00007fffff64d19e in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at src/core/ext/filters/client_channel/client_channel.cc:100
#4  0x00007fffff64d1cf in _GLOBAL__sub_I_client_channel.cc(void) () at src/core/ext/filters/client_channel/client_channel.cc:4046
#5  0x00007fffff7cf37a in call_init (l=<optimized out>, argc=argc@entry=1, argv=argv@entry=0x7ffffffedde8, env=env@entry=0x7ffffffeddf8) at dl-init.c:72
#6  0x00007fffff7cf476 in call_init (env=0x7ffffffeddf8, argv=0x7ffffffedde8, argc=1, l=<optimized out>) at dl-init.c:30
#7  _dl_init (main_map=0x7fffff7e9190, argc=1, argv=0x7ffffffedde8, env=0x7ffffffeddf8) at dl-init.c:119
#8  0x00007fffff7c10ca in _dl_start_user () from /lib64/ld-linux-x86-64.so.2
#9  0x0000000000000001 in ?? ()
#10 0x00007ffffffedfff in ?? ()
#11 0x0000000000000000 in ?? ()

(gdb) info sym 0x00007fffff64d1cf
_GLOBAL__sub_I_client_channel.cc + 19 in section .text of /usr/local/lib/libgrpc++.so.1


I confirm, this bug exists.
Working version of grpc++ is 1.24.3 (no bug). Tested by me.


Bug explaned:   here https://stackoverflow.com/questions/26547454/static-variable-is-initialized-twice

Reproducible: Always

Steps to Reproduce:
1. Clone example from https://github.com/grpc/grpc/tree/master/examples/cpp/helloworld

2. Build & run via strace & gdb.
3. Program hang in file greeter_server.cc line 60.
code line  "std::unique_ptr<Server> server(builder.BuildAndStart());"

library call ==>  BuildAndStart()

Actual Results:  
Infinity loop, cpu 100%

Expected Results:  
Should nice work.

version net-libs/grpc-1.24.3 work as expected.

versions 1.25 and later do not work.

Please hard mask.

In main branch patch exist few hours. May be version 1.27 are fixed.

[Comment 1 Yuriy Dmitriev 2020-01-16 16:41:59 UTC]

How to fix.
Downgrade to =net-libs/grpc-1.24.3 (now in portage)

[Comment 2 Yuriy Dmitriev 2020-01-16 17:06:51 UTC]

Fix in main branch here. We are wait new release.

https://github.com/grpc/grpc/commit/5a419c65b02b069f4c92623678743a3ddddc53cb

[Comment 3 Georgy Yakovlev 2020-01-17 09:55:35 UTC]

thanks for detailed report!

was able to reproduce and confirm backport below working

backport to 1.26.0  here

https://github.com/grpc/grpc/commit/72351f63fd650cc7acfcd2d0307e8e8e8f777283

[Comment 4 Larry the Git Cow 2020-01-17 10:22:00 UTC]

The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f5caa3775e78ca9e2501ab818692f5c22c31e46f

commit f5caa3775e78ca9e2501ab818692f5c22c31e46f
Author:     Georgy Yakovlev <[e-mail]>
AuthorDate: 2020-01-17 10:18:03 +0000
Commit:     Georgy Yakovlev <[e-mail]>
CommitDate: 2020-01-17 10:20:12 +0000

    dev-python/grpcio: revbump 1.26.0, fix odr violation
    
    Closes: https://bugs.gentoo.org/705574
    Package-Manager: Portage-2.3.84, Repoman-2.3.20
    Signed-off-by: Georgy Yakovlev <[e-mail]>

 .../grpcio/files/grpc-1.26-fix-odr-violation.patch | 421 +++++++++++++++++++++
 ...rpcio-1.26.0.ebuild => grpcio-1.26.0-r1.ebuild} |   2 +
 2 files changed, 423 insertions(+)

Additionally, it has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=62ca973339e55b3034c925dccc5cdaf9173bd8f2

commit 62ca973339e55b3034c925dccc5cdaf9173bd8f2
Author:     Georgy Yakovlev <[e-mail]>
AuthorDate: 2020-01-17 10:08:58 +0000
Commit:     Georgy Yakovlev <[e-mail]>
CommitDate: 2020-01-17 10:20:12 +0000

    dev-python/grpcio-testing: drop 1.25.0
    
    Bug: https://bugs.gentoo.org/705574
    Package-Manager: Portage-2.3.84, Repoman-2.3.20
    Signed-off-by: Georgy Yakovlev <[e-mail]>

 dev-python/grpcio-testing/Manifest                 |  1 -
 .../grpcio-testing/grpcio-testing-1.25.0.ebuild    | 22 ----------------------
 2 files changed, 23 deletions(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=523db0e9a7e7ccfa05bb25e2d50f2065cb1b268a

commit 523db0e9a7e7ccfa05bb25e2d50f2065cb1b268a
Author:     Georgy Yakovlev <[e-mail]>
AuthorDate: 2020-01-17 10:08:05 +0000
Commit:     Georgy Yakovlev <[e-mail]>
CommitDate: 2020-01-17 10:20:11 +0000

    dev-python/grpcio-tools: drop 1.25.0
    
    Bug: https://bugs.gentoo.org/705574
    Package-Manager: Portage-2.3.84, Repoman-2.3.20
    Signed-off-by: Georgy Yakovlev <[e-mail]>

 dev-python/grpcio-tools/Manifest                   |  1 -
 dev-python/grpcio-tools/grpcio-tools-1.25.0.ebuild | 29 ----------------------
 2 files changed, 30 deletions(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=d67f103d16e8a6fcc13e5b00dbe781506d44b2a4

commit d67f103d16e8a6fcc13e5b00dbe781506d44b2a4
Author:     Georgy Yakovlev <[e-mail]>
AuthorDate: 2020-01-17 10:07:19 +0000
Commit:     Georgy Yakovlev <[e-mail]>
CommitDate: 2020-01-17 10:20:10 +0000

    dev-python/grpcio: drop 1.25.0
    
    Bug: https://bugs.gentoo.org/705574
    Package-Manager: Portage-2.3.84, Repoman-2.3.20
    Signed-off-by: Georgy Yakovlev <[e-mail]>

 dev-python/grpcio/Manifest             |  1 -
 dev-python/grpcio/grpcio-1.25.0.ebuild | 38 ----------------------------------
 2 files changed, 39 deletions(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=52050abe050a3164c33b77ed2ed630e58f75133e

commit 52050abe050a3164c33b77ed2ed630e58f75133e
Author:     Georgy Yakovlev <[e-mail]>
AuthorDate: 2020-01-17 10:06:15 +0000
Commit:     Georgy Yakovlev <[e-mail]>
CommitDate: 2020-01-17 10:20:10 +0000

    net-libs/grpc: drop 1.25.0
    
    Bug: https://bugs.gentoo.org/705574
    Package-Manager: Portage-2.3.84, Repoman-2.3.20
    Signed-off-by: Georgy Yakovlev <[e-mail]>

 net-libs/grpc/Manifest           |   1 -
 net-libs/grpc/grpc-1.25.0.ebuild | 112 ---------------------------------------
 2 files changed, 113 deletions(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=259e0aaf3b1456fb350a8aef6555cb9696b08d81

commit 259e0aaf3b1456fb350a8aef6555cb9696b08d81
Author:     Georgy Yakovlev <[e-mail]>
AuthorDate: 2020-01-17 10:02:30 +0000
Commit:     Georgy Yakovlev <[e-mail]>
CommitDate: 2020-01-17 10:20:09 +0000

    net-libs/grpc: revbump 1.26.0, fix odr violation
    
    Bug: https://bugs.gentoo.org/705574
    Package-Manager: Portage-2.3.84, Repoman-2.3.20
    Signed-off-by: Georgy Yakovlev <[e-mail]>

 net-libs/grpc/Manifest                                      | 1 +
 net-libs/grpc/{grpc-1.26.0.ebuild => grpc-1.26.0-r1.ebuild} | 6 ++++--
 2 files changed, 5 insertions(+), 2 deletions(-)

[Comment 5 Yuriy Dmitriev 2020-01-17 12:26:09 UTC]

Thanks alot, folks!!!

Seems like working. As minimum helloworld sample now working)))))


Confirm, =net-libs/grpc-1.26-r1 are working.


With best wishes - Yuriy.