Bug 692986 – net-libs/grpc-1.23.0 - --grpc_out: protoc-gen-grpc: Plugin killed by signal 6.

• Product: Gentoo Linux
• Status: RESOLVED NEEDINFO

[Description jorgicio 2019-08-28 00:10:03 UTC]

Hi! I want to install tensorflow, which one of its dependencies is grpc.

However, this happens every time I want to build grpc:
``
x86_64-pc-linux-gnu-g++  -Wl,-O1 -Wl,--as-needed -g  -fPIC -Llibs/opt -pthread     /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/objs/opt/src/compiler/ruby_plugin.o /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/libs/opt/libgrpc_plugin_support.a  -lprotoc -lprotobuf -ldl -lrt -lm -lpthread -lz -lcares  -lprotoc -lprotobuf -o /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_ruby_plugin
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/channelz/channelz.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/health/v1/health.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo_messages.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/simple_messages.pb.cc`
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/channelz/channelz.proto
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/health/v1/health.proto
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/echo_messages.proto
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/simple_messages.proto
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/core/stats.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/payloads.pb.cc`
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/core/stats.proto
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/messages.pb.cc`
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/payloads.proto
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/messages.proto
third_party/protobuf/src: warning: directory does not exist.
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/status/status.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/reflection/v1alpha/reflection.pb.cc`
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/status/status.proto
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/reflection/v1alpha/reflection.proto
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/channelz/channelz.grpc.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/health/v1/health.grpc.pb.cc`
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo_messages.grpc.pb.cc`
protoc -Ithird_party/protobuf/src -I. --grpc_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens --plugin=protoc-gen-grpc=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_cpp_plugin src/proto/grpc/health/v1/health.proto
protoc -Ithird_party/protobuf/src -I. --grpc_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens --plugin=protoc-gen-grpc=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_cpp_plugin src/proto/grpc/channelz/channelz.proto
protoc -Ithird_party/protobuf/src -I. --grpc_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens --plugin=protoc-gen-grpc=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_cpp_plugin src/proto/grpc/testing/echo_messages.proto
mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo.pb.cc`
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
third_party/protobuf/src: warning: directory does not exist.
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
--grpc_out: protoc-gen-grpc: Plugin killed by signal 6.
make: *** [Makefile:2691: /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/health/v1/health.grpc.pb.cc] Error 1
make: *** Waiting for unfinished jobs....
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
--grpc_out: protoc-gen-grpc: Plugin killed by signal 6.
make: *** [Makefile:2837: /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo_messages.grpc.pb.cc] Error 1
protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/echo.proto
terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error -1
--grpc_out: protoc-gen-grpc: Plugin killed by signal 6.
make: *** [Makefile:2659: /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/channelz/channelz.grpc.pb.cc] Error 1
third_party/protobuf/src: warning: directory does not exist.
 * ERROR: net-libs/grpc-1.23.0::gentoo failed (compile phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=net-libs/grpc-1.23.0::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=net-libs/grpc-1.23.0::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/environment'.
 * Working directory: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0'
 * S: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0'

>>> Failed to emerge net-libs/grpc-1.23.0, Log file:

>>>  '/var/tmp/portage/net-libs/grpc-1.23.0/temp/build.log'

 * Messages for package net-libs/grpc-1.23.0:

 * ERROR: net-libs/grpc-1.23.0::gentoo failed (compile phase):
 *   emake failed
 * 
 * If you need support, post the output of `emerge --info '=net-libs/grpc-1.23.0::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=net-libs/grpc-1.23.0::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/environment'.
 * Working directory: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0'
 * S: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0'
``

Tried with previous versions and the error is the same.
Protobuf version installed is 3.8.0.

I'll attach the full log then.

Thanks.

Reproducible: Always

[Comment 1 jorgicio 2019-08-28 00:10:48 UTC]

Created attachment 588404 [details]
build.log

[Comment 2 jorgicio 2019-08-28 00:11:08 UTC]

Created attachment 588406 [details]
emerge --info

[Comment 3 Georgy Yakovlev 2019-09-01 07:58:09 UTC]

unfortunately I can't reproduce it on stable or unstable systems.

you have quite a lot overlays enabled, can you make sure that at least all the direct dependencies come from ::gentoo ?

also, do you have anything in LD_PRELOAD by any chance?

[Comment 4 jorgicio 2019-09-04 16:27:28 UTC]

(In reply to Georgy Yakovlev from comment #3)
> unfortunately I can't reproduce it on stable or unstable systems.
> 
> you have quite a lot overlays enabled, can you make sure that at least all
> the direct dependencies come from ::gentoo ?
> 
> also, do you have anything in LD_PRELOAD by any chance?

echo $LD_PRELOAD shows nothing and also all dependencies come from gentoo repo.

Also, I have openssl-1.1.1c-r1 installed. I don't know that's may be the cause.

[Comment 5 Georgy Yakovlev 2019-10-26 00:27:01 UTC]

i was able to reproduce similar crash in a low mem environment.

try setting MAKEOPTS="-j1 -l1" and see if it helps.

[Comment 6 jorgicio 2019-10-26 15:55:23 UTC]

(In reply to Georgy Yakovlev from comment #5)
> i was able to reproduce similar crash in a low mem environment.
> 
> try setting MAKEOPTS="-j1 -l1" and see if it helps.

Tried with MAKEOPTS="-j1" and MAKEOPTS="-j1 -l1" and failed in both cases.

1.24.3 didn't solve this issue.

[Comment 7 Bernd 2019-11-09 14:29:33 UTC]

Hi, I'm having a similar issue for a long time already (approx. 1 year, starting at around tensorflow-1.13, don't remember the grpc release at that time). Grpc always breaks with the same error, but it looks like it's not always from the same file(s). Over the time, I tried emerging several releases of tensorflow and they always stopped with such an issue from grpc.

Usually I build most packages using a tmpfs in RAM, and although I'm, strictly speaking, not on a low mem machine, I was following comment #5, and tried building grpc on disk and additionally using the MAKEOPTS from the comment, but it still fails with the described error. 

I also 'watch`ed `free` during the build and the memory got nowhere close to running out. At the time of the build failure, the `available` field in free output showed about 6.9Gi of free mem, the `free` field showed around 1.6Gi of free mem, whereas there were about 7.8Gi `used` out of 15Gi `total` mem.

I was now trying to emerge grpc-1.24.3 for tensorflow-2.0.0.

[Comment 8 Georgy Yakovlev 2020-10-09 22:54:39 UTC]

it it still happening?
we have  newer versions now and I haven't personally encountered this elusive bug.

the course of action if it still fails is try to catch the last command build system runs (it's now cmake based) and try to run it manually, under strace or even better gdb.

also try changing -march/-mtune flags, to lower model or something generic, or exact model. sometimes missing -march can actually miscompile code.

[Comment 9 Georgy Yakovlev 2021-12-20 05:14:49 UTC]

I'll close this for now, please reopen if more info becomes available.