Hi! I want to install tensorflow, which one of its dependencies is grpc. However, this happens every time I want to build grpc: `` x86_64-pc-linux-gnu-g++ -Wl,-O1 -Wl,--as-needed -g -fPIC -Llibs/opt -pthread /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/objs/opt/src/compiler/ruby_plugin.o /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/libs/opt/libgrpc_plugin_support.a -lprotoc -lprotobuf -ldl -lrt -lm -lpthread -lz -lcares -lprotoc -lprotobuf -o /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_ruby_plugin mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/channelz/channelz.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/health/v1/health.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo_messages.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/simple_messages.pb.cc` protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/channelz/channelz.proto protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/health/v1/health.proto protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/echo_messages.proto protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/simple_messages.proto third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/core/stats.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/payloads.pb.cc` protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/core/stats.proto mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/messages.pb.cc` protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/payloads.proto third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/messages.proto third_party/protobuf/src: warning: directory does not exist. mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/status/status.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/reflection/v1alpha/reflection.pb.cc` protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/status/status.proto protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/reflection/v1alpha/reflection.proto third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/channelz/channelz.grpc.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/health/v1/health.grpc.pb.cc` mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo_messages.grpc.pb.cc` protoc -Ithird_party/protobuf/src -I. --grpc_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens --plugin=protoc-gen-grpc=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_cpp_plugin src/proto/grpc/health/v1/health.proto protoc -Ithird_party/protobuf/src -I. --grpc_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens --plugin=protoc-gen-grpc=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_cpp_plugin src/proto/grpc/channelz/channelz.proto protoc -Ithird_party/protobuf/src -I. --grpc_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens --plugin=protoc-gen-grpc=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/bins/opt/grpc_cpp_plugin src/proto/grpc/testing/echo_messages.proto mkdir -p `dirname /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo.pb.cc` third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. third_party/protobuf/src: warning: directory does not exist. terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 --grpc_out: protoc-gen-grpc: Plugin killed by signal 6. make: *** [Makefile:2691: /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/health/v1/health.grpc.pb.cc] Error 1 make: *** Waiting for unfinished jobs.... terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 --grpc_out: protoc-gen-grpc: Plugin killed by signal 6. make: *** [Makefile:2837: /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/testing/echo_messages.grpc.pb.cc] Error 1 protoc -Ithird_party/protobuf/src -I. --cpp_out=/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens src/proto/grpc/testing/echo.proto terminate called after throwing an instance of 'std::system_error' what(): Unknown error -1 --grpc_out: protoc-gen-grpc: Plugin killed by signal 6. make: *** [Makefile:2659: /var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0/gens/src/proto/grpc/channelz/channelz.grpc.pb.cc] Error 1 third_party/protobuf/src: warning: directory does not exist. * ERROR: net-libs/grpc-1.23.0::gentoo failed (compile phase): * emake failed * * If you need support, post the output of `emerge --info '=net-libs/grpc-1.23.0::gentoo'`, * the complete build log and the output of `emerge -pqv '=net-libs/grpc-1.23.0::gentoo'`. * The complete build log is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/environment'. * Working directory: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0' * S: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0' >>> Failed to emerge net-libs/grpc-1.23.0, Log file: >>> '/var/tmp/portage/net-libs/grpc-1.23.0/temp/build.log' * Messages for package net-libs/grpc-1.23.0: * ERROR: net-libs/grpc-1.23.0::gentoo failed (compile phase): * emake failed * * If you need support, post the output of `emerge --info '=net-libs/grpc-1.23.0::gentoo'`, * the complete build log and the output of `emerge -pqv '=net-libs/grpc-1.23.0::gentoo'`. * The complete build log is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/build.log'. * The ebuild environment file is located at '/var/tmp/portage/net-libs/grpc-1.23.0/temp/environment'. * Working directory: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0' * S: '/var/tmp/portage/net-libs/grpc-1.23.0/work/grpc-1.23.0' `` Tried with previous versions and the error is the same. Protobuf version installed is 3.8.0. I'll attach the full log then. Thanks. Reproducible: Always
Created attachment 588404 [details] build.log
Created attachment 588406 [details] emerge --info
unfortunately I can't reproduce it on stable or unstable systems. you have quite a lot overlays enabled, can you make sure that at least all the direct dependencies come from ::gentoo ? also, do you have anything in LD_PRELOAD by any chance?
(In reply to Georgy Yakovlev from comment #3) > unfortunately I can't reproduce it on stable or unstable systems. > > you have quite a lot overlays enabled, can you make sure that at least all > the direct dependencies come from ::gentoo ? > > also, do you have anything in LD_PRELOAD by any chance? echo $LD_PRELOAD shows nothing and also all dependencies come from gentoo repo. Also, I have openssl-1.1.1c-r1 installed. I don't know that's may be the cause.
i was able to reproduce similar crash in a low mem environment. try setting MAKEOPTS="-j1 -l1" and see if it helps.
(In reply to Georgy Yakovlev from comment #5) > i was able to reproduce similar crash in a low mem environment. > > try setting MAKEOPTS="-j1 -l1" and see if it helps. Tried with MAKEOPTS="-j1" and MAKEOPTS="-j1 -l1" and failed in both cases. 1.24.3 didn't solve this issue.
Hi, I'm having a similar issue for a long time already (approx. 1 year, starting at around tensorflow-1.13, don't remember the grpc release at that time). Grpc always breaks with the same error, but it looks like it's not always from the same file(s). Over the time, I tried emerging several releases of tensorflow and they always stopped with such an issue from grpc. Usually I build most packages using a tmpfs in RAM, and although I'm, strictly speaking, not on a low mem machine, I was following comment #5, and tried building grpc on disk and additionally using the MAKEOPTS from the comment, but it still fails with the described error. I also 'watch`ed `free` during the build and the memory got nowhere close to running out. At the time of the build failure, the `available` field in free output showed about 6.9Gi of free mem, the `free` field showed around 1.6Gi of free mem, whereas there were about 7.8Gi `used` out of 15Gi `total` mem. I was now trying to emerge grpc-1.24.3 for tensorflow-2.0.0.
it it still happening? we have newer versions now and I haven't personally encountered this elusive bug. the course of action if it still fails is try to catch the last command build system runs (it's now cmake based) and try to run it manually, under strace or even better gdb. also try changing -march/-mtune flags, to lower model or something generic, or exact model. sometimes missing -march can actually miscompile code.
I'll close this for now, please reopen if more info becomes available.
