Bug 683928 – sci-libs/tensorflow-1.13.1 with dev-libs/protobuf-3.7.1 - In file included from ???: /usr/include/google/bigtable/v2/bigtable.pb.h:1869:30: error: 'google::protobuf::uint8* google::bigtable::v2::CheckAndMutateRowResponse::InternalSerializeWithCachedSizesT

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Harris Landgarten 2019-04-21 02:18:38 UTC]

Tensorflow is broken at runtime after the upgrade to dev-libs/protobuf-3.7.1 and dev-python/python-protobuf-3.7.1.

When attempting to rebuild tensorflow-1.13.1 with the upgraded protobufs the build fails with: 

/usr/include/google/bigtable/v2/bigtable.pb.h:1869:30: error: 'google::protobuf::uint8* google::bigtable::v2::CheckAndMutateRowResponse::InternalSerializeWithCachedSizesToArray(bool, google::protobuf::uint8*) const' marked 'final', but is not virtual
   ::google::protobuf::uint8* InternalSerializeWithCachedSizesToArray(
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/google/bigtable/v2/bigtable.pb.h:1972:30: error: 'google::protobuf::uint8* google::bigtable::v2::ReadModifyWriteRowRequest::InternalSerializeWithCachedSizesToArray(bool, google::protobuf::uint8*) const' marked 'final', but is not virtual
   ::google::protobuf::uint8* InternalSerializeWithCachedSizesToArray(
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/google/bigtable/v2/bigtable.pb.h:2126:30: error: 'google::protobuf::uint8* google::bigtable::v2::ReadModifyWriteRowResponse::InternalSerializeWithCachedSizesToArray(bool, google::protobuf::uint8*) const' marked 'final', but is not virtual
   ::google::protobuf::uint8* InternalSerializeWithCachedSizesToArray(
                              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/google/bigtable/v2/bigtable.pb.h: In member function 'void google::bigtable::v2::ReadRowsRequest::clear_rows_limit()':
/usr/include/google/bigtable/v2/bigtable.pb.h:2385:17: error: 'GOOGLE_LONGLONG' was not declared in this scope
   rows_limit_ = GOOGLE_LONGLONG(0);
                 ^~~~~~~~~~~~~~~
/usr/include/google/bigtable/v2/bigtable.pb.h:2385:17: note: suggested alternative: 'GOOGLE_DLOG'
   rows_limit_ = GOOGLE_LONGLONG(0);
              ^~~~~~~~~~~~~~~

Downgrading protobuf to 3.6.1.3 fixes the runtime breakage

Note build file is way too big to upload.

[Comment 1 Arfrever Frehtes Taifersar Arahesis 2019-04-21 05:38:40 UTC]

Which package is owner of /usr/include/google/bigtable/v2/bigtable.pb.h?

Create build log with MAKEOPTS="-j1" and compress this build log (`xz -9`).
If even compressed build log was too big, attach some last part of build log (e.g. `tail -n10000`).

[Comment 2 Jeroen Roovers (RETIRED) 2019-04-21 09:40:24 UTC]

(In reply to Arfrever Frehtes Taifersar Arahesis from comment #1)
> Which package is owner of /usr/include/google/bigtable/v2/bigtable.pb.h?

Apparently that's net-libs/google-cloud-cpp.

[Comment 3 Harris Landgarten 2019-04-21 12:46:55 UTC]

this upstream bug: https://github.com/tensorflow/tensorflow/issues/26155 deals with multiple issues but shows tensorflow-1.13.1 will not build with protobuf-3.7.1

This is from the above-referenced bug:

 To upgrade to protobuf 3.7.x one needs a bigger change to build environment as skylib build options have significantly changed.

[Comment 4 Harris Landgarten 2019-04-21 12:52:21 UTC]

Created attachment 573510 [details]
last 1000 lines of build log

-j1 build not possible for now. -j6 takes 7 hours to build now.

[Comment 5 Arfrever Frehtes Taifersar Arahesis 2019-04-21 20:25:45 UTC]

In file included from /usr/include/google/bigtable/v2/bigtable.grpc.pb.h:22:0,
                 from /usr/include/google/cloud/bigtable/data_client.h:21,
                 from ./tensorflow/contrib/bigtable/kernels/bigtable_lib.h:21,
                 from tensorflow/contrib/bigtable/kernels/bigtable_lib.cc:16:
/usr/include/google/bigtable/v2/bigtable.pb.h:17:2: error: #error This file was generated by an older version of protoc which is
 #error This file was generated by an older version of protoc which is
  ^~~~~
/usr/include/google/bigtable/v2/bigtable.pb.h:18:2: error: #error incompatible with your Protocol Buffer headers. Please
 #error incompatible with your Protocol Buffer headers.  Please
  ^~~~~
/usr/include/google/bigtable/v2/bigtable.pb.h:19:2: error: #error regenerate this file with a newer version of protoc.
 #error regenerate this file with a newer version of protoc.
  ^~~~~


Firstly rebuild net-libs/google-cloud-cpp with dev-libs/protobuf-3.7.1.
Next try building sci-libs/tensorflow again.

[Comment 6 LE GARREC Vincent 2019-04-22 21:28:33 UTC]

Rebuild google-cloud-cpp after updating protobuf solves the problem.

Thank.

Should we change RDEPEND in net-libs/google-cloud-cpp with the following ?
  =dev-libs/protobuf:=

[Comment 7 Arfrever Frehtes Taifersar Arahesis 2019-04-23 15:12:51 UTC]

:= operator should be used, but even this does not fully guarantee that users will upgrade packages in correct order.

[Comment 8 LE GARREC Vincent 2019-04-23 16:09:21 UTC]

Order should be respected because tensorflow depends on google-cloud-cpp if python enabled.

[Comment 9 Larry the Git Cow 2019-05-05 19:05:32 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=135c4ae14e323f2addeee483127104da87be9c83

commit 135c4ae14e323f2addeee483127104da87be9c83
Author:     Arfrever Frehtes Taifersar Arahesis <[e-mail]>
AuthorDate: 2019-05-05 02:30:03 +0000
Commit:     Mike Gilbert <[e-mail]>
CommitDate: 2019-05-05 19:04:02 +0000

    sci-libs/tensorflow: Depend on dev-libs/protobuf:=.
    
    Bug: https://bugs.gentoo.org/683928
    Signed-off-by: Arfrever Frehtes Taifersar Arahesis <[e-mail]>
    Signed-off-by: Mike Gilbert <[e-mail]>

 sci-libs/tensorflow/tensorflow-1.13.1.ebuild | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=697abaf217f98a7f450cd29e9d6ababb1b344936

commit 697abaf217f98a7f450cd29e9d6ababb1b344936
Author:     Arfrever Frehtes Taifersar Arahesis <[e-mail]>
AuthorDate: 2019-05-05 02:25:42 +0000
Commit:     Mike Gilbert <[e-mail]>
CommitDate: 2019-05-05 19:04:01 +0000

    net-libs/google-cloud-cpp: Depend on dev-libs/protobuf:=.
    
    Bug: https://bugs.gentoo.org/683928
    Signed-off-by: Arfrever Frehtes Taifersar Arahesis <[e-mail]>
    Signed-off-by: Mike Gilbert <[e-mail]>

 net-libs/google-cloud-cpp/google-cloud-cpp-0.2.0-r1.ebuild | 4 ++--
 net-libs/google-cloud-cpp/google-cloud-cpp-0.5.0.ebuild    | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

[Comment 10 Arfrever Frehtes Taifersar Arahesis 2019-05-05 20:21:58 UTC]

(In reply to Arfrever Frehtes Taifersar Arahesis from comment #5)
> In file included from
> /usr/include/google/bigtable/v2/bigtable.grpc.pb.h:22:0,
>                  from /usr/include/google/cloud/bigtable/data_client.h:21,
>                  from
> ./tensorflow/contrib/bigtable/kernels/bigtable_lib.h:21,
>                  from tensorflow/contrib/bigtable/kernels/bigtable_lib.cc:16:
> /usr/include/google/bigtable/v2/bigtable.pb.h:17:2: error: #error This file
> was generated by an older version of protoc which is
>  #error This file was generated by an older version of protoc which is
>   ^~~~~
> /usr/include/google/bigtable/v2/bigtable.pb.h:18:2: error: #error
> incompatible with your Protocol Buffer headers. Please
>  #error incompatible with your Protocol Buffer headers.  Please
>   ^~~~~
> /usr/include/google/bigtable/v2/bigtable.pb.h:19:2: error: #error regenerate
> this file with a newer version of protoc.
>  #error regenerate this file with a newer version of protoc.
>   ^~~~~
> 
> 
> Firstly rebuild net-libs/google-cloud-cpp with dev-libs/protobuf-3.7.1.
> Next try building sci-libs/tensorflow again.

If net-libs/google-cloud-cpp has been built with dev-libs/protobuf-3.7.1, and sci-libs/tensorflow still fails to build with dev-libs/protobuf-3.7.1, then re-open this bug and attach compressed, full build log with MAKEOPTS="-j1".