Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 631972 - sys-devel/crossdev-20160602-r1 should check overlay presence in when -oO is set and report nicer error
Summary: sys-devel/crossdev-20160602-r1 should check overlay presence in when -oO is s...
Status: UNCONFIRMED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal
Assignee: Gentoo Crossdev team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-09-25 04:49 UTC by R030t1
Modified: 2018-01-20 14:38 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
cross-aarch64-unknown-linux-gnu-info.log (cross-aarch64-unknown-linux-gnu-info.log,10.21 KB, text/plain)
2017-09-27 23:36 UTC, R030t1
Details
cross-aarch64-unknown-linux-gnu-binutils.log (cross-aarch64-unknown-linux-gnu-binutils.log,502 bytes, text/plain)
2017-09-27 23:37 UTC, R030t1
Details

Note You need to log in before you can comment on or make changes to this bug.
Description R030t1 2017-09-25 04:49:16 UTC
> crossdev -oO /usr/local/portage/odroid-c2 -t aarch64-unknown-linux-gnu
Results in the following:

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 [0;10m[1m[32m*[0;10m crossdev version:      20160602
 [0;10m[1m[32m*[0;10m Host Portage ARCH:     amd64
 [0;10m[1m[32m*[0;10m Target Portage ARCH:   arm64
 [0;10m[1m[32m*[0;10m Target System:         aarch64-unknown-linux-gnu
 [0;10m[1m[32m*[0;10m Stage:                 4 (C/C++ compiler)
 [0;10m[1m[32m*[0;10m ABIs:                  arm64

 [0;10m[1m[32m*[0;10m binutils:              binutils-[latest]
 [0;10m[1m[32m*[0;10m gcc:                   gcc-[latest]
 [0;10m[1m[32m*[0;10m headers:               linux-headers-[latest]
 [0;10m[1m[32m*[0;10m libc:                  glibc-[latest]

 [0;10m[1m[32m*[0;10m CROSSDEV_OVERLAY:      /usr/local/portage/odroid-c2/
 [0;10m[1m[32m*[0;10m PORT_LOGDIR:           /var/log/portage
 [0;10m[1m[32m*[0;10m PORTAGE_CONFIGROOT:    
 [0;10m[1m[32m*[0;10m Portage flags:         
  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -  _  -  ~  -

If PORTAGE_CONFIGROOT is set on the command line, the stage 1 merge of binutils fails stating that a profile must be selected. The profile "/usr/portage/profiles/default/linux/arm64/13.0/desktop" was selected per Sakaki's recommendation in one of his guides. The guide at https://wiki.gentoo.org/wiki/Raspberry_Pi_3_64_bit_Install does not include this information.

The above leads to:

* Refusing to create a cross-compiler using the same
* target as your host utils.

CHOST is not equal to BHOST.


Attempting the above with aarch64-unknown-linux-gnueabi informs me that there is no way to satisfy the request for the "binutils" package. If I modify make.conf to have the proper CHOST, I receive the last error above.

Respectfully,
     R0b0t1
Comment 1 Sergei Trofimovich (RETIRED) gentoo-dev 2017-09-25 07:36:44 UTC
crossdev tells you to attach log files to help investigation.
Can you do that?
Comment 2 R030t1 2017-09-27 23:36:53 UTC
Created attachment 496784 [details]
cross-aarch64-unknown-linux-gnu-info.log

(In reply to Sergei Trofimovich from comment #1)
> crossdev tells you to attach log files to help investigation.
> Can you do that?

For the problem in the title, no logs are produced. When a profile is not set logs are produced, but the only error states that a profile is not set. When a profile is set again, no logs are produced, crossdev immediately exits.

I have attached the informational log containing environment settings.
Comment 3 R030t1 2017-09-27 23:37:22 UTC
Created attachment 496786 [details]
cross-aarch64-unknown-linux-gnu-binutils.log
Comment 4 R030t1 2017-10-01 03:27:57 UTC
I have made some very invasive changes to the crossdev script without much luck. Forcefully setting the values used in the test relevant to the last message (https://gitweb.gentoo.org/proj/crossdev.git/tree/crossdev#n722) to presumed good values produces the notice that no ebuilds satisfy  $CTARGET/binutils.

At this point I can no longer continue using crossdev, which is stalling my progress on my SBC projects. I don't know how to continue from here.
Comment 5 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-01 22:41:06 UTC
It looks like too many things are conflated here.
For example cross-aarch64-unknown-linux-gnu-info.log output does not match
output in #comment1.

What does fail for you? Building initial cross-toolchain or using it?
Comment 6 R030t1 2017-10-02 03:22:42 UTC
My apologies if I am conflating anything. I listed 3 issues in series. I believe the root of the issue, if there is one, is whatever problem is causing crossdev to leave PORTAGE_CONFIGROOT unset when invoked (`crossdev -t aarch64-unknown-linux-gnu`). As I was not able to troubleshoot that issue directly, I detailed some of my attempts to work around it. The things I have detailed indicate to me that the automatic detection of various variables that crossdev does has been broken in some way.

I have recently tried to invoke crossdev on what was a working installation and it now fails in the exact same way as detailed in the bug report's topic. This leads me to believe there has been a major regression in the script.
Comment 7 R030t1 2017-10-02 03:25:09 UTC
If the original comment is not clear: simply invoking crossdev produces the quoted input and no other action is taken.
Comment 8 R030t1 2017-10-02 03:26:19 UTC
(In reply to R030t1 from comment #7)
> If the original comment is not clear: simply invoking crossdev produces the
> quoted input and no other action is taken.

"Quoted output," my apologies.
Comment 9 R030t1 2017-10-02 04:48:18 UTC
(In reply to Sergei Trofimovich from comment #5)
> It looks like too many things are conflated here.
> For example cross-aarch64-unknown-linux-gnu-info.log output does not match
> output in #comment1.
> 
> What does fail for you? Building initial cross-toolchain or using it?

Apologies again, I did not directly answer the question. Building the initial toolchain fails.
Comment 10 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-02 10:26:21 UTC
'crossdev -t aarch64-unknown-linux-gnu' should not set PORTAGE_CONFIGROOT variable. The toolchain is built on host system as a normal package (it runs on host and is tracked as host's ebuild in package database)

The main difference of cross-aarch64-unknown-linux-gnu/gcc from sys-devel/gcc
if the --target= (and sysroot, and other tweaks) flag.  But it's still a package
that is supposed to run on host, not target. It uses /etc/portage for PORTAGE_CONFIGROOT, not /usr/$CHOST.

Why do you need to set PORTAGE_CONFIGROOT? Can you show how things fail for you when you don't set PORTAGE_CONFIGROOT?
Comment 11 Roy Bamford gentoo-dev 2017-10-02 14:14:01 UTC
(In reply to R030t1 from comment #2)
> Created attachment 496784 [details]
> cross-aarch64-unknown-linux-gnu-info.log
> 
> (In reply to Sergei Trofimovich from comment #1)
> > crossdev tells you to attach log files to help investigation.
> > Can you do that?
> 
> For the problem in the title, no logs are produced. When a profile is not
> set logs are produced, but the only error states that a profile is not set.
> When a profile is set again, no logs are produced, crossdev immediately
> exits.
> 
> I have attached the informational log containing environment settings.

From your command 
crossdev -oO /usr/local/portage/odroid-c2 -t aarch64-unknown-linux-gnu

The cross ebuilds are saved in your /usr/local/portage/odroid-c2 overlay.


From your logs 

Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000

There are no settings for your /usr/local/portage/odroid-c2 overlay, so portage will not search there for the cross- ebuilds.  

Add your /usr/local/portage/odroid-c2 overlay to /etc/portage/repos.conf/
Comment 12 R030t1 2017-10-03 05:02:49 UTC
(In reply to Sergei Trofimovich from comment #10)
> 'crossdev -t aarch64-unknown-linux-gnu' should not set PORTAGE_CONFIGROOT
> variable. The toolchain is built on host system as a normal package (it runs
> on host and is tracked as host's ebuild in package database)
> 
> The main difference of cross-aarch64-unknown-linux-gnu/gcc from sys-devel/gcc
> if the --target= (and sysroot, and other tweaks) flag.  But it's still a
> package
> that is supposed to run on host, not target. It uses /etc/portage for
> PORTAGE_CONFIGROOT, not /usr/$CHOST.
> 
> Why do you need to set PORTAGE_CONFIGROOT? Can you show how things fail for
> you when you don't set PORTAGE_CONFIGROOT?

I did show how things fail for me if I do not set PORTAGE_CONFIGROOT: crossdev prints its initial configuration banner and then immediately exits. There was a Wiki page that showed sample output that had PORTAGE_CONFIGROOT set automatically. Sadly, I can't find it. If I set it manually crossdev will continue.

The files in PORTAGE_CONFIGROOT, typically /usr/$TARGET, configure the packages installed to that location (from memory).
Comment 13 R030t1 2017-10-03 05:10:07 UTC
Mr. Bamford, this is in reply to you, but quoting your message makes my browser unusable.

I have a file /etc/portage/repos.conf/odroid-c2.conf. There is a directory /usr/local/portage/odroid-c2/cross-aarch64-unknown-gnu that was presumably created by crossdev and populated with ebuilds for a select number of packages (binutils, gcc, gdb, glibc, and linux-headers).

Also important is that a previously configured system can no longer produce cross toolchains using newer versions of crossdev. It did successfully produce some before.

Thank you both for the help.
Comment 14 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-03 09:25:29 UTC
(In reply to R030t1 from comment #13)
> Mr. Bamford, this is in reply to you, but quoting your message makes my
> browser unusable.
> 
> I have a file /etc/portage/repos.conf/odroid-c2.conf. There is a directory
> /usr/local/portage/odroid-c2/cross-aarch64-unknown-gnu that was presumably
> created by crossdev and populated with ebuilds for a select number of
> packages (binutils, gcc, gdb, glibc, and linux-headers).
> 
> Also important is that a previously configured system can no longer produce
> cross toolchains using newer versions of crossdev. It did successfully
> produce some before.
> 
> Thank you both for the help.

It should be /etc/portage/repos.conf.d/odroid-c2.conf, not /etc/portage/repos.conf.d/odroid-c2.conf.
Comment 15 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-03 09:26:17 UTC
(In reply to Sergei Trofimovich from comment #14)
> (In reply to R030t1 from comment #13)

[trying again]
It should be /etc/portage/repos.conf.d/odroid-c2.conf, not /etc/portage/repos.conf/odroid-c2.conf.
Comment 16 Roy Bamford gentoo-dev 2017-10-05 10:46:58 UTC
Lets leave crossdev out of it meanwhile. I'll come back to that.
There are two steps to setting up a local overlay.
1. Creating the overlay directory structures and adding zero or more ebuilds.
2. Telling portage about the overlay, so it uses it for dependency resolution.
These separate steps apply to any overlay.

When you use crossdev --ov-output it will create new overlay for the cross ebuilds but not perform step 2, so portage will not use the overlay.

When step 2 has been performed, the overlay will be listed in emerge --info.
I have.

Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://192.168.10.119/gentoo-portage
    priority: -1000
    sync-rsync-extra-opts: --exclude=metadata/cache

gentoo_static
    location: /usr/local/gentoo-static
    masters: gentoo
    priority: 0

gentoo is the normal repository, gentoo_static is crossdev and everything else needed to install gentoo with a static /dev.
That second entry was missing in your emerge --info output above.

Nothing works until the overlay used by crossdev appears here. Even then, if the overlay directory structure is not correct, portage will silently fail to find anything. Having an empty overlay for whatever reason, is not an error.

As a further test, after the cross ebuilds are in place,
emerge -pv cross-aarch64-unknown-linux-gnu/gcc
should at least find the ebuild, (put your own tuple there).
Comment 17 R030t1 2017-10-06 02:16:10 UTC
(In reply to Sergei Trofimovich from comment #15)
> (In reply to Sergei Trofimovich from comment #14)
> > (In reply to R030t1 from comment #13)
> 
> [trying again]
> It should be /etc/portage/repos.conf.d/odroid-c2.conf, not
> /etc/portage/repos.conf/odroid-c2.conf.

Thank you both, and sorry Mr. Bamford. I forgot to put into practice Mr. Trofimovich's advice. I moved /etc/portage/repos.conf/ to /etc/portage/repos.conf.d/ and crossdev proceeded to complete up to stage 2 with the usual arguments.

Unfortunately, it did not finish and produce a C++ compiler. I attempted to merge @system in the crossroot and somehow /usr/bin/emerge was deleted. I am attempting to fix my system now.

Shall I open a new bug for the lack of C++ compiler generation?
Comment 18 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-06 09:03:08 UTC
> Unfortunately, it did not finish and produce a C++ compiler.
I don't understand that sentence. Did crossdev fail? When crossdev fails it asks to attach the logs.

> Shall I open a new bug for the lack of C++ compiler generation?
I think this bug is good enough to get you a cross-compiler.
Comment 19 R030t1 2017-10-07 06:37:50 UTC
(In reply to Sergei Trofimovich from comment #18)
> > Unfortunately, it did not finish and produce a C++ compiler.
> I don't understand that sentence. Did crossdev fail? When crossdev fails it
> asks to attach the logs.
> 
> > Shall I open a new bug for the lack of C++ compiler generation?
> I think this bug is good enough to get you a cross-compiler.

Yes, I think I misunderstood what was happening. A C++ compiler exists after crossdev completes. Thank you for the help!
Comment 20 R030t1 2017-10-07 15:39:14 UTC
There was actually a bug here, or maybe two: no place to my knowledge states that /etc/portage/repos.conf.d/ is needed over /etc/portage/repos.conf/, and crossdev fails messily instead of recognizing that configuration does not exist and warning the user about it.

I apologize for marking this bug as resolved, that was premature.
Comment 21 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-07 20:43:21 UTC
(In reply to R030t1 from comment #20)
> There was actually a bug here, or maybe two: no place to my knowledge states
> that /etc/portage/repos.conf.d/ is needed over /etc/portage/repos.conf/

Where and how would you like to see that documented? 'man portage'?

> crossdev fails messily instead of recognizing that configuration does not
> exist and warning the user about it.

emerge log reports quite a concise error of missing ebuild:
"""emerge: there are no ebuilds to satisfy "cross-s390-unknown-linux-gnu/binutils."""

And another log reports list of known repositories.

Is it a messy failure? What would you like to see instead?
Comment 22 R030t1 2017-10-08 06:52:42 UTC
(In reply to Sergei Trofimovich from comment #21)
> (In reply to R030t1 from comment #20)
> > There was actually a bug here, or maybe two: no place to my knowledge states
> > that /etc/portage/repos.conf.d/ is needed over /etc/portage/repos.conf/
> 
> Where and how would you like to see that documented? 'man portage'?
> 

I've read large sections of the man pages at some time or another, but I still keep coming across new information that I need for my day to day use of my system. I am not sure what a good solution is.

> > crossdev fails messily instead of recognizing that configuration does not
> > exist and warning the user about it.
> 
> emerge log reports quite a concise error of missing ebuild:
> """emerge: there are no ebuilds to satisfy
> "cross-s390-unknown-linux-gnu/binutils."""
> 
> And another log reports list of known repositories.
> 
> Is it a messy failure? What would you like to see instead?

Stating possible causes is usually a good idea. Before I understood what crossdev was doing, I had no idea that that message meant the configuration was incorrect in the way that it was. Perhaps it was right but there was no ebuild?

Cheers!
Comment 23 Mike Gilbert gentoo-dev 2017-10-09 16:15:05 UTC
I'm not sure where this "repos.conf.d" idea came from; portage looks at /etc/portage/repos.conf/, not /etc/portage/repos.conf.d/.
Comment 24 R030t1 2017-10-10 00:49:21 UTC
(In reply to Mike Gilbert from comment #23)
> I'm not sure where this "repos.conf.d" idea came from; portage looks at
> /etc/portage/repos.conf/, not /etc/portage/repos.conf.d/.

I'm glad you mentioned this, I saw portage "ignoring" my settings in repos.conf to use webrsync-gpg and was about to file bugs against portage and/or eix.

I have seen name.conf.d when the configuration file has been turned into a directory, but I didn't see anything else mention it (I even looked through crossdev itself).
Comment 25 Sergei Trofimovich (RETIRED) gentoo-dev 2017-10-10 10:18:27 UTC
(In reply to Mike Gilbert from comment #23)
> I'm not sure where this "repos.conf.d" idea came from; portage looks at
> /etc/portage/repos.conf/, not /etc/portage/repos.conf.d/.

All is my fault. I've conflated things up. My apologies.
Comment 26 Sergei Trofimovich (RETIRED) gentoo-dev 2017-12-10 09:53:24 UTC
The only thing left in this bug is to make crossdev error message nicer when overlay path is not known to emerge and a few hints on how to add it, right?