Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 759685

Summary: dev-util/catalyst-3.0.14 no module named 'portage'
Product: Gentoo Linux Reporter: Audiodef <online>
Component: Current packagesAssignee: Gentoo Catalyst Developers <catalyst>
Status: RESOLVED FIXED    
Severity: normal CC: bkohler, jstein, sam
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=519656
Whiteboard:
Package list:
Runtime testing required: ---

Description Audiodef 2020-12-13 00:55:17 UTC
When attempting to build a stage1, catalyst fails.

Previous versions of catalyst worked but they are no longer in Portage.

/etc/portage/make.profile link and package files are fine. They haven't changed since the last working version of catalyst.

Reproducible: Always

Steps to Reproduce:
1. eix-sync
2. catalyst -s latest
3. catalyst -f stage1.spec
Actual Results:  
12 Dec 2020 19:42:48 -05: NOTICE  : Loading configuration file: /etc/catalyst/catalyst.conf
12 Dec 2020 19:42:48 -05: NOTICE  : conf_values[options] = {'sticky-config'}
12 Dec 2020 19:42:48 -05: NOTICE  : Processing spec file: stage1.spec
12 Dec 2020 19:42:48 -05: NOTICE  : Using target: stage1
12 Dec 2020 19:42:48 -05: NOTICE  : Source file specification matching setting is: strict
12 Dec 2020 19:42:48 -05: NOTICE  : Accepted source file extensions search order: ['tar.xz', 'tpxz', 'xz', 'tar.bz2', 'bz2', 'tbz2', 'squashfs', 'sfs', 'tar.gz', 'gz', 'tar']
12 Dec 2020 19:42:48 -05: NOTICE  : Source path set to /home/audiodef/Documents/gentoostudio/catalyst/builds/stage3-amd64-seed.tar.xz
12 Dec 2020 19:42:48 -05: NOTICE  : stage1 stage path is /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/tmp/stage1root
12 Dec 2020 19:42:48 -05: NOTICE  : --- Running action sequence: unpack
12 Dec 2020 19:42:48 -05: NOTICE  : Clearing the chroot path ...
12 Dec 2020 19:42:48 -05: NOTICE  : Emptying directory: /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest
12 Dec 2020 19:42:48 -05: NOTICE  : Starting auto from /home/audiodef/Documents/gentoostudio/catalyst/builds/stage3-amd64-seed.tar.xz
12 Dec 2020 19:42:48 -05: NOTICE  : to /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest (this may take some time) ..
12 Dec 2020 19:43:02 -05: NOTICE  : --- Running action sequence: unpack_snapshot
12 Dec 2020 19:43:02 -05: NOTICE  : Unpacking portage tree (this can take a long time) ...
12 Dec 2020 19:43:05 -05: NOTICE  : --- Running action sequence: setup_confdir
12 Dec 2020 19:43:05 -05: NOTICE  : --- Running action sequence: portage_overlay
12 Dec 2020 19:43:05 -05: NOTICE  : --- Running action sequence: base_dirs
12 Dec 2020 19:43:05 -05: NOTICE  : --- Running action sequence: bind
12 Dec 2020 19:43:06 -05: NOTICE  : --- Running action sequence: chroot_setup
12 Dec 2020 19:43:06 -05: NOTICE  : Setting up chroot...
NOTICE:catalyst:Setting up chroot...
12 Dec 2020 19:43:06 -05: WARNING : Overriding certain env variables may cause catastrophic failure.
WARNING:catalyst:Overriding certain env variables may cause catastrophic failure.
12 Dec 2020 19:43:06 -05: WARNING : If your build fails look here first as the possible problem.
WARNING:catalyst:If your build fails look here first as the possible problem.
12 Dec 2020 19:43:06 -05: WARNING : Catalyst assumes you know what you are doing when setting these variables.
WARNING:catalyst:Catalyst assumes you know what you are doing when setting these variables.
12 Dec 2020 19:43:06 -05: WARNING : Catalyst Maintainers use VERY minimal envscripts, if used at all.
WARNING:catalyst:Catalyst Maintainers use VERY minimal envscripts, if used at all.
12 Dec 2020 19:43:06 -05: WARNING : You have been warned.
WARNING:catalyst:You have been warned.
12 Dec 2020 19:43:06 -05: NOTICE  : Writing the stage make.conf to: /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/etc/portage/make.conf
NOTICE:catalyst:Writing the stage make.conf to: /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/etc/portage/make.conf
12 Dec 2020 19:43:06 -05: NOTICE  : STICKY-CONFIG is enabled
NOTICE:catalyst:STICKY-CONFIG is enabled
12 Dec 2020 19:43:06 -05: NOTICE  : --- Running action sequence: setup_environment
NOTICE:catalyst:--- Running action sequence: setup_environment
12 Dec 2020 19:43:06 -05: NOTICE  : --- Running action sequence: run_local
NOTICE:catalyst:--- Running action sequence: run_local
copying make.conf to /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/tmp/stage1root/etc/portage
Copying stage1-chroot.sh to /tmp
copying stage1-chroot.sh to /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/tmp
copying chroot-functions.sh to /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/tmp
Ensure the file has the executable bit set
Running stage1-chroot.sh in chroot:
    chroot /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest /tmp/stage1-chroot.sh
!!! Section 'GentooStudio' in repos.conf has location attribute set to nonexistent directory: '/var/lib/GentooStudio'
!!! Section 'audio-overlay' in repos.conf has location attribute set to nonexistent directory: '/var/lib/audio-overlay'
!!! Section 'GentooStudio' in repos.conf has location attribute set to nonexistent directory: '/var/lib/GentooStudio'
!!! Section 'audio-overlay' in repos.conf has location attribute set to nonexistent directory: '/var/lib/audio-overlay'
>>> Regenerating /etc/ld.so.cache...
Traceback (most recent call last):
  File "/tmp/build.py", line 5, in <module>
    import portage
ModuleNotFoundError: No module named 'portage'
!!! Section 'GentooStudio' in repos.conf has location attribute set to nonexistent directory: '/var/lib/GentooStudio'
!!! Section 'audio-overlay' in repos.conf has location attribute set to nonexistent directory: '/var/lib/audio-overlay'
Your profile seems to be broken.
Could not build a list of build packages.
Double check your /etc/portage/make.profile link and the 'packages' files.
12 Dec 2020 19:43:09 -05: ERROR   : CatalystError: cmd(['/usr/share/catalyst/targets/stage1/stage1-controller.sh', 'run']) exited 1
ERROR:catalyst:CatalystError: cmd(['/usr/share/catalyst/targets/stage1/stage1-controller.sh', 'run']) exited 1
12 Dec 2020 19:43:09 -05: NOTICE  : /home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/var/tmp/portage is not a mount point. Skipping
NOTICE:catalyst:/home/audiodef/Documents/gentoostudio/catalyst/tmp/default/stage1-amd64-latest/var/tmp/portage is not a mount point. Skipping
12 Dec 2020 19:43:09 -05: ERROR   : CatalystError: Stage build aborting due to error.
ERROR:catalyst:CatalystError: Stage build aborting due to error.
12 Dec 2020 19:43:09 -05: ERROR   : Exception running action sequence run_local
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/catalyst/base/stagebase.py", line 1409, in run_local
    env=self.env)
  File "/usr/lib/python3.7/site-packages/catalyst/support.py", line 54, in cmd
    print_traceback=False)
catalyst.support.CatalystError: cmd(['/usr/share/catalyst/targets/stage1/stage1-controller.sh', 'run']) exited 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/catalyst/base/stagebase.py", line 1514, in run
    getattr(self, x)()
  File "/usr/lib/python3.7/site-packages/catalyst/base/stagebase.py", line 1418, in run_local
    print_traceback=False)
catalyst.support.CatalystError: Stage build aborting due to error.
ERROR:catalyst:Exception running action sequence run_local
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/catalyst/base/stagebase.py", line 1409, in run_local
    env=self.env)
  File "/usr/lib/python3.7/site-packages/catalyst/support.py", line 54, in cmd
    print_traceback=False)
catalyst.support.CatalystError: cmd(['/usr/share/catalyst/targets/stage1/stage1-controller.sh', 'run']) exited 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/catalyst/base/stagebase.py", line 1514, in run
    getattr(self, x)()
  File "/usr/lib/python3.7/site-packages/catalyst/base/stagebase.py", line 1418, in run_local
    print_traceback=False)
catalyst.support.CatalystError: Stage build aborting due to error.


Expected Results:  
Completed stage1 build.

My stage1.spec:

subarch: amd64
target: stage1
version_stamp: latest
rel_type: default
profile: default/linux/amd64/17.1
snapshot: latest
source_subpath: stage3-amd64-seed
update_seed: yes
update_seed_command: --update --deep --newuse --with-bdeps=y --backtrack=250 @system @world
portage_confdir: /etc/portage

catalyst.conf:

digests="sha512 whirlpool"
contents="auto"
distdir="/usr/portage/distfiles"
envscript="/etc/catalyst/catalystrc"
hash_function="crc32"
options="sticky-config"
portdir="/usr/portage"
repo_basedir="/usr"
repo_name="portage"
target_distdir="/usr/portage/distfiles"
target_pkgdir="/usr/portage/packages"
sharedir="/usr/share/catalyst"
shdir="%(sharedir)s/targets"
snapshot_cache="/home/audiodef/Documents/gentoostudio/catalyst/snapshot_cache"
storedir="/home/audiodef/Documents/gentoostudio/catalyst"
source_matching="strict"
Comment 1 Jonas Stein gentoo-dev 2020-12-13 01:14:41 UTC
is this a duplicate of 519656?
Comment 2 Audiodef 2020-12-13 17:52:04 UTC
No. I don't even use snapcache, and the issue I've reported has nothing to do with snapcache or running more than one instance.
Comment 3 Audiodef 2020-12-14 00:43:41 UTC
I had a couple of previous versions in distfiles, so I tossed in ebuilds for them, and the result is the same. The issue might be with python, not with catalyst, or perhaps with how catalyst handles python 3.7 or higher.
Comment 4 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2020-12-14 00:49:33 UTC
Please verify that the version of Python being used with Catalyst is aligned with PYTHON_TARGETS and such of Portage.
Comment 5 Audiodef 2020-12-14 00:51:56 UTC
I think the build.py in question is /root/catalyst/tmp/default/stage1-amd64-latest/tmp/build.py. Apparently there is no module named "portage" and maybe it's this that needs to be addressed. Not sure how that would happen, though. 

#!/usr/bin/python

import os
import sys
import portage

# this loads files from the profiles ...
# wrap it here to take care of the different
# ways portage handles stacked profiles
# last case is for portage-2.1_pre*
def scan_profile(path):
        if "grab_stacked" in dir(portage):
                return portage.grab_stacked(path, portage.settings.profiles, portage.grabfile, incremental_lines=1)
        else:
             	if "grab_multiple" in dir(portage):
                        return portage.stack_lists(portage.grab_multiple(path, portage.settings.profiles, portage.grabfile), incremental=1)
                else:
                     	return portage.stack_lists([portage.grabfile_package(os.path.join(x, path)) for x in portage.settings.profiles], incremental=1)

# loaded the stacked packages / packages.build files
pkgs = scan_profile("packages")
buildpkgs = scan_profile("packages.build")

# go through the packages list and strip off all the
# crap to get just the <category>/<package> ... then
# search the buildpkg list for it ... if it's found,
# we replace the buildpkg item with the one in the
# system profile (it may have <,>,=,etc... operators
# and version numbers)
for idx in range(0, len(pkgs)):
        try:
            	bidx = buildpkgs.index(portage.dep_getkey(pkgs[idx]))
                buildpkgs[bidx] = pkgs[idx]
                if buildpkgs[bidx][0:1] == "*":
                        buildpkgs[bidx] = buildpkgs[bidx][1:]
        except Exception:
                pass

for b in buildpkgs:
        sys.stdout.write(b + " ")
Comment 6 Audiodef 2020-12-14 00:58:49 UTC
(In reply to Sam James from comment #4)
> Please verify that the version of Python being used with Catalyst is aligned
> with PYTHON_TARGETS and such of Portage.

What do I need to do to verify this?
Comment 7 Ben Kohler gentoo-dev 2020-12-14 13:57:03 UTC
This bug has been hitting our releng builds the last couple of weeks.  The problem is that the stage1/build.py has a generic /usr/bin/python in its shebang, and is not wrapped by python-exec.  

It gets executed under whatever python interpreter version is first in "eselect python list" (/etc/python-exec/python-exec.conf), even if that version is not enabled in sys-apps/portage's PYTHON_TARGETS.

Most likely what has hapened is that python3.9 got newly installed, and it put itself at the top of the priority list.  When build.py gets executed under 3.9, it cannot import the portage module because sys-apps/portage has been built with PYTHON_TARGETS="python2_7 python3_8".

I'm still thinking about what the best fix might be.  A rough workaround would be to package.mask dev-lang/python:3.9 for now via portage_confdir.
Comment 8 Audiodef 2020-12-15 00:05:26 UTC
Thanks for confirming. On my system here, this happens regardless of which python is at the top of my eselect python list.
Comment 9 Larry the Git Cow gentoo-dev 2020-12-16 16:00:52 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/proj/catalyst.git/commit/?id=37cabf5c536fe8eef16532f18790c1da566aae15

commit 37cabf5c536fe8eef16532f18790c1da566aae15
Author:     Matt Turner <mattst88@gentoo.org>
AuthorDate: 2020-12-15 03:29:56 +0000
Commit:     Matt Turner <mattst88@gentoo.org>
CommitDate: 2020-12-16 03:01:23 +0000

    targets: Use interpreter with portage module for build.py
    
    Closes: https://bugs.gentoo.org/759685
    Signed-off-by: Matt Turner <mattst88@gentoo.org>

 targets/stage1/chroot.sh | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
Comment 10 Matt Turner gentoo-dev 2020-12-16 16:20:06 UTC
I cherry-picked these patches to the stable branch. They're included in catalyst-3.0.15.
Comment 11 Audiodef 2020-12-17 20:05:38 UTC
Thank you. I can confirm that catalyst now proceeds to build stage1.