Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 783957 - sys-apps/portage: lazily evaluate cnf_* variables in lib/portage/tests/__init__.py
Summary: sys-apps/portage: lazily evaluate cnf_* variables in lib/portage/tests/__init...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: All All
: Normal enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 782397 785484
  Show dependency tree
 
Reported: 2021-04-19 02:04 UTC by Zac Medico
Modified: 2021-07-05 01:39 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2021-04-19 02:04:41 UTC
I discovered this premature initialization of variables when it triggered this error running the unit tests in a venv on Ubuntu on WSL2:

> lib/portage/tests/__init__.py:  cnf_path = os.path.join(PORTAGE_BASE_PATH, 'cnf')
> lib/portage/tests/__init__.py:  cnf_etc_path = cnf_path
> lib/portage/tests/__init__.py:  cnf_bindir = PORTAGE_BIN_PATH
> lib/portage/tests/__init__.py:  cnf_sbindir = cnf_bindir
> lib/portage/tests/__init__.py:  cnf_path = os.path.join(EPREFIX or '/', GLOBAL_CONFIG_PATH)
> lib/portage/tests/__init__.py:  cnf_etc_path = os.path.join(EPREFIX or '/', 'etc')
> lib/portage/tests/__init__.py:  cnf_eprefix = EPREFIX
> lib/portage/tests/__init__.py:  cnf_bindir = os.path.join(EPREFIX or '/', 'usr', 'bin')
> lib/portage/tests/__init__.py:  cnf_sbindir = os.path.join(EPREFIX or '/', 'usr', 'sbin')
> lib/portage/tests/__init__.py:          self.cnf_path = cnf_path
> lib/portage/tests/__init__.py:          self.cnf_etc_path = cnf_etc_path
> lib/portage/tests/resolver/ResolverPlayground.py:       cnf_path_repoman = None
> repoman/lib/repoman/tests/__init__.py:  cnf_path = os.path.join(REPOMAN_BASE_PATH, 'cnf')
> repoman/lib/repoman/tests/__init__.py:  cnf_path_repoman = cnf_path
> repoman/lib/repoman/tests/__init__.py:  cnf_etc_path = cnf_path
> repoman/lib/repoman/tests/__init__.py:  cnf_bindir = os.path.join(REPOMAN_BASE_PATH, 'bin')
> repoman/lib/repoman/tests/__init__.py:  cnf_sbindir = cnf_bindir
> repoman/lib/repoman/tests/__init__.py:  cnf_path = os.path.join(EPREFIX or '/', GLOBAL_CONFIG_PATH)
> repoman/lib/repoman/tests/__init__.py:  cnf_path_repoman = os.path.join(EPREFIX or '/',
> repoman/lib/repoman/tests/__init__.py:  cnf_etc_path = os.path.join(EPREFIX or '/', 'etc')
> repoman/lib/repoman/tests/__init__.py:  cnf_eprefix = EPREFIX
> repoman/lib/repoman/tests/__init__.py:  cnf_bindir = os.path.join(EPREFIX or '/', 'usr', 'bin')
> repoman/lib/repoman/tests/__init__.py:  cnf_sbindir = os.path.join(EPREFIX or '/', 'usr', 'sbin')
> repoman/lib/repoman/tests/__init__.py:          self.cnf_path = cnf_path
> repoman/lib/repoman/tests/__init__.py:          self.cnf_etc_path = cnf_etc_path

> (portage-venv) $ python ~/portage-venv/lib/python3.8/site-packages/portage/tests/{runTests.py,ebuild/test_fetch.py}                                            
> testEbuildFetch (portage.tests.ebuild.test_fetch.EbuildFetchTestCase) ... ================================================================================================                    
> ======== Traceback for invalid call to portage.package.ebuild.config.config.__getitem__ ========                                                                                           
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/runTests.py", line 64, in <module>                                                                               
>     sys.exit(tests.main())                                                                                                                                                                    
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/__init__.py", line 62, in main                                                                                   
>     result = TextTestRunner(verbosity=2).run(suite)                                                                                                                                           
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/__init__.py", line 296, in run                                                                                   
>     test(result)                                                                                                                                                                              
>   File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__                                                                                                                           
>     return self.run(*args, **kwds)                                                                                                                                                            
>   File "/usr/lib/python3.8/unittest/suite.py", line 122, in run                                                                                                                               
>     test(result)                                                                                                                                                                              
>   File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__                                                                                                                           
>     return self.run(*args, **kwds)                                                                                                                                                            
>   File "/usr/lib/python3.8/unittest/suite.py", line 122, in run                                                                                                                               
>     test(result)                                                                                                                                                                              
>   File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__                                                                                                                           
>     return self.run(*args, **kwds)                                                                                                                                                            
>   File "/usr/lib/python3.8/unittest/suite.py", line 122, in run                                                                                                                               
>     test(result)                                                                                                                                                                              
>   File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__                                                                                                                           
>     return self.run(*args, **kwds)                                                                                                                                                            
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/__init__.py", line 209, in run                                                                                   
>     testMethod()                                                                                                                                                                              
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/ebuild/test_fetch.py", line 84, in testEbuildFetch                                                               
>     playground = ResolverPlayground(ebuilds=ebuilds_subst, distfiles=distfiles, user_config=user_config_subst)                                                                                
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/resolver/ResolverPlayground.py", line 171, in __init__                                                           
>     self._create_ebuild_manifests(ebuilds)                                                                                                                                                    
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/tests/resolver/ResolverPlayground.py", line 268, in _create_ebuild_manifests                                           
>     if not digestgen(mysettings=tmpsettings, myportdb=portdb):                                                                                                                                
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/package/ebuild/digestgen.py", line 47, in digestgen                                                                    
>     for myfile in fetchlist_dict[cpv]:                                                                                                                                                        
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/dbapi/porttree.py", line 1432, in __getitem__                                                                          
>     return list(self.portdb.getFetchMap(pkg_key, mytree=self.mytree))
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/dbapi/porttree.py", line 753, in getFetchMap
>     self.async_fetch_map(mypkg, useflags=useflags,
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/dbapi/porttree.py", line 807, in async_fetch_map
>     aux_get_future = self.async_aux_get(
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/dbapi/porttree.py", line 696, in async_aux_get
>     proc.start()
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/_emerge/AsynchronousTask.py", line 30, in start
>     self._start()
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/_emerge/EbuildMetadataPhase.py", line 104, in _start
>     retval = portage.doebuild(ebuild_path, "depend",
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/package/ebuild/doebuild.py", line 814, in doebuild
>     doebuild_environment(myebuild, mydo, myroot, mysettings, debug,
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/package/ebuild/doebuild.py", line 363, in doebuild_environment
>     mysettings["RPMDIR"]  = os.path.realpath(mysettings["RPMDIR"])
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/package/ebuild/config.py", line 2633, in __getitem__
>     return self._getitem(key)
>   File "/home/zmedico/portage-venv/lib/python3.8/site-packages/portage/package/ebuild/config.py", line 2694, in _getitem
>     raise KeyError(mykey)
> KeyError: 'RPMDIR'
> ================================================================================================
> ERROR
Comment 1 Larry the Git Cow gentoo-dev 2021-04-19 07:02:26 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c8dd97e8475686738b5c0c2c089f102a881b4820

commit c8dd97e8475686738b5c0c2c089f102a881b4820
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-04-19 02:28:09 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-04-19 06:50:21 +0000

    tests: lazily evaluate cnf_* vars (bug 783957)
    
    Lazily evaluate cnf_* variables, allowing for mock portage.const
    settings created by ResolverPlayground. In ResolverPlayground,
    create symlinks for PORTAGE_BIN_PATH scripts inside the mock
    EPREFIX, and restore portage.const.EPREFIX in the cleanup method.
    
    Bug: https://bugs.gentoo.org/783957
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/dispatch_conf.py                     |  4 +-
 lib/portage/tests/__init__.py                    | 59 +++++++++++++++++-------
 lib/portage/tests/resolver/ResolverPlayground.py | 39 +++++++++++++++-
 3 files changed, 83 insertions(+), 19 deletions(-)
Comment 2 Larry the Git Cow gentoo-dev 2021-05-24 09:05:16 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=cba21902d9a8d1ac07d2ff5b0d932e71fc1fac67

commit cba21902d9a8d1ac07d2ff5b0d932e71fc1fac67
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-05-24 07:47:11 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-05-24 09:05:08 +0000

    sys-apps/portage: Bump to version 3.0.19
    
     #520378: allow emerge --fetchonly to log to emerge-fetch.log
     #698244: portage(5) document user patch / eapply_user
     #781854: Suggest PORTAGE_LOG_FILTER_FILE_CMD cat fallback
     #782724: sort emerge --unmerge order for determinism
     #783957: lazily evaluate cnf_* variables in tests
     #784566: make emerge insensitive to relative order of optional
                      and positional arguments
     #787545: emerge CTRL C may be ignored when running pkg_pretend
     #787563: ebuild-ipc could handle KeyboardInterrupt
     #788967: emerge --jobs= triggers TypeError
    
    Bug: https://bugs.gentoo.org/785484
    Bug: https://bugs.gentoo.org/788967
    Bug: https://bugs.gentoo.org/787563
    Bug: https://bugs.gentoo.org/787545
    Bug: https://bugs.gentoo.org/784566
    Bug: https://bugs.gentoo.org/783957
    Bug: https://bugs.gentoo.org/782724
    Bug: https://bugs.gentoo.org/781854
    Bug: https://bugs.gentoo.org/698244
    Bug: https://bugs.gentoo.org/520378
    Package-Manager: Portage-3.0.18, Repoman-3.0.3
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-3.0.19.ebuild | 266 +++++++++++++++++++++++++++++++++
 2 files changed, 267 insertions(+)