Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 815871

Summary: sys-apps/portage-3.0.26: EbuildPhase.py mkdir FileExistsError: [Errno 17] File exists: b'/var/tmp/portage/sys-devel/gcc-11.2.0/files'
Product: Portage Development Reporter: fkhp <fkhp101>
Component: CoreAssignee: Portage team <dev-portage>
Status: RESOLVED FIXED    
Severity: normal CC: artyom.shinkaroff, ionen, mgorny, sam
Priority: Normal Keywords: InVCS, REGRESSION
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://github.com/gentoo/portage/pull/761
https://bugs.gentoo.org/show_bug.cgi?id=816291
https://bugs.gentoo.org/show_bug.cgi?id=820374
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on:    
Bug Blocks: 814857    
Attachments: build.log
environment
emerge --info

Description fkhp 2021-10-03 01:38:51 UTC
cpu: arm cortex-55, march=armv8.2a+crypto
platform: termux, proot-distro



++ emerge -v1 --nodeps =sys-devel/gcc-11.2.0
setlocale: unsupported locale setting
setlocale: unsupported locale setting
 * WARNING: The FEATURES variable contains one or more values that
 * should be disabled under normal circumstances: keepwork

 * IMPORTANT: 9 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.


These are the packages that would be merged, in order:

[ebuild  NS    ] sys-devel/gcc-11.2.0:11::gentoo [10.3.0-r2:10::gentoo] USE="(cxx) fortran nls nptl openmp pch (pie) sanitize ssp (-ada) -custom-cflags% -d -debug -doc (-fixed-point) -go -graphite (-hardened) -jit (-libssp) -lto (-multilib) -objc -objc++ -objc-gc -pgo -systemtap -test (-valgrind) -vanilla -vtv -zstd" 0 KiB

Total: 1 package (1 in new slot), Size of downloads: 0 KiB


>>> Verifying ebuild manifests

>>> Running pre-merge checks for sys-devel/gcc-11.2.0
>>> It appears that 'gcc-11.2.0' is already pretended; skipping.
>>> Remove '/var/tmp/portage/sys-devel/gcc-11.2.0/.pretended' to force pretend.

>>> Emerging (1 of 1) sys-devel/gcc-11.2.0::gentoo
 * gcc-11.2.0.tar.xz BLAKE2B SHA512 size ;-) ...                                                                       [ ok ]
 * gcc-11.2.0-patches-1.tar.bz2 BLAKE2B SHA512 size ;-) ...                                                            [ ok ]
>>> It appears that 'gcc-11.2.0' is already setup; skipping.
>>> Remove '/var/tmp/portage/sys-devel/gcc-11.2.0/.setuped' to force setup.
Exception in callback AsynchronousTask._exit_listener_cb(<bound method...0x73ee0175e0>>)
handle: <Handle AsynchronousTask._exit_listener_cb(<bound method...0x73ee0175e0>>)>
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.9/site-packages/_emerge/AsynchronousTask.py", line 210, in _exit_listener_cb
    listener(self)
  File "/usr/lib/python3.9/site-packages/_emerge/EbuildPhase.py", line 187, in _async_start_exit
    self._start_lock()
  File "/usr/lib/python3.9/site-packages/_emerge/EbuildPhase.py", line 210, in _start_lock
    self._start_ebuild()
  File "/usr/lib/python3.9/site-packages/_emerge/EbuildPhase.py", line 248, in _start_ebuild
    _prepare_fake_filesdir(self.settings)
  File "/usr/lib/python3.9/site-packages/portage/package/ebuild/prepare_build_dirs.py", line 486, in _prepare_fake_filesdir
    shutil.copytree(real_filesdir, filesdir, copy_function=copyfile)
  File "/usr/lib/python3.9/site-packages/portage/__init__.py", line 282, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
  File "/usr/lib/python3.9/shutil.py", line 557, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
  File "/usr/lib/python3.9/shutil.py", line 458, in _copytree
    os.makedirs(dst, exist_ok=dirs_exist_ok)
  File "/usr/lib/python3.9/os.py", line 225, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: b'/var/tmp/portage/sys-devel/gcc-11.2.0/files'
Terminated
++ : 0


Reproducible: Always

Steps to Reproduce:
1.create /etc/portage/env/sys-devel/gcc
pre_src_install () { 
    [[ $- = *x* ]] || { set -x; prev_xtrace="set +x"; }
echo EBUILD_PHASE=${EBUILD_PHASE}:EBUILD_PHASE_FUNC=${EBUILD_PHASE_FUNC}:EPREFIX=${EPREFIX}:
echo :$0:$@:$LINENO:
    p=${WORKDIR}/build/aarch64-unknown-linux-gnu/libatomic
    n=0
    for i in $(find $p -name Makefile); do
        grep armv8-a $i || continue
	echo FIX march: $i
	mv $i $i.bk && cp $i.bk $i && sed -i 's,armv8-a+lse,armv8.2-a+crypto,' $i
	grep ^IFUNC_OPTIONS ${i}*
	cd `dirname $i` && make clean
	: $((n++))
    done
    [[ $n == 0 ]] || {
        rm ${WORKDIR}/../.compiled
        echo "FIXed march, please run emerge for gcc again."
        exit 0
    }
    [[ -v prev_xtrace ]] && $prev_xtrace
}

2. emerge -v1 --nodeps =sys-devel/gcc-11.2.0
it will exit after patch the Makefile and make clean
3. emerge -v1 --nodeps =sys-devel/gcc-11.2.0

Actual Results:  
EbuildPhase.py mkdir FileExistsError: [Errno 17] File exists: b'/var/tmp/portage/sys-devel/gcc-11.2.0/files'

Expected Results:  
EbuildPhase.py need not mkdir b'/var/tmp/portage/sys-devel/gcc-11.2.0/files' if it existed.
and emerge should continue the compile phase till emerge success.

downgrade to sys-apps/portage-3.0.24 resolved the issue.
Comment 1 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-10-03 01:43:00 UTC
Could you please upload the full build.log, environment file, and emerge --info?
Comment 2 Ionen Wolkens gentoo-dev 2021-10-03 02:00:36 UTC
To clarify, it's a regression with FEATURES="keepwork"

The files/ dir may already exist then, but the new copy mechanism fails when it does.

To simplify the example without package.env, can run this twice (2nd will fail):

    FEATURES=keepwork emerge gzip

It's kinda nasty, but that did work with 3.0.24
Comment 3 fkhp 2021-10-03 02:50:07 UTC
Created attachment 742566 [details]
build.log

the log is not complete, it does not included the error message in the console.
Comment 4 fkhp 2021-10-03 02:50:40 UTC
Created attachment 742569 [details]
environment
Comment 5 fkhp 2021-10-03 02:51:06 UTC
Created attachment 742572 [details]
emerge --info
Comment 6 fkhp 2021-10-03 02:57:07 UTC
(In reply to fkhp from comment #3)
> Created attachment 742566 [details]
> build.log
> 
> the log is not complete, it does not included the error message in the
> console.

the error mesage in the console was not put into the build.log, it seems emerge exit bluntly when the exception happened without gracefully put all the following message into the build.log, and left a broken build.log.
Comment 7 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2021-10-03 07:02:45 UTC
I knew something like this is going to happen sooner or later.  How would you feel if I just reverted my original patch and restored the old Portage behavior for the time being?
Comment 8 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2021-10-03 07:46:06 UTC
(In reply to Michał Górny from comment #7)
> I knew something like this is going to happen sooner or later.  How would
> you feel if I just reverted my original patch and restored the old Portage
> behavior for the time being?

What about just ignoring if it already exists? I don’t think it’s possible/sane to make keepwork function correctly here anyway.
Comment 9 Larry the Git Cow gentoo-dev 2021-10-03 19:31:18 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f371f1804c5d2d263ca78aa79c3feed1bc5831ae

commit f371f1804c5d2d263ca78aa79c3feed1bc5831ae
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2021-10-03 17:48:48 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2021-10-03 19:30:33 +0000

    prepare_build_dirs: handle copytree FileExistsError
    
    Bug: https://bugs.gentoo.org/815871
    Reviewed-by: Michał Górny <mgorny@gentoo.org>
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 lib/portage/package/ebuild/prepare_build_dirs.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
Comment 10 Zac Medico gentoo-dev 2021-10-28 06:10:02 UTC
*** Bug 820374 has been marked as a duplicate of this bug. ***
Comment 11 wolfwood 2021-11-28 19:50:19 UTC
I hit this recently, when trying to update help2man emerge kept hitting an exception symlinking and exited.  I deleted the /var/tmp/portage/sys-apps/help2man-1.48.5 directory and the update succeeded.

>>> Emerging (4 of 19) sys-apps/help2man-1.48.5::gentoo
>>> Jobs: 1 of 19 complete, 2 running               Load avg: 3.75, 3.92, 3.96Exception in callback AsynchronousTask._exit_listener_cb(<bound method...7fb81cc50670>>)
handle: <Handle AsynchronousTask._exit_listener_cb(<bound method...7fb81cc50670>>)>
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/portage/package/ebuild/prepare_build_dirs.py", line 484, in _prepare_fake_filesdir
    link_target = os.readlink(symlink_path)
  File "/usr/lib/python3.9/site-packages/portage/__init__.py", line 282, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
OSError: [Errno 22] Invalid argument: b'/var/tmp/portage/sys-apps/help2man-1.48.5/files'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/lib/python3.9/site-packages/_emerge/AsynchronousTask.py", line 210, in _exit_listener_cb
    listener(self)
  File "/usr/lib/python3.9/site-packages/_emerge/EbuildPhase.py", line 187, in _async_start_exit
    self._start_lock()
  File "/usr/lib/python3.9/site-packages/_emerge/EbuildPhase.py", line 210, in _start_lock
    self._start_ebuild()
  File "/usr/lib/python3.9/site-packages/_emerge/EbuildPhase.py", line 248, in _start_ebuild
    _prepare_fake_filesdir(self.settings)
  File "/usr/lib/python3.9/site-packages/portage/package/ebuild/prepare_build_dirs.py", line 486, in _prepare_fake_filesdir
    os.symlink(real_filesdir, symlink_path)
  File "/usr/lib/python3.9/site-packages/portage/__init__.py", line 282, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
FileExistsError: [Errno 17] File exists: b'/usr/portage/sys-apps/help2man/files' -> b'/var/tmp/portage/sys-apps/help2man-1.48.5/files'
[1]    18104 terminated  sudo emerge -auvND --keep-going @world
Comment 12 Mike Gilbert gentoo-dev 2022-04-11 20:22:43 UTC
Released in portage-3.0.27.