Bug 699400 – sys-apps/portage: emirrordist DeletionTask rename error handling could be better

• Product: Portage Development
• Status: RESOLVED FIXED

[Description Zac Medico 2019-11-05 19:30:44 UTC]

After rename fails with ENOENT, it goes ahead and tries to copy the file anyway, as seen here (UnboundLocalError is handled in bug 699392):

> [ERROR] rename symlink-test from distfiles to recycle failed: [Errno 2] No such file or directory
> Traceback (most recent call last):
>   File "/usr/lib64/python2.7/site-packages/portage/util/_async/ForkProcess.py", line 55, in _spawn
>     rval = self._run()
>   File "/usr/lib64/python2.7/site-packages/portage/util/_async/FileCopier.py", line 16, in _run
>     shutil.copy(self.src_path, self.dest_path)
>   File "/usr/lib64/python2.7/site-packages/portage/__init__.py", line 246, in __call__
>     rval = self._func(*wrapped_args, **wrapped_kwargs)
>   File "/usr/lib64/python2.7/shutil.py", line 139, in copy
>     copyfile(src, dst)
>   File "/usr/lib64/python2.7/shutil.py", line 96, in copyfile
>     with open(src, 'rb') as fsrc:
> IOError: [Errno 2] No such file or directory: '/data/gmirror-distfiles/distfiles/ee/symlink-test'
> Traceback (most recent call last):
>   File "/usr/bin/emirrordist", line 86, in <module>
>     exec(data, new_globals)
>   File "<string>", line 23, in <module>
>   File "/usr/lib64/python2.7/site-packages/portage/_emirrordist/main.py", line 450, in emirrordist_main
>     signum = run_main_scheduler(MirrorDistTask(config))
>   File "/usr/lib64/python2.7/site-packages/portage/util/_async/run_main_scheduler.py", line 27, in run_main_scheduler
>     scheduler.wait()
>   File "/usr/lib64/python2.7/site-packages/_emerge/AsynchronousTask.py", line 83, in wait
>     self.scheduler.run_until_complete(self.async_wait())
>   File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 831, in run_until_complete
>     self.iteration()
>   File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 285, in iteration
>     return self._iteration(*args)
>   File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 306, in _iteration
>     if self._run_timeouts():
>   File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 601, in _run_timeouts
>     if self._run_idle_callbacks():
>   File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 560, in _run_idle_callbacks
>     if x._callback(*x._args):
>   File "/usr/lib64/python2.7/site-packages/portage/util/_eventloop/EventLoop.py", line 101, in __call__
>     self._callback(*self._args)
>   File "/usr/lib64/python2.7/site-packages/_emerge/AsynchronousTask.py", line 84, in wait
>     self._wait_hook()
>   File "/usr/lib64/python2.7/site-packages/_emerge/AsynchronousTask.py", line 195, in _wait_hook
>     self._exit_listener_stack.pop()(self)
>   File "/usr/lib64/python2.7/site-packages/portage/_emirrordist/DeletionTask.py", line 92, in _recycle_copier_exit
>     "to recycle failed: %s") % (self.distfile, e))
> UnboundLocalError: local variable 'e' referenced before assignment

[Comment 1 Zac Medico 2019-11-06 08:13:49 UTC]

Patches posted for review:

https://github.com/gentoo/portage/pull/475
https://archives.gentoo.org/gentoo-portage-dev/message/71872fbcf61541c8e468c05bcda745ae

[Comment 2 Larry the Git Cow 2019-11-07 06:29:48 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=e5c7b9e292c81c74c6b016aa03ef76214515f0bd

commit e5c7b9e292c81c74c6b016aa03ef76214515f0bd
Author:     Zac Medico <[e-mail]>
AuthorDate: 2019-11-06 08:03:48 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2019-11-06 20:05:27 +0000

    emaint: log FileCopier exceptions
    
    Since FileCopier now captures exceptions instead of showing
    a traceback, it's necessary to explicitly log them.
    
    Bug: https://bugs.gentoo.org/699400
    Signed-off-by: Zac Medico <[e-mail]>

 lib/portage/_emirrordist/DeletionTask.py | 2 +-
 lib/portage/_emirrordist/FetchTask.py    | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

https://gitweb.gentoo.org/proj/portage.git/commit/?id=7b8f57335c43054fe4008b7401d6ac2b3f710c1a

commit 7b8f57335c43054fe4008b7401d6ac2b3f710c1a
Author:     Zac Medico <[e-mail]>
AuthorDate: 2019-11-06 08:03:36 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2019-11-06 20:05:27 +0000

    FileCopier: capture exceptions
    
    Use ForkExecutor to capture exceptions instead of showing
    a full traceback. FileCopier callers will now be responsible
    for displaying relevant exception messages.
    
    Bug: https://bugs.gentoo.org/699400
    Signed-off-by: Zac Medico <[e-mail]>

 lib/portage/tests/util/test_file_copier.py | 44 ++++++++++++++++++++++++++++++
 lib/portage/util/_async/FileCopier.py      | 16 ++++++-----
 2 files changed, 53 insertions(+), 7 deletions(-)

[Comment 3 Zac Medico 2019-11-07 06:36:12 UTC]

If self.distfile_path refers to a broken symlink then we really should just remove it rather than try to rename the canonical path which doesn't exist.

[Comment 4 Zac Medico 2019-11-07 21:12:09 UTC]

(In reply to Zac Medico from comment #3)
> If self.distfile_path refers to a broken symlink then we really should just
> remove it rather than try to rename the canonical path which doesn't exist.

Actually, we shouldn't have a symlink here since self.distfile_path was added:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=af9c77bd037694164755d06034a74456ec45e3ab

[Comment 5 Larry the Git Cow 2019-11-09 05:22:55 UTC]

The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a95264dd3d846c10045644d5385d96826fbbae78

commit a95264dd3d846c10045644d5385d96826fbbae78
Author:     Zac Medico <[e-mail]>
AuthorDate: 2019-11-09 05:16:02 +0000
Commit:     Zac Medico <[e-mail]>
CommitDate: 2019-11-09 05:22:38 +0000

    sys-apps/portage: Bump to version 2.3.79
    
     #673400 emerge: fix error message for unknown options
     #699392 emirrordist: _recycle_copier_exit UnboundLocalError
     #699400 emirrordist: clean up FileCopier exception logging
     #699548 install.py: ignore -Z / --context
    
    Bug: https://bugs.gentoo.org/697734
    Bug: https://bugs.gentoo.org/673400
    Bug: https://bugs.gentoo.org/699392
    Bug: https://bugs.gentoo.org/699400
    Bug: https://bugs.gentoo.org/699548
    Package-Manager: Portage-2.3.78, Repoman-2.3.17
    Signed-off-by: Zac Medico <[e-mail]>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.79.ebuild | 261 +++++++++++++++++++++++++++++++++
 2 files changed, 262 insertions(+)