Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 717548 - app-emulation/qemu: qemu-alpha linux-user breaks emerge with python3.6+ inside alpha chroot
Summary: app-emulation/qemu: qemu-alpha linux-user breaks emerge with python3.6+ insid...
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Matthias Maier
URL: https://lists.nongnu.org/archive/html...
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-04-15 12:28 UTC by Barnabás Virágh
Modified: 2020-04-16 22:16 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
Python3 Exception from emerge (emerge-exception.txt,6.74 KB, text/plain)
2020-04-15 12:30 UTC, Barnabás Virágh
Details
emerge --info qemu on HOST system (emerge-info.txt,10.98 KB, text/plain)
2020-04-15 12:33 UTC, Barnabás Virágh
Details
0001-linux-user-syscall.c-add-target-to-host-mapping-for-.patch (0001-linux-user-syscall.c-add-target-to-host-mapping-for-.patch,1.39 KB, patch)
2020-04-15 18:32 UTC, Sergei Trofimovich
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Barnabás Virágh 2020-04-15 12:28:41 UTC
There is a bug in qemu-alpha (tested on amd64), which prevents emerge to run, with python 3.6 or higher, which was running OK with python 2.7. However, since 2.7 is phasing out, it's not an option anymore to run emerge with python 2.7.

It can be reproduced with a single-liner:
# python3 -c 'import selectors; selectors.DefaultSelector()'

qemu-alpha upstreams bug report:
https://bugs.launchpad.net/qemu/+bug/1872847

Reproducible: Always

Steps to Reproduce:
1. Chroot into a recent qemu-alpha gentoo tree
2. # emerge --update @world

Actual Results:  
Throws exception, and exits without emerging...

Expected Results:  
Updates packages in the alpha chroot tree.

qemu-mips64, qemu-arm, qemu-aarch64 was tested by me, and are not affected, nor in v4.2.0 nor with the git version of qemu.
Comment 1 Barnabás Virágh 2020-04-15 12:30:54 UTC
Created attachment 633068 [details]
Python3 Exception from emerge
Comment 2 Barnabás Virágh 2020-04-15 12:33:58 UTC
Created attachment 633070 [details]
emerge --info qemu on HOST system
Comment 3 Sergei Trofimovich gentoo-dev 2020-04-15 17:56:17 UTC
Posting logs as is:

Traceback (most recent call last):
  File "/usr/lib/python-exec/python3.7/emerge", line 53, in <module>
    retval = emerge_main()
  File "/usr/lib/python3.7/site-packages/_emerge/main.py", line 1309, in emerge_main
    return run_action(emerge_config)
  File "/usr/lib/python3.7/site-packages/_emerge/actions.py", line 3371, in run_action
    retval = action_build(emerge_config, spinner=spinner)
  File "/usr/lib/python3.7/site-packages/_emerge/actions.py", line 358, in action_build
    settings, trees, myopts, myparams, myaction, myfiles, spinner)
  File "/usr/lib/python3.7/site-packages/_emerge/depgraph.py", line 9912, in backtrack_depgraph
    myaction, myfiles, spinner)
  File "/usr/lib/python3.7/site-packages/_emerge/depgraph.py", line 9948, in _backtrack_depgraph
    backtrack_parameters=backtrack_parameters)
  File "/usr/lib/python3.7/site-packages/_emerge/depgraph.py", line 623, in __init__
    self._event_loop = asyncio._safe_loop()
  File "/usr/lib/python3.7/site-packages/portage/util/futures/_asyncio/__init__.py", line 269, in _safe_loop
    return _global_event_loop()
  File "/usr/lib/python3.7/site-packages/portage/util/_eventloop/global_event_loop.py", line 38, in global_event_loop
    instance = constructor()._asyncio_wrapper
  File "/usr/lib/python3.7/site-packages/portage/util/_eventloop/asyncio_event_loop.py", line 32, in __init__
    loop = loop or _real_asyncio.get_event_loop()
  File "/usr/lib/python3.7/asyncio/events.py", line 640, in get_event_loop
    self.set_event_loop(self.new_event_loop())
  File "/usr/lib/python3.7/asyncio/events.py", line 660, in new_event_loop
    return self._loop_factory()
  File "/usr/lib/python3.7/asyncio/unix_events.py", line 51, in __init__
    super().__init__(selector)
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 57, in __init__
    selector = selectors.DefaultSelector()
  File "/usr/lib/python3.7/selectors.py", line 349, in __init__
    self._selector = self._selector_cls()
OSError: [Errno 22] Invalid argument
Comment 4 Sergei Trofimovich gentoo-dev 2020-04-15 17:57:21 UTC
>     self._selector = self._selector_cls()
> OSError: [Errno 22] Invalid argument

Probably missing/incomplete syscall wiring for alpha. Let's track it down and write a small example that exhibits a problem.
Comment 5 Sergei Trofimovich gentoo-dev 2020-04-15 18:03:42 UTC
(In reply to Sergei Trofimovich from comment #4)
> >     self._selector = self._selector_cls()
> > OSError: [Errno 22] Invalid argument
> 
> Probably missing/incomplete syscall wiring for alpha. Let's track it down
> and write a small example that exhibits a problem.

Probably epoll:

$ fgrep -R _selector_cls /usr/lib/python3.7
/usr/lib/python3.7/selectors.py:        _selector_cls = select.poll
/usr/lib/python3.7/selectors.py:        _selector_cls = select.epoll
/usr/lib/python3.7/selectors.py:        _selector_cls = select.devpoll
Comment 6 Sergei Trofimovich gentoo-dev 2020-04-15 18:08:03 UTC
Checking on amd64:

$ strace -f python3 -c 'import selectors; selectors.DefaultSelector()' |& fgrep poll
epoll_create1(EPOLL_CLOEXEC)            = 3

Checking on qemu-alpha:

$ cat a.c
#include <sys/epoll.h>

int main() {
    epoll_create1(EPOLL_CLOEXEC);
}
$ alpha-unknown-linux-gnu-gcc a.c -o a
$ /usr/bin/qemu-alpha -strace -L /usr/alpha-unknown-linux-gnu/ ./a
epoll_create1(2097152,274886379576,274886379592,274889063496,274877908880,274886458400) = -1 errno=22 (Invalid argument)
4162665 exit_group(0)

Looks simple.
Comment 7 Sergei Trofimovich gentoo-dev 2020-04-15 18:24:55 UTC
My guess it's because EPOLL_CLOEXEC constant differs from target to target:

$ fgrep EPOLL_CLOEXEC glibc.git/
sysdeps/unix/sysv/linux/alpha/bits/epoll.h:    EPOLL_CLOEXEC  = 010000000
sysdeps/unix/sysv/linux/alpha/bits/epoll.h:#define EPOLL_CLOEXEC EPOLL_CLOEXEC
sysdeps/unix/sysv/linux/bits/epoll.h:    EPOLL_CLOEXEC = 02000000
sysdeps/unix/sysv/linux/bits/epoll.h:#define EPOLL_CLOEXEC EPOLL_CLOEXEC
sysdeps/unix/sysv/linux/hppa/bits/epoll.h:    EPOLL_CLOEXEC = 010000000
sysdeps/unix/sysv/linux/hppa/bits/epoll.h:#define EPOLL_CLOEXEC EPOLL_CLOEXEC
sysdeps/unix/sysv/linux/mips/bits/epoll.h:    EPOLL_CLOEXEC = 02000000
sysdeps/unix/sysv/linux/mips/bits/epoll.h:#define EPOLL_CLOEXEC EPOLL_CLOEXEC
sysdeps/unix/sysv/linux/sparc/bits/epoll.h:    EPOLL_CLOEXEC = 0x400000
sysdeps/unix/sysv/linux/sparc/bits/epoll.h:#define EPOLL_CLOEXEC EPOLL_CLOEXEC
sysdeps/unix/sysv/linux/test-errno-linux.c:  fails |= test_wrp (EINVAL, epoll_create1, EPOLL_CLOEXEC + 1);
sysdeps/unix/sysv/linux/x86/bits/epoll.h:    EPOLL_CLOEXEC = 02000000
sysdeps/unix/sysv/linux/x86/bits/epoll.h:#define EPOLL_CLOEXEC EPOLL_CLOEXEC
Comment 8 Sergei Trofimovich gentoo-dev 2020-04-15 18:32:54 UTC
Created attachment 633106 [details, diff]
0001-linux-user-syscall.c-add-target-to-host-mapping-for-.patch

Barnabás, can you try a 0001-linux-user-syscall.c-add-target-to-host-mapping-for-.patch patch to see if python can survive a bit more?
Comment 9 Sergei Trofimovich gentoo-dev 2020-04-15 22:25:31 UTC
Meanwhile proposed upstream as https://lists.nongnu.org/archive/html/qemu-devel/2020-04/msg02545.html
Comment 10 Barnabás Virágh 2020-04-16 01:18:55 UTC
This seems to solve the problem. Nice catch! Works nicely now, thank you!

As I see, as a side effect, ppc & sparc targets are also repaired, which are my next planned targets to experiment with. :)
Comment 11 Barnabás Virágh 2020-04-16 01:20:02 UTC
ppc -> hppa
Comment 12 Larry the Git Cow gentoo-dev 2020-04-16 22:16:01 UTC
The bug has been closed via the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=3a1b0a5e226c561f876d84210048b1fe3c95f2bb

commit 3a1b0a5e226c561f876d84210048b1fe3c95f2bb
Author:     Sergei Trofimovich <slyfox@gentoo.org>
AuthorDate: 2020-04-16 22:15:28 +0000
Commit:     Sergei Trofimovich <slyfox@gentoo.org>
CommitDate: 2020-04-16 22:15:55 +0000

    app-emulation/qemu: fix epoll_create1 handling, bug #717548
    
    Recent python started using epoll_create1() which had
    a bug of missing translation for flags from target to host.
    
    Pull upstreamed patch.
    
    Reported-by: Barnabás Virágh
    Closes: https://bugs.gentoo.org/717548
    Package-Manager: Portage-2.3.99, Repoman-2.3.22
    Signed-off-by: Sergei Trofimovich <slyfox@gentoo.org>

 .../qemu/files/qemu-4.2.0-epoll_create1.patch      |  39 +
 app-emulation/qemu/qemu-4.2.0-r4.ebuild            | 838 +++++++++++++++++++++
 2 files changed, 877 insertions(+)