At least python3, I don't know how to test this for python2. How it looks: $ python3 -c 'import concurrent.futures' Traceback (most recent call last): File "/usr/lib64/python3.3/multiprocessing/synchronize.py", line 27, in <module> from _multiprocessing import SemLock ImportError: cannot import name SemLock During handling of the above exception, another exception occurred: Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python3.3/concurrent/futures/__init__.py", line 17, in <module> from concurrent.futures.process import ProcessPoolExecutor File "/usr/lib64/python3.3/concurrent/futures/process.py", line 53, in <module> from multiprocessing.queues import SimpleQueue, Full File "/usr/lib64/python3.3/multiprocessing/queues.py", line 23, in <module> from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition File "/usr/lib64/python3.3/multiprocessing/synchronize.py", line 32, in <module> " function, see issue 3770.") ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770. Because of obscurity of problem it will be good to add check and warning in ebuild. Reproducible: Always
The same issue is present for python2. Honestly, I don't feel it is our responsibility to tell you how to properly configure a chroot build environment. Our installation handbook already properly documents that. So my initial reaction is to reject this request. However, I can see the value in preventing broken systems. What does the rest of the python team think on this?
I have the same problem with with python2.7 and python3.2 trying to run "meld" meld /etc/ddclient/ddclient.conf /etc/ddclient/._cfg0000_ddclient.conf Traceback (most recent call last): File "/usr/bin/meld", line 173, in <module> main() File "/usr/bin/meld", line 165, in main new_window = app.parse_args(sys.argv[1:]) File "/usr/lib64/meld/meld/meldapp.py", line 169, in parse_args tab = open_paths(args, options.auto_compare, options.auto_merge) File "/usr/lib64/meld/meld/meldwindow.py", line 796, in open_paths tab = self.append_diff(paths, auto_compare, auto_merge) File "/usr/lib64/meld/meld/meldwindow.py", line 746, in append_diff return self.append_filediff(paths, merge_output=merge_output) File "/usr/lib64/meld/meld/meldwindow.py", line 703, in append_filediff doc = filediff.FileDiff(app.prefs, len(files)) File "/usr/lib64/meld/meld/filediff.py", line 214, in __init__ self._cached_match = CachedSequenceMatcher() File "/usr/lib64/meld/meld/filediff.py", line 72, in __init__ None, matchers.init_worker, maxtasksperchild=1) File "/usr/lib64/python2.7/multiprocessing/__init__.py", line 232, in Pool return Pool(processes, initializer, initargs, maxtasksperchild) File "/usr/lib64/python2.7/multiprocessing/pool.py", line 138, in __init__ self._setup_queues() File "/usr/lib64/python2.7/multiprocessing/pool.py", line 232, in _setup_queues from .queues import SimpleQueue File "/usr/lib64/python2.7/multiprocessing/queues.py", line 48, in <module> from multiprocessing.synchronize import Lock, BoundedSemaphore, Semaphore, Condition File "/usr/lib64/python2.7/multiprocessing/synchronize.py", line 59, in <module> " function, see issue 3770.") ImportError: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.
Please do not turn this bug into a series of "me too" comments. We do not require any additional information.
I agree that it's not our responsibility to tell users how to configure a chroot. On the other hand, I do tend to think that with USE flags and everything, we try to provide users with predictable builds, in the sense that we try to replace automatic detection of dependency presence by explicit configuration through USE flags. Also, at least for me, I could agree that tracing this error message back to /dev/shm not being present at build time is... less than intuitive. As such, I think maybe adding a warning to our ebuild iff /dev/shm is not available would be a fairly cheap improvement.
Do we support any systems without /dev/shm? If not, we may just force it on. Also, I feel like this /dev/shm checking is really stupid and not cross-compile friendly. Wouldn't it be better to check some system functions or something? But I guess this is an upstream issue.
(In reply to Michał Górny from comment #5) > Also, I feel like this /dev/shm checking is really stupid and not > cross-compile friendly. Wouldn't it be better to check some system functions > or something? But I guess this is an upstream issue. So the configure check already compiles and runs a program which calls sem_open() to test for this. The problem is that the configure check is nonfatal; if sem_open fails, configure just sets the POSIX_SEMAPHORES_NOT_ENABLED macro and moves on. So, here are a couple of more ideas. 1. Patch configure to fail. I'm not sure if this would be accepted upstream. 2. Do a grep check on pyconfig.h and die if POSIX_SEMAPHORES_NOT_ENABLED has been defined.
Another idea: we could set the relevant autoconf cache vars to bypass the configure checks entirely, which would already happen for cross-compiles. export ac_cv_posix_semaphores_enabled=yes export ac_cv_broken_sem_getvalue=no
I went with the pyconfig.h check. + 30 Dec 2013; Mike Gilbert <floppym@gentoo.org> python-2.6.8-r3.ebuild, + python-2.6.9.ebuild, python-2.7.5-r3.ebuild, python-2.7.5-r4.ebuild, + python-2.7.6.ebuild, python-3.2.5-r3.ebuild, python-3.3.2-r2.ebuild, + python-3.3.3.ebuild: + Die if sem_open is detected as broken, bug 496328.
*** Bug 496332 has been marked as a duplicate of this bug. ***
Following the amd64 Handbook now, I encountered this bug. If broken chroots are what is being blamed, the handbooks should reflect this.
(In reply to Kyle Sanderson from comment #10) > Following the amd64 Handbook now, I encountered this bug. If broken chroots > are what is being blamed, the handbooks should reflect this. Did you miss this part in "Code Listing 1.4: Mounting /proc and /dev"? mount --rbind /dev /mnt/gentoo/dev
(In reply to Mike Gilbert from comment #11) > (In reply to Kyle Sanderson from comment #10) > > Following the amd64 Handbook now, I encountered this bug. If broken chroots > > are what is being blamed, the handbooks should reflect this. > > Did you miss this part in "Code Listing 1.4: Mounting /proc and /dev"? > > mount --rbind /dev /mnt/gentoo/dev shm is not mounted like that with the latest amd64 media. I'm not sure if it was a missing device or what, but shm wasn't present in my chroot after doing that.
(In reply to Kyle Sanderson from comment #12) > shm is not mounted like that with the latest amd64 media. I'm not sure if it > was a missing device or what, but shm wasn't present in my chroot after > doing that. It seems to work fine for me with install-amd64-minimal-20140116.iso. livecd ~ # mount --rbind /dev /mnt/gentoo/dev livecd ~ # grep shm /proc/mounts shm /mnt/gentoo/dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0
(In reply to Mike Gilbert from comment #13) I'm using the profile hardened/linux/amd64/no-multilib/selinux and getting the error "Broken sem_open function". ~ # grep shm /proc/mounts shm /dev/shm tmpfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ~ # stat /dev/shm File: ‘/dev/shm’ Size: 40 Blocks: 0 IO Block: 4096 directory Device: dh/13d Inode: 173 Links: 2 Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) Context: system_u:object_r:tmpfs_t Access: 2014-01-19 11:14:47.878608659 +0100 Modify: 2014-01-19 11:09:39.024129037 +0100 Change: 2014-01-19 11:09:39.024129037 +0100 Birth: - ~ # sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: strict Current mode: enforcing Mode from config file: enforcing Policy MLS status: disabled Policy deny_unknown status: denied Max kernel policy version: 28 ~ # grep POSIX_SEMAPHORES /var/tmp/portage/dev-lang/python-3.3.3/work/Python-3.3.3/pyconfig.h.in #undef HAVE_BROKEN_POSIX_SEMAPHORES #undef POSIX_SEMAPHORES_NOT_ENABLED
(In reply to Florian Steinel from comment #14) > (In reply to Mike Gilbert from comment #13) > > I'm using the profile hardened/linux/amd64/no-multilib/selinux and getting > the error "Broken sem_open function". This is a result of SELinux being to restrictive (tip: check your audit logs). I've send swift a patch which he'll likely include in the next policy revision; until then, you can add "fs_getattr_tmpfs(portage_sandbox_t)" to your local policy. PS: Next time, please open a separate bug instead of commenting on a related, but closed one.
(In reply to Mike Gilbert from comment #11) > (In reply to Kyle Sanderson from comment #10) > > Following the amd64 Handbook now, I encountered this bug. If broken chroots > > are what is being blamed, the handbooks should reflect this. > > Did you miss this part in "Code Listing 1.4: Mounting /proc and /dev"? > > mount --rbind /dev /mnt/gentoo/dev Indeed this does not work... /dev/shm need to be explicitly mounted as tmpfs in the chroot. See relevant catalyst commit http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=commit;h=e5a9e20376397f2714aa7a835b7f42f71fcc38d9 Could you please consider asking the docs team to update the handbook to take this into consideration?
Well, this test is really dumb then. I think we should consider just overriding the test result.
From the commit that Markos linked: > Mount /dev/shm in the chroot with the right options > Bind mounting /dev/shm into the chroot isn't a good idea, as there may > be collisions and result in weird side effects. Instead, we can just > mount a new tmpfs there, with the right options to ensure security. There is no chance of collisions during a single-user installation from a livecd. mount --rbind works fine for that use case. This bug is finding quite a few configurations that were simply broken, and nobody noticed. I would prefer to leave it as-is.
(In reply to Mike Gilbert from comment #18) > From the commit that Markos linked: > > There is no chance of collisions during a single-user installation from a > livecd. mount --rbind works fine for that use case. I know I use ssh to do all of my installs as my Java KVMoIP is pretty rough to use. I utilize multiple sessions to emerge multiple packages as well, such as configuring my kernel while upgrading something larger like gcc. This appears to be supported and recommended by the handbooks, or at least was when I did my first install. > This bug is finding quite a few configurations that were simply broken, and > nobody noticed. I would prefer to leave it as-is. Leaving it is fine (I guess updating my system before booting is foolish?), but the handbook(s) need to be updated.
(In reply to Kyle Sanderson from comment #19) > I know I use ssh to do all of my installs as my Java KVMoIP is pretty rough > to use. I utilize multiple sessions to emerge multiple packages as well, > such as configuring my kernel while upgrading something larger like gcc. > This appears to be supported and recommended by the handbooks, or at least > was when I did my first install. > Ok, and have you run into any concurrency issues with that? From your earlier comments, it seems that /dev/shm was not mounted on your system at all, which is a much different issue. I don't think the handbook needs to be updated, and I am waiting for someone to prove otherwise.
Ok, so after speaking with dwfreed about it in IRC, it does indeed seem safer to mount /dev/shm as a separate tmpfs instance in the chroot environment. However, bind-mounting does work 99.9% of the time; the exception being if you have the same application using shared memory inside and outside of the chroot environment. This could cause them to use the same "key" for a shared memory request. In any case, if you want the handbook updated, please file a separate bug for the docs team. It has nothing to do with the python team, other than we require a working /dev/shm for the sem_open configure check.
For other places where this also happens, I found that you can check the status of the /dev/shm tempfs with: mount | grep shm OR ls -ld /dev/shm If the permissions are not 1777 ... then as a temporary work around, you can just chmod 1777 /dev/shm and then the 'emerge python' should work.
If you are using Ubuntu to chroot from (because /dev/shm is a symlink on that): mount --rbind /run/shm /mnt/gentoo/run/shm It's possible this may work as well but I have not tested: mount -t tmpfs none /mnt/gentoo/run/shm Or this within the chroot: mount -t tmpfs none /run/shm
I've just hit this issue doing a catalyst 'run' on a debian host using a gentoo chroot. The failure is probably a sensible measure, but even rbind'ing /run seems to trigger the failure (probably for valid $reasons).
(In reply to Michael Everitt (IRC: veremitz) from comment #24) > I've just hit this issue doing a catalyst 'run' on a debian host using a > gentoo chroot. > > The failure is probably a sensible measure, but even rbind'ing /run seems to > trigger the failure (probably for valid $reasons). Workaround found via https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=694379#47 if anyone should hit this in future.
reopening bug as this now affects more user cases with the advent of podman and lxd containers, which don't mount a /dev/shm by default.
I'm going to proceed with the alternative approach of forcing cache vars. The mount situation of build host (arguably broken or not) should not really affect runtime behavior of Python running elsewhere, especially if that behavior boils down to being broken or not.
Ok, epsilonKNOT reported that this doesn't really fix build, just causes Python to fail at compile time. Let's forward-port Mike's fix to newer versions, I've apparently mistakenly removed it while killing USE=threads.
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=c6f51f5dc2fe040900180ae160d6b1e0985f17b9 commit c6f51f5dc2fe040900180ae160d6b1e0985f17b9 Author: Michał Górny <mgorny@gentoo.org> AuthorDate: 2021-01-27 14:51:37 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2021-01-27 15:10:03 +0000 dev-lang/python: Readd check for failed semaphore checks in 3.7+ Reintroduce the check for failing semaphore checks that result in broken multiprocessing in Python. This check has been mistakenly removed while removing support for USE=-threads. Closes: https://bugs.gentoo.org/496328 Signed-off-by: Michał Górny <mgorny@gentoo.org> dev-lang/python/python-3.7.9-r2.ebuild | 6 ++++++ dev-lang/python/python-3.8.7-r1.ebuild | 6 ++++++ dev-lang/python/python-3.9.1-r1.ebuild | 6 ++++++ 3 files changed, 18 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=56566e139e376bd02c170be73a56ba933766979a commit 56566e139e376bd02c170be73a56ba933766979a Author: Mike Gilbert <floppym@gentoo.org> AuthorDate: 2021-01-27 16:06:38 +0000 Commit: Mike Gilbert <floppym@gentoo.org> CommitDate: 2021-01-27 16:06:38 +0000 dev-lang/python: Readd check for failed semaphore checks in 3.10 Bug: https://bugs.gentoo.org/496328 Signed-off-by: Mike Gilbert <floppym@gentoo.org> dev-lang/python/python-3.10.0_alpha4.ebuild | 6 ++++++ 1 file changed, 6 insertions(+)
