If /var/tmp/portage (by default a directory) is replaced by an absolute symlink (or possibly a relative one) to a different directory (e.g. to work round low disk space in /var/tmp) the pyside6 build can fail. The failure message is detailed below; the build fails to find any of the Qt modules. The failure happens at step 253/964 (or thereabouts) when sources/pyside6/PySide6/QtCore/../support/generate_pyi.py is run for QtCore. QtCore has been built, it's in the right place (I think) but it's not found/known. The trivial work-round is to remove the symlink and let emerge create the /var/tmp/portage directory. It would probably also work to 'mount -o remount' a bigger disk on /var/tmp/portage. Reproducible: Always Steps to Reproduce: 1.rm -rf /var/tmp/portage # or rename 2.ln -s /bigdisc/tmp/portage /var/tmp/portage # i.e. an absolute symlink # NOTE: the target directory needs to exist :-) 3.emerge pyside6 # FAILS 4.rm /var/tmp/portage 5. emerge pyside6 # SUCCEEDS Actual Results: Search for 'QtCore' (with the single quotes) from the bottom of build.log; somehow the symlink is stopping sources/pyside6/PySide6/QtCore/../support/generate_pyi.py from finding the (shiboken?) Qt modules that have been build in the work directory: FAILED: PySide6/QtCore/CMakeFiles/QtCore_pyi /var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6_build-python3_11/PySide6/QtCore/CMakeFiles/QtCore_pyi cd /var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6_build-python3_11/PySide6/QtCore && /usr/bin/cmake -E env LD_LIBRARY_PATH=/var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6_build-python3_11/libpyside:/var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6_build-python3_11/libpysideqml:/usr/lib64 /var/tmp/portage/dev-python/pyside6-6.6.2-r1/temp/python3.11/bin/python3 /var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6/PySide6/QtCore/../support/generate_pyi.py QtCore --sys-path /var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6_build-python3_11 /usr/lib/python3.11/site-packages/shiboken6/.. Traceback (most recent call last): File "/var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6/PySide6/QtCore/../support/generate_pyi.py", line 91, in <module> generate_all_pyi(outpath, options=options) File "/var/tmp/portage/dev-python/pyside6-6.6.2-r1/work/pyside-setup-everywhere-src-6.6.2/sources/pyside6/PySide6/QtCore/../support/generate_pyi.py", line 49, in generate_all_pyi raise ImportError(f"The module(s) '{errors}' do not exist") ImportError: The module(s) 'QtCore' do not exist Expected Results: A clean build (i.e. as step (5) if the symlink isn't present). This may be because shiboken has code to follow symlinks; this was the cast in the antecedent trolltech code and that code failed when syspath (to generator/shiboken) contained '..' elements in the wrong place. Clearly it isn't that simple because I used an absolute symlink and it is way down the build tree. The bug is almost impossible to debug; I've spent the entire time since 6.6.2 replaced 6.6.1 trying to find out what was wrong and it was only when I recreated the 6.6.1 .ebuild (which had emerged fiine) that I realized it did happen there too so must have been the result of some other change on my system. Even then it took a while; I had introduced the symlink to allow webkit to build with -g and it only occurred to me that it might be a symlink issue because I had previously debugged the original trolltech code in PythonQt. I'm filing this bug not because I expect it to be fixed (I haven't checked shiboken but the trolltech code was completely rotted and unmaintainable) but because it might help someone else who encounters this problem.
Created attachment 888009 [details] build.log
Created attachment 888010 [details] emerge --info '=dev-python/pyside6-6.6.2-r1::gentoo'
Created attachment 888011 [details] emerge -pqv '=dev-python/pyside6-6.6.2-r1::gentoo'
While attempting to get to the bottom of this issue I did also attempt builds without -g (i.e. with no debugging) with -j1 (so ninja runs -j1 not -j12) and with a minimally configured pyside6; both variants of this package.use: # TEST: trying to get 6.6.2 to build #=dev-python/pyside6-6.6.2-r1 -bluetooth -concurrent -dbus -gui -network -opengl -printsupport -python_targets_python3_12 -qml -sql -svg -testlib -webchannel -widgets -xml =dev-python/pyside6-6.6.2-r1 -python_targets_python3_11 -bluetooth -concurrent -network -printsupport -qml -sql -svg -testlib -webchannel -xml =dev-python/shiboken6-6.6.2-r1 -python_targets_python3_11 They all failed the same way. I guess it could still be some kind of race; /var/tmp on my system is a pair of RAID NVME cards whereas the disc I used is a SATA drive.
Should really just set PORTAGE_TMPDIR when you need a different directory rather than a symlink (this can be changed per-package through package.env if needed too). Haven't looked at pyside6, but Qt in general had a lot of misc issues with symlinks (esp.. qtwebengine given Gn tries to get the absolute path and ends up confused).
(In reply to Ionen Wolkens from comment #5) I suspect it will also happen if /var/tmp or /var are symbolic links (but I haven't tested this). I've frequently used a symbolic link for /var/tmp in the past. > Should really just set PORTAGE_TMPDIR when you need a different directory > rather than a symlink (this can be changed per-package through package.env > if needed too). Yes, maybe, but the real problem is that the failure is incomprehensible. A minimal fix is to cause the emerge to fail with a meaningful message: test "$(realpath "$PORTAGE_TMPDIR")" = "$PORTAGE_TMPDIR" || echo "$PORTAGE_TMPDIR: pyside6 build will fail" > Haven't looked at pyside6, but Qt in general had a lot of misc issues with > symlinks (esp.. qtwebengine given Gn tries to get the absolute path and ends > up confused). Yes, I saw the qtwebengine fail too but it was easier to understand; /var/tmp/portage got repeated in some of the paths in the log file. Perhaps a general 'solution' is to do this for all the Qt ebuilds: PORTAGE_TMPDIR="$(realpath "${PORTAGE_TMPDIR}")" possibly with an ewarn that this was done. I don't know if 'realpath' is universally available in gentoo and I think it can sometimes fail (if one of the directories to the root has been removed for example).
(In reply to John Bowler from comment #6) > (In reply to Ionen Wolkens from comment #5) > I suspect it will also happen if /var/tmp or /var are symbolic links (but I > haven't tested this). I've frequently used a symbolic link for /var/tmp in > the past. It won't because portage already does realpath for PORTAGE_TMPDIR, but does not expect the last bit beyond the tmpdir (portage directory) to have been replaced by a symlink. Can be seen from an ebuild: mkdir /tmp/real ln -s /tmp/real /tmp/symlink PORTAGE_TMPDIR=/tmp/symlink emerge some-test-ebuild src_prepare() { die "${S}" } -> * ERROR: media-gfx/imv-4.5.0::gentoo failed (prepare phase): * /tmp/real/portage/media-gfx/imv-4.5.0/work/imv-v4.5.0 As you can see, there's no /symlink/ there.
(Qt did have other nonsense where it tried to reduce relative symlinks with some poor heuristics and got confused by /var/tmp -> /tmp by finding the same files there though, wouldn't happen if the directories were just named differently, bug #914195 -- realpath wouldn't even help there)
tl;dr, I'm not sure if the case where the portage-owned directory was replaced can be considered a bug -- feels more like misuse
(In reply to Ionen Wolkens from comment #9) > tl;dr, I'm not sure if the case where the portage-owned directory was > replaced can be considered a bug -- feels more like misuse (at most I guess portage could consider realpath'ing that bit too, but well)
(In reply to Ionen Wolkens from comment #10) > (In reply to Ionen Wolkens from comment #9) > > tl;dr, I'm not sure if the case where the portage-owned directory was > > replaced can be considered a bug -- feels more like misuse > > (at most I guess portage could consider realpath'ing that bit too, but well) There's also the question of where does this end, will dev-qt category directory get replaced by a symlink too? ;p
(In reply to Ionen Wolkens from comment #11) > (In reply to Ionen Wolkens from comment #10) > > (In reply to Ionen Wolkens from comment #9) > > > tl;dr, I'm not sure if the case where the portage-owned directory was > > > replaced can be considered a bug -- feels more like misuse > > > > (at most I guess portage could consider realpath'ing that bit too, but well) > There's also the question of where does this end, will dev-qt category > directory get replaced by a symlink too? ;p No, the path below PORTAGE_TMPDIR is pruned by a successful emerge. PORTAGE_TMPDIR is the only thing which isn't removed and, in fact, it always contains a sub-directory ._unmerge_ $PORTAGE_TMPDIR/portage can be removed; traditional UNIX would remove everything from /tmp on boot and I have certain used a memory file system for /tmp in the past (not that this works with webkit stuff of course :-( I tried with this in make.conf: PORTAGE_TMPDIR=/root/workspace/tmp Then (as root) whish /store being the mounted (SATA) disk:: ln -s /store/workspace /root/workspace mkdir -p /store/workspace/tmp chmod 1777 /store/workspace/tmp emerge pyside6 This works, apparently because $PORTAGE_TMPDIR *is* being 'realpathed'; build.log records that the source is unpacked to '/store/workspace/tmp/portage/dev-python/'... The behavior isn't Qt specific; emerge sys-devel/gcc shows the same behavior. Maybe someone encountered issues before this and implemented a general fix, but the nature of the fix (PORTAGE_TMPDIR, not PORTAGE_TMPDIR/portage) suggests that you care correct and what I did is "misuse". I verified that the behavior does still exist with this setup: cd /store/workspace/tmp rm -rf portage mkdir build ln -s build portage emerge pyside6 now fails. Ok, it's a "misuse" but non-Qt packages seem fine with the setup and, indeed, most of the Qt packages (dev-qt/) seemed ok too. The only one I saw failing was qtwebengine. Is there some way of getting input from the gentoo emerge guys? I *could* have just submitted a bug report; I almost did. So far as I can see there's nothing in the build information which reveals what I had done; no evidence that /var/tmp/portage was a symbolic link. So a bug report without the above analysis would have caused lots of confusion (though maybe not now you know about it :-) If what I did is wrong emerge should remove it as an error, but that's a judgement that needs to come from the emerge maintainers.
Sorry, but this isn't a supported configuration.
(In reply to John Bowler from comment #12) > > I *could* have just submitted a bug report; I almost did. So far as I can > see there's nothing in the build information which reveals what I had done; > no evidence that /var/tmp/portage was a symbolic link. So a bug report > without the above analysis would have caused lots of confusion (though maybe > not now you know about it :-) I don't think so, as it was obvious it was an odd configuration to begin with. Like Ionen said, we've dealt with this before. We know about it in Qt stuff, who knows what else breaks. I just don't get the point - why not set PORTAGE_TMPDIR properly?
(In reply to Sam James from comment #14) > I don't think so, as it was obvious it was an odd configuration to begin > with. To you, but not to me. > Like Ionen said, we've dealt with this before. We know about it in Qt stuff, > who knows what else breaks. I just don't get the point - why not set > PORTAGE_TMPDIR properly? It was a hack; a temporary measure I put in place while attempting to get 6.6.2 to build. It looks like I did it on Feb 27 (the directory datestamp) but it might have been a few days before that. So I was attempting to build pyside6-6.6.2, not pyside6-6.6.2-r1. I put the hack in to get qtwebengine6 to build; I don't use it but I thought not having it might be the problem. I completely accept that it's not a supported configuration; this isn't a "bug" it's a QoI issure. (I consider the upstream behavior a bug; DIY realpath, but I wasn't reporting that and upstream can dismiss it similarly.) At this point, having understood the issue, I'd say it is as worthy of an ewarn as webkit's massive disk space usage is. That's still QoI.
The supported way of doing that is per-package PORTAGE_TMPDIR via package.env: * https://wiki.gentoo.org/wiki/Knowledge_Base:Overriding_environment_variables_per_package * https://wiki.gentoo.org/wiki//etc/portage/package.env (In reply to John Bowler from comment #15) > (In reply to Sam James from comment #14) > > I don't think so, as it was obvious it was an odd configuration to begin > > with. > > To you, but not to me. > My point was that omitting any explanation wouldn't have made a difference.
Anyway, we should probably mention in the Portage man pages that symlinking PORTAGE_TMPDIR isn't a good idea.
(In reply to Sam James from comment #17) > Anyway, we should probably mention in the Portage man pages that symlinking > PORTAGE_TMPDIR isn't a good idea. Yes, although in fact a symlink for $PORTAGE_TMPDIR itself is fine since as @ionen observed and I unnecessarily repeated; $PORTAGE_TMPDIR is 'realpathed'. The problem is that "$PORTAGE_TMPDIR/portage" must not be a symlink (well, if it is at least some Qt packages will fail). So maybe make.conf should simply say that the structure *under* $PORTAGE_TMPDIR cannot be played with; not just ./portage but also, I assume, ./ccache? I've ended up with PORTAGE_TMPDIR=/tmp (which is tmpfs) FEATURES="fail-clean" (my /tmp only as 32G free - I don't use a swapfile) and /etc/portage/env/large-tmp which contains a definition of PORTAGE_TMPDIR pointing to somewhere on a large disk. I also set PORTAGE_LOGDIR as suggested by make.conf(5) It's confusing/difficult to explain because the default for PORTAGE_TEMPDIR is /var/tmp which is certainly a place where other applications and, indeed, users can create arbitrary files/directories/sockets/symlinks/etc. This is a well known source of bugs in traditional UNIX. In essence emerge/ebuild/gentoo is reserving a set of names under $PORTAGE_TMPDIR which cannot be used or changed by *anything* else. Meanwhile $PORTAGE_TMPDIR must exist, so: PORTAGE_TMPDIR=/tmp/reserved-for-portage # directory does not exist emerge --onshot sys-devel/gcc # immediately fails I want PORTAGE_TMPDIR to be a temporary directory under normal circumstances, but I can't make it a sub-directory of /tmp because /tmp is tmpfs (so the sub-directory disappears) On Linux /var/tmp is meant to be preserved across reboot so it's ok. I can think of other things I could do and maybe will do, e.g. making PORTAGE_TMPDIR be its own tmpfs (a mount point). My point is that it's tricky and more difficult to document.
