Since I have an old system, I downloaded the ebuild for dev-libs/mimalloc-2.0.3-r1 into a local overlay and gave it a try. The only thing I changed was the EAPI version (from 8 to 7). All went smoothly, up to the latest step of the installation, where I got: >>> Install dev-libs/mimalloc-2.0.3-r1 into /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image * Working in BUILD_DIR: "/XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/work/mimalloc-2.0.3_build" [0/1] Install the project... -- Install configuration: "Gentoo" -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/libmimalloc-gentoo.so.2.0 -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/libmimalloc-gentoo.so -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/cmake/mimalloc/mimalloc.cmake -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/cmake/mimalloc/mimalloc-gentoo.cmake -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/libmimalloc-gentoo.a -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/include/mimalloc.h -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/include/mimalloc-override.h -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/include/mimalloc-new-delete.h -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/cmake/mimalloc/mimalloc-config.cmake -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/cmake/mimalloc/mimalloc-config-version.cmake -- Installing: /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/mimalloc-gentoo.o rm: cannot remove '/XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/mimalloc.o': No such file or directory * ERROR: dev-libs/mimalloc-2.0.3-r1::YYYYYY failed (install phase): * (no error message) * * Call stack: * ebuild.sh, line 125: Called src_install * environment, line 1955: Called die * The specific snippet of code: * rm "${ED}/usr/$(get_libdir)/mimalloc.o" || die; Indeed, the ebuild is trying to remove mimalloc.o, or die - and it dies, because there is no mimalloc.o in the build directory: ll /XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/ total 920 drwxr-xr-x 3 root root 4096 Jan 16 19:10 cmake -rw-r--r-- 1 root root 394080 Jan 16 19:10 libmimalloc-gentoo.a lrwxrwxrwx 1 root root 25 Jan 16 19:10 libmimalloc-gentoo.so -> libmimalloc-gentoo.so.2.0 -rwxr-xr-x 1 root root 238072 Jan 16 19:10 libmimalloc-gentoo.so.2.0 -rw-r--r-- 1 root root 296704 Jan 16 19:10 mimalloc-gentoo.o There IS, however, a mimalloc-gentoo.o...and all other artifacts have this '-gentoo' suffix. I don't know if that's "normal". Is that an ebuild glitch? Or does it somehow come from my older EAPI? Reproducible: Always Steps to Reproduce: 1. emerge -1av =dev-libs/mimalloc-2.0.3-r1 Actual Results: 2. Compilation runs fine. 3. Installation runs fine too, up to the last step, where it tries rm "${ED}/usr/$(get_libdir)/mimalloc.o" || die; and dies, because there is no mimalloc.o, only a mimalloc-gentoo.o: rm: cannot remove '/XXXXXX/portage/dev-libs/mimalloc-2.0.3-r1/image/usr/lib64/mimalloc.o': No such file or directory Expected Results: Installation should go smoothly as well. Maybe with rm "${ED}/usr/$(get_libdir)/mimalloc-gentoo.o" || die; Some info: Portage 2.3.99 (python 3.7.7-final-0, default/linux/amd64/17.0/hardened, gcc-9.3.0, glibc-2.30-r8, 5.4.168-gentoo x86_64) ================================================================= System uname: Linux-5.4.168-gentoo-x86_64-Intel-R-_Core-TM-_i7-6700HQ_CPU_@_2.60GHz-with-gentoo-2.6 sh bash 5.0_p17 ld GNU ld (Gentoo 2.33.1 p2) 2.33.1 app-shells/bash: 5.0_p17::gentoo dev-java/java-config: 2.2.0-r4::gentoo dev-lang/perl: 5.30.1::gentoo dev-lang/python: 2.7.18::gentoo, 3.6.10-r2::gentoo, 3.7.7-r2::gentoo, 3.8.2-r2::gentoo dev-util/cmake: 3.16.5::gentoo dev-util/pkgconfig: 0.29.2::gentoo sys-apps/baselayout: 2.6-r1::gentoo sys-apps/openrc: 0.42.1::gentoo sys-apps/sandbox: 2.13::gentoo sys-devel/autoconf: 2.13-r1::gentoo, 2.69-r4::gentoo sys-devel/automake: 1.11.6-r3::gentoo, 1.12.6::gentoo, 1.13.4-r2::gentoo, 1.14.1::gentoo, 1.15.1-r2::gentoo, 1.16.1-r1::gentoo sys-devel/binutils: 2.33.1-r1::gentoo sys-devel/gcc: 7.3.0-r3::gentoo, 7.4.0-r2::gentoo, 7.5.0::gentoo, 8.3.0-r1::gentoo, 8.4.0::gentoo, 9.2.0-r2::gentoo, 9.3.0::gentoo sys-devel/gcc-config: 2.2.1::gentoo sys-devel/libtool: 2.4.6-r6::gentoo sys-devel/make: 4.2.1-r4::gentoo sys-kernel/linux-headers: 5.4-r1::gentoo (virtual/os-headers) sys-libs/glibc: 2.30-r8::gentoo
Please do include the full build.log. Also, I'd suggest you just work on upgrading your Portage: https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage.
(In reply to Sam James from comment #1) > Please do include the full build.log. > > Also, I'd suggest you just work on upgrading your Portage: > https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage. (Also, yes, it is possibly EAPI change related, as cmake.eclass changed behaviour a bit for EAPI 8, and I think by default, cmake-multilib.eclass will use cmake-utils.eclass by default unless you use a variable to specify cmake.eclass for EAPI 7).
(In reply to Sam James from comment #1) > Please do include the full build.log. > > Also, I'd suggest you just work on upgrading your Portage: > https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage. Hmm...if only things were that simple... By your own guide, it is very advisable to run emerge --sync *before* one tries to upgrade portage. Now, that's where my problems start: syncing will delete ebuilds that have been removed from Gentoo's tree, mostly due to a) lack of maintainer, b) Python 2 only compatibility, or c) whatever else. In older days I knew I could go to https://dev.gentoo.org/~swift/snapshots/ and find the ebuild that was deleted by the sync. Now that this is gone, I have to take care of it myself. O.K., suppose I do it. I make a huge backup of my portage tree before I sync it. Then I sync. But then I have to find out all those deleted ebuilds of programs that I do want to keep - and move them to a local overlay. I have no idea how to do this (how to locate deleted programs, so that I can filter out those that I need to keep), so it's probably a manual, read: tedious, procedure. I dread it. [OK, now that I think of it, I could do a find /usr/portage/ -maxdepth 2 before and after the sync, then take a 'diff -u' of the two lists and keep only the lines that start with a minus...I still dread the work that might come after that.] Only then, I can go on and "upgrade my portage". And we are talking only about portage, not the whole system - because each time I upgrade my system, I spend at least 2 months on it - blockers, Python hell, all *very* painful. Not the happy experience it used to be 15 years ago. ;-) Do you know what I think when I think about all this? I say to myself: "Oh well, some other day..." :-) P.S. Besides, there are other reasons that have kept me from upgrading. Instability of nvidia-drivers, for example. I thought, after I spent two (2!) months on this at the end of 2020, that it was the kernel - and swore to stay on 4.19 until the 5.x branch reaches 5.19! :-) But at the end of 2021, after yet another three (3!) months of searching, I found out it was NOT the kernel! It was the too low power level set by "Adaptive" or "Auto" modes of "Power Mizer" in nvidia-settings! This causes erratic crashes on my ASUS laptop! These 5 months I had to spend on this, are now missing from any "upgrade project"...You see, things are not that simple. But be assured, I *will* upgrade - some other day! :-) P.S. 2: Did I mention that I strongly prefer to *work* with my system, rather than updating, configuring, resolving issues? My notes on "issues" are half a MILLION lines - I just checked: cat /xxxxxx/yyyyyy/*.txt | wc -l 482489 distributed over almost 400 (four hundred!) files: ls /xxxxxx/yyyyyy/*.txt | wc -l 390 So it's not that I have not had my share of upgrade hell in the past quarter of a century - do you get me? :-)
(In reply to segmentation fault from comment #3) > (In reply to Sam James from comment #1) > > Please do include the full build.log. > > > > Also, I'd suggest you just work on upgrading your Portage: > > https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage. > > Hmm...if only things were that simple... > > By your own guide, it is very advisable to run > > emerge --sync > > *before* one tries to upgrade portage. Now, that's where my problems start: > syncing will delete ebuilds that have been removed from Gentoo's tree, > mostly due to a) lack of maintainer, b) Python 2 only compatibility, or c) > whatever else. In older days I knew I could go to Perhaps you could offer to maintain the things removed due to lack of maintainer (or port the Python 2 things to Python 3; most distributions have now dropped Python 2 -- note that I'm also open to restoring things which were dropped due to Python 2 if they've since been ported to Python 3). > > https://dev.gentoo.org/~swift/snapshots/ > > and find the ebuild that was deleted by the sync. Now that this is gone, I > have to take care of it myself. https://gentoo.osuosl.org/snapshots/ but note that everything is of course stored in git anyway, so nothing is ever truly gone. Your system is not adversely affected simply by old versions being removed from the repository and emerge will not remove things simply because the ebuild is now gone. You can clone the repository from https://anongit.gentoo.org/git/repo/gentoo.git or view it online at https://gitweb.gentoo.org/repo/gentoo.git/. Besides, if you really want to upgrade just Portage, grab its ebuild, put it in your own overlay, and let's try do it that way. But trying to use brand new ebuilds with an out of date package manager isn't going to end well. > > O.K., suppose I do it. I make a huge backup of my portage tree before I sync > it. Then I sync. But then I have to find out all those deleted ebuilds of > programs that I do want to keep - and move them to a local overlay. I have > no idea how to do this (how to locate deleted programs, so that I can filter > out those that I need to keep), so it's probably a manual, read: tedious, > procedure. I dread it. > Installed programs wouldn't be deleted just because the ebuilds are gone. > > [OK, now that I think of it, I could do a > > find /usr/portage/ -maxdepth 2 > > before and after the sync, then take a 'diff -u' of the two lists and keep > only the lines that start with a minus...I still dread the work that might > come after that.] > > > Only then, I can go on and "upgrade my portage". And we are talking only > about portage, not the whole system - because each time I upgrade my system, > I spend at least 2 months on it - blockers, Python hell, all *very* painful. > Not the happy experience it used to be 15 years ago. ;-) This isn't really normal and you should seek support when that happens. If it were normal, nobody would use Gentoo. We have support channels for exactly this reason. This sounds like an, as you've observed, unsustainable amount of work which will no doubt lead to various problems. > > Do you know what I think when I think about all this? I say to myself: > > "Oh well, some other day..." > > :-) > > P.S. Besides, there are other reasons that have kept me from upgrading. > Instability of nvidia-drivers, for example. I thought, after I spent two > (2!) months on this at the end of 2020, that it was the kernel - and swore > to stay on 4.19 until the 5.x branch reaches 5.19! :-) But at the end of > 2021, after yet another three (3!) months of searching, I found out it was > NOT the kernel! It was the too low power level set by "Adaptive" or "Auto" > modes of "Power Mizer" in nvidia-settings! This causes erratic crashes on my > ASUS laptop! These 5 months I had to spend on this, are now missing from any > "upgrade project"...You see, things are not that simple. But be assured, I > *will* upgrade - some other day! :-) > You could just mask newer nvidia-drivers and kernels then for now. > P.S. 2: Did I mention that I strongly prefer to *work* with my system, > rather than updating, configuring, resolving issues? My notes on "issues" > are half a MILLION lines - I just checked: I update regularly on most machines and don't experience problems, even fewer on stable boxes. And for a few machines, I update every ~4/6 months and they're fine given I take good care of them (depcleaned, clean world file, read news items, and so on). It sounds a lot like we just need to debug it in a support session at https://www.gentoo.org/support/. My preference is IRC. In any case, I can't do much with rants :(
(In reply to Sam James from comment #4) > Perhaps you could offer to maintain the things removed due to lack of > maintainer > (or port the Python 2 things to Python 3; most distributions have now > dropped Python 2 -- note that I'm also open to restoring things which > were dropped due to Python 2 if they've since been ported to Python 3). > In fact, I do plan to do it - when I'm done with upgrade hell! :-) Maintaining an ebuild is one thing, porting a Python 2 software to Python 3 is another. I have seen projects where there is a WONTFIX on requests to port to Python 3 - probably for a reason... > https://gentoo.osuosl.org/snapshots/ but note that everything is of course > stored in git anyway, so nothing is ever truly gone. > Been there. There are only snapshots of the last month or so at that URL. Here we are talking about years: Last emerge --sync was 1y 251d 16h 5m 53s ago. About git: yes, I synced it some time ago. Now what? How do I produce a snapshot of a certain date out of it? From my notes: ####################################################################### git clone https://github.com/gentoo/gentoo/ This creates a 2.5 GB directory 'gentoo' with ALL of gentoo, SINCE THE DAWN OF TIME! :facepalm: Now, trying the suggested command git checkout `git rev-list -n 1 --first-parent --before="2019-11-11" master` gives me: fatal: not a git repository (or any of the parent directories) ####################################################################### > Your system is not adversely affected simply by old versions being removed > from the repository and emerge will not remove things simply because > the ebuild is now gone. > I know that. I just want to have the ebuilds around, just in case. I don't feel well without them. What if I have to restore some package from scratch? > You can clone the repository from > https://anongit.gentoo.org/git/repo/gentoo.git > or view it online at https://gitweb.gentoo.org/repo/gentoo.git/. > See above. > Besides, if you really want to upgrade just Portage, grab its ebuild, put it > in your own overlay, and let's try do it that way. > > But trying to use brand new ebuilds with an out of date package manager > isn't going to end well. > O.K. this is something I wanted to give a try...and I've been struggling the past 3 hours with it, without success: I put portage-3.0.20-r6.ebuild in my local overlay. I changed PYTHON_COMPAT=( python3_{8..10} ) to PYTHON_COMPAT=( python3_8 ) in the ebuild (I do have python 3.8 installed, but not higher). Now portage-3.0.20-r6 wants >=sys-apps/file-5.41. So I put that one into my overlay too. But file-5.41 complains !!! The ebuild selected to satisfy "sys-apps/file" has unmet requirements. - sys-apps/file-5.41::XXXX USE="bzip2 python seccomp zlib -lzma -static-libs" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="(-python3_8)" The following REQUIRED_USE flag constraints are unsatisfied: python? ( python_targets_python3_8 ) The above constraints are a subset of the following complete expression: python? ( any-of ( python_targets_python3_8 ) ) ...no matter what I try! I tried already: - Remove PYTHON_TARGETS and PYTHON_SINGLE_TARGET from make.conf as per https://www.gentoo.org/support/news-items/2021-05-05-python3-9.html - emerge --ask -v1 sys-apps/file - Put */* PYTHON_TARGETS: -* python3_8 */* PYTHON_SINGLE_TARGET: -* python3_8 at the start, or the end of /etc/portage/package.use. Tried also any combinations, e.g. python2_7 python3_6 python3_7 python3_8 to no avail. - export PYTHON_TARGETS="python3_8"; emerge --ask -v1 sys-apps/file - PYTHON_TARGETS="python3_8" emerge --ask -v1 sys-apps/file - PYTHON_TARGETS="python2_7 python3_6 python3_7 python3_8" emerge --ask -v1 sys-apps/file - PYTHON_TARGETS="python3_8" PYTHON_SINGLE_TARGET="python3_8" emerge --ask -v1 sys-apps/file - USE="python_targets_python3_8" emerge --ask -v1 sys-apps/file - PYTHON_TARGETS="python3_8" USE="python_targets_python3_8" emerge --ask -v1 sys-apps/file - USE="-rsync-verify python_targets_python3_8" emerge --ask -v1 sys-apps/file in all combinations with or without =sys-apps/file-5.41 python_single_target_python3_8 or =sys-apps/file-5.41 python_targets_python3_8 (single or together) in /etc/portage/package.use. All of these bring the above message - which is totally idiotic in my eyes: I put all possible effort into telling portage to use Python 8 as its (single or not) target and it still complains that I have to choose one of "python 8" as its target? Huh? You understand now what I mean by "Python HELL"? That's not a rant, it's a bitter reality. Things should NEVER be allowed to go that wrong! But they have. And I do take care of my system. All is running fine and the system is not in any inconsistent state - it is just old. This, however, should not be handled as a deadly sin - sometimes there are reasons for this, as I said. But since I am talking to a specialist in portage upgrading, I have not lost all hope - any suggestions? :-) The plan is to upgrade to sys-apps/file-5.41, then to portage-3.0.20-r6 with USE="-rsync-verify python_targets_python3_8" and *without* doing "emerge --sync" first. > It sounds a lot like we just need to debug it in a support session at > https://www.gentoo.org/support/. My preference is IRC. > > In any case, I can't do much with rants :( Thank you for your offer. The first thing I will do as soon as I resolve this will be to warn others in the forum about a too low Power Level in nvidia-settings. And, just to be clear, all this was NOT a rant! I am actually in a good mood today...I have resolved a stability issue with my graphics driver that has cost me 5 months to hunt and nail down. And you know what? Today, Murphy's Law knocked my door: eating breakfast, the bread fell off my hand - on my beloved chinese red silk carpet! I thought "that was it! bye-bye carpet!", because, of course, as we all know, "the bread *always* falls on its marmalade side", right? That's Murphy's Law. I timidly turned my head and looked down. The bread was on the carpet - standing on its *thin* side, vertically!!! So I have all reasons to be in good mood today (maybe I should play lottery this week...). :-) P.S.: You have not seen my rants! :-) They are better kept in my personal notes (but rest assured: not all of their half-a-million lines is rant)! :-)))
(In reply to segmentation fault from comment #5) > Been there. There are only snapshots of the last month or so at that URL. > Here we are talking about years: > > Last emerge --sync was 1y 251d 16h 5m 53s ago. > That sounds inadvisable. There's a much larger archive at https://mirror.reenigne.net/gentoo-snapshots/ too. > About git: yes, I synced it some time ago. Now what? How do I produce a > snapshot of a certain date out of it? From my notes: Well, at worst, you go through git log, right? > > ####################################################################### > > git clone https://github.com/gentoo/gentoo/ > > This creates a 2.5 GB directory 'gentoo' with ALL of gentoo, > SINCE THE DAWN OF TIME! :facepalm: > You wanted full history, so... > Now, trying the suggested command > > git checkout `git rev-list -n 1 --first-parent --before="2019-11-11" master` > > gives me: > > fatal: not a git repository (or any of the parent directories) I don't think that error makes any sense. That command works fine for me in a checkout. > > ####################################################################### > > > Your system is not adversely affected simply by old versions being removed > > from the repository and emerge will not remove things simply because > > the ebuild is now gone. > > > > I know that. I just want to have the ebuilds around, just in case. I don't > feel well without them. What if I have to restore some package from scratch? > I can't do much about feelings. You could still recover them from git anyway. > > You can clone the repository from > > https://anongit.gentoo.org/git/repo/gentoo.git > > or view it online at https://gitweb.gentoo.org/repo/gentoo.git/. > > > > See above. > > > Besides, if you really want to upgrade just Portage, grab its ebuild, put it > > in your own overlay, and let's try do it that way. > > > > But trying to use brand new ebuilds with an out of date package manager > > isn't going to end well. > > > > O.K. this is something I wanted to give a try...and I've been struggling the > past 3 hours with it, without success: > > I put portage-3.0.20-r6.ebuild in my local overlay. > > I changed > > PYTHON_COMPAT=( python3_{8..10} ) > > to > > PYTHON_COMPAT=( python3_8 ) > > in the ebuild (I do have python 3.8 installed, but not higher). > > Now portage-3.0.20-r6 wants >=sys-apps/file-5.41. So I put that one into my > overlay too. But file-5.41 complains > > !!! The ebuild selected to satisfy "sys-apps/file" has unmet requirements. > - sys-apps/file-5.41::XXXX USE="bzip2 python seccomp zlib -lzma > -static-libs" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="(-python3_8)" > > The following REQUIRED_USE flag constraints are unsatisfied: > python? ( python_targets_python3_8 ) > >PYTHON_TARGETS="(-python3_8)" Says that python3_8 is masked for some reason. I can't tell you why because I don't know what your system state is. grep -rsin "python" /etc/portage > The above constraints are a subset of the following complete expression: > python? ( any-of ( python_targets_python3_8 ) ) > > ...no matter what I try! > > I tried already: > [snip] > > You understand now what I mean by "Python HELL"? That's not a rant, it's a > bitter reality. > You're not reading the output properly. It says that Python 3.8 is masked and you've likely masked it yourself as it's not masked in Gentoo. So, share that grep output I asked for, and let's go from there.
(In reply to Sam James from comment #6) > You're not reading the output properly. It says that Python 3.8 is masked > and you've likely masked it yourself as it's not masked in Gentoo. > Thank you for this pointer, very interesting...I will have to research this a bit myself. What did I do? Well, I followed advice that said to remove PYTHON_TARGETS and PYTHON_SINGLE_TARGET from /etc/portage/make.conf and put them like this */* PYTHON_TARGETS: -* python3_8 */* PYTHON_SINGLE_TARGET: python3_8 in /etc/portage/package.use. This did not work well at all, not matter what I put there...I started having the output you saw from emerge. The curious thing is, I was getting such messages also for installed packages and python3_6, for example. That is, if I tried to rebuild an already installed package whose single USE flag was python3_6, emerge would show it to me as (-python3_6) and would say that there were unmet requirements. But I quickly commented those two lines from my /etc/portage/package.use and put back my usual PYTHON_TARGETS="python2_7 python3_6 python3_7 python3_8" PYTHON_SINGLE_TARGET="python3_7" in make.conf. However, this did not change anything...After one day, and after installing some package that did not have to do with python at all... ...and after changing the PYTHON_COMPAT array in the ebuild to include python 3.7: PYTHON_COMPAT=( python3_{7..8} ) now I am allowed to install sys-apps/file at least on python3_7: [ebuild U ] sys-apps/file-5.41::XXXXXX [5.37-r1::gentoo] USE="bzip2%* python seccomp%* zlib -lzma% -static-libs" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python3_7 (-python3_8) (-python3_6%*)" 1,040 KiB Total: 1 package (1 upgrade), Size of downloads: 1,040 KiB Would you like to merge these packages? [Yes/No] So this is progress indeed and I still have to find out why I have (-python3_8) (-python3_6%*) python3_6 is excluded through the PYTHON_COMPAT array in the ebuild of 'file'. But python3_8? After sys-apps/file, I tried portage again, and got the unmet requirements message. I tried prepending PYTHON_TARGETS='python3_7 python3_8' to the emerge command and it went further, this time asking me a higher version of gemato. apps-portage/gemato had the same problem with python targets (after I downloaded the ebuild for 16.2 in my local overlay, of course), so I tried PYTHON_TARGETS='python3_7 python3_8' emerge --ask -v1 app-portage/gemato and it went further, this time asking me to upgrade app-crypt/gnupg. So I am seeing a pattern here. For some reason, I have to go with prepending PYTHON_TARGETS='python3_7 python3_8' to whatever package portage is missing, until I get them all - they can't be *that* many! I understand that, at this point you will say "why don't you 'emerge @world', instead of going package-for-package?" - and my answer is: because there are more than 4000 packages installed here and I dread the moment... :-))) > So, share that grep output I asked for, and let's go from there. That grep output is huge, contains lots of comments that I will have to filter out and shows nothing that hints to masking...any hints about what I should look at, before I embarrass the world with it? :-) Thank you for all, we are making some progress. I would have definitely given up on this, as I wouldn't dare to upgrade portage *only*, without your encouragement.
(In reply to segmentation fault from comment #7) > > I understand that, at this point you will say "why don't you 'emerge > @world', instead of going package-for-package?" - and my answer is: because > there are more than 4000 packages installed here and I dread the moment... > :-))) Heh. > > > So, share that grep output I asked for, and let's go from there. > > That grep output is huge, contains lots of comments that I will have to > filter out and shows nothing that hints to masking...any hints about what I > should look at, before I embarrass the world with it? :-) > Can you try this instead: grep -rsin "python3_8" /etc/portage | grep mask? There's definitely something masking it, but we're not yet sure what. It's likely to be /etc/portage/use.mask or similar, but can't be sure. Oh.. unless it's due to your very old sync. In which case, can you open (you might need to create /etc/portage/profile, it's *not* the same as make.profile) /etc/portage/profile/use.mask and /etc/portage/profile/use.stable.mask and put the same thing in both files: -python_targets_python3_8 -python_single_target_python3_8 # optionally, these too: #-python_targets_python3_9 #-python_single_target_python3_9 > Thank you for all, we are making some progress. I would have definitely > given up on this, as I wouldn't dare to upgrade portage *only*, without your > encouragement. No problem. :)
(In reply to Sam James from comment #8) > Can you try this instead: > grep -rsin "python3_8" /etc/portage | grep mask? > grep -rsin "python3_8" /etc/portage | grep mask Returns *nothing*. I told you this is a "good system". :-))) I don't mess around with such things - O.K., unless I have to. :-))) > There's definitely something masking it, but we're not yet sure what. It's > likely to be /etc/portage/use.mask or similar, but can't be sure. > O.K., let's see: ll /etc/portage/use.mask ls: cannot access '/etc/portage/use.mask': No such file or directory > Oh.. unless it's due to your very old sync. Maybe. But then it's a "bug" (or is it a feature? oops...:-)) in Gentoo's "matrix" - because: an old system, with an old portage and old ebuilds, all from the same point of time, should be consistent with each other, right? It should actually be consistent with new ebuild too - with the exception of EAPI version problems and...right, Python versions. ;-) O.K., after this small objection from me, let's go down that path: > In which case, can you open (you > might need to create /etc/portage/profile, /etc/portage/profile IS there: ll /etc/portage/profile total 8 -rw-r--r-- 1 root root 0 May 28 2017 package.provided drwxr-xr-x 2 root root 4096 Dec 12 2018 package.use.force drwxr-xr-x 2 root root 4096 Dec 12 2018 package.use.mask but contains nothing of interest to python: rg -F 'python' /etc/portage/profile Output: nothing. NOTE: rg (sys-apps/ripgrep) is a parallel version of grep, much faster than grep on multi-core machines. In this case, your "grep -rsin" wins the race though, with 2 vs. 8 milliseconds. :-) Both return nothing. > it's *not* the same as > make.profile) /etc/portage/profile/use.mask and > /etc/portage/profile/use.stable.mask and put the same thing in both files: > -python_targets_python3_8 > -python_single_target_python3_8 > > # optionally, these too: > #-python_targets_python3_9 > #-python_single_target_python3_9 > Well, this, did the job of freeing python 3.8 for emerge! Let's take, for example, sys-apps/file, which was installed in python 3.7 but not in python 3.8: eix sys-apps/file [I] sys-apps/file Available versions: 5.37-r1 ~5.38-r1 5.41[1] **9999*l {bzip2 lzma python seccomp static-libs zlib ABI_MIPS="n32 n64 o32" ABI_RISCV="lp64 lp64d" ABI_S390="32 64" ABI_X86="32 64 x32" PYTHON_TARGETS="python3_6 python3_7 python3_8"} Installed versions: 5.41[1](02:37:11 AM 01/19/2022)(bzip2 python seccomp zlib -lzma -static-libs ABI_MIPS="-n32 -n64 -o32" ABI_RISCV="-lp64 -lp64d" ABI_S390="-32 -64" ABI_X86="64 -32 -x32" PYTHON_TARGETS="python3_7 -python3_8") Homepage: https://www.darwinsys.com/file/ Description: identify a file's format by scanning binary data for patterns [1] "XXXXXX" /usr/local/portage/overlays/XXXXXX Let's try to rebuild it now (this is the brand new 5.41 which I put in my overlay, because portage needs a higher version of "file" in order to proceed with the upgrade of itself, i.e. sys-apps/portage): emerge -1av sys-apps/file * Last emerge --sync was 1y 253d 12h 8m 11s ago. These are the packages that would be merged, in reverse order: Calculating dependencies... done! [ebuild R ] sys-apps/file-5.41::XXXXXX USE="bzip2 python seccomp zlib -lzma -static-libs" ABI_X86="(64) -32 (-x32)" PYTHON_TARGETS="python3_7 python3_8*" 0 KiB [nomerge ] www-servers/varnish-6.3.2:0/2::gentoo USE="-jemalloc -jit -static-libs" PYTHON_TARGETS="python3_6 python3_7 -python3_8" [nomerge ] dev-python/sphinx-2.4.4::gentoo USE="-doc -latex -test" PYTHON_TARGETS="python3_6 python3_7 -pypy3 -python3_8" [nomerge ] dev-python/requests-2.23.0::gentoo USE="ssl -socks5 -test" PYTHON_TARGETS="python2_7 python3_6 python3_7 (-pypy3) -python3_8" [ebuild R ] dev-python/certifi-2019.11.28::gentoo PYTHON_TARGETS="python2_7 python3_6 python3_7 python3_8* -pypy3" 0 KiB [ebuild R ] dev-python/setuptools-44.1.0::gentoo USE="-test" PYTHON_TARGETS="python2_7 python3_6 python3_7 python3_8* (-pypy3)" 0 KiB Total: 3 packages (3 reinstalls), Size of downloads: 0 KiB Would you like to merge these packages? [Yes/No] This looks very exciting and not risky at all, so I'll go for it right now. There is another obstacle ahead, however, about which I would like to hear your opinion: As I was saying, portage needed a higher version of gemato (I have gemato-14.3 and the newest, in my overlay, is gemato-16.2) and gemato needed a higher version of app-crypt/gnupg, so I took gnupg-2.2.32-r1.ebuild into my overlay and merged it. But then, portage also asked for >=sec-keys/openpgp-keys-gentoo-release-20180706: emerge: there are no ebuilds to satisfy ">=sec-keys/openpgp-keys-gentoo-release-20180706". But there IS one package app-crypt/openpgp-keys-gentoo-release-20191030 pulled in by: sys-apps/portage-2.3.99-r2 requires >=app-crypt/openpgp-keys-gentoo-release-20180706 so this is due to the category change from app-crypt to sec-keys for openpgp-keys-gentoo-release. For this, I had to add sec-keys to my /etc/portage/categories. But still, I cannot install sec-keys/openpgp-keys-gentoo-release-20220101 due to file collisions: * Detected file collision(s): * * /usr/share/openpgp-keys/gentoo-release.asc * * Searching all installed packages for file collisions... * * Press Ctrl-C to Stop * * app-crypt/openpgp-keys-gentoo-release-20191030:0::gentoo * /usr/share/openpgp-keys/gentoo-release.asc * * Package 'sec-keys/openpgp-keys-gentoo-release-20220101' NOT merged due * to file collisions. If necessary, refer to your elog messages for the * whole content of the above message. I therefore plan to: Copy the *current* (read: old) ebuild of sys-apps/portage-2.3.99-r2 into my local overlay. Change "app-crypt/openpgp-keys-gentoo-release" in the copied ebuild to "sec-keys/openpgp-keys-gentoo-release". Copy my current (i.e. old) app-crypt/openpgp-keys-gentoo-release-20191030 as sec-keys/openpgp-keys-gentoo-release-20191030 (notice the change in category) in my overlay too. Rebuild my old sys-apps/portage-2.3.99-r2 with the version from my overlay, i.e. install sys-apps/portage-2.3.99-r2::XXXXXX - which now will pull openpgp-keys-gentoo-release-20191030 from sec-keys and not from app-crypt from the overlay. This "frees" app-crypt/openpgp-keys-gentoo-release, which is not needed anymore. Unmerge app-crypt/openpgp-keys-gentoo-release. Now, upgrade sec-keys/openpgp-keys-gentoo-release to the most current version, sec-keys/openpgp-keys-gentoo-release-20220101 (this is already in my overlay). This should be possible now, because there should be no file collisions from app-crypt/openpgp-keys-gentoo-release. Now I should have a still old portage (2.3.99-r2), but with a brand new openpgp-keys-gentoo-release (20220101), from the current sec-keys category. Is this a good plan? Are the problems you anticipate with -rsync-verify in https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage due to this keys upgrade? Will I be able to upgrade portage with the rsync-verify and in, say, python3.7? Or is something in portage-3.0.20-r6 that *needs* a higher python version? I am seeing lots of "conflicts", but I am optimistic because they seem to come from the newly enabled python3_8 USE flag (they are not version conflicts, just same package, same version, one "scheduled for merge" with the python3_8 USE flag, the other already installed, but not with python3_8 USE flag, and needed by some other python package that is also not installed with python 3_8 USE flag), so I will probably have to start emerge with the --newuse flag, to install all merged packages for Python 3.8 too... ...because currently I am trying to put all those packages on the command line, like this PYTHON_TARGETS='python3_7 python3_8' emerge --ask -v1 app-portage/gemato sys-apps/portage dev-python/six dev-python/chardet dev-python/PySocks dev-python/idna dev-python/urllib3 dev-python/cryptography dev-python/pyopenssl dev-python/requests dev-python/six dev-python/ply dev-python/pycparser dev-python/certifi dev-python/setuptools dev-python/html5-parser app-emulation/docker-compose dev-python/twisted dev-python/hyperlink dev-python/trustme but there seems to be no end in the list of packages... :-)
Well, the plan went well as outlined. :-) sys-apps/portage has been upgraded to version 3.0.20-r6 for Python 3.7 and 3.8 on this otherwise old (1 year 250+ days!) system! :-) What I did: - I let it upgrade 'file' to sys-apps/file-5.41 (see previous posts). - I followed the outlined plan for sec-keys/openpgp-keys-gentoo-release, with the only difference that, at some point just before the portage upgrade, I *had* to purge its copy in app-crypt hard with emerge -C app-crypt/openpgp-keys-gentoo-release-20191030 (I had already copied it under sec-keys in my overlay). - I did emerge --update --newuse --deep --with-bdeps=y @world many times and rebuilt 35 packages to include the python3_8 target. - For the portage-2.3.99-r2 ebuild, I had to prepend FEATURES='-userpriv', otherwise I would get "Permission denied" errors from tar during unpacking. This may be related to the fact that my portage build directory is a symbolic link to some other place in a RAM disk. - So at some point in time I had portage-2.3.99-r2 with sec-keys/openpgp-keys-gentoo-release-20220101 merged. - Finally, with PYTHON_TARGETS="python3_7 python3_8" FEATURES='-userpriv' emerge --ask -v1 sys-apps/portage app-portage/gemato app-portage/elicense app-admin/webapp-config app-portage/gentoolkit dev-java/java-config app-portage/layman app-portage/pfl i.e. by restricting PYTHON_TARGETS to the largest possible set that was supported by all packages on that command and disabling the userpriv feature (for the portage ebuild, see above), I managed to get portage upgraded: [ebuild R ] app-portage/pfl-3.1-r1::XXXXXX USE="network-cron" PYTHON_TARGETS="python3_7 python3_8 -python3_6*" 0 KiB [ebuild R ] app-portage/layman-2.4.3::gentoo USE="git subversion -cvs (-darcs) (-g-sorcery) -gpg -mercurial -sqlite -squashfs -sync-plugin-portage -test" PYTHON_TARGETS="python3_7 python3_8 -python3_6*" 0 KiB [ebuild R ] app-portage/gentoolkit-0.4.8::gentoo PYTHON_TARGETS="python3_7 python3_8 (-pypy3) -python3_6*" 0 KiB [ebuild R ] app-admin/webapp-config-1.55-r1::gentoo USE="portage" PYTHON_TARGETS="python3_7 python3_8 -python3_6*" 0 KiB [ebuild R ] app-portage/elicense-1.0.2::gentoo PYTHON_TARGETS="python3_7 -python3_6*" 0 KiB [ebuild R ] dev-java/java-config-2.2.0-r4:2::gentoo USE="-test" PYTHON_TARGETS="python3_7 python3_8 -python3_6*" 0 KiB [ebuild U ] sys-apps/portage-3.0.20-r6::XXXXXX [2.3.99-r2::XXXXXX] USE="(ipc) native-extensions rsync-verify (xattr) -apidoc -build -doc -gentoo-dev (-selinux) -test%" PYTHON_TARGETS="python3_7 python3_8 (-pypy3) (-python3_6%*)" 0 KiB [ebuild U ] app-portage/gemato-16.2::XXXXXX [14.3::gentoo] USE="gpg tools -test" PYTHON_TARGETS="python3_7 python3_8 (-pypy3) (-python3_6%*)" 79 KiB [ebuild N ] acct-user/portage-0::XXXXXX 0 KiB Total: 9 packages (2 upgrades, 1 new, 6 reinstalls), Size of downloads: 79 KiB The result is an-up-to-date portage: eix sys-apps/portage [I] sys-apps/portage Available versions: 2.3.8[1] 2.3.40-r1[1] (~)2.3.84-r1[1] 2.3.89-r3 2.3.99-r2 2.3.99-r2[1] 3.0.20-r6^t[1] **9999*l {apidoc binpkg-zstd build doc epydoc gentoo-dev +ipc +native-extensions +rsync-verify selinux test xattr KERNEL="linux" PYTHON_TARGETS="pypy3 python2_7 python3_6 python3_7 python3_8"} Installed versions: 3.0.20-r6^t[1](10:01:51 PM 01/19/2022)(ipc native-extensions rsync-verify xattr -apidoc -build -doc -gentoo-dev -selinux -test KERNEL="linux" PYTHON_TARGETS="python3_7 python3_8 -pypy3") Homepage: https://wiki.gentoo.org/wiki/Project:Portage Description: Portage is the package management and distribution system for Gentoo [1] "XXXXXX" /usr/local/portage/overlays/XXXXXX I guess, it works only on Python 3.7 and 3.8, since these were the targets of the upgrade command above. WOW! Unfortunately, this does not seem to suffice in order to compile mimalloc correctly: I reverted my change of the EAPI (I had set it to 7 and it was suspected that this was the reason behind the error reported here at the start) and set EAPI=8 in the ebuild. Now, however, I get the error "EAPI=8 not supported", even at trying to digest the ebuild: ebuild mimalloc-2.0.3-r1.ebuild digest * ERROR: dev-libs/mimalloc-2.0.3-r1::XXXXXX failed (depend phase): * EAPI=8 is not supported * * Call stack: * ebuild.sh, line 645: Called source '/usr/local/portage/overlays/XXXXXX/dev-libs/mimalloc/mimalloc-2.0.3-r1.ebuild' * mimalloc-2.0.3-r1.ebuild, line 6: Called inherit 'cmake' * ebuild.sh, line 329: Called __qa_source '/usr/portage/eclass/cmake.eclass' * ebuild.sh, line 114: Called source '/usr/portage/eclass/cmake.eclass' * cmake.eclass, line 99: Called die Looking into /usr/portage/eclass/cmake.eclass I indeed see: case ${EAPI} in 7) ;; *) die "EAPI=${EAPI:-0} is not supported" ;; esac and doing equery belongs /usr/portage/eclass/cmake.eclass * Searching for /usr/portage/eclass/cmake.eclass ... returns nothing, so the question is: which package owns /usr/portage/eclass/cmake.eclass why was it not upgraded with sys-apps/portage and why don't I see it in the output of 'equery belongs'? Obviously, "upgrade sys-apps/portage" was only part of the solution. We are there, but some piece is still missing from the puzzle... Any hints?
I start to suspect that the eclasses, like /usr/portage/eclass/cmake.eclass are updated through emerge --sync and not through any package - correct me if I am wrong. This would mean that the statement "Portage 3.0.20 and above supports EAPI 8" in https://wiki.gentoo.org/wiki/User:Sam/Portage_help/Upgrading_Portage is misleading: I do have sys-apps/portage-3.0.20-r6, but still cannot handle EAPI 8 ebuilds (like mimalloc-2.0.3-r1.ebuild) - I need the current eclasses too! Now what? Actually, as far as I can oversee it, I just need a way to sync my eclasses with those at, say https://gitweb.gentoo.org/repo/gentoo.git/tree/eclass and then all should be good? As I said previously, I would prefer to defer complete syncing to some later time of my choice (Gentoo is about choice, right? :-)). Is there an emerge option to sync eclasses only and not the ebuilds? That shouldn't be difficult to implement - and has its own use cases, for the daring, as we see here. ;-)
SUCCESS! Look at this: >>> Installing (1 of 1) dev-libs/mimalloc-2.0.3-r1::XXXXXX * checking 9 files for package collisions >>> Merging dev-libs/mimalloc-2.0.3-r1 to / --- /usr/ --- /usr/lib64/ --- /usr/lib64/cmake/ >>> /usr/lib64/cmake/mimalloc/ >>> /usr/lib64/cmake/mimalloc/mimalloc-config.cmake >>> /usr/lib64/cmake/mimalloc/mimalloc-config-version.cmake >>> /usr/lib64/cmake/mimalloc/mimalloc.cmake >>> /usr/lib64/cmake/mimalloc/mimalloc-relwithdebinfo.cmake >>> /usr/lib64/libmimalloc.so.2.0 --- /usr/include/ >>> /usr/include/mimalloc-new-delete.h >>> /usr/include/mimalloc-override.h >>> /usr/include/mimalloc.h >>> /usr/lib64/libmimalloc.so -> libmimalloc.so.2.0 >>> dev-libs/mimalloc-2.0.3-r1 merged. >>> Regenerating /etc/ld.so.cache... >>> Recording dev-libs/mimalloc in "world" favorites file... >>> Auto-cleaning packages... How did I manage? Here's how: As I said, I was missing the current eclasses. Just an updated sys-apps/portage-3.0.20-r6 alone would not cut it. But Gentoo and its portage system is flexible... :-) You can copy the current eclasses from https://gitweb.gentoo.org/repo/gentoo.git/plain/eclass/ into a directory 'eclass' of your local overlay - and they will take precedence for the ebuilds of your overlay over the ones in /usr/portage/eclass! This was exactly what I needed: eclasses that were current enough to read my EAPI 8 mimalloc ebuild, but would not interfere with my old system. So I did: cd /usr/local/portage/overlays/XXXXXX mkdir -p eclass wget -np -r -k -N -nc -l 0 https://gitweb.gentoo.org/repo/gentoo.git/plain/eclass/ find gitweb.gentoo.org/ -name index.html | xargs rm -f mv -i gitweb.gentoo.org/repo/gentoo.git/plain/eclass/* eclass/ rm -rf gitweb.gentoo.org/ cp -av eclass-save/* eclass/ [XXXXXX is my overlay and I had saved a few other eclasses, not relevant here, in eclass-save]. Now this should suffice for mimalloc, but for python-related packages you need some changes (remember, these are new eclasses in an almost two-year old system), otherwise Python Hell will strike again: Change _PYTHON_ALL_IMPLS to: _PYTHON_ALL_IMPLS=( pypy3 python2_7 python3_{6..8} ) and (for consistency with the above) _PYTHON_HISTORICAL_IMPLS to: _PYTHON_HISTORICAL_IMPLS=( pypy pypy1_{8,9} pypy2_0 python2_{5..6} python3_{1..5} ) in python-utils-r1.eclass. Thus, your ebuilds from your local overlay that use python2_7, python3_6 or python3_7 will not be invalidated. You will encounter other problems too: all ebuilds from your local overlay that use an EAPI less than 7 (say EAPI 5 or EAPI 6) will be invalid because the new eclasses will not be able to read this or inherit that. But your EAPI 7 and 8 ebuilds (like mimalloc's) should be fine. Since the new eclasses can read EAPI 8 ebuilds, I was able to digest mimalloc's ebuild ebuild mimalloc-2.0.3-r1.ebuild digest (remember, in order to 'digest' it previously, I had to change the EAPI to 7 in the ebuild - and this brought up the issue here). Now running emerge -av mimalloc was somewhat different: it required me to upgrade cmake to at least 3.20.5 (I had 3.16.5). So I put a fresh copy of https://gitweb.gentoo.org/repo/gentoo.git/plain/dev-util/cmake/ in my overlay and upgraded to dev-util/cmake-3.21.4. After that, merging mimalloc went smoothly and without the error message that I reported at the start. So this was due to a version of cmake (3.16.5) being too old and this fact being made explicit only after the upgrade of the eclasses (and sys-apps/portage itself of course). I learned a few things on the way, notably that it is NOT enough to keep ebuilds of your old software in your overlay - you must also have the right "set of eclasses" for each EAPI that is not officially supported but is used in those ebuilds and you must take care of Python Hell as shown above. Thanks to Sam James who has encouraged me to take this way and not give up.
