I am trying to cross emerge glibc as a normal user with FEATURES=fakeroot : SYSROOT=/usr/local/src/X-5000/root PKGDIR=/usr/local/src/X-5000/overlay/packages DISTDIR=/usr/local/src/X-5000/overlay/distfiles ROOT=/usr/local/src/X-5000/root emerge-x86_64-tm-linux-gnu -a --nodeps sys-libs/glibc This fails with: .... /usr/bin/install: cannot change ownership of '/usr/x86_64-tm-linux-gnu/tmp/portage/sys-libs/glibc-2.17/image//usr/lib/misc/glibc/pt_chown': Operation not permitted I don't know how to diagnose this. Any poniters? sys-apps/fakeroot-1.19 is installed
> cat sysroot-ebuild #: ${CROSS_CMD:=emerge --root-deps --onlydeps} #: ${CROSS_CMD:=emerge --root-deps --exclude "virtual/*"} : ${CROSS_CMD:=ebuild} CHOST=x86_64-tm-linux-gnu #SYSROOT=/usr/local/src/X-5000/root #PORTAGE_CONFIGROOT=/usr/local/src/X-5000/configroot #export SYSROOT PORTAGE_CONFIGROOT export CROSS_CMD CHOST cross-emerge "$@" > ./sysroot-ebuild /usr/portage/sys-apps/busybox/busybox-1.21.1.ebuild --debug install .... shift + local x + for x in '"$@"' + '[' amd64 = sep-usr ']' + for x in '"$@"' + '[' kernel_linux = sep-usr ']' + for x in '"$@"' + '[' make-symlinks = sep-usr ']' + for x in '"$@"' + '[' static = sep-usr ']' + for x in '"$@"' + '[' userland_GNU = sep-usr ']' + return 1 + ret=1 + [[ hxB == *f* ]] + set +f + return 1 + newbin busybox_unstripped busybox install: cannot change ownership of ‘/usr/x86_64-tm-linux-gnu/tmp/portage/sys-apps/busybox-1.21.1/image//bin/busybox’: Operation not permitted * ERROR: sys-apps/busybox-1.21.1::gentoo failed (install phase): * dobin failed This only happens when cross building busybox. Doing ebuild /usr/portage/sys-apps/busybox/busybox-1.21.1.ebuild --debug install works just fine
Adding PORTAGE_INST_UID=$(id -u) and PORTAGE_INST_GID=$(id -g) fixes this the cross ebuild problem. Othervise these are empty and install gets "-o 0" passed to it. Why are these not set when doing cross builds? However, this does not solve the problem with fakeroot. Had fakeroot worked it had ignored the chown so something else needs to be fixed there
even though PORTAGE_INST_UID=$(id -u) and PORTAGE_INST_GID=$(id -g) are set, I cannot build: ./sysroot-ebuild /usr/portage/app-admin/eselect/eselect-1.3.8.ebuild --debug install .... + return 1 + ret=1 + [[ hxB == *f* ]] + set +f + return 1 + fowners root:portage /var/lib/gentoo/news chown: changing ownership of ‘/usr/local/src/X-5000/root/var/tmp/portage/app-admin/eselect-1.3.8/image//var/lib/gentoo/news’: Operation not permitted * ERROR: app-admin/eselect-1.3.8::gentoo failed (install phase): * fowners failed Clearly FEATURES=fakeroot are broken in emerge/ebuild commands. Please advise, I am quite stuck now
Created attachment 363122 [details] ebuild --debug log
Now I know what the problem is, cross-emerge has: FAKEROOT= if [ $(id -u) -ne 0 ] ; then type -P fakeroot >/dev/null && FAKEROOT=fakeroot fi : ${CROSS_CMD:=emerge --root-deps=rdeps} exec ${FAKEROOT} ${CROSS_CMD} "$@" which makes emerge belive it is root and then this check disables fakeroot(in doebuild.py): fakeroot = fakeroot and uid != 0 and portage.process.fakeroot_capable since uid is 0 removing the uid check makes it work: fakeroot = fakeroot and portage.process.fakeroot_capable removing fakeroot from cross-emerge makes emerge bail with: "This action needs superuser access" Simplest way forward is just to remove the uid != 0 check. Please consider
Created attachment 370858 [details, diff] Patch to remove uid check Removes the uid check which disables fakeroot. However, now that fakeroot actually works another problem appears: #> FEATURES="fakeroot" ebuild /usr/portage/app-admin/eselect/eselect-1.3.8.ebuild clean package ... /usr/bin/install -c -m 644 config.bash core.bash default.eselect editor-variable.bash manip.bash multilib.bash output.bash package-manager.bash path-manipulation.bash skel.bash tests.bash '/var/tmp/portage/app-admin/eselect-1.3.8/image//usr/share/eselect/libs/' Making install in modules /bin/mkdir -p '/var/tmp/portage/app-admin/eselect-1.3.8/image//usr/share/eselect/modules/' /usr/bin/install -c -m 644 binutils.eselect editor.eselect env.eselect kernel.eselect locale.eselect modules.eselect news.eselect pager.eselect profile.eselect rc.eselect visual.eselect '/var/tmp/portage/app-admin/eselect-1.3.8/image//usr/share/eselect/modules/' >>> Completed installing eselect-1.3.8 into /var/tmp/portage/app-admin/eselect-1.3.8/image/ ecompressdir: bzip2 -9 /usr/share/doc ecompressdir: bzip2 -9 /usr/share/man ./ ./usr/ ./usr/share/ ./usr/share/man/ ./usr/share/man/man1/ ./usr/share/man/man1/eselect.1.bz2 tar: ./usr/share/man/man1/kernel-config.1.bz2: Cannot open: Too many levels of symbolic links ./usr/share/man/man1/profile-config.1.bz2 ./usr/share/man/man1/rc-config.1.bz2 ./usr/share/man/man5/ ./usr/share/man/man5/binutils.eselect.5.bz2 ./usr/share/man/man5/editor.eselect.5.bz2 ./usr/share/man/man5/env.eselect.5.bz2 There are symlinks in /usr/share/man/man1: # ls -l /var/tmp/portage/app-admin/eselect-1.3.8/image/usr/share/man/man1/ total 4 drwxr-xr-x 2 root root 101 Feb 20 09:58 ./ drwxr-xr-x 4 root root 28 Feb 20 09:58 ../ -rw-r--r-- 1 root root 952 Feb 20 09:58 eselect.1.bz2 lrwxrwxrwx 1 root root 28 Feb 20 09:58 kernel-config.1.bz2 -> ../man5/kernel.eselect.5.bz2 lrwxrwxrwx 1 root root 29 Feb 20 09:58 profile-config.1.bz2 -> ../man5/profile.eselect.5.bz2 lrwxrwxrwx 1 root root 24 Feb 20 09:58 rc-config.1.bz2 -> ../man5/rc.eselect.5.bz2 this becomes too much for tar when running under fakeroot, why I don't know
(In reply to Joakim Tjernlund from comment #6) .. > > ecompressdir: bzip2 -9 /usr/share/doc > ecompressdir: bzip2 -9 /usr/share/man > ./ > ./usr/ > ./usr/share/ > ./usr/share/man/ > ./usr/share/man/man1/ > ./usr/share/man/man1/eselect.1.bz2 > tar: ./usr/share/man/man1/kernel-config.1.bz2: Cannot open: Too many levels > of symbolic links > ./usr/share/man/man1/profile-config.1.bz2 > ./usr/share/man/man1/rc-config.1.bz2 > ./usr/share/man/man5/ > ./usr/share/man/man5/binutils.eselect.5.bz2 > ./usr/share/man/man5/editor.eselect.5.bz2 > ./usr/share/man/man5/env.eselect.5.bz2 > > There are symlinks in /usr/share/man/man1: > # ls -l /var/tmp/portage/app-admin/eselect-1.3.8/image/usr/share/man/man1/ > total 4 > drwxr-xr-x 2 root root 101 Feb 20 09:58 ./ > drwxr-xr-x 4 root root 28 Feb 20 09:58 ../ > -rw-r--r-- 1 root root 952 Feb 20 09:58 eselect.1.bz2 > lrwxrwxrwx 1 root root 28 Feb 20 09:58 kernel-config.1.bz2 -> > ../man5/kernel.eselect.5.bz2 > lrwxrwxrwx 1 root root 29 Feb 20 09:58 profile-config.1.bz2 -> > ../man5/profile.eselect.5.bz2 > lrwxrwxrwx 1 root root 24 Feb 20 09:58 rc-config.1.bz2 -> > ../man5/rc.eselect.5.bz2 > > this becomes too much for tar when running under fakeroot, why I don't know Did some debugging and found: After install some other cmds are executed from this variable: _post_phase_cmds = { "install" : [ "install_qa_check", "install_symlink_html_docs", "install_hooks"], "preinst" : [ "preinst_sfperms", "preinst_selinux_labels", "preinst_suid_scan", ] } install_qa_check does the ecompressdir: bzip2 -9 /usr/share/doc ecompressdir: bzip2 -9 /usr/share/man However all the post_phase cmds are executed WITHOUT the fakeroot wrapper, this obviosly confuses fakeroot later on. Adding the below hack did the trick, but this is just a hack: diff --git a/bin/ebuild-helpers/ecompress b/bin/ebuild-helpers/ecompress index 71287b4..918f31d 100755 --- a/bin/ebuild-helpers/ecompress +++ b/bin/ebuild-helpers/ecompress @@ -18,6 +18,9 @@ if [[ ${PORTAGE_COMPRESS_FLAGS+set} != "set" ]] ; then bzip2|gzip) PORTAGE_COMPRESS_FLAGS="-9";; esac fi +fr () { + FAKEROOT_DEBUG=5 fakeroot -i ${T}/fakeroot.state -s ${T}/fakeroot.state "$@" +} # decompress_args(suffix, binary) # - suffix: the compression suffix to work with @@ -153,7 +156,7 @@ case $1 in [[ -n ${suffix} ]] && echo -n "${@/%/${suffix}$'\001'}" | \ tr '\001' '\000' | ${XARGS} -0 rm -f # Finally, let's actually do some real work - "${PORTAGE_COMPRESS}" ${PORTAGE_COMPRESS_FLAGS} "$@" + fr "${PORTAGE_COMPRESS}" ${PORTAGE_COMPRESS_FLAGS} "$@" ret=$? [[ $ret -ne 0 ]] && __helpers_die "${0##*/} failed" exit $ret
Also, could the portage team comment? At least say something so I know you have seen this bug.
Yes, we've seen your recent posts. Most of the team are new since after you started this bug. Unfortunately this bug is in areas most of us are unfamiliar with the portage code. That was some great debugging work. My bash foo is too poor to comment on it. Mike, Sebastian, Arfrever, could you please chime in on his solution so far?
(In reply to Brian Dolbec from comment #9) > Yes, we've seen your recent posts. Most of the team are new since after you > started this bug. Unfortunately this bug is in areas most of us are > unfamiliar with the portage code. Great to hear from you though :) > > That was some great debugging work. My bash foo is too poor to comment on Same here, my python foo is even worse > it. Mike, Sebastian, Arfrever, could you please chime in on his solution so > far? I don't think my fix is generic enough, one woul have to wrap every call wich modifies the FS with fakeroot. Preferably one should add fakeroot at a higher level so every FS op is covered. I don't know how to do that. Jocke PS. minor hack improvement: fr () { if has fakeroot ${FEATURES}; then fakeroot -i ${T}/fakeroot.state -s ${T}/fakeroot.state "$@" else "$@" fi }
Pure speculation but here goes: fakeroot support needs to be added to MiscFunctionsProcess
Guys, I really don't know how to move this forward
Mike, the portage folks were interested in having you look at this. -A
Any ideas how to improve fakeroot in portage?
So I did some digging and came up with a more generic solution: diff --git a/pym/_emerge/MiscFunctionsProcess.py b/pym/_emerge/MiscFunctionsProcess.py index bada79d..8fe24e4 100644 --- a/pym/_emerge/MiscFunctionsProcess.py +++ b/pym/_emerge/MiscFunctionsProcess.py @@ -20,8 +20,13 @@ class MiscFunctionsProcess(AbstractEbuildProcess): portage_bin_path = settings["PORTAGE_BIN_PATH"] misc_sh_binary = os.path.join(portage_bin_path, os.path.basename(portage.const.MISC_SH_BINARY)) + fr_args = "" + if "fakeroot" in settings.features: + fr_args = "-i ${T}/fakeroot.state -s ${T}/fakeroot.state -- " + fr_args += misc_sh_binary + misc_sh_binary = portage.const.FAKEROOT_BINARY - self.args = [portage._shell_quote(misc_sh_binary)] + self.commands + self.args = [portage._shell_quote(misc_sh_binary)] + [fr_args] + self.commands if self.logfile is None and \ self.settings.get("PORTAGE_BACKGROUND") != "subprocess": self.logfile = settings.get("PORTAGE_LOG_FILE") This works fine for me
Created attachment 381352 [details, diff] Wrap fakeroot around MISC_SH_BINARY calls
Created attachment 381354 [details, diff] Do not force "pretend" if fakeroot
So the mailing list seems nonfunctional so I attach two patches here that I think are ready. I have built @system with profile /usr/portage/profiles/default/linux/amd64/13.0/ with fakeroot. Please apply
Hallo Joakim. The dev-portage mailing list should be working fine. I will look into testing and merging your patches this weekend or so. Thank you for your work, and apologies on behalf of the team for not paying more attention to this bug.
(In reply to Alexander Berntsen from comment #19) > Hallo Joakim. The dev-portage mailing list should be working fine. I will > look into testing and merging your patches this weekend or so. > > Thank you for your work, and apologies on behalf of the team for not paying > more attention to this bug. Thanks, I got some python knownledge out of it :) If you use cross-emerge(from crossdev) when you test, rember to remove the fakeroot wrapper inside cross-emerge.
(In reply to Alexander Berntsen from comment #19) > Hallo Joakim. The dev-portage mailing list should be working fine. I will > look into testing and merging your patches this weekend or so. > > Thank you for your work, and apologies on behalf of the team for not paying > more attention to this bug. Any progress?
Hallo Joakim, I have been on vacation for a week and have not progressed yet. But "bump" posts will not be necessary -- I *will* get to it eventually.
Created attachment 384624 [details, diff] Cleaner version I made the patch somewhat prettier.
(In reply to Joakim Tjernlund from comment #23) > Created attachment 384624 [details, diff] [details, diff] > Cleaner version > > I made the patch somewhat prettier. The patch looks good. I just have a couple of minor corrections: 1) In self.commands, fr_file and misc_sh_binary should be quoted with portage._shell_quote(), since self.args is later joined with spaces and passed to the shell. These paths typically don't contain spaces, but it's always good to be vigilant with our quoting. 2) There is so much conditional logic for fakeroot inside the spawn() function that it's hard to tell when it's really enabled. So, maybe we should make your new code conditional on os.path.exists(fr_file), as a sanity check.
(In reply to Zac Medico from comment #24) > (In reply to Joakim Tjernlund from comment #23) > > Created attachment 384624 [details, diff] [details, diff] [details, diff] > > Cleaner version > > > > I made the patch somewhat prettier. > > The patch looks good. I just have a couple of minor corrections: > > 1) In self.commands, fr_file and misc_sh_binary should be quoted with > portage._shell_quote(), since self.args is later joined with spaces and > passed to the shell. These paths typically don't contain spaces, but it's > always good to be vigilant with our quoting. > > 2) There is so much conditional logic for fakeroot inside the spawn() > function that it's hard to tell when it's really enabled. So, maybe we > should make your new code conditional on os.path.exists(fr_file), as a > sanity check. Could I please get you help with that? I really don't know python so just to get to this stage took quite alot of time. Pretty please? :) I do need this merged really soon so if 2) is non trivial I hope this could be a later patch.
Created attachment 384634 [details, diff] Wrap fakeroot around MISC_SH_BINARY calls I've updated your patch to use spawn's fakeroot keyword argument. Please test.
(In reply to Zac Medico from comment #26) > Created attachment 384634 [details, diff] [details, diff] > Wrap fakeroot around MISC_SH_BINARY calls > > I've updated your patch to use spawn's fakeroot keyword argument. Please > test. I have tested it now and it works. I can build the problematic eselect and also @system(100 pkgs) on amd64. I don't understand how this patch works though, seems like magic :)
(In reply to Joakim Tjernlund from comment #27) > I don't understand how this patch works though, seems like magic :) The spawn function supports a boolean "fakeroot" parameter, which causes it to call a function called spawn_fakeroot that handles the rest.
(In reply to Zac Medico from comment #28) > (In reply to Joakim Tjernlund from comment #27) > > I don't understand how this patch works though, seems like magic :) > > The spawn function supports a boolean "fakeroot" parameter, which causes it > to call a function called spawn_fakeroot that handles the rest. Figured it has be something like that but I don't see it. Nothing to worry about as I dont' speak python yet :) Do not forget the [Do not force "pretend" if fakeroot] patch, it is trivial but you cannot use fakeroot without it Getting these 2 patches into portage should close bug 447638 and maybe bug 492724
(In reply to Joakim Tjernlund from comment #29) > Do not forget the [Do not force "pretend" if fakeroot] patch, it is trivial > but you cannot use fakeroot without it I don't like the looks of that patch, because it enables lots of actions without checking for appropriate privileges. Maybe what we should really do is create a new mode which allows a non-root user to run emerge (with or without fakeroot). The new mode would be nearly identical to the existing prefix mode (which does not require root privileges), except that packages are not necessarily installed into a prefix. This is what bug 433453 is about.
Also, since you're using cross-emerge, I guess you are assuming that no dependencies need to be merged to ROOT=/ ? If so, note that default behavior is for emerge to install dependencies from DEPEND into ROOT=/, and that this behavior is controllable with the --root-deps option. However, the --root-deps option is really a nasty hack, and using something like EAPI 5-hdepend should be preferred: https://blogs.gentoo.org/zmedico/2012/09/25/experimental-eapi-5-hdepend/
(In reply to Zac Medico from comment #30) > (In reply to Joakim Tjernlund from comment #29) > > Do not forget the [Do not force "pretend" if fakeroot] patch, it is trivial > > but you cannot use fakeroot without it > > I don't like the looks of that patch, because it enables lots of actions > without checking for appropriate privileges. How will that hurt anything? If you don't have reuqired privs it will simply fail. Running with real root privs is what one should be careful with. > > Maybe what we should really do is create a new mode which allows a non-root > user to run emerge (with or without fakeroot). The new mode would be nearly > identical to the existing prefix mode (which does not require root > privileges), except that packages are not necessarily installed into a > prefix. This is what bug 433453 is about. I don't know what the difference is between my patch and prefix, I just need to merge into my own ROOT is a normal user.
(In reply to Zac Medico from comment #31) > Also, since you're using cross-emerge, I guess you are assuming that no > dependencies need to be merged to ROOT=/ ? If so, note that default behavior > is for emerge to install dependencies from DEPEND into ROOT=/, and that this > behavior is controllable with the --root-deps option. However, the > --root-deps option is really a nasty hack, and using something like EAPI > 5-hdepend should be preferred: > > https://blogs.gentoo.org/zmedico/2012/09/25/experimental-eapi-5-hdepend/ I know, I wish HDEPEND would get approved but it looks like it isn't going to?
(In reply to Joakim Tjernlund from comment #32) > (In reply to Zac Medico from comment #30) > > (In reply to Joakim Tjernlund from comment #29) > > > Do not forget the [Do not force "pretend" if fakeroot] patch, it is trivial > > > but you cannot use fakeroot without it > > > > I don't like the looks of that patch, because it enables lots of actions > > without checking for appropriate privileges. > > How will that hurt anything? If you don't have reuqired privs it will simply > fail. But will it fail gracefully, with a meaningful error message? > Running with real root privs is what one should be careful with. > > > > > Maybe what we should really do is create a new mode which allows a non-root > > user to run emerge (with or without fakeroot). The new mode would be nearly > > identical to the existing prefix mode (which does not require root > > privileges), except that packages are not necessarily installed into a > > prefix. This is what bug 433453 is about. > > I don't know what the difference is between my patch and prefix, I just need > to merge into my own ROOT is a normal user. I think that we can consider your request as being a special case of bug 433453, where the user wishes to install to some ROOT other than ROOT=/, and ROOT=/ is not necessarily writable by that user. (In reply to Joakim Tjernlund from comment #33) > (In reply to Zac Medico from comment #31) > > Also, since you're using cross-emerge, I guess you are assuming that no > > dependencies need to be merged to ROOT=/ ? If so, note that default behavior > > is for emerge to install dependencies from DEPEND into ROOT=/, and that this > > behavior is controllable with the --root-deps option. However, the > > --root-deps option is really a nasty hack, and using something like EAPI > > 5-hdepend should be preferred: > > > > https://blogs.gentoo.org/zmedico/2012/09/25/experimental-eapi-5-hdepend/ > > I know, I wish HDEPEND would get approved but it looks like it isn't going > to? It seems to be listed in the Rejected section here: https://wiki.gentoo.org/wiki/Future_EAPI/EAPI_6_tentative_features I haven't been following the discussions. It may just be a matter of someone writing a PMS patch and garnering enough support on the gentoo-pms mailing list.
(In reply to Zac Medico from comment #34) > (In reply to Joakim Tjernlund from comment #32) > > (In reply to Zac Medico from comment #30) > > > (In reply to Joakim Tjernlund from comment #29) > > > > Do not forget the [Do not force "pretend" if fakeroot] patch, it is trivial > > > > but you cannot use fakeroot without it > > > > > > I don't like the looks of that patch, because it enables lots of actions > > > without checking for appropriate privileges. > > > > How will that hurt anything? If you don't have reuqired privs it will simply > > fail. > > But will it fail gracefully, with a meaningful error message? > > > Running with real root privs is what one should be careful with. > > > > > > > > Maybe what we should really do is create a new mode which allows a non-root > > > user to run emerge (with or without fakeroot). The new mode would be nearly > > > identical to the existing prefix mode (which does not require root > > > privileges), except that packages are not necessarily installed into a > > > prefix. This is what bug 433453 is about. > > > > I don't know what the difference is between my patch and prefix, I just need > > to merge into my own ROOT is a normal user. > > I think that we can consider your request as being a special case of bug > 433453, where the user wishes to install to some ROOT other than ROOT=/, and > ROOT=/ is not necessarily writable by that user. Maybe, would binpkgs still list root etc. as owner of the files? I want to build binpkgs as myself and then be able to install there on a real target using real root.
(In reply to Joakim Tjernlund from comment #35) > Maybe, would binpkgs still list root etc. as owner of the files? Yes, but only when you have FEATURES=fakeroot enabled. Otherwise, it would behave like prefix mode does now, and all files would be owned by the user running emerge. > I want to build binpkgs as myself and then be able to install there on > a real target using real root. Using FEATURES=fakeroot together with the feature discussed in bug 433453 should enable you to do that.
(In reply to Zac Medico from comment #36) > (In reply to Joakim Tjernlund from comment #35) > > Maybe, would binpkgs still list root etc. as owner of the files? > > Yes, but only when you have FEATURES=fakeroot enabled. Otherwise, it would > behave like prefix mode does now, and all files would be owned by the user > running emerge. > > > I want to build binpkgs as myself and then be able to install there on > > a real target using real root. > > Using FEATURES=fakeroot together with the feature discussed in bug 433453 > should enable you to do that. Great, could you refresh the patch in 433453 so I can test ?
(In reply to Joakim Tjernlund from comment #37) > Great, could you refresh the patch in 433453 so I can test ? If just attached a refreshed patch to bug 433453.
I think we can change the summary of this bug to refer to the "Too many levels of symbolic links" issue with tar that was solved by the "Wrap fakeroot around MISC_SH_BINARY calls" patch. Then we can close this bug, since the fix is already released in portage-2.2.13.
"Wrap fakeroot around MISC_SH_BINARY calls" is in this commit: https://github.com/gentoo/portage/commit/dd2f7f91713bb3a5a14635c7930885fab99c5b17
(In reply to Zac Medico from comment #39) > I think we can change the summary of this bug to refer to the "Too many > levels of symbolic links" issue with tar that was solved by the "Wrap > fakeroot around MISC_SH_BINARY calls" patch. Then we can close this bug, > since the fix is already released in portage-2.2.13. Yes, this bug is fixed and can be closed.
This is fixed in 2.2.13.
*** Bug 447638 has been marked as a duplicate of this bug. ***
