| Summary: | add the possibility to install packages as non-root user | ||
|---|---|---|---|
| Product: | Portage Development | Reporter: | Sébastien Fabbro (RETIRED) <bicatali> |
| Component: | Enhancement/Feature Requests | Assignee: | Portage team <dev-portage> |
| Status: | RESOLVED FIXED | ||
| Severity: | normal | CC: | joakim.tjernlund |
| Priority: | Normal | Keywords: | InVCS, PATCH |
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | Linux | ||
| See Also: | https://bugs.gentoo.org/show_bug.cgi?id=489016 | ||
| Whiteboard: | |||
| Package list: | Runtime testing required: | --- | |
| Bug Depends on: | |||
| Bug Blocks: | 484436 | ||
| Attachments: |
Support unprivileged mode
Support unprivileged mode create_trees: whitelist PORTAGE_DEPCACHEDIR |
||
|
Description
Sébastien Fabbro (RETIRED)
2012-08-30 19:08:35 UTC
We might be able to auto-detect this case, without the need for an option. For example, see how the master branch automatically generates variables for prefix in this commit: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=9ba122ee2242369aed344e3756d29c9b153bdbc8 So, if $ROOT (which defaults to "/") is owned by the calling user, can we just use the UID and GID of $ROOT to initialize the relevant variables? Those variables would be PORTAGE_INST_UID, PORTAGE_INST_GID, PORTAGE_USERNAME, and PORTAGE_GRPNAME. at least in my particular uses, that would perfectly do what i need. Created attachment 323232 [details, diff]
Support unprivileged mode
The unprivileged mode is automatically enabled if the current user is not root, but has write access to the EROOT directory (not due to the 0002 bit).
This patch makes fowners failures non-fatal for unprivileged mode, since that's how it has always behaved for prefix. Is that okay? We could also add chown wrappers if that helps.
so if i get it right, the user would only have to set the ROOT directory, and may be some other variables that one has to set, which i forget (log directories, lockfiles?) what was the rationale behind fowners failures non fatal? (In reply to comment #4) > so if i get it right, the user would only have to set the ROOT directory, > and may be some other variables that one has to set, which i forget (log > directories, lockfiles?) Right. Are you planning to install everything into a ROOT? If you are, and you don't have write access to /, then the emerge --root-deps option may be useful, since that will allow you to avoid having it try to install DEPEND in / where you don't have write access. > what was the rationale behind fowners failures non fatal? For prefix, they typically don't care for this kind of failure, since they expect everything to be owned and executed by a single user. If your use case is different, then we can handle it differently. (In reply to comment #5) > > Right. Are you planning to install everything into a ROOT? If you are, and > you don't have write access to /, then the emerge --root-deps option may be > useful, since that will allow you to avoid having it try to install DEPEND > in / where you don't have write access. to be precise in my case, i would need to do two things actually: a) install everything into a user ROOT, and b) install a very thin system as root (using it later on as a very thin virtual machine), and install a huge software stack as a non-root user which i will later distribute to the thin VMs as an on-demand read-only https based mounted file system > > what was the rationale behind fowners failures non fatal? > > For prefix, they typically don't care for this kind of failure, since they > expect everything to be owned and executed by a single user. If your use > case is different, then we can handle it differently. so i think i would rather have some chown wrappers. (In reply to comment #6) > to be precise in my case, i would need to do two things actually: a) install > everything into a user ROOT, and b) install a very thin system as root > (using it later on as a very thin virtual machine), and install a huge > software stack as a non-root user which i will later distribute to the thin > VMs as an on-demand read-only https based mounted file system To clarify, the software that you install as a non-root user is not intended to be under a prefix? > so i think i would rather have some chown wrappers. I've added some chown and chgrp wrappers: http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit;h=45bedf9cdc10fafd94858f67d62b36d35dd99544 Will these wrappers suit your needs? (In reply to comment #7) > To clarify, the software that you install as a non-root user is not intended > to be under a prefix? right now it is under a prefix, but ultimately it won't. > > I've added some chown and chgrp wrappers: > > http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=commit; > h=45bedf9cdc10fafd94858f67d62b36d35dd99544 > > Will these wrappers suit your needs? they do. thanks! Created attachment 384764 [details, diff] Support unprivileged mode This is a rebase of the previous patch, so that it applies to the current master. I'll be maintaining this patch in the following branch: https://github.com/zmedico/portage/tree/bug_433453 This branch currently also includes the MiscFunctionsProcess.py fakeroot patch from bug 489016. (In reply to Zac Medico from comment #9) > Created attachment 384764 [details, diff] [details, diff] > Support unprivileged mode > > This is a rebase of the previous patch, so that it applies to the current > master. I'll be maintaining this patch in the following branch: > > https://github.com/zmedico/portage/tree/bug_433453 > > This branch currently also includes the MiscFunctionsProcess.py fakeroot > patch from bug 489016. Initial tests show that FEATURES=fakeroot ... works as before but removing fakeroot yields This action requires superuser access... Would you like to add --pretend to options? [Yes/No] Should not this work too if my ROOT is writeable by me? (In reply to Joakim Tjernlund from comment #10) > (In reply to Zac Medico from comment #9) > > Created attachment 384764 [details, diff] [details, diff] [details, diff] > > Support unprivileged mode > > > > This is a rebase of the previous patch, so that it applies to the current > > master. I'll be maintaining this patch in the following branch: > > > > https://github.com/zmedico/portage/tree/bug_433453 > > > > This branch currently also includes the MiscFunctionsProcess.py fakeroot > > patch from bug 489016. > > Initial tests show that FEATURES=fakeroot ... works as before but > removing fakeroot yields > This action requires superuser access... > Would you like to add --pretend to options? [Yes/No] > > Should not this work too if my ROOT is writeable by me? Oops, used my old portage install. This is your new: ./cross-emerge-system -1a eselect Traceback (most recent call last): File "/usr/local/src/zmedico/portage/bin/emerge", line 51, in <module> retval = emerge_main() File "/usr/local/src/zmedico/portage/pym/_emerge/main.py", line 1042, in emerge_main action=myaction, args=myfiles, opts=myopts) File "/usr/local/src/zmedico/portage/pym/portage/proxy/objectproxy.py", line 31, in __call__ return result(*args, **kwargs) File "/usr/local/src/zmedico/portage/pym/_emerge/actions.py", line 3238, in load_emerge_config setconfig = load_default_config(settings, root_trees) File "/usr/local/src/zmedico/portage/pym/portage/_sets/__init__.py", line 316, in load_default_config return SetConfig(_getfiles(), settings, trees) File "/usr/local/src/zmedico/portage/pym/portage/_sets/__init__.py", line 65, in __init__ for p in paths: File "/usr/local/src/zmedico/portage/pym/portage/_sets/__init__.py", line 308, in _getfiles dbapi = trees["porttree"].dbapi File "/usr/local/src/zmedico/portage/pym/portage/util/__init__.py", line 1472, in __getitem__ result = lazy_item.func(*pargs, **kwargs) File "/usr/local/src/zmedico/portage/pym/portage/dbapi/porttree.py", line 1089, in __init__ self.dbapi = portdbapi(mysettings=settings) File "/usr/local/src/zmedico/portage/pym/portage/dbapi/porttree.py", line 240, in __init__ self.depcachedir, x, self._known_keys, **cache_kwargs) File "/usr/local/src/zmedico/portage/pym/portage/cache/sqlite.py", line 47, in __init__ self._db_init_connection(config) File "/usr/local/src/zmedico/portage/pym/portage/cache/sqlite.py", line 82, in _db_init_connection raise cache_errors.InitializationError(self.__class__, "can't ensure perms on %s" % self._dbpath) InitializationError: Creation of instance <class 'portage.cache.sqlite.database'> failed due to can't ensure perms on /var/cache/edb/dep/usr/portage.sqlite (In reply to Joakim Tjernlund from comment #11) > (In reply to Joakim Tjernlund from comment #10) > > (In reply to Zac Medico from comment #9) > > > Created attachment 384764 [details, diff] [details, diff] [details, diff] [details, diff] > > > Support unprivileged mode > > > > > > This is a rebase of the previous patch, so that it applies to the current > > > master. I'll be maintaining this patch in the following branch: > > > > > > https://github.com/zmedico/portage/tree/bug_433453 > > > > > > This branch currently also includes the MiscFunctionsProcess.py fakeroot > > > patch from bug 489016. > > > > Initial tests show that FEATURES=fakeroot ... works as before but > > removing fakeroot yields > > This action requires superuser access... > > Would you like to add --pretend to options? [Yes/No] > > > > Should not this work too if my ROOT is writeable by me? > > Oops, used my old portage install. This is your new: > ./cross-emerge-system -1a eselect > Traceback (most recent call last): > File "/usr/local/src/zmedico/portage/bin/emerge", line 51, in <module> > retval = emerge_main() > File "/usr/local/src/zmedico/portage/pym/_emerge/main.py", line 1042, in > emerge_main > action=myaction, args=myfiles, opts=myopts) > File "/usr/local/src/zmedico/portage/pym/portage/proxy/objectproxy.py", > line 31, in __call__ > return result(*args, **kwargs) > File "/usr/local/src/zmedico/portage/pym/_emerge/actions.py", line 3238, > in load_emerge_config > setconfig = load_default_config(settings, root_trees) > File "/usr/local/src/zmedico/portage/pym/portage/_sets/__init__.py", line > 316, in load_default_config > return SetConfig(_getfiles(), settings, trees) > File "/usr/local/src/zmedico/portage/pym/portage/_sets/__init__.py", line > 65, in __init__ > for p in paths: > File "/usr/local/src/zmedico/portage/pym/portage/_sets/__init__.py", line > 308, in _getfiles > dbapi = trees["porttree"].dbapi > File "/usr/local/src/zmedico/portage/pym/portage/util/__init__.py", line > 1472, in __getitem__ > result = lazy_item.func(*pargs, **kwargs) > File "/usr/local/src/zmedico/portage/pym/portage/dbapi/porttree.py", line > 1089, in __init__ > self.dbapi = portdbapi(mysettings=settings) > File "/usr/local/src/zmedico/portage/pym/portage/dbapi/porttree.py", line > 240, in __init__ > self.depcachedir, x, self._known_keys, **cache_kwargs) > File "/usr/local/src/zmedico/portage/pym/portage/cache/sqlite.py", line > 47, in __init__ > self._db_init_connection(config) > File "/usr/local/src/zmedico/portage/pym/portage/cache/sqlite.py", line > 82, in _db_init_connection > raise cache_errors.InitializationError(self.__class__, "can't ensure > perms on %s" % self._dbpath) > InitializationError: Creation of instance <class > 'portage.cache.sqlite.database'> failed due to can't ensure perms on > /var/cache/edb/dep/usr/portage.sqlite ls -l /var/cache/edb/dep/usr/portage.sqlite* -rw-rw-rw- 1 root portage 303104 Sep 12 17:27 /var/cache/edb/dep/usr/portage.sqlite deleting this file and redo I get a new cache file owed by me: -rw-rw-r-- 1 jocke users 4096 Sep 15 10:20 /var/cache/edb/dep/usr/portage.sqlite I guess one have used another cache for unpriv operation. Now fakeroot works as before and no fakeroot give me: /cross-emerge-system -1a eselect .... * chown failure ignored with unprivileged user: * chown root:portage ${D}/var/lib/gentoo/news * /bin/chown: changing ownership of ‘${D}/var/lib/gentoo/news’: Operation not permitted >>> Completed installing eselect-1.4.1 into /usr/x86_64-tm-linux-gnu/tmp/portage/app-admin/eselect-1.4.1/image/ I guess this is expected? (In reply to Joakim Tjernlund from comment #12) > I guess one have used another cache for unpriv operation. You can use the PORTAGE_DEPCACHEDIR environment variable to customize the cache location. > Now fakeroot works as before and no fakeroot give me: > /cross-emerge-system -1a eselect > .... > > * chown failure ignored with unprivileged user: > * chown root:portage ${D}/var/lib/gentoo/news > * /bin/chown: changing ownership of ‘${D}/var/lib/gentoo/news’: > Operation not permitted > >>> Completed installing eselect-1.4.1 into /usr/x86_64-tm-linux-gnu/tmp/portage/app-admin/eselect-1.4.1/image/ > > I guess this is expected? Yes, it's supposed to use the unprivileged chown/chgrp wrappers that prefix mode has been using. Those failures can typically be ignored because chown/chgrp calls are typically only relevant when running as root (or fakeroot). (In reply to Zac Medico from comment #13) > (In reply to Joakim Tjernlund from comment #12) > > I guess one have used another cache for unpriv operation. > > You can use the PORTAGE_DEPCACHEDIR environment variable to customize the > cache location. > > > Now fakeroot works as before and no fakeroot give me: > > /cross-emerge-system -1a eselect > > .... > > > > * chown failure ignored with unprivileged user: > > * chown root:portage ${D}/var/lib/gentoo/news > > * /bin/chown: changing ownership of ‘${D}/var/lib/gentoo/news’: > > Operation not permitted > > >>> Completed installing eselect-1.4.1 into /usr/x86_64-tm-linux-gnu/tmp/portage/app-admin/eselect-1.4.1/image/ > > > > I guess this is expected? > > Yes, it's supposed to use the unprivileged chown/chgrp wrappers that prefix > mode has been using. Those failures can typically be ignored because > chown/chgrp calls are typically only relevant when running as root (or > fakeroot). The error is becuase _ensure_access does apply_permissions(path, gid=self._gid, mode=self._perms) which wants to change group from "portage" to my default group "users". This feels a bit odd as the whole dir structure has "portage" as group and and the containing dir has perms drwxrwsr-x 4 root portage 72 Sep 16 19:03 ./ It is resonable to require the user to be a member of portagte group I think in wich case self._gid should be set to "portage" somehow or drop "gid=self._gid" part. Using PORTAGE_DEPCACHEDIR=/tmp/jocke USE=internal-glib ./cross-emerge-system -1a glibc gives the same error, portage wont switch cache dir, /tmp/jocke/usr/portage/ is created though. I added a print(uid, gid , mode, mask, filename); to apply_permissions (-1, 100, 1512, 2, u'/usr/x86_64-tm-linux-gnu/var/lib/portage') (-1, 100, 493, 2, u'/usr/x86_64-tm-linux-gnu/var/cache/edb') (-1, 100, 1080, 2, u'/var/cache/edb/dep') (-1, 100, 1080, 2, u'/var/cache/edb/dep') (-1, 100, 436, -1, u'/var/cache/edb/dep/usr/portage.sqlite') So portage is looking in CHOST first but then goes on to the HOST cache dirs. Created attachment 384948 [details, diff]
create_trees: whitelist PORTAGE_DEPCACHEDIR
Hopefully this patch will fix PORTAGE_DEPCACHEDIR to work for you. You should not need to be in the portage group.
(In reply to Zac Medico from comment #16) > Created attachment 384948 [details, diff] [details, diff] > create_trees: whitelist PORTAGE_DEPCACHEDIR > > Hopefully this patch will fix PORTAGE_DEPCACHEDIR to work for you. You > should not need to be in the portage group. Yes, that did it :) PS. Having a print(filename, uid, gid, mode, mask); in apply_permissions() prints alot, are they all needed? (In reply to Zac Medico from comment #16) > Created attachment 384948 [details, diff] [details, diff] > create_trees: whitelist PORTAGE_DEPCACHEDIR > > Hopefully this patch will fix PORTAGE_DEPCACHEDIR to work for you. You > should not need to be in the portage group. Possibly PORTAGE_DEPCACHEDIR could default to ROOT so one doesn't have to set it? (In reply to Joakim Tjernlund from comment #17) > PS. > Having a > print(filename, uid, gid, mode, mask); > in apply_permissions() prints alot, are they all needed? A lot of them probably become redundant after portage has adjusted them once, but it's good to validate permissions with each run. The function does not actually call chmod or chown unless changes are necessary, so it's pretty harmless. (In reply to Joakim Tjernlund from comment #18) > Possibly PORTAGE_DEPCACHEDIR could default to ROOT so one doesn't have to > set it? Well, I'm not so sure that polluting ROOT is a good default. For unprivileged mode though, it might make sense to use ~/.portage/cache or something like that. > (In reply to Joakim Tjernlund from comment #18)
> > Possibly PORTAGE_DEPCACHEDIR could default to ROOT so one doesn't have to
> > set it?
>
> Well, I'm not so sure that polluting ROOT is a good default. For
> unprivileged mode though, it might make sense to use ~/.portage/cache or
> something like that.
Well, in a way you alredy pollute ROOT(for ROOT = /) and there are
other pkgs which already put stuff under ROOT/var/cache.
My cross ROOT already have portage stuff there anyway:
ls new-root-system/var/cache/edb/ -l
total 276
drwxr-xr-x 3 jocke users 96 Sep 17 21:04 ./
drwxr-xr-x 4 jocke users 26 Jul 21 18:31 ../
-rw-r--r-- 1 jocke users 3 Sep 17 21:04 counter
drwxrwsr-x 3 jocke portage 16 Jul 24 17:53 dep/
-rw-r--r-- 1 jocke users 1348 Sep 17 21:04 mtimedb
-rw-r--r-- 1 jocke users 5452 Sep 15 10:20 vdb_blockers.pickle
-rw-r--r-- 1 jocke users 265218 Sep 15 10:20 vdb_metadata.pickle
BTW, are you ready to move this to master now?
(In reply to Joakim Tjernlund from comment #20) > Well, in a way you alredy pollute ROOT(for ROOT = /) and there are > other pkgs which already put stuff under ROOT/var/cache. Yes, you're right. So, I'll make it default to $ROOT/var/cache/edb/dep > BTW, are you ready to move this to master now? Soon I'll post it to the gentoo-portage-dev mailing list for feedback. (In reply to Zac Medico from comment #21) > (In reply to Joakim Tjernlund from comment #20) > > Well, in a way you alredy pollute ROOT(for ROOT = /) and there are > > other pkgs which already put stuff under ROOT/var/cache. > > Yes, you're right. So, I'll make it default to $ROOT/var/cache/edb/dep > > > BTW, are you ready to move this to master now? > > Soon I'll post it to the gentoo-portage-dev mailing list for feedback. ping? I've just sent the "unprivileged" patch to the list for review: http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4607 I haven't written a patch to change the PORTAGE_DEPCACHEDIR default yet. Hopefully I will find some time for that soon. Also, I'm not sure how the unprivileged mode will behave if it needs to install DEPEND (or HDEPEND) into ROOT=/ for some build-time dependency. I still need to look into that. (In reply to Zac Medico from comment #23) > I've just sent the "unprivileged" patch to the list for review: > > http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4607 > > I haven't written a patch to change the PORTAGE_DEPCACHEDIR default yet. > Hopefully I will find some time for that soon. > > Also, I'm not sure how the unprivileged mode will behave if it needs to > install DEPEND (or HDEPEND) into ROOT=/ for some build-time dependency. I > still need to look into that. Haven't seen any feedback on list, how to move this forward? Here's what Brian said about another patch set recently posted to the list, and the same probably applies to the "unprivileged" patch: http://gentoo.2317880.n4.nabble.com/PATCH-per-package-environment-generalize-the-mechanism-to-be-profile-specific-td285220i40.html#a286544 > We will be having a team meeting soon to decide recent patches, etc.. > > Although I think for the 2.2.14 final release we will not include this, > just a few bug fixes. After 2.2.14 final is released we will likely > merge it so it will be in 9999 for awhile and then the 2.2.15 release. (In reply to Zac Medico from comment #25) > Here's what Brian said about another patch set recently posted to the list, > and the same probably applies to the "unprivileged" patch: > > http://gentoo.2317880.n4.nabble.com/PATCH-per-package-environment-generalize- > the-mechanism-to-be-profile-specific-td285220i40.html#a286544 > > > We will be having a team meeting soon to decide recent patches, etc.. > > > > Although I think for the 2.2.14 final release we will not include this, > > just a few bug fixes. After 2.2.14 final is released we will likely > > merge it so it will be in 9999 for awhile and then the 2.2.15 release. 2.2.14 has been out for a while, now feels like a good time :) The unprivileged mode patch is in git now: https://github.com/gentoo/portage/commit/1364fcd89384c9f60e6d72d7057dc00d8caba175 Now I'll look into doing the PORTAGE_DEPCACHEDIR changes that we've discussed. (In reply to Zac Medico from comment #27) > The unprivileged mode patch is in git now: > > https://github.com/gentoo/portage/commit/ > 1364fcd89384c9f60e6d72d7057dc00d8caba175 > > Now I'll look into doing the PORTAGE_DEPCACHEDIR changes that we've > discussed. This commit is causing this error locally: >>> Emerging (1 of 1) app-misc/beep-1.2.2-r1::gentoo Unable to unshare: EINVAL * beep-1.2.2.tar.gz SHA256 SHA512 WHIRLPOOL size ;-) ... [ ok ] Unable to unshare: EINVAL local variable 'portage_gid' referenced before assignment: /usr/bin/sandbox /home/michael/dev/gentoo/portage/bin/ebuild.sh unpack Traceback (most recent call last): File "/home/michael/dev/gentoo/portage/pym/portage/process.py", line 317, in spawn unshare_net, unshare_ipc, cgroup) File "/home/michael/dev/gentoo/portage/pym/portage/process.py", line 504, in _exec if groups: File "/home/michael/dev/gentoo/portage/pym/portage/proxy/objectproxy.py", line 89, in __bool__ return bool(object.__getattribute__(self, '_get_target')()) File "/home/michael/dev/gentoo/portage/pym/portage/data.py", line 250, in _get_target return _get_global(object.__getattribute__(self, '_name')) File "/home/michael/dev/gentoo/portage/pym/portage/data.py", line 160, in _get_global v = [portage_gid] UnboundLocalError: local variable 'portage_gid' referenced before assignment * The ebuild phase 'unpack' has exited unexpectedly. This type of behavior (In reply to Michael Palimaka (kensington) from comment #28) > UnboundLocalError: local variable 'portage_gid' referenced before assignment > * The ebuild phase 'unpack' has exited unexpectedly. This type of behavior Thanks, it's fixed in git now: https://github.com/gentoo/portage/commit/b26d3aad771a6ca177228ba984c3c98130e3c60a (In reply to Zac Medico from comment #27) > The unprivileged mode patch is in git now: > > https://github.com/gentoo/portage/commit/ > 1364fcd89384c9f60e6d72d7057dc00d8caba175 > > Now I'll look into doing the PORTAGE_DEPCACHEDIR changes that we've > discussed. ping? (In reply to Zac Medico from comment #16) > Created attachment 384948 [details, diff] [details, diff] > create_trees: whitelist PORTAGE_DEPCACHEDIR I'm working on an updated patch that will fix create_trees to automatically generate a PORTAGE_DEPCACHEDIR setting relative to EROOT if PORTAGE_DEPCACHEDIR is unset and the default PORTAGE_DEPCACHEDIR is not writable. I will post this patch later today. The new PORTAGE_DEPCACHEDIR patch is in this branch: https://github.com/zmedico/portage/tree/bug_433453 I've posted it for review here: http://thread.gmane.org/gmane.linux.gentoo.portage.devel/4762 (In reply to Zac Medico from comment #32) > The new PORTAGE_DEPCACHEDIR patch is in this branch: It's in the master branch now: https://github.com/gentoo/portage/commit/690b9deaf22f85a3cc7b366f5bd126b9e603c90d This is fixed in 2.2.15. |