Bug 620524 – sys-apps/install-xattr: setxattr() failed: Operation not permitted - on hardened profile with FEATURES="userpriv" during src_compile as opposed to src_install

• Product: Gentoo Linux
• Status: CONFIRMED ---

[Description Alessandro Barbieri 2017-06-03 17:22:52 UTC]

emerge fails with

install    -m0755 -d ../bin
install -m0755 unibmp2hex unicoverage unidup unibdf2hex unifontpic unigencircles unigenwidth unihex2bmp unihexgen unipagecount ../bin
install-xattr: setxattr() failed: Operation not permitted
make[1]: *** [Makefile:43: bin-stamp] Error 1
make[1]: Leaving directory '/var/tmp/notmpfs/portage/media-fonts/unifont-9.0.06/work/unifont-9.0.06/src'
make: *** [Makefile:76: bindir] Error 2
 * ERROR: media-fonts/unifont-9.0.06::gentoo failed (compile phase):
 *   emake failed

I have an hardened profile with non hardened kernel (with the patch for xattr and tmpfs)
the build dir is on btrfs (on tmpfs emerges)

[Comment 1 Alessandro Barbieri 2017-06-03 17:23:28 UTC]

Created attachment 475112 [details]
info

info

[Comment 2 Alessandro Barbieri 2017-06-03 17:24:05 UTC]

Created attachment 475114 [details]
unifont-9.0.06:20170603-163016.log

log

[Comment 3 Mike Gilbert 2017-06-03 21:25:46 UTC]

This is not a bug in unifont.

[Comment 4 Alessandro Barbieri 2017-06-04 14:56:43 UTC]

Retried in tmpfs (ramfs) and fails too

[Comment 5 Alessandro Barbieri 2017-06-04 23:15:28 UTC]

with FEATURES="-xattr" I managed to emerge it

[Comment 6 Alessandro Barbieri 2017-06-10 01:28:21 UTC]

I suspect it's pypy fault

[Comment 7 Alessandro Barbieri 2017-06-10 23:46:06 UTC]

It's not pypy

[Comment 8 matt black 2017-07-05 19:13:55 UTC]

This happens to me building rust on a non-hardened desktop profile system on ext4 hd. I got a successful build by disabling kernel option EXT4_FS_SECURITY. Unfortunately that breaks other things.

[Comment 9 Tim Climis 2017-07-23 17:15:30 UTC]

I also ran into this with unifont-10 and rust.  Taking EXT4_FS_SECURITY out of my kernel also allowed me to build those packages.

[Comment 10 Alessandro Barbieri 2017-07-30 19:16:22 UTC]

(In reply to Tim Climis from comment #9)
> I also ran into this with unifont-10 and rust.  Taking EXT4_FS_SECURITY out
> of my kernel also allowed me to build those packages.

Ok but I'm on BTRFS and doesn't exist a similar option (e.g. BTRFS_FS_SECURITY)
must be something else

[Comment 11 Nils Freydank 2017-10-01 11:20:04 UTC]

Hi,
I have this issue with literally *every* package in a gentoo hardened, non-selinux container (lxd/lxc) on btrfs. I purged SELinux on the host kernel a day ago and enabled SMACK for elogind. I don’t know if that somehow interferes.

[Comment 12 Anthony Basile 2017-10-01 11:36:36 UTC]

(In reply to Nils from comment #11)
> Hi,
> I have this issue with literally *every* package in a gentoo hardened,
> non-selinux container (lxd/lxc) on btrfs. I purged SELinux on the host
> kernel a day ago and enabled SMACK for elogind. I don’t know if that somehow
> interferes.

Can you try the following on any of your btrfs volumes as root:

# touch zzz
# setfattr -n user.test -v abc zzz
# getfattr -d zzz

You should get

# file: zzz
user.test="abc"

If that fails, then attach your kernel config file, along with details of what version it is and any custom patches.

[Comment 13 Nils Freydank 2017-10-01 14:55:14 UTC]

(In reply to Anthony Basile from comment #12)
> (In reply to Nils from comment #11)
> [...]
> 
> Can you try the following on any of your btrfs volumes as root:
> 
> [...]

I tried, and this works both on the host where I don’t see issues
and the container. All systems are hardened-64bit-glibc ones.

[Comment 14 Anthony Basile 2017-10-01 17:41:49 UTC]

(In reply to Nils from comment #13)
> (In reply to Anthony Basile from comment #12)
> > (In reply to Nils from comment #11)
> > [...]
> > 
> > Can you try the following on any of your btrfs volumes as root:
> > 
> > [...]
> 
> I tried, and this works both on the host where I don’t see issues
> and the container. All systems are hardened-64bit-glibc ones.

This is perplexing.  Why would setxattr() fail in install-xattr but not fail when called from setfattr?  Are there any clues dmesg.

Also, I'd like to reproduce this at my end.  Can you tell me what kernel to use and attach your kernel config?

[Comment 15 Nils Freydank 2017-10-01 19:37:10 UTC]

(In reply to Anthony Basile from comment #14)
> (In reply to Nils from comment #13)
> > (In reply to Anthony Basile from comment #12)
> > > (In reply to Nils from comment #11)
> > > [...]
> > > 
> > > Can you try the following on any of your btrfs volumes as root:
> > > 
> > > [...]
> > 
> > I tried, and this works both on the host where I don’t see issues
> > and the container. All systems are hardened-64bit-glibc ones.
> 
> This is perplexing.  Why would setxattr() fail in install-xattr but not fail
> when called from setfattr?  Are there any clues dmesg.
> 
> Also, I'd like to reproduce this at my end.  Can you tell me what kernel to
> use and attach your kernel config?

I’m using vanilla upstream (currently 4.13.4) with the following patches from
4.13.0 genpatches:

- 1500_XATTR_USER_PREFIX.patch
- 4567_distro-Gentoo-Kconfig.patch
- 1510_fs-enable-link-security-restrictions-by-default.patch
- 5010_enable-additional-cpu-optimizations-for-gcc.patch
- 2900_dev-root-proc-mount-fix.patch

I’ll attach the output of "emerge --info lxd" and my current kernel config.

[Comment 16 Nils Freydank 2017-10-01 19:40:16 UTC]

Created attachment 497362 [details]
emerge --info lxd

[Comment 17 Nils Freydank 2017-10-01 19:40:45 UTC]

Created attachment 497364 [details]
used kernel config

[Comment 18 Nils Freydank 2017-10-01 20:02:25 UTC]

There are no further clues in dmesg, but I recgonised something else: While moving a file from or to a tmpfs as as non-root user I get

mv: setting attribute 'security.SMACK64' for 'security.SMACK64': Operation not permitted

while the file is still moved. Same issue with moving from one btrfs to another (I have no other FS to test right now.)

All tests worked by executing as root. (Running 'mv test1 test2' as a normal user inside one btrfs doesn’t count with CoW, I guess. FTR, that works, too.)

[Comment 19 Anthony Basile 2017-10-01 23:11:03 UTC]

(In reply to Nils from comment #18)
> There are no further clues in dmesg, but I recgonised something else: While
> moving a file from or to a tmpfs as as non-root user I get
> 
> mv: setting attribute 'security.SMACK64' for 'security.SMACK64': Operation
> not permitted
> 
> while the file is still moved. Same issue with moving from one btrfs to
> another (I have no other FS to test right now.)
> 
> All tests worked by executing as root. (Running 'mv test1 test2' as a normal
> user inside one btrfs doesn’t count with CoW, I guess. FTR, that works, too.)

This is probably the root cause of your issue.  Try creating a file with user.test="abc" xattrs as in comment #12 and move it between btrfs volumes and see if you get the same error.

[Comment 20 Nils Freydank 2017-10-02 10:41:08 UTC]

(In reply to Anthony Basile from comment #19)
> (In reply to Nils from comment #18)
> > There are no further clues in dmesg, but I recgonised something else: While
> > moving a file from or to a tmpfs as as non-root user I get
> > 
> > mv: setting attribute 'security.SMACK64' for 'security.SMACK64': Operation
> > not permitted
> > 
> > while the file is still moved. Same issue with moving from one btrfs to
> > another (I have no other FS to test right now.)
> > 
> > All tests worked by executing as root. (Running 'mv test1 test2' as a normal
> > user inside one btrfs doesn’t count with CoW, I guess. FTR, that works, too.)
> 
> This is probably the root cause of your issue.  Try creating a file with
> user.test="abc" xattrs as in comment #12 and move it between btrfs volumes
> and see if you get the same error.

cp works fine, while mv doesn’t save the xattr, i.e. getfattr’s output is empty on the target file.

[Comment 21 Anthony Basile 2017-10-02 15:16:44 UTC]

(In reply to Nils from comment #20)
> 
> cp works fine, while mv doesn’t save the xattr, i.e. getfattr’s output is
> empty on the target file.

Okay something strange is going on here.  Make sure you don't have any aliasing of cp (to something like 'cp --preserve') and then produce an strace of cp, mv and install-xattr as you copy/move the file from one btrfs volume to another.

[Comment 22 Nils Freydank 2017-10-03 20:33:33 UTC]

FTR, I tested 4.13.3 and 4.13.4, where 4.13.3 was my last SELinux kernel without SMACK, and 4.13.3 is my current SMACK kernel without SELinux. Results were identical (just to rule out a recent regression plus SELinux and SMACK themselves).

File path according to which:
  /bin/cp.................. sys-apps/coreutils-8.28 with USE="xattr"
  /bin/mv.................. same
  /usr/bin/install-xattr... sys-apps/install-xattr-0.5-r1

Strace generation:
% touch {cp,mv,install-xattr}-test
% setfattr -n user.test -v abc {cp,mv,install-xattr}-test
% getfattr -d {cp,mv,install-xattr}-test
# file: cp-test
user.test="abc"

# file: mv-test
user.test="abc"

# file: install-xattr-test
user.test="abc"

% strace cp cp-test /mnt/btrfs_usb/data/ >& cp-test.strace # will be attached
% strace mv mv-test /mnt/btrfs_usb/data >& mv-test.strace # will be attached
% strace install-xattr install-xattr-test /mnt/btrfs/usb_data >& install-xattr-test.strace # will be attached

% getfattr -d /mnt/btrfs_usb/data/{cp,mv,install-xattr}-test
getfattr: Removing leading '/' from absolute path names
getfattr: /mnt/btrfs_usb/data/install-xattr-test: No such file or directory
# file: mnt/btrfs_usb/data/mv-test
user.test="abc"

[Comment 23 Nils Freydank 2017-10-03 20:35:11 UTC]

Created attachment 497560 [details]
strace for cp (btrfs to btrfs)

[Comment 24 Nils Freydank 2017-10-03 20:35:37 UTC]

Created attachment 497562 [details]
strace for mv (btrfs to btrfs)

[Comment 25 Nils Freydank 2017-10-03 20:36:04 UTC]

Created attachment 497564 [details]
strace for install-xattr (btrfs to btrfs)

[Comment 26 Nils Freydank 2017-10-03 20:41:27 UTC]

Created attachment 497566 [details]
strace for install-xattr (btrfs to btrfs) - fixed

I had a typo in my command. install-xattr seems to work:

% getfattr -d /mnt/btrfs_usb/data/install-xattr-test
# file: mnt/btrfs_usb/data/install-xattr-test
user.test="abc"

[Comment 27 Anthony Basile 2017-10-04 05:07:53 UTC]

(In reply to Nils from comment #26)
> Created attachment 497566 [details]
> strace for install-xattr (btrfs to btrfs) - fixed
> 
> I had a typo in my command. install-xattr seems to work:
> 
> % getfattr -d /mnt/btrfs_usb/data/install-xattr-test
> # file: mnt/btrfs_usb/data/install-xattr-test
> user.test="abc"

Well this partially answers my question in comment #14.  setfattr() is working in install-xattr as expected.  There is nothing abnormal in the straces.  (BTW, I don't need it, but since install-xattr forks a child, to get the full strace you'd need the -f flag.)

I assume you ran the commands in comment #22 as root.  I'm wondering if you get the same failure you did in comment #18 with user.pax labels as when you run the commands as a non-root user.  Since portage runs as root:portage, it may be that  SMACK is preventing a non privileged user from copying user.pax labels over like security.SMACK.  Also make sure that portage is in the wheel group.

So far it appears that install-xattr is not failing in general, but it is failing to setxattr() in the portage environment.  I'm not sure why.

[Comment 28 Nils Freydank 2017-10-06 14:47:31 UTC]

(In reply to Anthony Basile from comment #27)
> (In reply to Nils from comment #26)
> > Created attachment 497566 [details]
> > strace for install-xattr (btrfs to btrfs) - fixed
> > 
> > I had a typo in my command. install-xattr seems to work:
> > 
> > % getfattr -d /mnt/btrfs_usb/data/install-xattr-test
> > # file: mnt/btrfs_usb/data/install-xattr-test
> > user.test="abc"
> 
> Well this partially answers my question in comment #14.  setfattr() is
> working in install-xattr as expected.  There is nothing abnormal in the
> straces.  (BTW, I don't need it, but since install-xattr forks a child, to
> get the full strace you'd need the -f flag.)
> 
> I assume you ran the commands in comment #22 as root.  I'm wondering if you
> get the same failure you did in comment #18 with user.pax labels as when you
> run the commands as a non-root user.  Since portage runs as root:portage, it
> may be that  SMACK is preventing a non privileged user from copying user.pax
> labels over like security.SMACK.  Also make sure that portage is in the
> wheel group.
> 
> So far it appears that install-xattr is not failing in general, but it is
> failing to setxattr() in the portage environment.  I'm not sure why.

Sorry, I partially mixed up things:
- without SMACK, everything works fine as expected
- with SMACK, only inside the LX-containers install-xattrs fails for portage,
regardless if portage is in wheel group or not. By the way, why should it be there? I run no "real", non-system, users there so I could not test it as non-root.

I was’t the original bug reporter though. If the original reporter (Alessandro) hasn’t SMACK enabled our issue’s roots might be different and I suggest to open a new bug report for SMACK + install-xattr.

[Comment 29 sachse 2017-10-07 23:41:20 UTC]

It looks like it happens something to do with the actual portage package.
The previous version works fine, but after updating it, the gcc package can not be longer installed.
I have add "=sys-apps/portage-2.3.8 -xattr" to package.use.
This skip the problem temporally.

[Comment 30 Anthony Basile 2017-10-11 14:43:36 UTC]

(In reply to sachse from comment #29)
> It looks like it happens something to do with the actual portage package.
> The previous version works fine, but after updating it, the gcc package can
> not be longer installed.
> I have add "=sys-apps/portage-2.3.8 -xattr" to package.use.
> This skip the problem temporally.

Maybe the portage team knows what changed wrt xattrs between the two versions.

[Comment 31 Mike Gilbert 2017-10-11 15:05:22 UTC]

(In reply to sachse from comment #29)

I think we would need more information (log output) about your install failure to diagnose it.

[Comment 32 Zac Medico 2017-12-09 22:08:43 UTC]

You can use a command like this to dump all of the xattrs:

getfattr -m- -d /var/tmp/notmpfs/portage/media-fonts/unifont-9.0.06/work/unifont-9.0.06/src/*

With portage-2.3.17, the doins helper copies xattrs itself, and is will display the specific xattr name that triggers the EOPNOTSUPP error. For example, the doins out from bug 640290 looks like this:

portage.exception.OperationNotSupported: Filesystem containing file '/var/tmp/portage/app-admin/systemrescuecd-x86-5.1.2/image//usr/share/systemrescuecd/systemrescuecd-x86-5.1.2.iso' does not support extended attribute 'user.xdg.origin.url'

[Comment 33 Alessandro Barbieri 2019-08-05 05:59:38 UTC]

Seems that adding FEATURES="-userpriv" solves the problem, then I managed to emerge glibc and unifont. Could you check?

[Comment 34 Terra 2021-02-18 20:35:22 UTC]

Profile:  default/linux/amd64/17.1/hardened

I hit this today after turning on SMACK in the kernel.

When I hit the glitch I was emerging:  sys-fs/multipath-tools

After some debugging, I figured out the solution to the problem was adding the following to /etc/portage/make.conf

PORTAGE_XATTR_EXCLUDE="security.SMACK64 ${PORTAGE_XATTR_EXCLUDE}"

FWIW - the 'man make.conf' entry states it defaults to "security.*" which is not true.  The default setting can be found in:  /usr/share/portage/config/make.globals

PING: Zac Medico <zmedico@g.o>

[Comment 35 Zac Medico 2021-02-19 01:16:38 UTC]

(In reply to Terra from comment #34)
> Profile:  default/linux/amd64/17.1/hardened
> 
> I hit this today after turning on SMACK in the kernel.
> 
> When I hit the glitch I was emerging:  sys-fs/multipath-tools
> 
> After some debugging, I figured out the solution to the problem was adding
> the following to /etc/portage/make.conf
> 
> PORTAGE_XATTR_EXCLUDE="security.SMACK64 ${PORTAGE_XATTR_EXCLUDE}"
> 
> FWIW - the 'man make.conf' entry states it defaults to "security.*" which is
> not true.  The default setting can be found in: 
> /usr/share/portage/config/make.globals
> 
> PING: Zac Medico <zmedico@g.o>

Thanks for the report. Are we sure that a change to PORTAGE_XATTR_EXCLUDE will address the root cause of the problem? Note that portage has a special preinst_selinux_labels function:

https://gitweb.gentoo.org/proj/portage.git/tree/bin/misc-functions.sh?h=portage-3.0.14#n428

It says here that at privileged process can use setxattr to set the Smack label, so maybe we should escalate privileges instead:

https://www.kernel.org/doc/Documentation/security/Smack.txt

[Comment 36 Zac Medico 2021-02-19 01:36:02 UTC]

I've opened bug 771540 about adding support for PORTAGE_XATTR_EXCLUDE="security.* -security.capability" so that it's easy to make exceptions like it is for INSTALL_MASK.

[Comment 37 Alessandro Barbieri 2021-02-28 01:43:05 UTC]

The point here is that only root can set extended attributes (that's why without userpriv it works). As a workaround I've set 
 %portage ALL=(ALL) NOPASSWD: /usr/bin/install-xattr
inside /etc/sudoers

[Comment 38 Alessandro Barbieri 2021-02-28 01:43:38 UTC]

I think that a more elegant solution will involve capabilities

[Comment 39 Zac Medico 2021-02-28 01:49:23 UTC]

The src_install phase runs as root (regardless of FEATURES=userpriv), so if you can rearrange the ebuild to call the install commands during src_install, then you'll have a working solution.

[Comment 40 Zac Medico 2021-02-28 01:55:30 UTC]

I think we should simply fix media-fonts/unifont to call install during src_install, and reassign this bug to the media-fonts/unifont maintainers.

[Comment 41 Zac Medico 2021-02-28 02:00:24 UTC]

If portage supports PORTAGE_XATTR_EXCLUDE="security.* -security.capability" as proposed in bug 771540, then do we want to exclude security.SMACK64 or not? If the setxattr call succeeds with root privileges in src_install, then is there no need to exclude it?

[Comment 42 Mike Gilbert 2021-02-28 03:04:38 UTC]

(In reply to Zac Medico from comment #40)
> I think we should simply fix media-fonts/unifont to call install during
> src_install, and reassign this bug to the media-fonts/unifont maintainers.

The unifont build system uses "install" in place of mkdir and cp to copy files around in ${S}.

This would probably work just fine if it called /usr/bin/install without the install-xattr implicit magic.

[Comment 43 Zac Medico 2021-02-28 04:11:14 UTC]

Where did the security.SMACK64 xattr come from anyway, and is there any reason to preserve it?