The init script provided by app-emulation/qemu, /etc/init.d/qemu-binfmt, can be used to define the needed prerequisites to be able to launch other arch (supported by QEmu) binaries like and so for example by chroot, bash etc. With a qemu is builded with USE="static static-softmmu static-user" QEMU_SOFTMMU_TARGETS="arm i386 x86_64" QEMU_USER_TARGETS="arm i386 x86_64" the binary /usr/bin/qemu-arm is installed. Then you can launch for example armhf binaries # /etc/init.d/qemu-binfmt start # /usr/bin/qemu-arm -L /data/build/armhf /data/build/armhf/bin/uname -a Linux otis.scabb 3.10.41-longterm #1 SMP Sat Jun 7 17:01:06 CEST 2014 armv7l armv7l armv7l GNU/Linux But you cannot use chroot : # chroot /data/build/armhf /bin/bash /bin/bash: /bin/bash: cannot execute binary file This is because the qemu-binfmt helper script define the flag 'P' (last position) : if [ $cpu != "arm" -a -x "/usr/bin/qemu-armeb" ] ; then echo ':armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:P' > /proc/sys/fs/binfmt_misc/register fi That flag is documented by the kernel (https://www.kernel.org/doc/Documentation/binfmt_misc.txt) as : - 'flags' is an optional field that controls several aspects of the invocation of the interpreter. It is a string of capital letters, each controls a certain aspect. The following flags are supported - 'P' - preserve-argv[0]. Legacy behavior of binfmt_misc is to overwrite the original argv[0] with the full path to the binary. When this flag is included, binfmt_misc will add an argument to the argument vector for this purpose, thus preserving the original argv[0]. 'O' - open-binary. Legacy behavior of binfmt_misc is to pass the full path of the binary to the interpreter as an argument. When this flag is included, binfmt_misc will open the file for reading and pass its descriptor as an argument, instead of the full path, thus allowing the interpreter to execute non-readable binaries. This feature should be used with care - the interpreter has to be trusted not to emit the contents of the non-readable binary. 'C' - credentials. Currently, the behavior of binfmt_misc is to calculate the credentials and security token of the new process according to the interpreter. When this flag is included, these attributes are calculated according to the binary. It also implies the 'O' flag. This feature should be used with care as the interpreter will run with root permissions when a setuid binary owned by root is run with binfmt_misc. When using flag 'OC', everything is then correct. This is the way Debian and Ubuntu use it. # /usr/bin/qemu-arm -L /data/build/armhf /data/build/armhf/bin/uname -a Linux otis.scabb 3.10.41-longterm #1 SMP Sat Jun 7 17:01:06 CEST 2014 armv7l armv7l armv7l GNU/Linux # chroot /data/build/armhf /bin/bash root@localhost:/# That 'P' flag is present to all arch define in init script Should we change this ? Reproducible: Always
@jer, ARM is just an example here, it's the same for every arch
I too am running into this issue, and it's killing me because qemu-user was removed from the tree. To replicate: echo 'QEMU_USER_TARGETS="aarch64 arm"' >> /etc/portage/make.conf echo "app-emulation/qemu static-user" >> /etc/portage/package.use/qemu emerge debootstrap qemu /etc/init.d/qemu-binfmt start cd tmp debootstrap --verbose --arch armel --variant=minbase --foreign squeeze rootfs/ http://ftp.debian.org/debian mkdir -p rootfs/dev/pts mount -t devpts devpts rootfs/dev/pts mount -t proc proc rootfs/proc cp /usr/bin/qemu-arm rootfs/usr/bin chroot rootfs will then fail with /bin/bash: /bin/bash: cannot execute binary file So something is missing, and qemu with static-user is NOT a drop-in replacement for qemu-user as was claimed in the mask message. Completely aside from this, could we also maybe move towards the same as other distributions and name the file qemu-arm-static to differentiate?
And, I'm unsure why exactly, but when switching to OC as Bertrand suggests, I can no longer launch new processes on my machines, so OC doesn't seem to be 100% correct either.
Check my patch to qemu-user in bug: https://bugs.gentoo.org/show_bug.cgi?id=516454 Should sort out this, just don't forget to change binfmt flag from P to PO too.
I can confirm that this bug still exists. I was trying to chroot into an arm environment after following http://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot and http://www.gentoo.org/proj/en/base/embedded/handbook/?part=1&chap=5 I would get the same error, and modifying the /etc/init.d/qemu-binfmt script to use OC instead of P fixed it for me.
(In reply to Nathan Shearer from comment #5) > I can confirm that this bug still exists. I was trying to chroot into an arm > environment after following > http://wiki.gentoo.org/wiki/Crossdev_qemu-static-user-chroot and > http://www.gentoo.org/proj/en/base/embedded/handbook/?part=1&chap=5 > > I would get the same error, and modifying the /etc/init.d/qemu-binfmt script > to use OC instead of P fixed it for me. Removing P is a fragile fix. Some apps require argv0 to be diffrent. You are better off using the patch in bug 516454 and fixing the binfmt flag to be PO instead.
there isn't any info in this bug that covers the lower level problems, so i'll summarize them here binfmt_misc passes the full resolved path of the emulated program. this way the interp (qemu-xxx) can open & run it. the problem is that the original argv[0] value is lost, so when you run something like `ls`, the ls program should be given argv[0]="ls", but instead it receives argv[0]="/bin/ls". when the P flag is enabled, the kernel will pass along the original argv[0]. this way the interp can open the file it needs to, and then before calling the program, swap back in the original argv[0]. that means the qemu interp receives both "/bin/ls" and "ls". unfortunately, qemu doesn't know about this additional thing, so it passes along both to the target program leading to confusing output. when you run `ls`, you get back: $ ls /bin/ls: cannot access ls: No such file or directory that's because it behaves as if you ran it like so: $ /bin/ls ls bug 516454 is not a blocker as it affects few programs and is not a new issue. we can resolve that w/out making things work now. the O flag is not needed normally, but it does make execution work when it might otherwise not (per the documentation wrt unreadable files by the user). the security issues i don't think matter in these cases as the loader is trusted (we emerged it after all). the C flag falls into the same bucket. if you're trusting qemu to chroot somewhere, then you're also trusting the capabilities in there.
should be all set now in the tree; thanks for the report! Commit message: Add aarch64 to the init script, and switch flags from P to OC until qemu itself can understand the extra argv[0] http://sources.gentoo.org/app-emulation/qemu/files/qemu-binfmt.initd-r1?r1=1.3&r2=1.4 http://sources.gentoo.org/app-emulation/qemu/qemu-2.1.0.ebuild?rev=1.1