Bug 396275 – www-client/firefox 9.0 and xpcshell on grsec/pax kernel run into infinite loop doing mmap/munmap, need paxctl -r

• Product: Gentoo Linux
• Status: RESOLVED FIXED

[Description Klaus Kusche 2011-12-28 09:42:10 UTC]

I just tried to emerge firefox 9.0 on an amd64 system with a grsec/pax kernel
with execute protection and mmap randomization enabled (but no hardened
userland and no PIE executables).

Emerge runs into a 100 % CPU infinite loop in the GENERATE_CACHE command
of packager.mk, just after successfully creating the omni.jar zip.

top shows xpcshell eating all the CPU, and attaching an strace to it shows that
xpcshell is performing the same two mmap/munmap infinitely:
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a0e3afe000
munmap(0x2a0e3afe000, 2097152)          = 0
mmap(0x2a0e3b00000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a0e3bfe000
munmap(0x2a0e3bfe000, 1048576)          = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a0e3afe000
munmap(0x2a0e3afe000, 2097152)          = 0
mmap(0x2a0e3b00000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a0e3bfe000
munmap(0x2a0e3bfe000, 1048576)          = 0
mmap(NULL, 2097152, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a0e3afe000
munmap(0x2a0e3afe000, 2097152)          = 0
mmap(0x2a0e3b00000, 1048576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a0e3bfe000
munmap(0x2a0e3bfe000, 1048576)          = 0

Setting paxctl -r on xpcshell solves the problem,
but is only a quick hack, not a solution.

I guess that xpcshell assumes that giving an address to mmap will result
in the map being placed at that address (the second mmap explicitely requests
address 0x2a0e3b00000). However, this assumption is fundamentally wrong:
By definition, the address given to mmap is only a hint.
The kernel may or may not obey that hint, it is allowed to use any other address
(and it always uses an address differing from the one requested when
mmap randomization is in effect).

So, if upsteam code makes any such assumptions about mmap addresses,
it is broken and needs to be fixed.

[Comment 1 Klaus Kusche 2011-12-28 10:36:48 UTC]

The problem does not only affect xpcshell during install.

It also affects firefox itself:
Firefox 9.0 loops infinitely when started (before showing any window),
paxctl -r on the firefox binary solves the problem.

[Comment 2 PaX Team 2011-12-31 13:03:34 UTC]

Created attachment 297447 [details, diff]
possible fix for ff9/aslr

zakalwe, one of our users, posted the attached patch on irc the other day, it's the adaptation of my fix for jemalloc from a few years ago (yes, looks like bad code spreads by copy-pasting). i didn't try it myself yet but if it works for others, please include it in gentoo, upstream will be unlikely to react in a timely manner (the jemalloc/aslr bug is still not fix in ff itself, afaik).

[Comment 3 Klaus Kusche 2011-12-31 21:17:29 UTC]

The proposed patch fixes the problem for me.

Most likely, it needs to be applied to thunderbird 9.x, too?

[Comment 4 Wirt Wolff 2012-01-01 05:17:37 UTC]

Attachment 297447 [details, diff] patch also works for me, with hardened kernel ~amd64.  FWIW firefox USE="alsa custom-optimization dbus ipc libnotify linguas_en system-sqlite webm". Built with hardened profile x86_64-pc-linux-gnu-4.6.2

[Comment 5 Attila Tóth 2012-01-02 16:36:30 UTC]

(In reply to comment #4)
> Attachment 297447 [details, diff] [details, diff] patch also works for me, with hardened kernel ~amd64.  FWIW
> firefox USE="alsa custom-optimization dbus ipc libnotify linguas_en
> system-sqlite webm". Built with hardened profile x86_64-pc-linux-gnu-4.6.2

Am I correct, that this diff must be applied to js/src/jsgcchunk.cpp?

[Comment 6 Klaus Kusche 2012-01-02 16:44:50 UTC]

(In reply to comment #5)
> Am I correct, that this diff must be applied to js/src/jsgcchunk.cpp?

Yes.

[Comment 7 Attila Tóth 2012-01-02 23:40:30 UTC]

(In reply to comment #6)
> (In reply to comment #5)
> > Am I correct, that this diff must be applied to js/src/jsgcchunk.cpp?
> 
> Yes.

Works for me.

[Comment 8 Jory A. Pratt (RETIRED) 2012-01-03 22:56:31 UTC]

hardened team feel free to make the adjustment or I will do so on thursday when I return.

[Comment 9 Nirbheek Chauhan (RETIRED) 2012-01-04 20:46:47 UTC]

Anarchy bumped the patch tarball. This is now fixed in the tree, please sync and re-merge 9.0. Thanks!

[Comment 10 Jaak Ristioja 2012-01-13 02:35:46 UTC]

Same problem with www-client/icecat-9.0.1, I think. Could this bug be reopened for icecat please?

[Comment 11 Anthoine Bourgeois 2012-01-21 01:35:58 UTC]

(In reply to comment #10)
> Same problem with www-client/icecat-9.0.1, I think. Could this bug be reopened
> for icecat please?

I had a similar issue (infinite loop on firefox 9.0 build on xpcshell) but paxctl -r doesn't fix it neither. After a strace on xpcshell I find that the issue is more like #390911. May be you should read https://bugs.gentoo.org/show_bug.cgi?id=390911#c11 and http://forums.gentoo.org/viewtopic-p-6934508.html .