Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 329499

Summary: dev-lang/python-2.6 'rwx' mmap() calls prevent loading of ctypes module (possibly others) under new PaX kernels
Product: Gentoo Linux Reporter: Radoslaw Madej (radegand) <radegand>
Component: [OLD] DevelopmentAssignee: Python Gentoo Team <python>
Status: RESOLVED FIXED    
Severity: normal CC: andybaker.brfc, azamat.hackimov, capsel, csingley, dschridde+gentoobugs, hardened-kernel+disabled, hardened, hppa, jaak, jah, kanelxake, kfm, lejonet, nikoli, pageexec, pavel.labushev, subscryer, tamiko
Priority: High Keywords: InVCS
Version: unspecified   
Hardware: All   
OS: Linux   
URL: http://bugs.python.org/issue5504
See Also: https://bugs.gentoo.org/show_bug.cgi?id=559502
Whiteboard: 2.6.5_p20100801, 2.7_p20100801, 3.1.2_p20100801, 3.2_pre20100801
Package list:
Runtime testing required: ---
Bug Depends on: 334425, 436626    
Bug Blocks: 339066    
Attachments: proposed patch
emerge --info
Patch for 2.6.5-r3
Patch for 2.7
Patch for 3.1.2-r4
strace output from twistd
python part of the fix (2.7.2)
libffi part of the fix (3.0.10)
kernel EMUTRAMP patch (tested on 3.1.8-grsec-201201062207)
ctypes test results
python3 patch
simple conditional libffi fix
simple conditional python fix

Description Radoslaw Madej (radegand) 2010-07-22 21:54:07 UTC
Due to the recent change in PaX behaviour, which does not allow for a 'rwx' mappings to be created (even if the can be reduced to 'r-x' only by PaX, which was previously accepted) it is not possible to load the ctypes module. This has been confirmed for hardened-sources-2.6.32-r11 as well as hand-patched kernel 2.6.34.1 with grsecurity-2.2.0-2.6.34.1-201007162107.patch

Steps to reproduce:
1. emerge and boot hardened-sources-2.6.32-r11 (or use hand patched 2.6.34.1 as described above)
2. make sure using paxctl that mprotect is not disabled for python
3. run: python -c "import modules"

Results:
# python -c "import ctypes"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
    CFUNCTYPE(c_int)(lambda: None)
MemoryError

Relevant strace snippet: 

open("/usr/lib64/python2.6/ctypes/_endian.pyc", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=2286, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x31b9d421000
read(5, "\321\362\r\n3r+Lc\0\0\0\0\0\0\0\0\5\0\0\0@\0\0\0s\307\0\0\0d\0"..., 4096) = 2286
fstat(5, {st_mode=S_IFREG|0644, st_size=2286, ...}) = 0
read(5, "", 4096)                       = 0
close(5)                                = 0
munmap(0x31b9d421000, 4096)             = 0
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x31b9d2b5000
close(4)                                = 0
mmap(NULL, 4080, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = -1 EPERM (Operation not permitted)
close(3)                                = 0
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):
) = 35

Expected result:
successful import of the ctypes module.

Additional info:
In Python source under Modules/_ctypes/malloc_closure.c there's a following code snippet:

#ifdef MS_WIN32
        item = (ITEM *)VirtualAlloc(NULL,
                                               count * sizeof(ITEM),
                                               MEM_COMMIT,
                                               PAGE_EXECUTE_READWRITE);
        if (item == NULL)
                return;
#else
        item = (ITEM *)mmap(NULL,
                            count * sizeof(ITEM),
                            PROT_READ | PROT_WRITE | PROT_EXEC,
                            MAP_PRIVATE | MAP_ANONYMOUS,
                            -1,
                            0);
        if (item == (void *)MAP_FAILED)
                return;
#endif

Losing the 'PROT_EXEC' seems to solve the issue which has also been confirmed by running the ctypes tests. 

strace output for the patched version:
...
fstat(4, {st_mode=S_IFREG|0644, st_size=2041, ...}) = 0
open("/usr/lib64/python2.6/ctypes/_endian.pyc", O_RDONLY) = 5
fstat(5, {st_mode=S_IFREG|0644, st_size=2286, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x26fa4a7f000
read(5, "\321\362\r\nz\275HLc\0\0\0\0\0\0\0\0\5\0\0\0@\0\0\0s\307\0\0\0d\0"..., 4096) = 2286
fstat(5, {st_mode=S_IFREG|0644, st_size=2286, ...}) = 0
read(5, "", 4096)                       = 0
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x26fa4913000
close(5)                                = 0
munmap(0x26fa4a7f000, 4096)             = 0
close(4)                                = 0
mmap(NULL, 4080, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x26fa4a7f000
close(3)                                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x26fa42a8110}, {0x26fa45c5975, [], SA_RESTORER, 0x26fa42a8110}, 8) = 0
munmap(0x26fa4913000, 266240)           = 0
exit_group(0)
Comment 1 Radoslaw Madej (radegand) 2010-07-22 21:55:13 UTC
Created attachment 239871 [details, diff]
proposed patch
Comment 2 Radoslaw Madej (radegand) 2010-07-22 21:57:26 UTC
Created attachment 239875 [details]
emerge --info
Comment 3 Radoslaw Madej (radegand) 2010-07-22 22:09:02 UTC
Sorry, I made few mistakes:

- mappings were obviously reduced to 'rw-' not 'r-x'

- steps to reproduce: point 3, should be: python -c "import ctypes"

this affects python 2.6 and 3.1 (2.6.5-r3 and 3.1.2-r3). The proposed patch has only been tested with 2.6
Comment 4 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-22 22:41:23 UTC
We won't include this experimental patch until upstream approves it.
Comment 5 PaX Team 2010-07-24 22:37:55 UTC
(In reply to comment #4)
> We won't include this experimental patch until upstream approves it.

why? it fixes a bug and is needed for hardened users right now. if you're unable to verify its correctness (a simple removal of PROT_EXEC) then maybe you shouldn't be sticking your nose into things you don't understand.
Comment 6 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-24 22:53:34 UTC
Report the patch to upstream [1] instead of complaining.

[1] http://bugs.python.org/
Comment 7 Anthony Basile gentoo-dev 2010-07-24 23:43:39 UTC
(In reply to comment #4)
> We won't include this experimental patch until upstream approves it.
> 

From what I recall, that code consistently (ab)used PROT_EXEC where unnecessary and potentially dangerous.  I would put some pressure on upstream to explain why all their mmap()'s use it.
Comment 8 solar (RETIRED) gentoo-dev 2010-07-25 16:45:45 UTC
(In reply to comment #4)
> We won't include this experimental patch until upstream approves it.

If you don't we will. python has to be patched one way or another.

Comment 9 solar (RETIRED) gentoo-dev 2010-07-25 17:18:16 UTC
This has been fixed in the tree 2.6.x, 2.7.x, 3.x. I used a sed statement vs patching so it gets carried onto version bumps w/o being lost. 

Radoslaw Madej please also file a bug with the upstream python team.
Comment 10 Radoslaw Madej (radegand) 2010-07-26 17:08:46 UTC
Upstream bug logged:
http://bugs.python.org/issue9385
Comment 11 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-31 15:26:53 UTC
Upstream has better patch, which will be applied.
Comment 12 Guy Martin (RETIRED) gentoo-dev 2010-07-31 16:10:30 UTC
The current fix (sed in the ebuild) makes test_ctypes segv on hppa using vanilla kernel, gcc 4.2.4 or 4.4.4-r1 and python 2.6.5-r2.
Comment 13 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-31 22:08:35 UTC
The usage of sed in the ebuilds has been reverted.

HPPA maintainers:
Please test the patches from ctypes maintainer. If the patches work correctly, then I will include them in the new patchsets for new revisions of dev-lang/python.
Comment 14 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-31 22:10:35 UTC
Created attachment 240887 [details, diff]
Patch for 2.6.5-r3
Comment 15 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-31 22:12:11 UTC
Created attachment 240889 [details, diff]
Patch for 2.7
Comment 16 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-31 22:13:16 UTC
Created attachment 240891 [details, diff]
Patch for 3.1.2-r4
Comment 17 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-07-31 22:18:37 UTC
Alpha / HPPA maintainers:
Has the reversion of wrong changes fixed bug #330323 for you?

Alpha maintainers:
Please check if these patches work for you.
Comment 18 Tobias Klausmann (RETIRED) gentoo-dev 2010-08-01 14:08:22 UTC
I've now tested the most recent versions of Python as mentioned in bugs 14-16, with and without the patches of said comments. They all pass the ctypes test now.
Comment 19 Guy Martin (RETIRED) gentoo-dev 2010-08-01 16:15:23 UTC
The test_ctypes passes fine as well on hppa with the patch for 2.6.5-r3.
Comment 20 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-08-02 15:07:21 UTC
Fixed in r365.
Comment 21 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-08-02 19:29:24 UTC
Fixed in 2.6.5_p20100801, 2.7_p20100801, 3.1.2_p20100801 and 3.2_pre20100801.
Comment 22 kfm 2010-08-04 16:40:57 UTC
(In reply to comment #21)
> Fixed in 2.6.5_p20100801, 2.7_p20100801, 3.1.2_p20100801 and 3.2_pre20100801.

Afrever, as these versions had to be dropped - for unrelated reasons as I understand - would there be any chance of getting the patch folded in for a simple revision bump of the existing instances? Naturally, 2.6.5 is of particular importance, being the current stable version.
Comment 23 Brian Harring (RETIRED) gentoo-dev 2010-08-19 08:41:35 UTC
(In reply to comment #22)
> (In reply to comment #21)
> > Fixed in 2.6.5_p20100801, 2.7_p20100801, 3.1.2_p20100801 and 3.2_pre20100801.
> 
> Afrever, as these versions had to be dropped - for unrelated reasons as I
> understand - would there be any chance of getting the patch folded in for a
> simple revision bump of the existing instances? Naturally, 2.6.5 is of
> particular importance, being the current stable version.
Yes, it'll be in.
Comment 24 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-09-04 16:36:26 UTC
*** Bug 332419 has been marked as a duplicate of this bug. ***
Comment 25 Daniel Kuehn (RETIRED) gentoo-dev 2010-09-04 22:00:27 UTC
I have tested the 2.6.5 patch on AMD64 gentoo hardened.
uname -r
2.6.34-hardened-r1-UL30VT
and it works like a charm, strace confirms that it doesnt attempt any RWX callings, just RX or RW.
Comment 26 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-09-10 19:20:08 UTC
*** Bug 336722 has been marked as a duplicate of this bug. ***
Comment 27 kfm 2010-09-24 15:03:18 UTC
Not to sound like a stuck record, but will the patch be committed any time soon? I realise that it serves a limited purpose for those not using Hardened Gentoo but I, for one, would greatly appreciate a 2.6.5-r4 bump to this end. Are there other bugs that need to be addressed for a new revision to be issued?
Comment 28 Mattias Merilai 2010-09-27 13:49:02 UTC
(In reply to comment #27)
> Not to sound like a stuck record, but will the patch be committed any time
> soon? I realise that it serves a limited purpose for those not using Hardened
> Gentoo but I, for one, would greatly appreciate a 2.6.5-r4 bump to this end.
> Are there other bugs that need to be addressed for a new revision to be issued?
> 

+1, needed for iotop
Comment 29 Magnus Granberg gentoo-dev 2010-09-29 17:58:13 UTC
*** Bug 339152 has been marked as a duplicate of this bug. ***
Comment 30 Andy Baker 2010-10-03 21:01:32 UTC
+1. Currently cannot use coherence. Will appreciated patched source.
Comment 31 blub bla 2010-10-05 15:41:18 UTC
(In reply to comment #28)
> (In reply to comment #27)
> > Not to sound like a stuck record, but will the patch be committed any time
> > soon? I realise that it serves a limited purpose for those not using Hardened
> > Gentoo but I, for one, would greatly appreciate a 2.6.5-r4 bump to this end.
> > Are there other bugs that need to be addressed for a new revision to be issued?
> > 
> 
> +1, needed for iotop
> 

+1 issue exists on 
2.6.35-r1
as well...

Comment 32 Anthony Basile gentoo-dev 2010-10-05 18:42:27 UTC
(In reply to comment #31)
> (In reply to comment #28)
> > (In reply to comment #27)
> > > Not to sound like a stuck record, but will the patch be committed any time
> > > soon? I realise that it serves a limited purpose for those not using Hardened
> > > Gentoo but I, for one, would greatly appreciate a 2.6.5-r4 bump to this end.
> > > Are there other bugs that need to be addressed for a new revision to be issued?
> > > 
> > 
> > +1, needed for iotop
> > 
> 
> +1 issue exists on 
> 2.6.35-r1
> as well...
> 

This issue will continue on all hardened-sources from now on because of the improved restrictions on mmap in grsec-2.2.0.  The last hardened-sources where this is not an issue is 2.6.32-r9 which was based on the last grsec-2.1.14 patch.  However, since certain configurations of that kernel are exploitable on amd64 (see bug #338273), this bug has become critical.

I know some users that are now resorting to overlays to get past this.  Been there done that --- not good from QA point of view etc.

So ... adding to the above +1's.
Comment 33 Rick Farina (Zero_Chaos) gentoo-dev 2010-10-08 04:09:11 UTC
gnuplot-py-1.8 is marked stable but can't build because of this bug.

I tested the 2.6 patch and it fixed the issue.  I put it in my overlay so other people can have it but as several people pointed out, this is a horrible solution.

Please don't let this desperately needed and confirmed working fix rot in bug tracker.

Thanks
Comment 34 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2010-10-08 05:13:40 UTC
python herd (other than Arfrever):
can we please get this included ASAP?

With the stabilization of new kernels for the recent IA32 exploits, it's starting to bite everybody.

If there no response by the end of Oct 9th, I'll look at bumping them myself, because this will break Python for infra as well.
Comment 35 kfm 2010-10-08 08:11:48 UTC
Thanks for your intervention, Robin.
Comment 36 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-10-11 09:47:34 UTC
python-2.6.6, including a patch to fix this issue, has been released in unstable. I presume that means this bug can be closed, though feel free to reopen if I've missed something.
Comment 37 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2010-10-11 17:57:29 UTC
djc: so we're fast-tracking 2.6.6 to stable for this?
Comment 38 Magnus Granberg gentoo-dev 2010-10-11 17:59:18 UTC
jasmin ~ # python -c "import ctypes"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib64/python2.6/ctypes/__init__.py", line 546, in <module>
    CFUNCTYPE(c_int)(lambda: None)
MemoryError
jasmin ~ #
jasmin ~ # eselect python list
Available Python interpreters:
  [1]   python2.6 *
  [2]   python3.1
jasmin ~ #
[ebuild   R   ] dev-lang/python-2.6.6  USE="berkdb gdbm ncurses readline ssl threads (wide-unicode) xml -build -doc -examples -ipv6 -sqlite -tk -wininst" 0 kB
Comment 39 kfm 2010-10-11 18:11:38 UTC
python-2.6.6 doesn't include the patch ...
Comment 40 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-10-12 20:42:37 UTC
Yeah, I failed there. Please give 2.6.6-r1 a shot.
Comment 41 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-10-12 20:43:08 UTC
Robin, I'd be happy to fast-track 2.6.6 to stable for this.
Comment 42 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-10-12 20:46:39 UTC
Sorry to be spammy: the reason I wanted 2.6.6 instead of updating 2.6.5 is that 2.6.5 isn't actually upstream 2.6.5, it has a huge amount of backports, and it turned out people (including me, btw) didn't really like that way of doing it, so I figured that we should get a quick 2.6.6 instead of another broken 2.6.5 revision with this patch included.
Comment 43 Magnus Granberg gentoo-dev 2010-10-12 21:42:09 UTC
The -r1 works for me  thank you for fixing it.
Comment 44 Anthony Basile gentoo-dev 2010-10-13 20:17:09 UTC
(In reply to comment #43)
> The -r1 works for me  thank you for fixing it.
> 

Same here.  Thanks.
Comment 45 Pavel Labushev 2010-10-23 22:06:40 UTC
I encountered another rwx mapping with 2.6.6-r1 inside read-only chroot on a Hardened system:

It Oct 24 05:57:44 [kernel] grsec: denied RWX mmap of <anonymous mapping> by /tmp/readonly_chroot/usr/bin/python2.6[python2.6:31335] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:20184] uid/euid:0/0 gid/egid:0/0
Oct 24 05:57:44 [kernel] grsec: Segmentation fault occurred at 7465725f in /tmp/readonly_chroot/usr/bin/python2.6[python2.6:31335] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:20184] uid/euid:0/0 gid/egid:0/0



And another issue is due to TPE restrictions when python tries to map executable some untrusted temporary .so file generated by libffi:

testing@worm3 ~ $ id
uid=1219(testing) gid=1221(testing) groups=1221(testing)
testing@worm3 ~ $ python -c "import ctypes"
Segmentation fault

Oct 24 06:01:48 [kernel] grsec: denied untrusted exec of /home/testing/ffiCigB2E by /usr/bin/python2.6[python2.6:31420] uid/euid:1219/1219 gid/egid:1221/1221, parent /bin/bash[bash:31406] uid/euid:1219/1219 gid/egid:1221/1221
Oct 24 06:01:48 [kernel] grsec: Segmentation fault occurred at 7465725f in /usr/bin/python2.6[python2.6:31420] uid/euid:1219/1219 gid/egid:1221/1221, parent /bin/bash[bash:31406] uid/euid:1219/1219 gid/egid:1221/1221

Workaround is to add user testing to trusted TPE group:

$ id
uid=1219(testing) gid=1221(testing) groups=1221(testing),500(tpe)
$ python -c "import ctypes"
$ 
Comment 46 Pavel Labushev 2010-10-23 22:08:38 UTC
(In reply to comment #45)
> I encountered another rwx mapping with 2.6.6-r1 inside read-only chroot on a
> Hardened system:

To reproduce run python -c "import python" inside read-only chroot.
Comment 47 Pavel Labushev 2010-10-23 22:21:09 UTC
> To reproduce run python -c "import python" inside read-only chroot.

Sorry, it's python -c "import ctypes"

Comment 48 Pavel Labushev 2010-10-23 22:46:52 UTC
python-3.2_pre20101010, unlike 2.6.6-r1, doesn't do any rwx mappings inside read-only chroot, but just segfaults instead:

# python3.2 -c "import ctypes"
Segmentation fault

Oct 24 06:33:09 [kernel] grsec: Segmentation fault occurred at 00000074 in /home/root/python3.2_readonly_chroot/usr/bin/python3.2[python3.2:31705] uid/euid:0/0 gid/egid:0/0, parent /home/root/python3.2_readonly_chroot/bin/bash[bash:31192] uid/euid:0/0 gid/egid:0/0


The TPE issue is the same for 3.2_pre20101010 as for 2.6.6-r1:

$ id
uid=1219(testing) gid=1221(testing) groups=1221(testing)
$ python3.2 -c "import ctypes"
Segmentation fault

Oct 24 06:41:12 [kernel] grsec: denied untrusted exec of /home/root/python3.2_chroot/home/testing/ffixA4YwV by /home/root/python3.2_chroot/usr/bin/python3.2[python3.2:31801] uid/euid:1219/1219 gid/egid:1221/1221, parent /home/root/python3.2_chroot/bin/bash[bash:31790] uid/euid:1219/1219 gid/egid:1221/1221
Oct 24 06:41:12 [kernel] grsec: Segmentation fault occurred at 00000074 in /home/root/python3.2_chroot/usr/bin/python3.2[python3.2:31801] uid/euid:1219/1219 gid/egid:1221/1221, parent /home/root/python3.2_chroot/bin/bash[bash:31790] uid/euid:1219/1219 gid/egid:1221/1221

And with the same workaround:

$ id
uid=1219(testing) gid=1221(testing) groups=1221(testing),500(tpe)
$ python3.2 -c "import ctypes"
$ 
Comment 49 Dirkjan Ochtman (RETIRED) gentoo-dev 2010-10-27 13:12:19 UTC
2.6.6-r1 stabilization requested in bug 342927.
Comment 50 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-11-15 12:32:03 UTC
*** Bug 345591 has been marked as a duplicate of this bug. ***
Comment 51 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-11-28 19:03:47 UTC
This bug was fixed long time ago.
Comment 52 Dennis Schridde 2010-11-28 19:25:04 UTC
(In reply to comment #51)
> This bug was fixed long time ago.
Are you sure?

# python -c "import ctypes"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.6/ctypes/__init__.py", line 546, in <module>
    CFUNCTYPE(c_int)(lambda: None)
MemoryError

# python --version
Python 2.6.5

[I] dev-lang/python (2.6.5-r3(2.6)@14/10/10 3.1.2-r4(3.1)@14/10/10)

The installed version 2.6.5 is the latest stable.

Requesting REOPEN.
Comment 53 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2010-11-28 19:27:17 UTC
Bugs should be closed after they have been fixed in the newest (potentially not yet stable) versions.
Comment 54 kfm 2010-11-29 05:31:06 UTC
> Bugs should be closed after they have been fixed in the newest (potentially not
> yet stable) versions.

Well, it depends. I don't mean to sound ungrateful for your help in finding the correct solution (I'm aware of the effort that you put into python maintenance at large) but the trouble is this: from the hardened user's perspective, this bug has now been in existence for over four months and there is still no solution in the stable branch. The purpose of distinguishing between stable and testing is, quite naturally, to provide a reasonable assurance to any user that chooses the stable branch that what they install will actually be as stable and bug-free as is possible.

By closing this bug as it stands, the implication is that affected users should have to upgrade to an ~arch package in order to resolve the issue, which I find odd because inserting the patch into the stable branch in this case would have been trivial. Granted, I can see that a stable request exists for python-2.6.6 but there is no timeframe (will it go stable in a week, a month or longer?). 

That is why I asked back in Comment #22 whether we could simply fold in the patch to the *current* stable branch, revision bump and be done with it. The bug could have been closed at the time under the assumption that the patch would be retained in future ebuilds for as long as upstream haven't fixed it and then we could have stopped nagging about it.

I upgraded to python-2.6.6 as soon as I knew it was carrying the patch but my point is that not everyone should have to if they would rather stick to the stable branch. It's not as though it needs testing either; we know the patch fixes the problem.
Comment 55 Dennis Schridde 2010-11-29 09:04:31 UTC
I switched to ~ia64 for dev-lang/python-2.6.6, but the issue seems not to be fixed:
# python --version
Python 2.6.6
# python -c "import ctypes"
<no output, good so far>
# /etc/init.d/pyicq-t restart
 * Starting ICQ Jabber Transport ...
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/pyicq-t/pyicq-t.py", line 11, in <module>
    import main
  File "/usr/lib/python2.6/site-packages/pyicq-t/src/main.py", line 129, in <module>
    from twisted.words.protocols.jabber import component
  File "/usr/lib/python2.6/site-packages/twisted/words/protocols/jabber/component.py", line 23, in <module>
    from twisted.application import service
  File "/usr/lib/python2.6/site-packages/twisted/application/service.py", line 23, in <module>
    from twisted.plugin import IPlugin
  File "/usr/lib/python2.6/site-packages/twisted/plugin.py", line 34, in <module>
    from twisted.python.modules import getModule
  File "/usr/lib/python2.6/site-packages/twisted/python/modules.py", line 68, in <module>
    from twisted.python.filepath import FilePath, UnlistableError
  File "/usr/lib/python2.6/site-packages/twisted/python/filepath.py", line 29, in <module>
    from twisted.python.win32 import ERROR_FILE_NOT_FOUND, ERROR_PATH_NOT_FOUND
  File "/usr/lib/python2.6/site-packages/twisted/python/win32.py", line 168, in <module>
    formatError = _ErrorFormatter.fromEnvironment().formatError
  File "/usr/lib/python2.6/site-packages/twisted/python/win32.py", line 128, in fromEnvironment
    from ctypes import WinError
  File "/usr/lib/python2.6/ctypes/__init__.py", line 546, in <module>
    CFUNCTYPE(c_int)(lambda: None)
MemoryError
 * start-stop-daemon: failed to start `/usr/bin/python'                                                                                                                                                       [ !! ]
 * ERROR: pyicq-t failed to start

I do not understand what is happening here, but it seems that there are some conditions under which it is still not possible to import ctypes.

Again I request to reopen this bug.

[I] dev-lang/python (2.6.6-r1(2.6)@28/11/10 3.1.2-r4(3.1)@28/11/10): Python is an interpreted, interactive, object-oriented programming language.
[I] net-im/pyicq-t (0.8.1.5@29/11/10): Python based jabber transport for ICQ


A similar error happens when starting pymsn-t:
# /etc/init.d/pymsn-t restart
 * Starting MSN Jabber Transport ...
ImportError! You probably forgot to install Twisted Words or Web. Have a look at the docs. You may also be using an unsupported version of Twisted.
 * start-stop-daemon: failed to start `/usr/bin/python'                                                                                                                                                       [ !! ]
 * ERROR: pymsn-t failed to start

It is at 0.11.3-r2 + more patches from upstream:
[I] net-im/pymsn-t (0.11.3_p1[1]@29/11/10): Python based jabber transport for MSN
Comment 56 kfm 2010-11-30 07:10:02 UTC
Re Comment 55 - I'm unable to reproduce this on amd64 with:

  dev-lang/python-2.6.6-r1
  dev-python/twisted-web-10.1.0
  dev-python/twisted-words-10.1.0
  dev-python/imaging-1.1.7
  dev-python/pycrypto-2.3
  dev-python/pyopenssl-0.10-r1
  net-im/pymsn-t-0.11.3-r2

Perhaps the issue concerns ia64 specifically.



Comment 57 PaX Team 2011-01-08 16:25:07 UTC
(In reply to comment #55)
> I switched to ~ia64 for dev-lang/python-2.6.6, but the issue seems not to be
> fixed:

can you "strace -f" these failing apps and provide the logs here please?
Comment 58 Dennis Schridde 2011-04-29 22:33:33 UTC
(In reply to comment #57)
> (In reply to comment #55)
> > I switched to ~ia64 for dev-lang/python-2.6.6, but the issue seems not to be
> > fixed:
> 
> can you "strace -f" these failing apps and provide the logs here please?

Seems fixed with sys-kernel/hardened-sources-2.6.32-r42 and dev-lang/python-2.7.1-r1.
Comment 59 Burcin Erocal 2011-10-03 12:00:57 UTC
Created attachment 288671 [details]
strace output from twistd

I can reproduce this problem on amd64 while starting twistd for a buildbot master. The strace output indicates that it tries to create a temporary file and mmap() it with PROT_EXEC in /tmp, /var/tmp and /dev/shm. All of these are mounted with the noexec flag on my system. I can start the server after removing this flag from one of these mount points. 


7885  open("/tmp/ffiS3iyaG", O_RDWR|O_CREAT|O_EXCL, 0600) = 13
7885  unlink("/tmp/ffiS3iyaG")          = 0
7885  ftruncate(13, 4096)               = 0
7885  mmap(NULL, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 13, 0) = -1 EPERM (Operation not permitted)
7885  close(13)                         = 0
7885  open("/var/tmp/ffi5ASZqt", O_RDWR|O_CREAT|O_EXCL, 0600) = 13
7885  unlink("/var/tmp/ffi5ASZqt")      = 0
7885  ftruncate(13, 4096)               = 0
7885  mmap(NULL, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 13, 0) = -1 EPERM (Operation not permitted)
7885  close(13)                         = 0
7885  open("/dev/shm/ffiANqtHg", O_RDWR|O_CREAT|O_EXCL, 0600) = 13
7885  unlink("/dev/shm/ffiANqtHg")      = 0
7885  ftruncate(13, 4096)               = 0
7885  mmap(NULL, 4096, PROT_READ|PROT_EXEC, MAP_SHARED, 13, 0) = -1 EPERM (Operation not permitted)
7885  close(13)                         = 0
Comment 60 Agostino Sarubbo gentoo-dev 2011-10-17 13:45:15 UTC
Re-opened. I see in my grsec.log:

Oct 14 04:58:37 devil kernel: grsec: denied RWX mmap of <anonymous mapping> by /usr/bin/python2.7[python2.7:18717] uid/euid:250/250 gid/egid:250/250, parent /usr/bin/python2.7[python2.7:18716] uid/euid:250/250 gid/egid:250/250

Feel free to ask other info if are needed.
Comment 61 Anthony Basile gentoo-dev 2011-10-17 18:13:54 UTC
(In reply to comment #60)
> Re-opened. I see in my grsec.log:
> 
> Oct 14 04:58:37 devil kernel: grsec: denied RWX mmap of <anonymous mapping> by
> /usr/bin/python2.7[python2.7:18717] uid/euid:250/250 gid/egid:250/250, parent
> /usr/bin/python2.7[python2.7:18716] uid/euid:250/250 gid/egid:250/250
> 
> Feel free to ask other info if are needed.

I have seen this problem "randomly" pop up on hardened systems and I don't know why.  It is not fixable by the original patch so it is not the same bug, but it is an rwx mmap killed by pax and it does come from "import ctypes".

Ago, can you test python -c "import ctypes" and see if it gets killed in comment 1.  We may want to open other bug if we isolate this to some other issue.
Comment 62 PaX Team 2011-10-17 20:16:34 UTC
(In reply to comment #61)
> I have seen this problem "randomly" pop up on hardened systems and I don't know
> why.  It is not fixable by the original patch so it is not the same bug, but it
> is an rwx mmap killed by pax and it does come from "import ctypes".

if the culprit process get a SIGKILL from PaX then all you need to do is enable coredumping (ulimit -c unlimited) in your shell and run it again and you'll get a nice coredump where you can look up the backtrace, etc and figure out which code tried to execute runtime generated code.
Comment 63 Markos Chandras (RETIRED) gentoo-dev 2011-10-19 07:09:11 UTC
*** Bug 369789 has been marked as a duplicate of this bug. ***
Comment 64 Larry 2011-12-27 13:49:23 UTC
We have confirmed this with mod_wsgi and CPython 2.7.2-r3 and a third-party is also having trouble. libffi's closures are to blame for the /tmp/ffiXXXXX RX/RWX mmap behavior. The result is a SIGSEGV in the Apache process in some dubious PyMalloc call which clearly isn't the culprit.

Someone please give Drepper incentives to take some long vacations as far as possible from a networked computer.
Comment 65 Larry 2011-12-27 13:52:40 UTC
(In reply to comment #48)
> python-3.2_pre20101010, unlike 2.6.6-r1, doesn't do any rwx mappings inside
> read-only chroot, but just segfaults instead:
> 
> # python3.2 -c "import ctypes"
> Segmentation fault
> 
> Oct 24 06:33:09 [kernel] grsec: Segmentation fault occurred at 00000074 in
> /home/root/python3.2_readonly_chroot/usr/bin/python3.2[python3.2:31705]
> uid/euid:0/0 gid/egid:0/0, parent
> /home/root/python3.2_readonly_chroot/bin/bash[bash:31192] uid/euid:0/0
> gid/egid:0/0
> 
> 
> The TPE issue is the same for 3.2_pre20101010 as for 2.6.6-r1:
> 
> $ id
> uid=1219(testing) gid=1221(testing) groups=1221(testing)
> $ python3.2 -c "import ctypes"
> Segmentation fault
> 
> Oct 24 06:41:12 [kernel] grsec: denied untrusted exec of
> /home/root/python3.2_chroot/home/testing/ffixA4YwV by
> /home/root/python3.2_chroot/usr/bin/python3.2[python3.2:31801]
> uid/euid:1219/1219 gid/egid:1221/1221, parent
> /home/root/python3.2_chroot/bin/bash[bash:31790] uid/euid:1219/1219
> gid/egid:1221/1221
> Oct 24 06:41:12 [kernel] grsec: Segmentation fault occurred at 00000074 in
> /home/root/python3.2_chroot/usr/bin/python3.2[python3.2:31801]
> uid/euid:1219/1219 gid/egid:1221/1221, parent
> /home/root/python3.2_chroot/bin/bash[bash:31790] uid/euid:1219/1219
> gid/egid:1221/1221
> 
> And with the same workaround:
> 
> $ id
> uid=1219(testing) gid=1221(testing) groups=1221(testing),500(tpe)
> $ python3.2 -c "import ctypes"
> $ 


This is exactly the current situation on amd64 with 2.7.2-r3.
Comment 66 Anthony Basile gentoo-dev 2011-12-27 19:52:06 UTC
(In reply to comment #65)
> (In reply to comment #48)
> > python-3.2_pre20101010, unlike 2.6.6-r1, doesn't do any rwx mappings inside
> > read-only chroot, but just segfaults instead:
> > 
> > # python3.2 -c "import ctypes"
> > Segmentation fault
> > 
> > Oct 24 06:33:09 [kernel] grsec: Segmentation fault occurred at 00000074 in
> > /home/root/python3.2_readonly_chroot/usr/bin/python3.2[python3.2:31705]
> > uid/euid:0/0 gid/egid:0/0, parent
> > /home/root/python3.2_readonly_chroot/bin/bash[bash:31192] uid/euid:0/0
> > gid/egid:0/0
> > 
> > 
> > The TPE issue is the same for 3.2_pre20101010 as for 2.6.6-r1:
> > 
> > $ id
> > uid=1219(testing) gid=1221(testing) groups=1221(testing)
> > $ python3.2 -c "import ctypes"
> > Segmentation fault
> > 
> > Oct 24 06:41:12 [kernel] grsec: denied untrusted exec of
> > /home/root/python3.2_chroot/home/testing/ffixA4YwV by
> > /home/root/python3.2_chroot/usr/bin/python3.2[python3.2:31801]
> > uid/euid:1219/1219 gid/egid:1221/1221, parent
> > /home/root/python3.2_chroot/bin/bash[bash:31790] uid/euid:1219/1219
> > gid/egid:1221/1221
> > Oct 24 06:41:12 [kernel] grsec: Segmentation fault occurred at 00000074 in
> > /home/root/python3.2_chroot/usr/bin/python3.2[python3.2:31801]
> > uid/euid:1219/1219 gid/egid:1221/1221, parent
> > /home/root/python3.2_chroot/bin/bash[bash:31790] uid/euid:1219/1219
> > gid/egid:1221/1221
> > 
> > And with the same workaround:
> > 
> > $ id
> > uid=1219(testing) gid=1221(testing) groups=1221(testing),500(tpe)
> > $ python3.2 -c "import ctypes"
> > $ 
> 
> 
> This is exactly the current situation on amd64 with 2.7.2-r3.

This is the undead bug that just won't die. Try enabling PAX_MPROTECT_COMPAT.  It is less secure but should work.
Comment 67 Pavel Labushev 2011-12-28 10:36:24 UTC
Created attachment 297183 [details, diff]
python part of the fix (2.7.2)
Comment 68 Pavel Labushev 2011-12-28 10:37:18 UTC
Created attachment 297185 [details, diff]
libffi part of the fix (3.0.10)
Comment 69 Arfrever Frehtes Taifersar Arahesis 2011-12-28 22:00:43 UTC
(In reply to comment #67 and comment #68)

Please send patches to upstreams.
Comment 70 Pavel Labushev 2012-01-06 05:15:13 UTC
These patches implement workaround for several basic ctypes use cases, so they won't be accepted by the upstreams. The solution is to disable mprotect restrictions for python executable files by default.
Comment 71 Matt Summers (RETIRED) gentoo-dev 2012-01-09 15:30:28 UTC
(In reply to comment #70)
> These patches implement workaround for several basic ctypes use cases, so they
> won't be accepted by the upstreams. The solution is to disable mprotect
> restrictions for python executable files by default.

The solution is to get upstream to implement mmap'd functions correctly, however they seem to be reluctant for reasons I cannot fathom.

Please use PAX_MPROTECT_COMPAT=y as it will work, albeit with a bit less robustness in protection.
Comment 72 Pavel Labushev 2012-01-09 16:17:26 UTC
(In reply to comment #71)
> The solution is to get upstream to implement mmap'd functions correctly,
> however they seem to be reluctant for reasons I cannot fathom.

Maybe the reason is some parts of the code do need W|X mappings.

> Please use PAX_MPROTECT_COMPAT=y as it will work, albeit with a bit less
> robustness in protection.

The following ctypes test cases will fail regardless:

test_libc
test_functions
test_as_parameter
test_random_things
test_pointers
test_refcounts
test_callbacks
test_simplesubclasses
test_funcptr

PAX_MPROTECT_COMPAT=y and my patches basically do the same thing: make common ctypes use cases not to crash python. Sad, but they are just partial workarounds.

Btw, PAX_MPROTECT_COMPAT=y might deceive some apps into believing they got executable memory and consequentially fail instead of interpreting EACCES/EPERM and act accordingly.
Comment 73 Anthony Basile gentoo-dev 2012-01-09 16:26:43 UTC
(In reply to comment #72)

> PAX_MPROTECT_COMPAT=y and my patches basically do the same thing: make common
> ctypes use cases not to crash python. Sad, but they are just partial
> workarounds.

Let me understand this clearly, are you saying that with your patches you can retain the more restrictive PAX_MPROTECT and still have python work?  Or do you need both PAX_MPROTECT_COMPAT=y and your patches.

Looking over what you did, it looks like the former, but I have not tested.

> Btw, PAX_MPROTECT_COMPAT=y might deceive some apps into believing they got
> executable memory and consequentially fail instead of interpreting EACCES/EPERM
> and act accordingly.

Yeah there are issues in this direction.

I'm not opposed to asking the python folks to add these patches conditional on USE="pax_kernel" but I'm afraid this is going to become a nightmare.  Upstream is showing no signs of moving in a direction where W|X is avoided.  If this gets worse then I'm afraid we may have to move in the direction of pax marking python like we do with other misbehaving interpreters.  I hate the idea but what other options do we have?
Comment 74 Pavel Labushev 2012-01-09 17:33:36 UTC
(In reply to comment #73)
> Let me understand this clearly, are you saying that with your patches you can
> retain the more restrictive PAX_MPROTECT and still have python work?

Yes. Some of the basic ctypes use cases work. Examples:
$ python -c 'import uuid; uuid.uuid4()'
$ python -c 'import ctypes; ctypes.CDLL("libc.so.6")["getuid"]()'

That's with a small overhead of one more syscall per (infrequent) closure allocation when MPROTECT is on and with no such overhead when it's off.

> I'm not opposed to asking the python folks to add these patches conditional on
> USE="pax_kernel" but I'm afraid this is going to become a nightmare.

Even if you decide to supply the workaround, it should be documented very clearly that in some use cases ctypes *will* crash the interpreter. Maybe something like USE="ctypes_pax_workaround" with a clear description will be more appropriate than USE="pax_kernel".

> python like we do with other misbehaving interpreters.  I hate the idea but
> what other options do we have?

MPROTECT doesn't prevent python bytecode injection, so maybe native code injection shouldn't be much of a concern. Guess we should ask pipacs or somebody experienced in exploitation.
Comment 75 PaX Team 2012-01-09 21:12:48 UTC
(In reply to comment #74)
> Even if you decide to supply the workaround, it should be documented very
> clearly that in some use cases ctypes *will* crash the interpreter.

what i'd like to know is why ctypes has to resort to runtime codegen, is that some fundamental need or just a programmer's choice that could be changed?

> MPROTECT doesn't prevent python bytecode injection, so maybe native code
> injection shouldn't be much of a concern. Guess we should ask pipacs or
> somebody experienced in exploitation.

bytecode injection is definitely a possible way of exploiting a python process but at the same time we'd also have to worry about the more mundane ret2libc style attacks as well (as there's no defense against either out there yet).

with all that said, after discussion with Pavel i implemented EMUTRAMP emulation for the libffi trampolines in the latest patches, so just have the ebuild enable it on python and everything should work fine.
Comment 76 Pavel Labushev 2012-01-10 00:51:25 UTC
There are minor issues.

1. Kernel should be compiled with PAX_MPROTECT_COMPAT or python should be compiled with this patch: https://bugs.gentoo.org/attachment.cgi?id=297183 - or else the mmap in Modules/_ctypes/malloc_closure.c:more_core will fail because of MPROTECT and ctypes won't work.

2. To apply grsec TPE restrictions to python users and/or to run python ctypes code in a read-only chroot, one should compile libffi with this patch: https://bugs.gentoo.org/attachment.cgi?id=297185 - or else libffi will try to avoid MPROTECT restrictions by creating a temporary file like /tmp/ffiS3iyaG, mmap it RW and RX and execute trampoline code from the RX mapping (MPROTECT will deny that trick).

The patches make python and libffi fall back to RW mappings when RWX ones fail. This behavior is compatible: one can disable MPROTECT restrictions for python and get vanilla ctypes behavior and performance (the overhead of EMUTRAMP is about 25% here, but YMMV).
Comment 77 Pavel Labushev 2012-01-10 01:01:12 UTC
Created attachment 298441 [details, diff]
kernel EMUTRAMP patch (tested on 3.1.8-grsec-201201062207)
Comment 78 Radoslaw Madej (radegand) 2012-01-13 13:49:40 UTC
The patches also seem to be working as expected on a x64 system:

1) I've tested the patches provided in c76 and c77, applied to python-2.7.2-r3, libffi-3.0.10 and hardened-sources-3.0.7. Grsec config was the gentoo-virtualization profile with emutramp enabled (without TPE though)

2) Ran: 
python2.7 /usr/lib/python2.7/ctypes/test/runtests.py -v

All tests finished fine :)
Comment 79 Radoslaw Madej (radegand) 2012-01-13 13:54:08 UTC
Created attachment 298841 [details]
ctypes test results

Test results attached. Grsec denies RWX mapping but the tests finish fine, from the kern.log:

grsec: denied RWX mmap of <anonymous mapping> by /usr/bin/python2.7[python2.7:9030] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:8048] uid/euid:0/0 gid/egid:0/0
Comment 80 Anthony Basile gentoo-dev 2012-01-14 17:31:52 UTC
Would the python team consider adding Pavel's patches conditional to USE="pax_kernel"?
Comment 81 Pavel Labushev 2012-01-18 00:02:07 UTC
Maybe I should make it clearer. Now, with libffi trampolines support in EMUTRAMP, the proposed patches are not workaround anymore - they are a part of the solution, but only for hardened-sources (because of dependency on EMUTRAMP) and thus cannot be upstreamed. Please, note that the issue is in the stable python and, imho, it should be fixed asap, as it keeps impacting more and more people. 

Please, if you could, don't delay resolution.
Comment 82 Arfrever Frehtes Taifersar Arahesis 2012-01-18 13:52:27 UTC
Firstly, you should discuss changes in dev-libs/libffi with maintainers of dev-libs/libffi (toolchain team).

Secondly, in patch for dev-lang/python please use '#ifdef some_definition_defined_only_on_systems_which_need_this_patch'.
Comment 83 Anthony Basile gentoo-dev 2012-01-19 16:03:13 UTC
(In reply to comment #82)
> Firstly, you should discuss changes in dev-libs/libffi with maintainers of
> dev-libs/libffi (toolchain team).
> 
> Secondly, in patch for dev-lang/python please use '#ifdef
> some_definition_defined_only_on_systems_which_need_this_patch'.

I can push this through, but there is just one important difference between this and other #ifdefs which are only defined on systems that need it.  Our criterion does NOT live in userland.  So its not something that a ./configure script should automagically define/undefine based on the current running kernel.  While you can tell if the *current* kernel is PaX enabled,

   cat /proc/self/status  | grep -q PaX && echo "PaX" || echo "No PaX"

the user can easily get into a situation where, they build on a system where PaX is not running and don't get the patch.  Then boot into a PaX kernel and fail!

I recommend a ./configure flag called --with-pax-compatibility (as you suggested), which is disabled by default, will only be enabled by the user adding the flag, and comes with a help that says, enable this if you *ever* anticipate running python under a PaX enabled kernel.

@Pavel.  They're your patches, do you want to integrate them into the build system as Arfrever suggests, or shall I?
Comment 84 Dennis Schridde 2012-02-23 00:50:16 UTC
(In reply to comment #83)
> I recommend a ./configure flag called --with-pax-compatibility (as you
> suggested), which is disabled by default, will only be enabled by the user
> adding the flag, and comes with a help that says, enable this if you *ever*
> anticipate running python under a PaX enabled kernel.
Ain't that what IUSE=pax_kernel is for?

Has there been any recent progress with this issue?
Comment 85 PaX Team 2012-02-24 13:29:36 UTC
before you dig too deep into trying to make all this depend on something: what's the whole point of that? non-PaX kernel users won't be executing this code at all, and PaX kernel users need this now unconditionally. so why go all the trouble to finding yet another superfluous switch to toggle?
Comment 86 Pavel Labushev 2012-03-04 12:48:44 UTC
Created attachment 304207 [details, diff]
python3 patch
Comment 87 Magnus Granberg gentoo-dev 2012-03-13 11:08:51 UTC
*** Bug 407681 has been marked as a duplicate of this bug. ***
Comment 88 UpstreamPragmatist 2012-04-05 09:34:40 UTC
I am having this problem on amd64, kernel version 3.2.2-hardened-r1 (Mar 19).
The problem exists under python2.7.2, python3.1.4 and python 3.2.2. These are the only versions I have installed.

Pavel's patch above (from 4th of March) fixes the issue.

Can someone please integrate it with python + pax_kernel?
Comment 89 Matt Summers (RETIRED) gentoo-dev 2012-04-06 00:03:17 UTC
Pavel's patch is a nice implementation of solar's sed from the end of july, 2010. 

I include it here for reference;
sed -i -e s/'PROT_READ | PROT_WRITE | PROT_EXEC'/'PROT_READ | PROT_WRITE'/g Modules/_ctypes/malloc_closure.c

Is there no way to get this patch upstream?
Comment 90 UpstreamPragmatist 2012-04-06 05:29:23 UTC
I have applied this patch to all of my deployments now using the following in /etc/portage/env/dev-lang/python (make a new file, no need to mark it +x):

post_src_prepare (){
sed -i -e s/'PROT_WRITE\s*|\s*PROT_EXEC'/'PROT_WRITE'/g Modules/_ctypes/mallo*.c
}

Thanks to everyone for identifying the issue and to Matt for isolating the code and giving me the idea how to fix this. Since upstream Python people apparently don't care about -hardened users, or the security of their other users, I think this a decent solution if you need a usable Python installation. The fact that this is issue has not been solved after 20 months is sad.
Comment 91 Dirkjan Ochtman (RETIRED) gentoo-dev 2012-04-12 09:39:05 UTC
I've added the proposed fix to the python 2.7.3 ebuild, hitting the tree shortly.
Comment 92 Pavel Labushev 2012-04-12 10:56:27 UTC
(In reply to comment #91)
In-tree python uses system libffi that needs to be patched, or else the python part of the fix won't work.
Comment 93 Dirkjan Ochtman (RETIRED) gentoo-dev 2012-04-12 11:08:40 UTC
This is the "libffi part of the fix (3.0.10)"? Has that been discussed with libffi upstream at all?
Comment 94 Pavel Labushev 2012-04-12 12:14:04 UTC
(In reply to comment #93)
> This is the "libffi part of the fix (3.0.10)"?
That depends on which behavior you choose.

"libffi part of the fix (3.0.10)" is a dumb compatible fix that causes annoying "denied RWX mmap" kernel log messages per every closure allocation on kernels compiled with CONFIG_GRKERNSEC_RWXMAP_LOG.

"python3 patch" contains a better fix: mmap RW by default unless the FFI_DISABLE_EMUTRAMP environment variable does exist (empty or not).

> Has that been discussed with libffi upstream at all?
No.
Comment 95 PaX Team 2012-04-12 13:42:17 UTC
(In reply to comment #94)
> "libffi part of the fix (3.0.10)" is a dumb compatible fix that causes
> annoying "denied RWX mmap" kernel log messages per every closure allocation
> on kernels compiled with CONFIG_GRKERNSEC_RWXMAP_LOG.
> 
> "python3 patch" contains a better fix: mmap RW by default unless the
> FFI_DISABLE_EMUTRAMP environment variable does exist (empty or not).

err, somewhere above i questioned this 'better' fix already and got no response so here it is again: what's the point of the runtime detection of EMUTRAMP again? and why would anyone have to set a *userland* variable for it when it's a *kernel* feature? (if you really want this kind of detection then check for 'E' in /proc/self/status:PaX)
Comment 96 Pavel Labushev 2012-04-12 23:46:27 UTC
(In reply to comment #95)
> err, somewhere above i questioned this 'better' fix already and got no

It's better in comparison with "libffi part of the fix (3.0.10)". It's no way better than the simple conditional patches I use on my systems. Upstreaming this wasn't my idea, and I don't like it, if you ask me.

> response so here it is again: what's the point of the runtime detection of
> EMUTRAMP again? and why would anyone have to set a *userland* variable for

FFI_DISABLE_EMUTRAMP *disables* EMUTRAMP support (read: enables PROT_EXEC use) in _ctypes and libffi. I tried to predict what upstream would want, keeping in mind their usual ways to resolve such issues, and at the same time keep the fix secure by default yet tunable at runtime, for performance and compatibility.

> it when it's a *kernel* feature? (if you really want this kind of detection
> then check for 'E' in /proc/self/status:PaX)

I refused this idea because /proc isn't always available. For example, I don't mount it inside chroot environments yet I do run chrooted python apps. And someone on #grsecurity have the entire /proc hidden using RBAC. So I had to keep that in mind.
Comment 97 Pavel Labushev 2012-04-12 23:48:55 UTC
Created attachment 308707 [details, diff]
simple conditional libffi fix
Comment 98 Pavel Labushev 2012-04-12 23:50:03 UTC
Created attachment 308711 [details, diff]
simple conditional python fix
Comment 99 Pavel Labushev 2012-04-17 10:21:38 UTC
This bug is still not resolved. Please, change the status.
Comment 100 Arfrever Frehtes Taifersar Arahesis 2012-04-17 23:42:50 UTC
(In reply to comment #99)
> This bug is still not resolved. Please, change the status.

This bug was fixed, but hardened kernel some months later invented a new way of breaking ctypes.
Comment 101 Arfrever Frehtes Taifersar Arahesis 2012-04-28 05:24:31 UTC
A simpler solution was implemented and backported to dev-lang/python-{2.6.8,2.7.3-r1,3.1.5,3.2.3}::gentoo.

If somebody wants to use a different solution, then he should send patches to libffi/CPython upstreams. When patches are accepted by upstreams, then a separate bug for libffi and a separate bug for CPython can be created in Gentoo bugzilla.
Comment 102 Nikoli 2014-04-08 09:33:24 UTC
Updated to libffi-3.0.13-r1 and recompiled python, now dmesg prints 'grsec: denied RWX mmap of <anonymous mapping> by' every time i run emerge or equery:

[169075.733809] grsec: denied RWX mmap of <anonymous mapping> by /usr/lib64/portage/bin/emerge[emerge:24515] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:18955] uid/euid:0/0 gid/egid:0/0
[169076.268528] grsec: denied RWX mmap of <anonymous mapping> by /usr/lib64/portage/bin/portageq[portageq:24524] uid/euid:0/0 gid/egid:0/0, parent /usr/bin/gcc-config[gcc-config:24523] uid/euid:0/0 gid/egid:0/0
[169081.002196] grsec: denied RWX mmap of <anonymous mapping> by /usr/lib64/python-exec/python2.7/equery[equery:24542] uid/euid:0/0 gid/egid:0/0, parent /bin/bash[bash:18955] uid/euid:0/0 gid/egid:0/0

emerge and equery do not crash and are not killed.

# paxctl-ng -v /usr/bin/python2.7
/usr/bin/python2.7:
        open(O_RDWR) failed: cannot change PT_PAX flags
        PT_PAX    : -e---
        XATTR_PAX : -E---


[ebuild   R    ] sys-apps/portage-2.2.8-r1  USE="(ipc) python2 (xattr) -build -doc -epydoc (-pypy2_0) -python3 (-selinux)" LINGUAS="ru" PYTHON_TARGETS="python2_7 (-pypy2_0) -python2_6 -python3_2 -python3_3 (-python3_4)" 0 kB
[ebuild  NS    ] dev-lang/python-3.3.3:3.3 [2.7.5-r3:2.7] USE="hardened ipv6 ncurses readline sqlite ssl threads xml -build -doc -examples -gdbm -tk -wininst" 0 kB
[ebuild   R    ] dev-libs/libffi-3.0.13-r1  USE="pax_kernel {test} -debug -static-libs" ABI_X86="(64) (-32) (-x32)" 0 kB
Comment 103 Magnus Granberg gentoo-dev 2015-07-04 15:16:28 UTC
I don't see any more problems, reopen if.