Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 528200

Summary: media-libs/x265 - /usr/lib64/libx265.a has executable stack
Product: Gentoo Linux Reporter: Andrew John Hughes <gnu_andrew>
Component: Current packagesAssignee: Gentoo Media-video project <media-video>
Status: RESOLVED FIXED    
Severity: normal CC: anthonyryan1, atoth, fturco, hardened, nikoli, paolo.pedroni
Priority: Normal Keywords: PATCH
Version: unspecified   
Hardware: AMD64   
OS: Linux   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 666486    
Bug Blocks:    
Attachments: Mark assembly files with no executable stack.
x265-1.4-noEXEstack.patch
x265-1.5-noEXEstack.patch
x265-1.7-noEXEstack.patch
modified patch to be clean on some systems
x265-1.8-noEXEstack.patch
x265-1.8-noEXEstack.patch modified version
note-GNU-stack fix
x265-1.9-noEXEstack.patch
x265-2.0-noEXEstack.patch
x265-2.5-noEXEstack.patch
Backported fix from x265 2.7

Description Andrew John Hughes 2014-11-04 03:29:27 UTC
* QA Notice: The following files contain writable and executable sections
 *  Files with such sections will not work properly (or at all!) on some
 *  architectures/operating systems.  A bug should be filed at
 *  http://bugs.gentoo.org/ to make sure the issue is fixed.
 *  For more information, see http://hardened.gentoo.org/gnu-stack.xml
 *  Please include the following list of files in your report:
 *  Note: Bugs should be filed for the respective maintainers
 *  of the package in question and not hardened@g.o.
 * !WX --- --- usr/lib64/libx265.a:const-a.asm.o
 * !WX --- --- usr/lib64/libx265.a:cpu-a.asm.o
 * !WX --- --- usr/lib64/libx265.a:ssd-a.asm.o
 * !WX --- --- usr/lib64/libx265.a:mc-a.asm.o
 * !WX --- --- usr/lib64/libx265.a:mc-a2.asm.o
 * !WX --- --- usr/lib64/libx265.a:pixel-util8.asm.o
 * !WX --- --- usr/lib64/libx265.a:blockcopy8.asm.o
 * !WX --- --- usr/lib64/libx265.a:pixeladd8.asm.o
 * !WX --- --- usr/lib64/libx265.a:dct8.asm.o
 * !WX --- --- usr/lib64/libx265.a:sad16-a.asm.o
 * !WX --- --- usr/lib64/libx265.a:intrapred16.asm.o
 * !WX --- --- usr/lib64/libx265.a:ipfilter16.asm.o
 * !WX --- --- usr/lib32/libx265.a:const-a.asm.o
 * !WX --- --- usr/lib32/libx265.a:cpu-a.asm.o
 * !WX --- --- usr/lib32/libx265.a:ssd-a.asm.o
 * !WX --- --- usr/lib32/libx265.a:mc-a.asm.o
 * !WX --- --- usr/lib32/libx265.a:mc-a2.asm.o
 * !WX --- --- usr/lib32/libx265.a:pixel-util8.asm.o
 * !WX --- --- usr/lib32/libx265.a:blockcopy8.asm.o
 * !WX --- --- usr/lib32/libx265.a:pixeladd8.asm.o
 * !WX --- --- usr/lib32/libx265.a:dct8.asm.o
 * !WX --- --- usr/lib32/libx265.a:sad16-a.asm.o
 * !WX --- --- usr/lib32/libx265.a:intrapred16.asm.o
 * !WX --- --- usr/lib32/libx265.a:ipfilter16.asm.o
 * !WX --- --- usr/lib32/libx265.a:pixel-32.asm.o


Reproducible: Always
Comment 1 Andrew John Hughes 2014-11-04 03:36:37 UTC
Created attachment 388496 [details, diff]
Mark assembly files with no executable stack.
Comment 2 Samuli Suominen (RETIRED) gentoo-dev 2014-11-09 09:12:34 UTC
Any chance you could post this patch to upstream?
Comment 3 Paolo Pedroni 2015-02-13 13:10:57 UTC
Created attachment 396360 [details, diff]
x265-1.4-noEXEstack.patch

The original patch did not apply cleanly to x265-1.4. This version is rebased against that version.
Comment 4 Paolo Pedroni 2015-03-18 10:09:44 UTC
Created attachment 399182 [details, diff]
x265-1.5-noEXEstack.patch

Patch rebased against x265-1.5
Comment 5 Alexis Ballier gentoo-dev 2015-03-18 12:02:00 UTC
(In reply to Samuli Suominen from comment #2)
> Any chance you could post this patch to upstream?

please do; as far as I am concerned, I will not apply this patch until upstream merges it
Comment 6 Attila Tóth 2015-04-22 21:13:59 UTC
(In reply to Paolo Pedroni from comment #4)
> Created attachment 399182 [details, diff] [details, diff]
> x265-1.5-noEXEstack.patch
> 
> Patch rebased against x265-1.5

Seems to do the trick for me! Thanks!
Comment 7 Paolo Pedroni 2015-05-20 08:11:09 UTC
Created attachment 403650 [details, diff]
x265-1.7-noEXEstack.patch

Patch rebased against x265-1.7
Comment 8 Attila Tóth 2015-05-20 11:38:04 UTC
(In reply to Paolo Pedroni from comment #7)
> Created attachment 403650 [details, diff] [details, diff]
> x265-1.7-noEXEstack.patch
> 
> Patch rebased against x265-1.7

Thank you so much!
Any chance to push this upstream?
Comment 9 Paolo Pedroni 2015-05-20 12:01:26 UTC
(In reply to Attila Tóth from comment #8)
> (In reply to Paolo Pedroni from comment #7)
> > Created attachment 403650 [details, diff] [details, diff] [details, diff]
> > x265-1.7-noEXEstack.patch
> > 
> > Patch rebased against x265-1.7
> 
> Thank you so much!
> Any chance to push this upstream?

I have no idea if the original poster of the patch (Andrew John Hughes, see comment #1) is taking care of that.
Comment 10 Andrew John Hughes 2015-05-20 22:08:02 UTC
I haven't had the time to look into upstreaming it.
Comment 11 Attila Tóth 2015-05-20 22:15:54 UTC
(In reply to Paolo Pedroni from comment #7)
> Created attachment 403650 [details, diff] [details, diff]
> x265-1.7-noEXEstack.patch
> 
> Patch rebased against x265-1.7

The patch did not applied cleanly for me. I attach a version, which was modified a little bit to be clean on my systems.
Comment 12 Attila Tóth 2015-05-20 22:16:45 UTC
Created attachment 403700 [details, diff]
modified patch to be clean on some systems
Comment 13 Paolo Pedroni 2015-10-12 08:32:09 UTC
Created attachment 414412 [details, diff]
x265-1.8-noEXEstack.patch

Patch for x265-1.8
Comment 14 Attila Tóth 2015-10-14 12:36:44 UTC
(In reply to Paolo Pedroni from comment #13)
> Created attachment 414412 [details, diff] [details, diff]
> x265-1.8-noEXEstack.patch
> 
> Patch for x265-1.8

Thanks!
Comment 15 Attila Tóth 2015-10-14 12:38:59 UTC
Created attachment 414556 [details, diff]
x265-1.8-noEXEstack.patch modified version

I've made some cleanup and added a chunk to handle loopfilter.asm as well.
Comment 16 Attila Tóth 2015-10-14 12:41:21 UTC
(In reply to Alexis Ballier from comment #5)
> (In reply to Samuli Suominen from comment #2)
> > Any chance you could post this patch to upstream?
> 
> please do; as far as I am concerned, I will not apply this patch until
> upstream merges it

Isn't it possible to include the patch in a way making it conditional to either pic or hardened USE flags?
Comment 17 Magnus Granberg gentoo-dev 2015-10-14 19:58:52 UTC
Created attachment 414576 [details, diff]
note-GNU-stack fix

cleaner note-GNU-stack patch
Comment 18 Paolo Pedroni 2016-02-01 14:24:07 UTC
Created attachment 424412 [details, diff]
x265-1.9-noEXEstack.patch

Patch for x265-1.9
Comment 19 Attila Tóth 2016-02-03 21:58:31 UTC
(In reply to Paolo Pedroni from comment #18)
> Created attachment 424412 [details, diff] [details, diff]
> x265-1.9-noEXEstack.patch
> 
> Patch for x265-1.9

Works for me! Thanks again: Dw.
Comment 20 Paolo Pedroni 2016-08-01 12:50:06 UTC
Created attachment 442190 [details, diff]
x265-2.0-noEXEstack.patch

This is all that's really needed to fix this problem in 2.0 (it probably works in 1.8 and 1.9 as well).
Comment 21 Attila Tóth 2016-08-01 14:30:50 UTC
(In reply to Paolo Pedroni from comment #20)
> Created attachment 442190 [details, diff] [details, diff]
> x265-2.0-noEXEstack.patch
> 
> This is all that's really needed to fix this problem in 2.0 (it probably
> works in 1.8 and 1.9 as well).

Works for me (portage user patch mechanism), thx again!
Comment 22 Agostino Sarubbo gentoo-dev 2016-12-31 22:14:36 UTC
did someone forward the patch upstream?
Comment 23 Anthony Ryan 2017-01-04 01:14:55 UTC
Since nobody else has done it, I've opened a PR upstream here: https://bitbucket.org/multicoreware/x265/pull-requests/30/ensure-x86-asm-is-marked-nowrite-noexec-on/diff
Comment 24 Attila Tóth 2017-03-16 00:40:53 UTC
(In reply to Paolo Pedroni from comment #20)
> Created attachment 442190 [details, diff] [details, diff]
> x265-2.0-noEXEstack.patch
> 
> This is all that's really needed to fix this problem in 2.0 (it probably
> works in 1.8 and 1.9 as well).

Patch works for x265-2.3 as well. I used the user patch mechanism, simple. Thx!
Comment 25 Attila Tóth 2017-07-17 17:53:28 UTC
Created attachment 485360 [details, diff]
x265-2.5-noEXEstack.patch

Former patch for x265-2.0 ported to x265-2.5.
Comment 26 gen2dev 2018-02-25 22:14:07 UTC
Created attachment 521050 [details, diff]
Backported fix from x265 2.7

x265-2.5-noEXEstack.patch also applies to x265 2.6 cleanly. However it's only partially fixing the upstream bug, which is that the assembler defines either "elf32" or "elf64" but the x86 assembly source is still using "elf". If I'm reading it right the unfixed part causes some symbols that should be hidden inside the library to be visible.

Upstream fixed the bug in their 2.7 release in a slightly different way, in the same commit where they changed from the YASM assembler to NASM. here:
  https://bitbucket.org/multicoreware/x265/commits/9eabffb26dd62e4f48c5679594ae13690eb9d221

Here's a backport of that upstream 2.7 fix to 2.6. It is only the part that fixes the "elf" issue, without the other changes in the same file in that 2.7 commit.
Comment 27 Alexis Ballier gentoo-dev 2020-06-05 15:43:33 UTC
(In reply to gen2dev from comment #26)
> Upstream fixed the bug in their 2.7 release in a slightly different way, in
> the same commit where they changed from the YASM assembler to NASM. here:
>  
> https://bitbucket.org/multicoreware/x265/commits/
> 9eabffb26dd62e4f48c5679594ae13690eb9d221
> 
> Here's a backport of that upstream 2.7 fix to 2.6. It is only the part that
> fixes the "elf" issue, without the other changes in the same file in that
> 2.7 commit.

should be fixed now then