Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 202465 - media-libs/xvid-1.1.3-r1: pic fixes cause xvid decoder to segfault
Summary: media-libs/xvid-1.1.3-r1: pic fixes cause xvid decoder to segfault
Status: VERIFIED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Gentoo Media-video project
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-12-16 14:28 UTC by Alexis Ballier
Modified: 2007-12-18 11:27 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
fixed textrel patch for xvid-1.1.3 (xvid-1.1.3-textrel.patch,190.41 KB, patch)
2007-12-16 18:32 UTC, PaX Team
Details | Diff
fixed textrel patch for xvid-1.1.3 (xvid-1.1.3-textrel.patch,190.31 KB, patch)
2007-12-16 22:51 UTC, PaX Team
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alexis Ballier gentoo-dev 2007-12-16 14:28:08 UTC
I didn't catch it at first because default decoder is usually ffmpeg not xvid, but forcing xvid causes a segfault:

take foo.avi any xvid encoded file:
# mplayer -vo yuv4mpeg -vc xvid foo.avi 
MPlayer dev-SVN-rUNKNOWN-4.2.2 (C) 2000-2007 MPlayer Team
CPU: AMD Athlon(tm) 64 Processor 3200+ (Family: 15, Model: 31, Stepping: 0)
CPUflags:  MMX: 1 MMX2: 1 3DNow: 1 3DNow2: 1 SSE: 1 SSE2: 1
Compiled for x86 CPU with extensions: MMX MMX2 3DNow 3DNowEx SSE SSE2

Playing foo.avi.
AVI file format detected.
[aviheader] Video stream found, -vid 0
AVI: No audio stream found -> no sound.
VIDEO:  [XVID]  720x480  24bpp  29.970 fps  2002.1 kbps (244.4 kbyte/s)
Clip info:
 Software: transcode-1.0.4
Using (default) progressive frame mode.==========================================================================
Forced video codec: xvid
Opening video decoder: [xvid] XviD 1.0 decoder
xvid: using library version 1.1.3 (build xvid-1.1.3)
Selected video codec: [xvid] vfm: xvid (XviD (MPEG-4))
==========================================================================
Audio: no sound
Starting playback...
VDec: vo config request - 720 x 480 (preferred colorspace: Planar YV12)
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.50:1 - prescaling to correct movie aspect.
VO: [yuv4mpeg] 720x480 => 720x480 Planar YV12 
V:   0.7  21/ 21 23%  8%  0.0% 0 0 

MPlayer interrupted by signal 11 in module: decode_video
- MPlayer crashed by bad usage of CPU/FPU/RAM.
  Recompile MPlayer with --enable-debug and make a 'gdb' backtrace and
  disassembly. Details in DOCS/HTML/en/bugreports_what.html#bugreports_crash.
- MPlayer crashed. This shouldn't happen.
  It can be a bug in the MPlayer code _or_ in your drivers _or_ in your
  gcc version. If you think it's MPlayer's fault, please read
  DOCS/HTML/en/bugreports.html and follow the instructions there. We can't and
  won't help unless you provide this information when reporting a possible bug.

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xf7bfd6c0 (LWP 18765)]
0x01010101 in ?? ()
(gdb) bt
#0  0x01010101 in ?? ()
#1  0x01010101 in ?? ()
#2  0xf7d60df8 in decoder_bf_interpolate_mbinter (dec=0x8810500, forward=
      {y = 0xf79c5480 '!' <repeats 16 times>, "########$$$$$$$$", '#' <repeats 16 times>, '$' <repeats 16 times>, '&' <repeats 32 times>, '\'' <repeats 24 times>, "&&&&&&&&''''''''", ')' <repeats 32 times>, "********))))))))'3?GOSNENQTSONPS"..., u = 0x88a6520 "\211\211\211\211\211\211\211\211", '\212' <repeats 48 times>, '\211' <repeats 16 times>, '\212' <repeats 16 times>, "\211\211\211\211\211\211\211\211\207\207\207\207\207\207\207\207", '\205' <repeats 16 times>, "\207\207\207\207\207\207\207\207", '\204' <repeats 16 times>, "\205\205\205\205\205\205\205\205", '\204' <repeats 16 times>, '\205' <repeats 32 times>..., v = 0x88c5d20 "ssssssss", 'q' <repeats 24 times>, 'o' <repeats 16 times>, 'n' <repeats 24 times>, "mmmmmmmmlllllllliiiiiiiihhhhhhhhjjjjjjjjkkkkkkkkiiiiiiiioooooooonnnnnnnn", 'l' <repeats 16 times>, "nnnnnnnnoooooooonnnnnnnnllllllllnnnnnnnn"...}, backward=
      {y = 0xf7a44480 "\034\034\034\034\034\034\034\034", '\036' <repeats 16 times>, ' ' <repeats 48 times>, "\036\036\036\036\036\036\036\036", ' ' <repeats 32 times>, "!!!!!!!!%%%%%%%%&&&&&&&&", '\'' <repeats 16 times>, "((((((((%%%%%%%%", '&' <repeats 16 times>, " 1AGJKJGKLLIEFKP"..., u = 0x88674e0 '\212' <repeats 24 times>, "\213\213\213\213\213\213\213\213", '\212' <repeats 32 times>, '\211' <repeats 16 times>, "\212\212\212\212\212\212\212\212\211\211\211\211\211\211\211\211\207\207\207\207\207\207\207\207", '\205' <repeats 16 times>, "\206\207\207\207\207\207\207\207\205", '\204' <repeats 14 times>, "\205\205\205\205\205\205\205\205\205", '\204' <repeats 17 times>, '\205' <repeats 31 times>..., v = 0x8886d20 "vvvvvvvvtttttttt", 's' <repeats 16 times>, 't' <repeats 16 times>, "ssssssssqqqqqqqq", 'n' <repeats 16 times>, "mmmmmmmmmlkjhgfeeeeeeeeejjjjjjjjghhhhhhhgffffffflooooooonnnnnnnm", 'l' <repeats 16 times>, "mnnnnnnnnooooooonnnnnnnmlllllllmnnnnnnnn"...}, pMB=0xf771bf94, 
    x_pos=9, y_pos=0, bs=0xfff539d4, direct=0) at ../../src/image/interpolate8x8.h:332
#3  0xf7d65d70 in decoder_decode (dec=0x88676d1, frame=0xfff53a94, stats=0xfff53acc)
    at ../../src/decoder.c:1473
#4  0xf7d6b9ad in xvid_decore (handle=0x8810500, opt=424, param1=0xfff53a94, param2=0xfff53acc)
    at ../../src/xvid.c:825

(gdb) print $pc
$1 = (void (*)()) 0x1010101
(gdb) x/9i $pc
0x1010101:	Cannot access memory at address 0x1010101
(gdb) i r
eax            0x88676d1	143029969
ecx            0x88290c0	142774464
edx            0x1a8	424
ebx            0xf7df7ff4	-136347660
esp            0xfff537f8	0xfff537f8
ebp            0xf771bf94	0xf771bf94
esi            0x3	3
edi            0xfffff4b1	-2895
eip            0x1010101	0x1010101
eflags         0x210206	[ PF IF RF ID ]
cs             0x23	35
ss             0x2b	43
ds             0x2b	43
es             0x2b	43
fs             0x0	0
gs             0x63	99


of course, this is fine with 1.1.3 (without textrel patch)
Comment 1 PaX Team 2007-12-16 18:32:13 UTC
Created attachment 138642 [details, diff]
fixed textrel patch for xvid-1.1.3

i fixed a missing stack readjustment in 3dn code (can you confirm that your box is an amd?) that could result in the stack trace you showed, let me know if that was it. also you could have reused the old bugzilla entry as this is a bug related to that patch... now someone's gonna have to invalidate the patch there and redirect people here (i'm off to the pub myself ;).
Comment 2 PaX Team 2007-12-16 18:34:23 UTC
(In reply to comment #1)
> Created an attachment (id=138642) [edit]
> fixed textrel patch for xvid-1.1.3
> 
> i fixed a missing stack readjustment in 3dn code (can you confirm that your box
> is an amd?)

silly me, it's an athlon so i think i did get it right. really need that beer now.
Comment 3 Alexis Ballier gentoo-dev 2007-12-16 22:08:38 UTC
(In reply to comment #1)
> Created an attachment (id=138642) [edit]
> fixed textrel patch for xvid-1.1.3
> 
> i fixed a missing stack readjustment in 3dn code (can you confirm that your box
> is an amd?) that could result in the stack trace you showed, let me know if
> that was it.

yes its an athlon64 running a 32bits chroot, but I get a similar failure with a p3

> also you could have reused the old bugzilla entry as this is a bug
> related to that patch... now someone's gonna have to invalidate the patch there
> and redirect people here (i'm off to the pub myself ;).

bah the only valid patch will be the one in the tree, that everyone will use ;)



However, with your new patch I still get the same error on the athlon64 & the p3 :/


for completeness, here is what I get on the P3:
(gdb) bt
#0  0x01010101 in ?? ()
#1  0x01010101 in ?? ()
#2  0xb77e3d7d in interpolate8x8_add_switch (cur=<value optimized out>, refn=<value optimized out>, 
    x=<value optimized out>, y=0, dx=1, dy=1, stride=424, rounding=0) at ../../src/image/interpolate8x8.h:332
#3  0xb77e5e14 in decoder_bf_interpolate_mbinter (dec=0x87bfb00, forward=
      {y = 0xb7403480 '!' <repeats 16 times>, "########$$$$$$$$", '#' <repeats 16 times>, '$' <repeats 16 times>, '&' <repeats 32 times>, '\'' <repeats 24 times>, "&&&&&&&&''''''''", ')' <repeats 32 times>, "********))))))))'3?GOSNENQTSONPS"..., u = 0x8855ae0 "\211\211\211\211\211\211\211\211", '\212' <repeats 48 times>, '\211' <repeats 16 times>, '\212' <repeats 16 times>, "\211\211\211\211\211\211\211\211\207\207\207\207\207\207\207\207", '\205' <repeats 16 times>, "\207\207\207\207\207\207\207\207", '\204' <repeats 16 times>, "\205\205\205\205\205\205\205\205", '\204' <repeats 16 times>, '\205' <repeats 32 times>..., v = 0x8875320 "ssssssss", 'q' <repeats 24 times>, 'o' <repeats 16 times>, 'n' <repeats 24 times>, "mmmmmmmmlllllllliiiiiiiihhhhhhhhjjjjjjjjkkkkkkkkiiiiiiiioooooooonnnnnnnn", 'l' <repeats 16 times>, "nnnnnnnnoooooooonnnnnnnnllllllllnnnnnnnn"...}, backward=
      {y = 0xb7482480 "\034\034\034\034\034\034\034\034", '\036' <repeats 16 times>, ' ' <repeats 48 times>, "\036\036\036\036\036\036\036\036", ' ' <repeats 32 times>, "!!!!!!!!%%%%%%%%&&&&&&&&", '\'' <repeats 16 times>, "((((((((%%%%%%%%", '&' <repeats 16 times>, " 1AGJKJGKLLIEFKP"..., u = 0x8816ae0 '\212' <repeats 24 times>, "\213\213\213\213\213\213\213\213", '\212' <repeats 32 times>, '\211' <repeats 16 times>, "\212\212\212\212\212\212\212\212\211\211\211\211\211\211\211\211\207\207\207\207\207\207\207\207", '\205' <repeats 16 times>, "\206\207\207\207\207\207\207\207\205", '\204' <repeats 15 times>, "\205\205\205\205\205\205\205\205", '\204' <repeats 17 times>, '\205' <repeats 31 times>..., v = 0x88362e0 "vvvvvvvvtttttttt", 's' <repeats 16 times>, 't' <repeats 16 times>, "ssssssssqqqqqqqq", 'n' <repeats 16 times>, "mmmmmmmmmlkjhgfeeeeeeeeejjjjjjjjghhhhhhhgffffffflooooooonnnnnnnm", 'l' <repeats 16 times>, "mnnnnnnnnooooooonnnnnnnmlllllllmnnnnnnnn"...}, pMB=0xb7159f94, x_pos=9, 
    y_pos=0, bs=0xbf999c94, direct=0) at ../../src/decoder.c:1295
#4  0xb77e8711 in decoder_decode (dec=0x87bfb00, frame=0xbf999d54, stats=0xbf999d8c)
    at ../../src/decoder.c:1473
#5  0xb77ebf02 in xvid_decore (handle=0x87bfb00, opt=2, param1=0x87d86c0, param2=0xbf999d8c)
    at ../../src/xvid.c:825
#6  0x080e1cc4 in ?? ()
#7  0x087bfb00 in ?? ()
#8  0x00000002 in ?? ()
#9  0xbf999d54 in ?? ()
#10 0xbf999d8c in ?? ()
Comment 4 PaX Team 2007-12-16 22:51:18 UTC
Created attachment 138664 [details, diff]
fixed textrel patch for xvid-1.1.3

i've found another unbalanced stack problem.
Comment 5 Alexis Ballier gentoo-dev 2007-12-17 09:16:58 UTC
seems all good for me, thanks a lot!
Comment 6 Jakub Moc (RETIRED) gentoo-dev 2007-12-18 11:27:56 UTC
WFM, thanks!