Bug 282707 – segfault in =media-libs/libtheora-1.1_alpha2 [with fix]

• Status: RESOLVED FIXED
• Product: Gentoo Linux

[Description Marcin Kościelnicki 2009-08-25 15:54:09 UTC]

mplayer on ogg theora files catches SIGSEGV in decode_video. The same thing would probably happen with any other media player using libtheora.

I traced the problem to a memset being called on wrong array due to internal structure changes between 1.0 and 1.1. Theoras now play fine with the following fix:

diff -ur libtheora-1.1alpha2/lib/dec/decode.c libtheora-1.1alpha2-fixed/lib/dec/decode.c
--- libtheora-1.1alpha2/lib/dec/decode.c        2009-05-27 00:53:06.000000000 +0200
+++ libtheora-1.1alpha2-fixed/lib/dec/decode.c  2009-08-25 17:52:16.198866987 +0200
@@ -1958,7 +1958,7 @@
       cheight=yheight>>!(info->pixel_fmt&2);
       yplane_sz=yhstride*(size_t)yheight;
       cplane_sz=chstride*(size_t)cheight;
-      memset(_dec->state.ref_frame_data,0x80,yplane_sz+2*cplane_sz);
+      memset(_dec->state.ref_frame_data[0],0x80,3*(yplane_sz+2*cplane_sz));
     }
     else{
       for(refi=0;refi==_dec->state.ref_frame_idx[OC_FRAME_GOLD]||

[Comment 1 Samuli Suominen (RETIRED) 2009-08-27 09:37:00 UTC]

Is this still valid with libtheora 1.1 beta3? Added it to tree few minutes ago.

[Comment 2 Marcin Kościelnicki 2009-08-27 10:19:05 UTC]

Works in libtheora 1.1 beta3. Thanks.