vlc crashes when hwaccel is used: * tools->preferences->codec->hardware-accelerated decoding set to "automatic" or "VDPAU video decoder" * a suitable video is played (almost everything except h265) Reproducible: Always Steps to Reproduce: 1. ensure hwaccel is not disabled 2. play any suitable video 3. get segfault Thread 24 "av:h264:df0" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff9f3646c0 (LWP 3230)] ff_hwaccel_frame_priv_alloc (avctx=0x7fffa0c240c0, hwaccel_picture_private=0x7fffa0da99c0) at src/libavcodec/decode.c:1848 warning: 1848 src/libavcodec/decode.c: No such file or directory (gdb) bt #0 ff_hwaccel_frame_priv_alloc (avctx=0x7fffa0c240c0, hwaccel_picture_private=0x7fffa0da99c0) at src/libavcodec/decode.c:1848 #1 0x00007fffad93319d in alloc_picture (h=0x7fffa0cfe3c0, pic=0x7fffa0da9950) at src/libavcodec/h264_slice.c:209 #2 0x00007fffad934456 in h264_frame_start (h=0x7fffa0cfe3c0) at src/libavcodec/h264_slice.c:528 #3 0x00007fffad937a81 in h264_field_start (h=0x7fffa0cfe3c0, sl=0x7fffa0db9080, nal=0x7fff98045990, first_slice=1) at src/libavcodec/h264_slice.c:1601 #4 0x00007fffad939c75 in ff_h264_queue_decode_slice (h=0x7fffa0cfe3c0, nal=0x7fff98045990) at src/libavcodec/h264_slice.c:2132 #5 0x00007fffad946612 in decode_nal_units (h=0x7fffa0cfe3c0, buf=0x7fffa4007ef0 "", buf_size=2094) at src/libavcodec/h264dec.c:651 #6 0x00007fffad947c34 in h264_decode_frame (avctx=0x7fffa0c240c0, pict=0x7fffa0c06800, got_frame=0x7fffa0c23a50, avpkt=0x7fffa0c1ea40) at src/libavcodec/h264dec.c:1047 #7 0x00007fffadcdf8f1 in frame_worker_thread (arg=0x7fffa0c23940) at src/libavcodec/pthread_frame.c:223 #8 0x00007ffff7e1fd09 in start_thread () at /lib64/libc.so.6 #9 0x00007ffff7e938a8 in clone3 () at /lib64/libc.so.6
Created attachment 879675 [details] emerge --info
Hm, out of curiosity.. CFLAGS="-g3 -fno-strict-aliasing" CXXFLAGS="-march=native -O2 -pipe -fno-strict-aliasing" What's happened there?
Notes: 1. This is possibly a dup of #918964 but that one lacks enough details to be sure 2. This is very likely not a bug in VLC after all, but the whole chain vlc->libavcodec->vdpau->video driver is not entirely clear to me 3. This reproduces with all vlc versions currently available in portage (earliest is 3.0.18-r3) 4. The problem goes away when downgrading ffmpeg to 6.0.1
(In reply to Sam James from comment #2) > Hm, out of curiosity.. > > CFLAGS="-g3 -fno-strict-aliasing" > CXXFLAGS="-march=native -O2 -pipe -fno-strict-aliasing" > > What's happened there? Same as before: Thread 25 "av:h264:df0" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fff9e8136c0 (LWP 21728)] ff_hwaccel_frame_priv_alloc (avctx=0x7fffa4c2eb00, hwaccel_picture_private=0x7fffa4da9a00) at src/libavcodec/decode.c:1848 warning: 1848 src/libavcodec/decode.c: No such file or directory (gdb) bt #0 ff_hwaccel_frame_priv_alloc (avctx=0x7fffa4c2eb00, hwaccel_picture_private=0x7fffa4da9a00) at src/libavcodec/decode.c:1848 #1 0x00007fff9eca119d in alloc_picture (h=0x7fffa4cfe400, pic=0x7fffa4da9990) at src/libavcodec/h264_slice.c:209 #2 0x00007fff9eca2456 in h264_frame_start (h=0x7fffa4cfe400) at src/libavcodec/h264_slice.c:528 #3 0x00007fff9eca5a81 in h264_field_start (h=0x7fffa4cfe400, sl=0x7fffa4db90c0, nal=0x7fff90045990, first_slice=1) at src/libavcodec/h264_slice.c:1601 #4 0x00007fff9eca7c75 in ff_h264_queue_decode_slice (h=0x7fffa4cfe400, nal=0x7fff90045990) at src/libavcodec/h264_slice.c:2132 #5 0x00007fff9ecb4612 in decode_nal_units (h=0x7fffa4cfe400, buf=0x7fff98007550 "", buf_size=2094) at src/libavcodec/h264dec.c:651 #6 0x00007fff9ecb5c34 in h264_decode_frame (avctx=0x7fffa4c2eb00, pict=0x7fffa4c06840, got_frame=0x7fffa4c36d10, avpkt=0x7fffa4c05180) at src/libavcodec/h264dec.c:1047 #7 0x00007fff9f04d8f1 in frame_worker_thread (arg=0x7fffa4c36c00) at src/libavcodec/pthread_frame.c:223 #8 0x00007ffff7e1fd09 in start_thread () at /lib64/libc.so.6 #9 0x00007ffff7e938a8 in clone3 () at /lib64/libc.so.6 A wild guess based on a very quick look at the vlc code makes me feel this might originate in some api/abi change in ffmpeg. I'm masking it for now...
(In reply to acab from comment #4) > (In reply to Sam James from comment #2) > > Hm, out of curiosity.. > > > > CFLAGS="-g3 -fno-strict-aliasing" > > CXXFLAGS="-march=native -O2 -pipe -fno-strict-aliasing" > > > > What's happened there? > > Same as before: > To be clear, I meant "the CFLAGS look inconsistent, why?" Did you fix them to be the same as CXXFLAGS (possibly including -g3) or did you do something else? > Thread 25 "av:h264:df0" received signal SIGSEGV, Segmentation fault. > [Switching to Thread 0x7fff9e8136c0 (LWP 21728)] > ff_hwaccel_frame_priv_alloc (avctx=0x7fffa4c2eb00, > hwaccel_picture_private=0x7fffa4da9a00) > at src/libavcodec/decode.c:1848 > warning: 1848 src/libavcodec/decode.c: No such file or directory > (gdb) bt > #0 ff_hwaccel_frame_priv_alloc (avctx=0x7fffa4c2eb00, > hwaccel_picture_private=0x7fffa4da9a00) > at src/libavcodec/decode.c:1848 hmm... https://libera.irclog.whitequark.org/ffmpeg-devel/2023-11-21#35282468;.
try https://patchwork.ffmpeg.org/project/ffmpeg/patch/1700546252-17490-1-git-send-email-dmitry.v.rogozhkin@intel.com/?
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=cef2e06f2fabd2cb77dd2432a15d1a32c187813e commit cef2e06f2fabd2cb77dd2432a15d1a32c187813e Author: Sam James <sam@gentoo.org> AuthorDate: 2023-12-18 04:50:05 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-12-18 04:50:35 +0000 media-video/ffmpeg: backport fixes from 6.1 branch for 6.1 ``` $ git shortlog d4ff0020b40b524a490cf62eccbd3a318f4c0e58..HEAD Anton Khirnov (1): lavc/dvdsubenc: only check canvas size when it is actually set Benjamin Cheng (1): hwcontext_vulkan: guard unistd.h include Dmitry Rogozhkin (1): avcodec/decode: validate hw_frames_ctx when AVHWAccel.free_frame_priv is used Lynne (1): lavc/Makefile: build vulkan decode code if vulkan_av1 has been enabled Sebastian Ramacher (2): avcodec/fft: Use av_mallocz to avoid invalid free/uninit avcoded/fft: Fix memory leak if ctx2 is used ``` Closes: https://bugs.gentoo.org/920185 Signed-off-by: Sam James <sam@gentoo.org> media-video/ffmpeg/ffmpeg-6.1-r3.ebuild | 588 +++++++++++++++++++++ ...-Use-av_mallocz-to-avoid-invalid-free-uni.patch | 54 ++ ...coded-fft-Fix-memory-leak-if-ctx2-is-used.patch | 26 + ...ode-validate-hw_frames_ctx-when-AVHWAccel.patch | 64 +++ ...enc-only-check-canvas-size-when-it-is-act.patch | 31 ++ ...le-build-vulkan-decode-code-if-vulkan_av1.patch | 31 ++ ...6-hwcontext_vulkan-guard-unistd.h-include.patch | 33 ++ 7 files changed, 827 insertions(+)
Please try out ffmpeg-6.1-r3. I think this should sort it.
(In reply to Sam James from comment #5) > To be clear, I meant "the CFLAGS look inconsistent, why?" > > Did you fix them to be the same as CXXFLAGS (possibly including -g3) or did > you do something else? Sorry i was particularly dense yesterday, even for my standards. (In reply to Sam James from comment #8) > Please try out ffmpeg-6.1-r3. I think this should sort it. It does! Thanks a lot
(In reply to acab from comment #9) > (In reply to Sam James from comment #5) > > To be clear, I meant "the CFLAGS look inconsistent, why?" > > > > Did you fix them to be the same as CXXFLAGS (possibly including -g3) or did > > you do something else? > > Sorry i was particularly dense yesterday, even for my standards. > I was being (accidentally) terse, don't worry about it > (In reply to Sam James from comment #8) > > Please try out ffmpeg-6.1-r3. I think this should sort it. > > It does! Thanks a lot Yay! Thanks for confirming!
