Compiling obs-studio with the nvenc use flag enabled does not pull nvidia-video-codec, which causes nvenc encoding to not work. Emerged nvidia-video-codec and it straight out worked, so I'm supposing it is a bug.
Interesting. I haven't had this package installed myself, according to my logs, and I don't have NVIDIA hardware on my main machine any longer.
I wonder what changed.
In any case, will look into it. Thanks for reporting!
My two cents:
At least in my machine pulling nvidia-video-codec is not necessary. To enable video (de)en-coding it is sufficient to enable "nvenc" USE flag in obs-studio AND "uvm" USE flag in nvidia-drivers. My setup:
x11-drivers/nvidia-drivers-455.28-r1::gentoo was built with the following:
USE="X driver gtk3 kms (libglvnd) multilib static-libs tools uvm -compat -wayland" ABI_X86="32 (64) (-x32)"
media-video/obs-studio-25.0.8-r1::gentoo was built with the following:
USE="alsa fdk jack nvenc pulseaudio ssl truetype v4l vlc -imagemagick -luajit -python -speex" ABI_X86="(64)" PYTHON_SINGLE_TARGET="-python3_6 -python3_7"
and I can (de)en-code with NVENC with no problems.