I writing this in case someone see this bug and will be looking for fix here. Upstream is aware of it , and this is ffmpeg bug. This was already fixed in ffmpeg, but unfortunately this fix isn't in version 5.0.1 (the newest version in portage tree right now), so right now the only way to fix it is to install ffmpeg-9999
You (and we) should be free to apply the fix as a user patch to released versions of media-video/ffmpeg.
Upstream patch: https://git.ffmpeg.org/gitweb/ffmpeg.git/patch/68595b46cb374658432fff998e82e5ff434557ac
Well, from what I can tell this is a ffmpeg issue that would happen regardless of the yt-dlp context if converting such videos -- so don't think much for yt-dlp to do here.
Maybe ffmpeg would be interested in backporting the fix given patch looks trivial, I assume comment #2 implies it works?