This is because the version of ffmpeg shipped with xdtv is not compatible with the recents api changes in x264. Here is a patch, which is no more than the fix for ffmpeg. Beware, its not compatible with the older snapshots of x264-svn, so it might be applied to a p.masked version of xdtv because x264-svn-20060810 is still p.masked. Regards, Alexis.
Created attachment 95272 [details, diff] The patch
I've removed x264 support. As per VLC, x264 is not API stable so it should really be used only within ffmpeg that can be updated to use the new x264 snapshot every time. For who needs x264 support in XdTV, I would suggest to investigate how can XdTV be made to support external ffmpeg, that would solve this (and others) problem.