When I use video=vesafb:ywrap,mtrr,1280x800-32 the screen goes black once it loads fb system. and stays black after that all the time. The system does work and I know the boot continues and it even loads X, but X appears black too. The same line works when I use radeonfb instead (and replace vesafb: with radeonfb: in the kernel line). Maybe vesafb-tng doesn't work at all with wide screen resolutions?
vesafb-tng works with what the video BIOS can provide. Can you get the standard vesafb driver to initialize this resolution? (you would have to find a mode number for 1280x800-32 -- this can be done, for example, with the vbetest program from sys-libs/lrmi)
I haven't yet tried with vesafb, but when I use radeonfb with that resolution it works. Furthermore, when I use radeonfb without specifying in kernel line any specific resolution, it gets that native resolution from BIOS or something (it's a laptop LCD). So yes - I would guess the BIOS has this resolution, and as it's the native for that screen I'd like to use it. But indeed when I run vbetest it outputs a list of resolutions, and 1280x800 is not between those. Does that mean that even though 1280x800 is the native screen resolution I can't use it with vesafb-tng? only with radeonfb? should I use radeonfb then rather than vesafb-tng?
vesafb-tng can only use these resolutions that are provided by the video BIOS. These usually don't include any non-standard ones (ie. other than 640x480, 800x600, 1024x768, 1280x1024, 1600x1200), even if they happen to be the native resolution for the used display device. If you want 1280x800, you will have to use radeonfb.