xfce works just fine without either gnome or kde, but the ebuild forces both. I would suggest maintaining two ebuilds - one with gnome and kde and one with just plain gtk support.
Check the USE variable HOWTO document.