Beware I am new to this! :) Anyway, I've got an X-Less box, use X is not in the make.conf. When trying to emerge jed, it tries to grab X anyway. I fixed my ebuild in a very ugly way, making it only use textmode jed.
Make certain you have not commented out USE=, you want to set it to USE="" if you want to turn off all USE vars.