In the ebuild for imlib2 there is a USE flag X/-X, but it is not used. No matter what you do, there is a dependency on X. I think the DEPEND section needs "X? (virtual/x11)" instead of just "virtual/x11". The package's ./configure supports a --without-x.
imlib2 needs X libs to compile even if you disable x support ;)