icedtea has two sets of X11 dependencies that are selected by USE="X": $X_COMMON_DEP and $X_DEPEND. However, line 98 of the ebuild adds $X_COMMON_DEP to $DEPEND without checking USE flags. This causes dev-java/icedtea to pull in X11 dependencies regardless of the configured USE flags.
Bah nevermind. It appears that icedtea actually has build-time dependencies on these libraries that can't be carved out easily. Abandoning.