Right now, doxygen provides two ineffective USE flags: 'dot' and 'latex'. These flags only pull in runtime dependencies, while not affecting the doxygen code at all.
Thus, for users having either graphviz or respective latex parts installed, they only force needless USE change and rebuild of doxygen.
I think that the flags should be removed (or possibly transformed into GLEP-62 IUSE_RUNTIME at some point) and the packages should depend on the required packages directly instead.
Sometimes, this is already a case because of the output format choice. Namely, dependency on doxygen[dot] does not enforce the DOT_IMAGE_FORMAT. Effectively, the package should instead depend on e.g.:
IIRC, the original request (and meaning) of the dot USE flag for doxygen itself was to build its own manual during install. Building the PDF doxygen manual requires both graphviz and a latex tool to complete "make refman". Building the HTML doxygen manual would only need graphviz itself. SVG graphics in the doxygen output in general should be optional (I think it defaults to .png images, but .svg allows interactive zoom/pan for large graphs now). PNG is probably a better default for install.