app-text/scrollkeeper and app-text/gnome-doc-utils dependencies pull the whole tree of packages on non gnome system, while those package are not obligately needed neither for building nor for runnning (moreover i see --disable-scrollkeeper by default in pkg_setup() which looks strange). would it be possible to drop those or put them under doc use flag?
no and no. Both are needed for build time as is reflected in the build. The fact that --disable-scrollkeeper is passed doesn't mean we don't use scrollkeeper afterwards which is in fact the case (in postinst from the eclass). gnome-doc-utils is used for user documentation which is not optional.
closing invalid per previous comment.
