there is no use flag to indicate to portage to build vim with clientserver support.
Running vim's ./configure with the --with-x flag causes this feature to be enabled. In fact, this is the only way to enable it (unless you happen to be running Windows). That switch is controlled by the existing vim-with-x use flag. So, to get clientserver support, you must build vim with the vim-with-x use flag.