I feel like I may be missing something here but I've not found any reference or explanation for this issue.
We run headless systems with -X to do GPGPU computations using Cuda. For <x11-drivers/nvidia-drivers-355.11-r2 this was not a problem.
From version 355.11-r2 libcuda.so and other libraries require USE=X to be deployed. Furthermore with USE=-X nvidia-smi is deployed (required for GPU enumeration I believe) but it cannot run as it depends on libnvidia-ml.so which is not deployed with USE=-X.
This is based on the rules in the ebuilds so I've not included my specific build config. It seems unecessary that we should now need to install X on our systems simply to get libcuda. Especially when it was not required in previous versions.
I confirm this issue installing nvidia-driver to headless server for cuda computing not possible.