when trying to install libgd it requires xfree. On a server with libgd support it should be possible to install it without X.
Found the problem :( .. I overlooked something in the USE variabel