At least the following two smartcard related files are installed whereas they shouldn't be due to USE="-smartcard": /usr/share/doc/openssh-4.6_p1-r1/README.smartcard.bz2 /usr/share/openssh/Ssh.bin See this URL for Ssh.bin related information: http://linuxfromscratch.org/pipermail/blfs-dev/2005-November/012039.html
the conclusion they came to at in that thread is to "leave it alone" ...