I noticed that portage will regenerate ld.so.cache (probably through calling env-update?) every time a package is merged/unmerged. This is often unnecessary, since there are a lot of small packages that don't install libs, and the overhead is in that case quite a lot (while I don't mind losing 2 minutes for a OpenOffice.org twelve-hours-long merging, it's annoying to lose 2 mins when installing, e.g, portage-manpages). For example I made a couple of calculations for the libperl_rebuilder script: on my machine (p4 2.4Ghz 256ram) it took >= 6 hours (and it was the only task), instead of ~2 hour, due to a >1.5min ld.so.cache regeneration per package merged. So I guess the best solution would be NOT to update ld.so.cache if: a) [dirs used for install] != [dirs from ld.so.conf] AND b) after merging we need an unmerging of an old package, because we need to do it only once. Hopefully, this will help people with large /usr/lib dirs (mine is ~1.4Gb). Please note that ld.so.conf lacks listing some dirs I believe are added in automatic: most notably /usr/lib, /lib and /usr/local/lib. In http://www.daemon-systems.org/man/ld.so.conf.5.html they're called as "standard set". Purpose for further optimization: call also env-update only if necessary, e.g if calling at least one time "doenvd" in the ebuild[s].
you would also need to check the timestamp of /etc/ld.so.conf before and after the merge in order to catch changes in that file aswell. But I agree. Doing a check between all the directories in /etc/ld.so.conf agains the files would be were nice, and save a lot of building time, at you mentioned with the perl example.
This has been done in the past and only caused problems. Too many exceptions.