/usr should not be used for something that is considered a cache. /var/cache is probably better for this. I suggest the default place for distfiles is moved to some place below /var/cache
This is changable in make.conf. Move it where you like. Attach to a FHS bug, if you feel this is still an issue.