When emerge fetches a package and the checksum fails, it will create a file in distfiles with ._checksum_failure_.[randomstring] appended to the name, and try the next mirror. This leads to a DoS problem, as a large file that is correct, but with an incorrect checksum or size will then be fetched umpteen times, filling up the distfiles directory. Reproducible: Always Steps to Reproduce: emerge xulrunner # (just an example, since this had an incorrect sum/length this time -- please note that this is NOT a bug report against xulrunner, cause the problem will reappear with ANY package when there's a mismatch) Downloading 'xulrunner-1.8.1.9-source.tar.bz2'... see /var/log/emerge-fetch.log for details. Actual Results: cd /usr/portage/distfiles du -sk *checksum* 34608 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.0oUHaW 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.5NdZgb 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.C2BpyQ 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.K-w10a 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.O5DS5c 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.VCmXr_ 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_._hcRwY 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.awzjck 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.iGJCzf 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.l1MGTD 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.o6Ru2b 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.oAv7N8 34604 xulrunner-1.8.1.9-source.tar.bz2._checksum_failure_.yXN2Kz That's almost half a gigabyte from one single package. If the package had been bigger, or more than one ebuild had incorrect size/checksums, the problem would have been worse. Expected Results: Leave at most ONE copy of a checksum_failure fetch. Not one for each attempt. Also, and this is secondary to the main problem, these were HTTP fetches, and I noticed that the whole file was fetched before the file size was checked. If the HTTP reply returns a Size header, why not check against it before proceeding to download the whole file? Lots of bandwidth could be saved that way.
(In reply to comment #0) > Expected Results: > Leave at most ONE copy of a checksum_failure fetch. Not one for each attempt. This patch fixes that. > Also, and this is secondary to the main problem, these were HTTP fetches, and I > noticed that the whole file was fetched before the file size was checked. If > the HTTP reply returns a Size header, why not check against it before > proceeding to download the whole file? Lots of bandwidth could be saved that > way. That would be nice, but unfortunately FETCHCOMMAND (see `man make.conf`) isn't flexible like that. I suppose we could implement it using the python libraries and have a way to disable FETCHCOMMAND. The attached patch should help by making it bail out automatically after 5 checksum failures. After that many failures, it usually doesn't help to keep trying additional mirrors. It's not ideal, but it's a lot better than the previous behavior.
Created attachment 135125 [details, diff] detect duplicate files to save disk space and quit after 5 checksum failures
This has been released in 2.1.3.18. I also added some code to make it prefer non mirror:// uris after 2 checksum failures, to increase the probability of success before reaching the maximum of 5 checksum failures.
