Entries in /data/cached/buglist-*.html do not include a last-modified timestamp. Therefore, if the bugs were actually accessible (see bug #457782), bots would have to read each bug report each time it crawled the site, resulting in much greater server load than necessary. Perhaps a better solution would be to generate a compressed sitemap.xml to replace the current buglist-*.html files. This has many advantages, one of which is that robots will know which bug reports are stale in its cache and which are still good and, thus, only download bugs reports which have been modified. Additionally, the bandwidth, I/O & CPU load on the server will be reduced, as the compressed file will be smaller than an uncompressed html.
This is 8 years old, please re-open if its an actual problem for you.
