BinaryWolf

Yahoo! Search blog

Cool site. A sample of the type of articles found here is Yahoo! Search Tips for Webmasters: Saving Bandwidth.

Crawl-Delay: There’s one last trick you can use to help reduce the bandwidth requirements of your site. You can use a special robots.txt directive, crawl-delay, to reduce the speed at which our crawlers make requests to your site. This allows webmasters to manage their bandwidth without restricting content on their site from crawlers and is being used effectively by sites like Slashdot. A safe value for this would be a delay that would allow us to fetch every page on your site in about five days. So a five second delay (crawl-delay: 5.0) would be fine for a site with 2,000 pages, but not for a site with 100,000 or more.