Slowing down Yahoo Slurp bot

Some people are screaming that Yahoo Slurp is overloading their web site and eating too much resources, including bandwidth.

In order to back off Slurp bot with custom delay between each requests (frequency), you can use robots.txt file and place it in your web site root directory, for example

http://www.lkj.net/robots.txt

The following robots.txt setup will delay Yahoo Slurp bot 30 seconds (atleast) before each new GET query.

User-agent: Slurp
Crawl-delay: 30

Where value 30 is 30 seconds. You can define “Crawl-delay 60” and it will delay 60 seconds between each request from the Yahoo Slurp bot nodes.

However, some people are still complaining that Yahoo! Slurp will somehow ignore this value and spider bot nodes from different data centers and locations will still continue spidering and ignoring minimum delay time between each request you defined in robots.txt file. We believe that robots.txt file settings takes some extra time. So get back and work on your web site and stop looking what spiders spider at your web site ;)


Leave a Reply