About Web Robots

Aren't robots bad for the web?
There are a few reasons people believe robots are bad for the Web:

• Certain robot implementations can (and have in the past) overloaded networks and servers. This happens especially with people who are just starting to write a robot; these days there is sufficient information on robots to prevent some of these mistakes.
• Robots are operated by humans, who make mistakes in configuration, or simply don't consider the implications of their actions. This means people need to be careful, and robot authors need to make it difficult for people to make mistakes with bad effects
• Web-wide indexing robots build a central database of documents, which doesn't scale too well to millions of documents on millions of sites.

But at the same time the majority of robots are well designed, professionally operated, cause no problems, and provide a valuable service in the absence of widely deployed better solutions. So no, robots aren't inherently bad, nor inherently brilliant, and need careful attention.

Where do I find out more about robots?

There is a Web robots home page on: http://www.robotstxt.org/wc/robots.html Of course the latest version of this FAQ is there. You'll also find details and an archive of the robots mailing list, which is intended for technical discussions about robots.