Indexing robots

How does a robot decide where to visit?

This depends on the robot, each one uses different strategies.
In general they start from a historical list of URLs, especially of documents with many links elsewhere, such as server lists, "What's New" pages, and the most popular sites on the Web.

About Web Robots

Aren't robots bad for the web?
There are a few reasons people believe robots are bad for the Web: