Search Engine Spiders and Googlebot

Search engines use a variety of methods to find websites—links, user submissions, and domain registrar information all help them discover new sites and new pages. But they all use a technology called a spider to scan these sites and pages for their index.

A search spider is an automated piece of software that acts much like a web browser. The difference is that instead of displaying web pages on a screen, it captures the content of the page into an index. This index is used to drive the search results. If your page mentions widgets, then the index entry for your page will include widgets, and a search for widgets may show your page as one of the results.

Another difference is that typically, spiders will follow every link on a page, building up a network of links both internal to a site and outward to other sites. This network is what drives The Value of a Link.

Spiders have grown in complexity and sophistication in recent years. It used to be that only static text would be captured. Now, spiders such as Google's own Googlebot spider are actually executing and interpreting javascript, and, to some degree, flash content.