web+crawlers

It sends a request to a web server for a web page, downloading the entire page, then handing it off to Google’s indexer.
==== Googlebot consists of many computers requesting and fetching pages and can request thousands of different pages simultaneously. To avoid overwhelming web servers, or crowding out requests from human users, Googlebot makes requests of each individual web server more slowly than it’s capable of doing. ==== ==== Google rejects those URLs submitted through its Add URL form that it suspects are trying to deceive users by employing tactics such as including hidden text or links on a page now the Add URL form also has a test: it displays some squiggly letters designed to fool automated “ letter-guessers  ”; it asks you to enter the letters you see — something like an eye-chart test to stop spambots. ==== ==== When Googlebot fetches a page, it culls all the links appearing on the page and adds them to a queue for subsequent crawling. Googlebot tends to encounter little spam because most web authors link only to what they believe are high-quality pages. ====