Google Generation

These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this is not the case, a robot simply visits sites by requesting documents from them. The concept of robots or spiders is very important to understanding search results from today’s Web browsers.

In 1994 Jerry Yang and David Filo created Yahoo!. It started out as a collection of their favorite Web sites, but Yahoo! also contained descriptions of what the user would find on the page. Later in 1994, WebCrawler was introduced, developed by University of Washington researcher Brian Pinkerton. WebCrawler was of major importance in that it did not simply search and index the Internet for document titles; it indexed the entire document and made the entire document available to search. AOL used WebCrawler, its full-text search, and a simple browser-based interface to make the Web fit for mainstream consumption beyond academics and tech geeks. The following quote reveals just how small the Internet was in its infancy: When the Internet was young, when the Web comprised less than 10 million pages, when Yahoo! was a funky set of links and “Google” was just a common misspelling for a very large number, Louis Monier put the entire Web on a single computer.