Search engines need to index the massive amount of content that exists online, so they use automated programs to crawl the World Wide Web. Web crawlers identify, analyze, and add information to search engine indexes.
Web crawlers are invaluable tools for indexing content on the Internet. However, they are not equipped to handle technical hurdles posed by dynamic databases and specialized interfaces. For example, databases provide answers based on the queries submitted by users. However, content that can't be accessed and indexed by search crawlers can't be found by search engines, and therefore, appears invisible to the typical search engine user.
With so many Americans using the major search engines as their main entry to Internet content, this is a critical obstacle to better, more complete information. The irony here is that these databases can be easily configured to make their content available to users through search engines.
Previous section: Frustrating Searches | Next section: Robots.txt Files