Definition Web Crawler:

A web-crawler (sometimes known as an indexing spider) is a program used by search engines such as (Google, Yahoo and MSN (as well as many others) to index and catalog the world-wide-web. Twenty-four hours a day, seven days a week these web-crawlers follow links from one website to another. When the web-crawler arrives at a new site it analyzes the construction of the website and assuming everything is in compliance the web-crawler then catalogs said website the the search engine's directory.

Close Window