Editing Web crawler (section)

==Crawler identification==
Web crawlers typically identify themselves to a Web server by using the [[user agent|User-agent]] field of an [[HTTP]] request. Web site administrators typically examine their [[Web server]]s' log and use the user agent field to determine which crawlers have visited the web server and how often. The user agent field may include a [[Uniform Resource Locator|URL]] where the Web site administrator may find out more information about the crawler. Examining Web server log is tedious task, and therefore some administrators use tools to identify, track and verify Web crawlers. [[Spambots]] and other malicious Web crawlers are unlikely to place identifying information in the user agent field, or they may mask their identity as a browser or other well-known crawler.

Web site administrators prefer Web crawlers to identify themselves so that they can contact the owner if needed. In some cases, crawlers may be accidentally trapped in a [[spider trap|crawler trap]] or they may be overloading a Web server with requests, and the owner needs to stop the crawler. Identification is also useful for administrators that are interested in knowing when they may expect their Web pages to be indexed by a particular [[Web search engine|search engine]].