Editing Robots.txt (section)

===Crawl-delay directive===
The crawl-delay value is supported by some crawlers to throttle their visits to the host. Since this value is not part of the standard, its interpretation is dependent on the crawler reading it. It is used when the multiple burst of visits from bots is slowing down the host. Yandex interprets the value as the number of seconds to wait between subsequent visits.<ref name="yandex-robots"/> Bing defines crawl-delay as the size of a time window (from 1 to 30 seconds) during which BingBot will access a web site only once.<ref name="bing-crawl-delay">{{cite web |url=https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ |title=To crawl or not to crawl, that is BingBot's question |date=3 May 2012 |access-date=9 February 2016 |archive-url=https://web.archive.org/web/20160203142822/https://blogs.bing.com/webmaster/2012/05/03/to-crawl-or-not-to-crawl-that-is-bingbots-question/ |archive-date=2016-02-03 |url-status=live }}</ref> Google ignores this directive,<ref>{{cite web |title=How Google interprets the robots.txt specification |url=https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt |website=Google Search Central |access-date=2024-10-06 |date=2024-05-23}}</ref> but provides an interface in its [[Google Search Console|search console]] for webmasters, to control the [[Googlebot]]'s subsequent visits.<ref>{{cite web |title=Change Googlebot crawl rate - Search Console Help |url=https://support.google.com/webmasters/answer/48620?hl=en |website=support.google.com |access-date=22 October 2018 |language=en |archive-url=https://web.archive.org/web/20181118205747/https://support.google.com/webmasters/answer/48620?hl=en |archive-date=2018-11-18 |url-status=live }}</ref>

<pre>
User-agent: bingbot
Allow: /
Crawl-delay: 10
</pre>