Editing Robots.txt (section)

==History==
The standard was proposed by [[Martijn Koster]],<ref>{{cite web |url=http://www.greenhills.co.uk/historical.html |title=Historical |website=Greenhills.co.uk |access-date=2017-03-03 |archive-url=https://web.archive.org/web/20170403152037/http://www.greenhills.co.uk/historical.html |archive-date=2017-04-03 |url-status=live }}</ref><ref>{{cite web |title=Maintaining Distributed Hypertext Infostructures: Welcome to MOMspider's Web |first=Roy |last=Fielding |work=First International Conference on the World Wide Web |year=1994 |place=Geneva |url=http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps |access-date=September 25, 2013 |format=PostScript |archive-url=https://web.archive.org/web/20130927093658/http://www94.web.cern.ch/WWW94/PapersWWW94/fielding.ps |archive-date=2013-09-27 |url-status=live }}</ref> when working for [[Nexor]]<ref name=":0">{{cite web |url=http://www.robotstxt.org/orig.html#status |title=The Web Robots Pages |publisher=Robotstxt.org |date=1994-06-30 |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140112090633/http://www.robotstxt.org/orig.html#status |archive-date=2014-01-12 |url-status=live }}</ref> in February 1994<ref>{{cite web|title=Important: Spiders, Robots and Web Wanderers |first=Martijn |last=Koster |work=www-talk mailing list |date=25 February 1994 |url=http://inkdroid.org/tmp/www-talk/4113.html |format=[[Hypermail]] archived message |url-status=dead |archive-url=https://web.archive.org/web/20131029200350/http://inkdroid.org/tmp/www-talk/4113.html |archive-date=October 29, 2013 }}</ref> on the ''www-talk'' mailing list, the main communication channel for WWW-related activities at the time. [[Charles Stross]] claims to have provoked Koster to suggest robots.txt, after he wrote a badly behaved web crawler that inadvertently caused a [[denial-of-service attack]] on Koster's server.<ref>{{cite web |url=http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html |title=How I got here in the end, part five: "things can only get better!" |work=Charlie's Diary |date=19 June 2006 |access-date=19 April 2014 |archive-url=https://web.archive.org/web/20131125220913/http://www.antipope.org/charlie/blog-static/2009/06/how_i_got_here_in_the_end_part_3.html |archive-date=2013-11-25 |url-status=live }}</ref>

The standard, initially RobotsNotWanted.txt, allowed [[web developer]]s to specify which bots should not access their website or which pages bots should not access. The internet was small enough in 1994 to maintain a complete list of all bots; [[Server (computing)|server]] overload was a primary concern. By June 1994 it had become a [[de facto standard|''de facto'' standard]];<ref name="Verge">{{cite web|url=https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders|title=The text file that runs the internet|work=[[The Verge]]|last=Pierce|first=David|date=14 February 2024|accessdate=16 March 2024}}</ref> most complied, including those operated by search engines such as [[WebCrawler]], [[Lycos]], and [[AltaVista]].<ref name="sear_Robo">{{cite web |title=Robots.txt Celebrates 20 Years Of Blocking Search Engines |author=Barry Schwartz |work=Search Engine Land |date=30 June 2014 |access-date=2015-11-19 |url=http://searchengineland.com/robots-txt-celebrates-20-years-blocking-search-engines-195479 |archive-url=https://web.archive.org/web/20150907000430/http://searchengineland.com/robots-txt-celebrates-20-years-blocking-search-engines-195479 |archive-date=2015-09-07 |url-status=live }}</ref>

On July 1, 2019, Google announced the proposal of the Robots Exclusion Protocol as an official standard under [[Internet Engineering Task Force]].<ref>{{Cite web|url=https://webmasters.googleblog.com/2019/07/rep-id.html|title=Formalizing the Robots Exclusion Protocol Specification|website=Official Google Webmaster Central Blog|language=en|access-date=2019-07-10|archive-date=2019-07-10|archive-url=https://web.archive.org/web/20190710060436/https://webmasters.googleblog.com/2019/07/rep-id.html|url-status=live}}</ref> A proposed standard{{Ref RFC|9309}} was published in September 2022 as RFC 9309.