Editing Robots.txt (section)

==Compliance==
The robots.txt protocol is widely complied with by bot operators.<ref name="Verge"/> <!--It entered the court as part of ''[[eBay v. Bidder's Edge]]'',<ref name=":1">{{Cite news |last= |first= |date=2000-07-31 |title=EBay Fights Spiders on the Web |url=https://www.wired.com/2000/07/ebay-fights-spiders-on-the-web/ |access-date=2024-08-02 |work=[[Wired (magazine)|Wired]] |language=en-US |issn=1059-1028}}</ref> where eBay attempted to block a bot, and the company operating the crawler was ordered to stop crawling eBay's servers using any automatic means, by [[Injunction|legal injunction]]  the basis of [[Trespass to chattels|trespassing]].<ref name="case">{{cite court|litigants=eBay v. Bidder's Edge|vol=100|reporter=F. Supp. 2d|opinion=1058|pinpoint=|court=[[N.D. Cal.]]|date=2000|quote=|url=http://www.cand.uscourts.gov/cand/tentrule.nsf/3979517dd11390ce8825690a007c1b9e/d0fc1406324de0cd882568e90081ebf4/$FILE/Ebay.pdf|archive-url=https://web.archive.org/web/20000817173849/http://www.cand.uscourts.gov/cand/tentrule.nsf/3979517dd11390ce8825690a007c1b9e/d0fc1406324de0cd882568e90081ebf4/$FILE/Ebay.pdf|url-status=dead|accessdate=2000-08-17}}</ref><ref>{{Cite web |last=Hoffmann |first=Jay |date=2020-09-15 |title=Chapter 4: Search |url=https://thehistoryoftheweb.com/book/search/ |access-date=2024-08-02 |website=The History of the Web |language=en-US}}</ref><ref name=":1" />-->

===Search engines===
Some major search [[Facebook Paper|engines]] following this standard include Ask,<ref name="ask-webmasters">{{cite web |title=About Ask.com: Webmasters |url=http://about.ask.com/docs/about/webmasters.shtml |website=About.ask.com |access-date=16 February 2013 |archive-date=27 January 2013 |archive-url=https://web.archive.org/web/20130127134025/http://about.ask.com/docs/about/webmasters.shtml |url-status=live }}</ref> AOL,<ref name="about-aol-search">{{cite web |title=About AOL Search |url=http://search.aol.com/aol/about |website=Search.aol.com |access-date=16 February 2013 |archive-date=13 December 2012 |archive-url=https://web.archive.org/web/20121213134546/http://search.aol.com/aol/about |url-status=dead }}</ref> Baidu,<ref name="baidu-spider">{{cite web |title=Baiduspider |url=http://www.baidu.com/search/spider_english.html |website=Baidu.com |access-date=16 February 2013 |archive-date=6 August 2013 |archive-url=https://web.archive.org/web/20130806131031/http://www.baidu.com/search/spider_english.html |url-status=live }}</ref> Bing,<ref name="bing-blog-robots">{{cite web|url=https://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/|title=Robots Exclusion Protocol: joining together to provide better documentation|website=Blogs.bing.com|date=3 June 2008 |archive-url=https://web.archive.org/web/20140818025412/http://blogs.bing.com/webmaster/2008/06/03/robots-exclusion-protocol-joining-together-to-provide-better-documentation/|archive-date=2014-08-18|url-status=live|access-date=16 February 2013}}</ref>  DuckDuckGo,<ref name="duckduckgo-bot">{{cite web|url=https://duckduckgo.com/duckduckbot|website=DuckDuckGo.com|title=DuckDuckGo Bot|access-date=25 April 2017|archive-date=16 February 2017|archive-url=https://web.archive.org/web/20170216043103/https://duckduckgo.com/duckduckbot|url-status=live}}</ref> Kagi,<ref name="KagiBot">{{cite web|url=https://kagi.com/bot|website=Kagi Search|title=Kagi Search KagiBot|access-date=20 November 2024|archive-date=12 April 2024|archive-url=https://web.archive.org/web/20240412192855/https://kagi.com/bot|url-status=live}}</ref> Google,<ref name="google-webmasters-spec">{{cite web |url=https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt |title=Webmasters: Robots.txt Specifications |work=Google Developers |access-date=16 February 2013 |archive-url=https://web.archive.org/web/20130115214137/https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt |archive-date=2013-01-15 |url-status=live }}</ref> Yahoo!,<ref name="yahoo-search-is-bing">{{cite web |url=http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true |title=Submitting your website to Yahoo! Search |access-date=16 February 2013 |archive-url=https://web.archive.org/web/20130121035801/http://help.yahoo.com/kb/index?page=content&y=PROD_SRCH&locale=en_US&id=SLN2217&impressions=true |archive-date=2013-01-21 |url-status=live }}</ref> and Yandex.<ref name="yandex-robots">{{cite web |url=http://help.yandex.com/webmaster/?id=1113851 |title=Using robots.txt |website=Help.yandex.com |access-date=16 February 2013 |archive-url=https://web.archive.org/web/20130125040017/http://help.yandex.com/webmaster/?id=1113851 |archive-date=2013-01-25 |url-status=live }}</ref>

===Archival sites===
Some web archiving projects ignore robots.txt. [[Archive Team]] uses the file to discover more links, such as [[sitemap]]s.<ref>{{cite web |title=ArchiveBot: Bad behavior |url=https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior |website=wiki.archiveteam.org |publisher=Archive Team |access-date=10 October 2022 |archive-date=10 October 2022 |archive-url=https://web.archive.org/web/20221010034711/https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior |url-status=live }}</ref> Co-founder [[Jason Scott]] said that "unchecked, and left alone, the robots.txt file ensures no mirroring or reference for items that may have general use and meaning beyond the website's context."<ref>{{cite web|url=http://www.archiveteam.org/index.php?title=Robots.txt|publisher=Archive Team|title=Robots.txt is a suicide note|author=Jason Scott|access-date=18 February 2017|archive-url=https://web.archive.org/web/20170218044527/http://www.archiveteam.org/index.php?title=Robots.txt|archive-date=2017-02-18|url-status=live|author-link=Jason Scott}}</ref> In 2017, the [[Internet Archive]] announced that it would stop complying with robots.txt directives.<ref>{{Cite web|url=https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/|title=Robots.txt meant for search engines don't work well for web archives {{!}} Internet Archive Blogs|website=blog.archive.org|date=17 April 2017|language=en-US|access-date=2018-12-01|archive-url=https://web.archive.org/web/20181204130028/http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/|archive-date=2018-12-04|url-status=live}}</ref><ref name="Verge"/> According to ''[[Digital Trends]]'', this followed widespread use of robots.txt to remove historical sites from search engine results, and contrasted with the nonprofit's aim to archive "snapshots" of the internet as it previously existed.<ref name="Internet Archive">{{cite news |url=https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi |title=The Internet Archive Will Ignore Robots.txt Files to Maintain Accuracy |newspaper=[[Digital Trends]] |first=Brad |last=Jones |date=24 April 2017 |access-date=8 May 2017 |archive-url=https://web.archive.org/web/20170516130029/https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi |archive-date=2017-05-16 |url-status=live }}</ref>

===Artificial intelligence===
Starting in the 2020s, web operators began using robots.txt to deny access to bots collecting training data for [[generative artificial intelligence|generative AI]]. In 2023, Originality.AI found that 306 of the thousand most-visited websites blocked [[OpenAI]]'s GPTBot in their robots.txt file and 85 blocked [[Google]]'s Google-Extended. Many robots.txt files named GPTBot as the only bot explicitly disallowed on all pages. Denying access to GPTBot was common among news websites such as the [[BBC]] and ''[[The New York Times]]''. In 2023, blog host [[Medium (website)|Medium]] announced it would deny access to all artificial intelligence web crawlers as "AI companies have leached value from writers in order to spam Internet readers".<ref name="Verge"/>

GPTBot complies with the robots.txt standard and gives advice to web operators about how to disallow it, but ''[[The Verge]]''{{'}}s David Pierce said this only began after "training the underlying models that made it so powerful". Also, some bots are used both for search engines and artificial intelligence, and it may be impossible to block only one of these options.<ref name="Verge"/> ''[[404 Media]]'' reported that companies like [[Anthropic]] and [[Perplexity.ai]] circumvented robots.txt by renaming or spinning up new scrapers to replace the ones that appeared on popular [[blocklist]]s.<ref>{{Cite web |last=Koebler |first=Jason |date=2024-07-29 |title=Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones) |url=https://www.404media.co/websites-are-blocking-the-wrong-ai-scrapers-because-ai-companies-keep-making-new-ones/ |access-date=2024-07-29 |website=404 Media}}</ref>