Editing Robots.txt (section)

===Archival sites===
Some web archiving projects ignore robots.txt. [[Archive Team]] uses the file to discover more links, such as [[sitemap]]s.<ref>{{cite web |title=ArchiveBot: Bad behavior |url=https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior |website=wiki.archiveteam.org |publisher=Archive Team |access-date=10 October 2022 |archive-date=10 October 2022 |archive-url=https://web.archive.org/web/20221010034711/https://wiki.archiveteam.org/index.php/ArchiveBot#Bad_behavior |url-status=live }}</ref> Co-founder [[Jason Scott]] said that "unchecked, and left alone, the robots.txt file ensures no mirroring or reference for items that may have general use and meaning beyond the website's context."<ref>{{cite web|url=http://www.archiveteam.org/index.php?title=Robots.txt|publisher=Archive Team|title=Robots.txt is a suicide note|author=Jason Scott|access-date=18 February 2017|archive-url=https://web.archive.org/web/20170218044527/http://www.archiveteam.org/index.php?title=Robots.txt|archive-date=2017-02-18|url-status=live|author-link=Jason Scott}}</ref> In 2017, the [[Internet Archive]] announced that it would stop complying with robots.txt directives.<ref>{{Cite web|url=https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/|title=Robots.txt meant for search engines don't work well for web archives {{!}} Internet Archive Blogs|website=blog.archive.org|date=17 April 2017|language=en-US|access-date=2018-12-01|archive-url=https://web.archive.org/web/20181204130028/http://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/|archive-date=2018-12-04|url-status=live}}</ref><ref name="Verge"/> According to ''[[Digital Trends]]'', this followed widespread use of robots.txt to remove historical sites from search engine results, and contrasted with the nonprofit's aim to archive "snapshots" of the internet as it previously existed.<ref name="Internet Archive">{{cite news |url=https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi |title=The Internet Archive Will Ignore Robots.txt Files to Maintain Accuracy |newspaper=[[Digital Trends]] |first=Brad |last=Jones |date=24 April 2017 |access-date=8 May 2017 |archive-url=https://web.archive.org/web/20170516130029/https://www.digitaltrends.com/computing/internet-archive-robots-txt/#ixzz4gQYOqpUi |archive-date=2017-05-16 |url-status=live }}</ref>