Editing Robots.txt (section)

==Alternatives==
Many robots also pass a special [[user-agent]] to the web server when fetching content.<ref>{{cite web |url=http://www.user-agents.org/ |title=List of User-Agents (Spiders, Robots, Browser) |publisher=User-agents.org |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140107154205/http://user-agents.org/ |archive-date=2014-01-07 |url-status=live }}</ref> A web administrator could also configure the server to automatically return failure (or [[Cloaking|pass alternative content]]) when it detects a connection using one of the robots.<ref>{{cite web |url=https://httpd.apache.org/docs/2.2/howto/access.html |title=Access Control - Apache HTTP Server |publisher=Httpd.apache.org |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20131229110831/http://httpd.apache.org/docs/2.2/howto/access.html |archive-date=2013-12-29 |url-status=live }}</ref><ref>{{cite web |url=http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings |title=Deny Strings for Filtering Rules : The Official Microsoft IIS Site |publisher=Iis.net |date=2013-11-06 |access-date=2013-12-29 |archive-url=https://web.archive.org/web/20140101112730/http://www.iis.net/configreference/system.webserver/security/requestfiltering/filteringrules/filteringrule/denystrings |archive-date=2014-01-01 |url-status=live }}</ref>

Some sites, such as [[Google]], host a <code>humans.txt</code> file that displays information meant for humans to read.<ref>{{Cite web |url=https://www.google.com/humans.txt |title=Google humans.txt |access-date=October 3, 2019 |archive-date=January 24, 2017 |archive-url=https://web.archive.org/web/20170124121422/https://www.google.com/humans.txt |url-status=live }}</ref> Some sites such as [[GitHub]] redirect humans.txt to an ''About'' page.<ref>{{Cite web |url=https://github.com/humans.txt |title=Github humans.txt |website=[[GitHub]] |access-date=October 3, 2019 |archive-date=May 30, 2016 |archive-url=https://web.archive.org/web/20160530160942/https://github.com/humans.txt |url-status=live }}</ref>

Previously, Google had a joke file hosted at <code>/killer-robots.txt</code> instructing [[Terminator (character)|the Terminator]] not to kill the company founders [[Larry Page]] and [[Sergey Brin]].<ref>{{Cite web|url=https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html|title=Is This a Google Easter Egg or Proof That Skynet Is Actually Plotting World Domination?|last=Newman|first=Lily Hay|date=2014-07-03|website=Slate Magazine|language=en|access-date=2019-10-03|archive-date=2018-11-18|archive-url=https://web.archive.org/web/20181118104127/https://slate.com/technology/2014/07/a-killer-robots-txt-google-easter-egg.html|url-status=live}}</ref><ref>{{Cite web|url=https://www.google.com/killer-robots.txt|title=/killer-robots.txt|date=2018-01-10|url-status=dead|archive-url=https://web.archive.org/web/20180110160916/https://www.google.com/killer-robots.txt|archive-date=2018-01-10|access-date=2018-05-25}}</ref>