Editing Robots.txt (section)

==Security==
Despite the use of the terms ''allow'' and ''disallow'', the protocol is purely advisory and relies on the compliance of the [[web robot]]; it cannot enforce any of what is stated in the file. <ref>{{cite web |title=Block URLs with robots.txt: Learn about robots.txt files |url=https://support.google.com/webmasters/answer/6062608 |access-date=2015-08-10 |archive-url=https://web.archive.org/web/20150814013400/https://support.google.com/webmasters/answer/6062608 |archive-date=2015-08-14 |url-status=live }}</ref> Malicious web robots are unlikely to honor robots.txt; some may even use the robots.txt as a guide to find disallowed links and go straight to them. While this is sometimes claimed to be a security risk,<ref>{{cite web |url=https://www.theregister.co.uk/2015/05/19/robotstxt/ |title=Robots.txt tells hackers the places you don't want them to look |work=The Register |access-date=August 12, 2015 |archive-url=https://web.archive.org/web/20150821063759/http://www.theregister.co.uk/2015/05/19/robotstxt/ |archive-date=2015-08-21 |url-status=live }}</ref> this sort of ''[[security through obscurity]]'' is discouraged by standards bodies. The [[National Institute of Standards and Technology]] (NIST) in the United States specifically recommends against this practice: "System security should not depend on the secrecy of the implementation or its components."<ref>{{cite journal |last1=Scarfone |first1=K. A. |last2=Jansen |first2=W. |last3=Tracy |first3=M. |date=July 2008 |title=Guide to General Server Security |url=http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf |url-status=live |journal=National Institute of Standards and Technology |doi=10.6028/NIST.SP.800-123 |archive-url=https://web.archive.org/web/20111008115412/http://csrc.nist.gov/publications/nistpubs/800-123/SP800-123.pdf |archive-date=2011-10-08 |access-date=August 12, 2015}}</ref> In the context of robots.txt files, security through obscurity is not recommended as a security technique.<ref>{{cite book |author=Sverre H. Huseby |title=Innocent Code: A Security Wake-Up Call for Web Programmers |publisher=John Wiley & Sons |year=2004 |pages=91–92 |isbn=9780470857472 |url=https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92 |access-date=2015-08-12 |archive-url=https://web.archive.org/web/20160401193437/https://books.google.com/books?id=RjVjgPQsKogC&pg=PA92 |archive-date=2016-04-01 |url-status=live }}</ref>