Editing Robots.txt (section)

==Meta tags and headers==
In addition to root-level robots.txt files, robots exclusion directives can be applied at a more granular level through the use of [[Robots meta tag]]s and X-Robots-Tag HTTP headers. The robots meta tag cannot be used for non-HTML files such as images, text files, or PDF documents. On the other hand, the X-Robots-Tag can be added to non-HTML files by using [[.htaccess]] and [[httpd.conf]] files.<ref name="google-meta">{{cite web |url=https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag |title=Robots meta tag and X-Robots-Tag HTTP header specifications - Webmasters — Google Developers |access-date=2013-08-17 |archive-url=https://web.archive.org/web/20130808020946/https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag |archive-date=2013-08-08 |url-status=live }}</ref>

===A "noindex" meta tag===
<syntaxhighlight lang="html">
<meta name="robots" content="noindex" />
</syntaxhighlight>

===A "noindex" HTTP response header===
<syntaxhighlight lang="html">
X-Robots-Tag: noindex
</syntaxhighlight>

The X-Robots-Tag is only effective after the page has been requested and the server responds, and the robots meta tag is only effective after the page has loaded, whereas robots.txt is effective before the page is requested. Thus if a page is excluded by a robots.txt file, any robots meta tags or X-Robots-Tag headers are effectively ignored because the robot will not see them in the first place.<ref name="google-meta"/>

===Maximum size of a robots.txt file===
The Robots Exclusion Protocol requires crawlers to parse at least 500 kibibytes (512000 bytes) of robots.txt files,{{Ref RFC|9309|section=2.5: Limits}} which Google maintains as a 500 kibibyte file size restriction for robots.txt files.<ref>{{Cite web |title=How Google Interprets the robots.txt Specification {{!}} Documentation |url=https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt |access-date=2022-10-17 |website=Google Developers |language=en |archive-date=2022-10-17 |archive-url=https://web.archive.org/web/20221017101925/https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt |url-status=live }}</ref>