Editing Robots.txt (section)

=== The use of the wildcard * in rules ===
The directive <code>Disallow: /something/</code> blocks all files and subdirectories starting with <code>/something/</code>.

In contrast using a wildcard, (if supported by the crawler), allows for more complex patterns in specifying paths and files to allow or disallow from crawling, for example <code>Disallow: /something/*/other</code> blocks URLs such as:
<pre>/something/foo/other
/something/bar/other
</pre>

It would not prevent the crawling of <code>/something/foo/else</code>, as that would not match the pattern.

The wildcard <code>*</code> allows greater flexibility but may not be recognized by all crawlers, although it is part of the Robots Exclusion Protocol RFC <ref>{{Cite report |url=https://www.rfc-editor.org/rfc/rfc9309.html#name-special-characters |title=Robots Exclusion Protocol |last=Koster |first=Martijn |last2=Illyes |first2=Gary |last3=Zeller |first3=Henner |last4=Sassman |first4=Lizzi |date=September 2022 |publisher=Internet Engineering Task Force |issue=RFC 9309}}</ref>

A wildcard at the end of a rule in effect does nothing, as that is the standard behaviour.