disallow directive

Robots.txt Directives: A Guide to All Standard & Hidden Rules

An in-depth guide to all robots.txt directives. Learn the standard syntax (Allow, Disallow) and lesser-known parameters like Crawl-delay and Yandex's Clean-para