Robots file

Topics

Pages

• Basic
• Instagram robots
• Jekyll

Content

About

A /robots.txt file your domain gives directives for SEO crawlers like GoogleBot on how to crawl your site. Crawlers can be requested to ignore the /admin path or the entire domain for example. Note that directives are suggestions and do not have to be obeyed by crawlers.

Resources

• robotstxt.org homepage.
  • Robots database
• Google
  • Introduction to robots.txt in Google’s Docs.
  • Test your robots.txt with the robots.txt Tester - requires Google account.