What is robots.txt?
Definition
robots.txt is a plain text file at the root of a website that instructs search engine crawlers which pages or directories to crawl or skip. It uses a simple directive syntax: `User-agent` (which crawler), `Allow`, and `Disallow`. It does not guarantee pages won't be indexed — only that they won't be actively crawled.
Why robots.txt matters for Shopify stores
Shopify auto-generates a robots.txt that blocks `/admin`, `/cart`, `/checkout`, and similar paths. Merchants generally don't need to customise it, but third-party apps or themes occasionally inject directives that accidentally block product or collection pages — a crawl-killing mistake that's difficult to spot.
Shopify example
A Shopify store's robots.txt correctly contains `Disallow: /admin/` and `Disallow: /cart` but shouldn't contain `Disallow: /collections/` — that would block all collection pages from being crawled.
How Sauron handles robots.txt
Sauron's store health check fetches and parses the robots.txt file, flagging any Disallow rules that would prevent crawling of product, collection, or blog pages.
Common mistakes
- ✕ Confusing robots.txt Disallow with noindex — robots.txt prevents crawling, not indexing
- ✕ Accidentally blocking entire sections with overly broad Disallow paths
- ✕ Forgetting to add the sitemap URL at the bottom of robots.txt
- ✕ Thinking robots.txt keeps pages secret — it's publicly readable by anyone
Related terms
Automate your Shopify SEO
Understanding robots.txt is step one. Sauron handles the implementation — automatically, for your entire Shopify store.