How to Translate Shopify Store Robots.txt and Crawl Directives for International Markets
Most Shopify merchants launching into new languages obsess over translation quality and hreflang tags, then ship a robots.txt that silently blocks half their translated URLs from Google. Getting shopify robots.txt multilingual configuration right is unglamorous work, but a single misplaced Disallow rule can undo weeks of translation effort before a single international visitor ever lands on your store.
Why Robots.txt Matters More When You Go Multilingual
When your store serves one language, a misconfigured robots.txt is a problem. When it serves five, it multiplies into five problems — each affecting a separate audience and a separate set of rankings.
Shopify's native multilingual storefront uses language subfolders: your French store lives at /fr/, your German store at /de/, and so on. These paths are real, indexable URLs that Google treats as distinct pages. If your robots.txt blocks /fr/ or any pattern that happens to match those paths, Googlebot stops crawling them entirely — no crawl, no index, no rankings in France.
This matters especially because Shopify's default robots.txt (which you can now customise via robots.txt.liquid) includes several blanket Disallow rules for internal search, cart, and account paths. The risk is that well-intentioned customisations — blocking /search to avoid thin content, for example — can inadvertently match /fr/search, /de/search, and every other locale's equivalent in one line. Understanding Shopify's URL structure for multilingual stores before you edit robots.txt is a prerequisite, not an afterthought.
Understanding Shopify's Default Robots.txt Behaviour
Shopify generates your robots.txt file automatically. Since Shopify 2.0, you can override it by creating a robots.txt.liquid template in your theme. The default file typically disallows:
/admin/cart/orders/checkouts/checkout/carts/*?*(query strings, which catches internal search results)
The query-string rule is the most dangerous for multilingual stores. A rule like Disallow: /*?* blocks URLs with any query parameter — which sounds sensible until you remember that some localisation apps append ?locale=fr or similar parameters. If you're using Shopify's native Markets (which uses path-based subfolders rather than parameters), this specific conflict is less likely, but it's worth auditing your actual URLs before assuming you're safe.
What Shopify does not generate: separate per-locale sitemaps. Despite what you may read elsewhere, Shopify produces a single sitemap.xml that includes all language variants with hreflang annotations embedded — not /fr/sitemap.xml or /de/sitemap.xml as separate files. When you reference your sitemap in robots.txt, there is one URL to list: https://yourdomain.com/sitemap.xml. Listing invented locale-specific sitemap URLs will cause crawlers to return a 404 and simply ignore the directive.
How to Edit robots.txt.liquid for Multilingual Stores
Step 1: Create the Template
If you haven't already, duplicate your existing robots.txt logic into a robots.txt.liquid file via Online Store → Themes → Edit Code → Templates. Shopify's documentation covers the exact steps. Once the file exists, Shopify uses it instead of the auto-generated version.
Step 2: Audit Your Disallow Rules Against Locale Paths
Go through every Disallow rule and ask: does this pattern also match /fr/[pattern], /de/[pattern], and so on?
A rule like Disallow: /search will block /search but — importantly — not /fr/search by default, because path matching is literal. However, a rule like Disallow: /*search* would block /fr/search. Check your rules carefully with this lens.
Step 3: Explicitly Allow Locale Subfolders If Needed
If you have broad rules that might interfere with language paths, add explicit Allow directives above the conflicting Disallow rules. In robots.txt, Allow takes precedence over Disallow when both match and the Allow path is equal or longer. For example:
Allow: /fr/
Allow: /de/
Allow: /es/
Disallow: /*?*
Step 4: Handle Locale-Specific Disallows Manually
A natural question: can you loop through locales dynamically in robots.txt.liquid using Liquid? Shopify's robots.txt.liquid does not expose a native locales array or any shop-level locale object you can iterate. Any per-locale path rules you write are hardcoded strings you maintain yourself — they do not auto-sync when you add or remove a language in Shopify Markets. Build this into your process: whenever you activate a new language, update your robots.txt template at the same time.
Here is what a manually maintained multilingual block looks like:
{% for group in robot_groups %}
User-agent: {{ group.user_agent }}
{% for rule in group.rules %}
{{ rule.directive }}: {{ rule.value }}
{% endfor %}
{% endfor %}
# Locale-specific rules — update manually when adding/removing languages
User-agent: *
Allow: /fr/
Allow: /de/
Allow: /es/
Allow: /nl/
Disallow: /cart
Disallow: /checkout
Sitemap: https://{{ shop.permanent_domain }}/sitemap.xml
Crawl Budget: When Does It Actually Matter?
Crawl budget — the number of URLs Googlebot crawls on your site within a given timeframe — becomes a meaningful concern in specific circumstances. Google's own documentation states that crawl budget is "not something most publishers have to think about" for sites under a few thousand URLs, and that it primarily matters for "large sites" with hundreds of thousands of pages or sites with crawling problems like redirect chains and duplicate content.
For a typical Shopify store, multiplying your product count by your locale count gives you an order-of-magnitude sense of your crawlable universe, but this alone doesn't mean you have a crawl budget problem. Where multilingual stores do encounter real crawl waste is through:
- Faceted navigation URLs (filters, sort orders) across every locale
- Duplicate checkout and account paths that weren't properly disallowed
- Thin or near-duplicate content that Google chooses to crawl but not index
If you're concerned about duplicate content across language versions, the solution is correct hreflang implementation and genuine localisation — not aggressive robots.txt blocking that removes content from the index entirely.
Verifying Your Configuration in Google Search Console
After updating your robots.txt, verification has two layers.
First, use the robots.txt tester inside Google Search Console (Legacy tools → robots.txt Tester) to test specific multilingual URLs against your rules. Enter /fr/collections/summer-sale and confirm Googlebot: Allowed.
Second, monitor the Coverage report for each language's URL patterns. If you see a spike in "Blocked by robots.txt" errors after publishing changes, that's your signal to roll back and re-audit. For detailed guidance on reading these reports for multilingual stores, using Google Search Console to monitor your multilingual store covers the specific filters and views that surface per-locale crawl data.
What Not to Block: Common Multilingual Mistakes
These are the robots.txt errors that most reliably kill international SEO:
- Blocking
/fr/or any locale subfolder root — this removes the entire language from Google's index - Blocking translated collection pages — your collection page SEO investment becomes invisible to crawlers
- Blocking pages with structured data — if Googlebot can't crawl a page, it can't read the schema markup that powers rich snippets in international search results
- Disallowing blog content paths — translated blog content is high-value crawl territory, particularly if you're running pillar pages and topic clusters in multiple languages
Connecting Robots.txt to Your Broader International SEO Stack
Robots.txt is one layer of a larger technical foundation. It works alongside hreflang (which tells Google which language version to serve to which audience — see the complete hreflang guide for Shopify), your sitemap, and your URL structure. Getting these three components consistent with each other is what prevents the confused indexing that shows up as ranking in the wrong country or the wrong language.
If you're earlier in the process of building out your multilingual store, Shopify multilingual SEO: how to rank in every language gives a fuller picture of how these technical elements fit together with content and translation quality.
For the translation side of this work — ensuring every product, collection, and page that you're allowing crawlers to index is actually translated with accurate, on-brand language — StoreLingo handles AI translation across 47 languages with glossary support to keep brand terms consistent, and change detection so only updated content gets re-translated.
Add StoreLingo on the Shopify App Store →
FAQ
Does Shopify generate separate robots.txt files for each language?
No. Shopify generates a single robots.txt file at the root of your domain that applies to all language subfolders. Any locale-specific crawl rules must be written into your robots.txt.liquid template and maintained manually.
Will blocking /search in robots.txt also block /fr/search and other locale search paths?
A literal Disallow: /search matches only that exact path prefix — it blocks /search and /search/results but not /fr/search. However, wildcard rules like Disallow: /*search* will match locale search paths across every language, so test any wildcard rule against your multilingual URL patterns before publishing.
If Googlebot can't crawl my translated pages, will hreflang still work?
No. Hreflang annotations — whether in your sitemap or in page <head> tags — are only processed if Googlebot can actually crawl and render the page. Blocking a locale subfolder in robots.txt means Google neither indexes the content nor processes the hreflang signals pointing to it, which can cause your original-language pages to rank incorrectly in markets you've translated for.
Choose from 47 languages
StoreLingo translates products, collections, pages and articles with AI — review and approve by default. On paid plans from Starter, auto-translate is off by default and publishes new or updated products automatically. Keep your brand terms consistent.
Add to Shopify →