site/templates/robots.txt

User-agent: *
Sitemap: {{ get_url(path="sitemap.xml") }}
Disallow: /formats/*/index.html
Disallow: /formats/index.html
Initial commit 2020-08-11 16:14:38 +02:00			`User-agent: *`
			`Sitemap: {{ get_url(path="sitemap.xml") }}`
Use more refined robots.txt template The previous one possibly also blocks RSS feeds. 2021-12-11 17:44:14 +01:00			`Disallow: /formats/*/index.html`
			`Disallow: /formats/index.html`