Crawl rules & sitemap
A step of the Cloud application tree on Gradus, in the Frontend area: Second app, UI, browser tests.
Decide who may crawl the product, say so in robots.txt, list the public pages in a sitemap and an llms.txt and keep every other environment out of search. llms.txt is a proposed convention no major AI vendor has confirmed reading, so give it little time.
Usually comes after: Custom domain.
Leads to: Search engines.
Prompt for your AI assistant
Read docs/product/idea.md, docs/product/pitch.md and this product's public routes. List the crawlers that train AI models and the ones that serve search, which use separate user agents, recommend a policy for this product and write it as the next docs/decisions record. Publish robots.txt with that policy and the absolute sitemap URL, generate sitemap.xml from the public routes at build time with only canonical addresses and a lastmod from when each page last changed, send a noindex header on private routes and disallow all crawling in every environment except production. Generate llms.txt from the same docs following the proposal at llmstxt.org: the product name, a one-paragraph summary and links to the pages that explain it best, with nothing private. Add a post-deploy check that all three files answer with their own content types, because a single-page host answers a missing file with the application's HTML and a 200.This is https://gradus.kazet.blog/tree/cloud-application/step/crawl-rules as plain HTML. In a browser the page also lets you mark the step forged and copy a message for your AI assistant.