# Crawl rules & sitemap

A step of the [Cloud application](https://gradus.kazet.blog/tree/cloud-application) tree on Gradus, in the Frontend area: Second app, UI, browser tests.

Page: https://gradus.kazet.blog/tree/cloud-application/step/crawl-rules

Decide who may crawl the product, say so in robots.txt, list the public pages in a sitemap and an llms.txt and keep every other environment out of search. llms.txt is a proposed convention no major AI vendor has confirmed reading, so give it little time.

Usually comes after: [Custom domain](https://gradus.kazet.blog/tree/cloud-application/step/custom-domain).

Leads to: [Search engines](https://gradus.kazet.blog/tree/cloud-application/step/search-engines).

## Prompt for your AI assistant

```
Read docs/product/idea.md, docs/product/pitch.md and this product's public routes. List the crawlers that train AI models and the ones that serve search, which use separate user agents, recommend a policy for this product and write it as the next docs/decisions record. Publish robots.txt with that policy and the absolute sitemap URL, generate sitemap.xml from the public routes at build time with only canonical addresses and a lastmod from when each page last changed, send a noindex header on private routes and disallow all crawling in every environment except production. Generate llms.txt from the same docs following the proposal at llmstxt.org: the product name, a one-paragraph summary and links to the pages that explain it best, with nothing private. Add a post-deploy check that all three files answer with their own content types, because a single-page host answers a missing file with the application's HTML and a 200.
```

Every step of the tree: https://gradus.kazet.blog/tree/cloud-application/llms.txt