Alternate solution proposed was to parse robots.txt.

Having an editor for robots.txt inside Composr would not hurt.

7h time estimate added, for making a robots.txt editor, and moving some hard-coded rules in v8 into a default robots.txt. Should be an addon, and installer would need to be very careful to not overwrite an existing robots.txt (so the default rules should probably be stored in PHP code and saved into there via a function call run during installation).

I'm closing this.

A simple robots.txt editor is now implemented for v11, which will make editing robots.txt a little easier.

I thought about what was written here about keeping robots.txt and the XML Sitemap in sync, or about having content options to exclude it via robots.txt and the XML Sitemap. The problem with it is that it assumes a binary - that the content is either not to be crawled by anything, or it is. robots.txt allows specifying which crawlers have access to content, and the XML Sitemap is not specifically for crawlers (it could be used by an HTML Validation tool for example). So it doesn't line up very well with the necessary flexibility of these formats.

Because the implementation is problematic, I think just putting robots.txt in the hands of the user, but helping them to edit it, is the correct approach. Easier for us to do too.