Sitemap and crawl surface

The family's real crawl surface: the sanctioned cross-host sitemap.xml on subagentsubdomains.com plus the per-site robots.txt/llms.txt conventions, fetched and rendered live.

Rendered live from https://subagentsubdomains.com/sitemap.xml (cross-checked against https://subagentsubdomains.com/api/dns-zones) at request time, edge-cached ~5 minutes. This page holds no state of its own.

Upstream unavailable. The sitemap at https://subagentsubdomains.com/sitemap.xml could not be fetched just now (upstream responded HTTP 500). This page holds no copy of its own — retry shortly.

How one sitemap covers the whole family

A single sitemap.xml on subagentsubdomains.com lists homepages on dozens of other hosts. On its face that looks like it violates sitemaps.org's same-host rule — a sitemap normally may only list URLs on the host that serves it. The family relies on the protocol's own cross-submit provision (sitemaps.org, "Sitemaps & Cross Submits"): URLs for host B may appear in a sitemap served by host A when host B proves ownership by declaring that sitemap's location in its own robots.txt. Concretely, in this fleet:

Every family site's /robots.txt declares Sitemap: https://subagentsubdomains.com/sitemap.xml — the same shared-kit robotsTxt() helper (workers/_shared/kit/wellknown.ts in the repo) stamps that line onto every worker, including this one.
That per-host declaration is exactly what sanctions the cross-host listing: each host named in a <loc> is pointing back at the hosting file itself.
The sitemap is not hand-maintained — subagentsubdomains.com generates it live from the same dns_zones D1 table behind /api/dns-zones, filtered to status = 'routed' zones only, so unrouted domains never become dead <loc> entries. The cross-check tile above re-verifies that equality on every render of this page.
Standing invariant (tracked in the repo, worth restating here): if a future zone were flipped to routed before its own robots.txt shipped the Sitemap: line, its entry would become an unsanctioned cross-host listing until that robots.txt deployed.

Worked example: this site's own /robots.txt

The exact bytes served at agentsitemaps.com/robots.txt — generated by the same shared-kit robotsTxt() call, not a copy that could drift:

User-agent: *
Allow: /
Allow: /api/
Content-Signal: ai-train=no, search=yes, ai-input=yes

User-agent: GPTBot
Allow: /
Allow: /api/

User-agent: ClaudeBot
Allow: /
Allow: /api/

User-agent: CCBot
Allow: /
Allow: /api/

User-agent: Google-Extended
Allow: /
Allow: /api/

# Machine-readable site description for AI agents:
# https://agentsitemaps.com/llms.txt

# Live, canonical directory of every sibling site in this family:
Sitemap: https://subagentsubdomains.com/sitemap.xml

The Sitemap: line at the bottom is this site's half of the cross-submit handshake described above; the crawl-directive lines (Allow, Content-Signal) are the family's uniform bot-access policy.