Public Sites and Private Portals Need Opposite Crawler Policies

In late June I synchronized crawler and discovery surfaces across the x70 projects. The tempting version of that task was a checklist: add a sitemap, open robots.txt, publish structured data, and create llms.txt everywhere.

That would have been simple and wrong.

The ecosystem contains public marketing sites, a personal portfolio, free web tools, authenticated client applications, and an internal Admin. Those surfaces do not want the same kind of discovery.

Start with the audience, not the crawler

A public product site benefits when a search engine can find its real pages, understand the canonical domain, and follow contextual links to related products. Its sitemap can enumerate routes intended for public search. Its structured data can describe the organization, product, article, or FAQ that a visitor actually sees.

An internal dashboard has the opposite job. It should authenticate people, authorize commands, and avoid presenting operational routes as public content. Its crawler policy therefore blocks the application rather than advertising a map of it.

That distinction sounds obvious, but template inheritance makes it easy to miss. When a marketing site and portal live in the same monorepo, copying the public layout metadata into the app can produce an indexable sign-in shell or private route names in a sitemap.

Discovery files have different consumers

robots.txt, sitemaps, canonical tags, structured data, and llms.txt are not interchangeable SEO decorations.

• robots.txt communicates crawl permissions; it is not an authorization mechanism.
• A sitemap should list canonical pages that are intentionally eligible for search, not every URL the server can answer.
• Structured data should match visible content rather than add claims that the page never makes.
• llms.txt is a proposed machine-readable guide, not a web standard or a guarantee that an AI system will cite the site.
• Internal links help people and crawlers understand relationships between public pages.

Google’s official link guidance recommends ordinary anchor elements with resolvable href values and says that important pages should have a link from another page. That supports the x70 ecosystem directory as navigation. It does not prove a ranking improvement.

Canonical URLs have to follow production

Each sibling repository can be deployed to a generated Vercel domain before its public domain is attached. If metadata reads the wrong environment value, a perfectly valid build can announce the preview hostname as canonical.

The reliable sequence is operational:

1. configure the public domain;
2. deploy the source that knows its production hostname;
3. inspect the rendered canonical, Open Graph URL, robots output, and sitemap;
4. fetch representative routes from the public domain.

Source code can show intent. Only the deployed HTML shows what a crawler receives.

Shared organization, distinct site purpose

The public sites all belong to the same x70 ecosystem, so their organization identity and cross-links can come from shared sources. Their page schemas and route inventories remain local.

Skirack describes a physical-product project. Tools lists browser utilities. The Portfolio describes my work. Social describes a publishing and discussion product. Replacing those meanings with one generic Organization block would make the markup consistent but less accurate.

The same principle applies to titles. A shared format can enforce the middle-dot separator while each page still names its own purpose.

No indexing claim without evidence

Publishing a sitemap does not prove that Google indexed the pages. Allowing an AI crawler does not prove that a model used the content. Adding JSON-LD does not award a rich result.

Those are remote outcomes and have to be checked separately in Search Console, crawl logs, or the live product that consumes the data.

The work completed in June was narrower and still worthwhile: the maintained x70 surfaces then in scope had a clearer statement of whether they were public, what their canonical routes were, and how they related to the rest of the ecosystem.

Good discoverability begins by deciding what should be discovered.