What an answer engine actually fetches from your site

Three consumers, three capabilities

The strategy question — how to get cited when the answer replaces the click — is covered in the AI search optimization guide. This one is about the build: what you actually ship so that the strategy has something to work with. And that starts with being precise about who is fetching.

That third row is the one that changes how you build. When someone asks an assistant a question and it fetches your page to answer, there is no render queue and no second pass. It gets one HTTP response, extracts what it can, and moves on. Whatever is not in that response does not exist.

Test what a non-rendering fetcher sees

One command settles the question. Fetch your page with no JavaScript engine, strip the tags, and count the words:

curl -sL https://yoursite.com/some-page \ | python3 -c "import sys,re,html;s=sys.stdin.read(); s=re.sub(r'(?is)<(script|style|noscript).*?',' ',s); print(len(html.unescape(re.sub('<[^>]+>',' ',s)).split()),'words')" # A 2,000-word article should report roughly 2,000 words. # If it reports 12, everything a reader sees is being drawn by JavaScript.

The failure looks like this in the source: <div id="root"></div> and a script tag. That is the entire page as far as a non-rendering client is concerned. Every word of your content, every heading, every link exists only after a bundle downloads, parses, executes, and fetches data — none of which happens for an answer engine on a two-second budget.

The fix is not clever, it is structural: generate real HTML files. Static site generation, server-side rendering with a real HTML response, or a prerender step in your build. All three produce the same thing that matters — a document with the content already in it. The page you are reading is a generated static file: a Python script holds the content as data, renders the markup and structured data, and writes an HTML file. There is no client-side hydration involved and there is nothing to miss.

Semantic HTML over div soup

Extractors — the readability-style algorithms that strip navigation and boilerplate to find the article — score elements on their tags and structure. Content wrapped in generic containers scores worse than the same content in semantic ones, and in a few cases the meaning is lost entirely rather than merely downgraded.

• One <h1>, then a heading hierarchy that matches the outline. Heading structure is how an extractor segments the page into topics, and it is how a retrieval system decides which section answers a question.
• <main> and <article> around the real content. The cheapest possible signal for "this is the page, the rest is furniture."
• Real <table> markup for tabular data, with <th>. A comparison table built from nested flex divs is not a table to anything but a browser — the row and column relationships are gone, and those relationships were the information.
• Real <ol> and <ul> for lists. A sequence of styled divs is a sequence of paragraphs to an extractor.
• Descriptive link text. "Click here" tells a machine nothing about the destination; the link text is a large part of how the target page's topic is inferred.
• Text in text, not in images. A chart rendered as a PNG with the numbers baked in is invisible. Put the numbers in a table next to it.

llms.txt: cheap, useful, and not a standard

llms.txt is a proposed convention: a Markdown file at your domain root giving language models a curated, machine-readable index of your site. The format is an H1 with the site name, an optional blockquote summary, then H2 sections of links, each with a short description after a colon.

# Your Name > One-paragraph description of who you are and what this site contains. ## Guides - [Choosing an embedding model for RAG](https://site.com/guides/embedding-model-rag.html): dimensions vs cost, recall@k evaluation, when a reranker wins. - [Prompt caching](https://site.com/guides/prompt-caching.html): prefix matching, TTL, break-even math. ## Projects - [DIRA](https://site.com/dira.html): zero-dependency Python security scanner.

A companion llms-full.txt holds the same material expanded — full descriptions rather than one-liners — for a consumer that wants everything in one fetch. Serve both at the root, and copy llms.txt to /.well-known/llms.txt as well, since that is where a second convention looks for it.

Be honest about what this buys you: no major provider has publicly committed to consuming these files, and adoption is uneven. It is cheap insurance rather than a ranking lever. The reason to do it anyway is that it is genuinely useful independent of any crawler — it is a machine-readable index of your own site that you control, and it takes minutes to generate.

The important part is generating it from the same source your pages come from. Two hand-maintained lists diverge within a month. On this site the guide registry is a Python data structure; the pages, the hub, the sitemap, and both llms files are all rendered from it, so a new guide cannot exist on one surface and be missing from another.

robots.txt and the named AI user-agents

AI companies operate several crawlers with distinct names and distinct purposes, and the single most useful thing to understand here is that they are not interchangeable. Blocking the wrong one removes you from citations while doing nothing about training.

So a policy of "opt out of training, stay visible in answers" is expressible, and it looks like this:

# opt out of bulk training crawls User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / # stay retrievable and citable User-agent: OAI-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://yoursite.com/sitemap.xml

robots.txt is a request, not an access control. It is honoured by the major operators and ignored by anything that does not want to honour it. If content genuinely must not be fetched, it needs authentication, not a directive. Related: a stray X-Robots-Tag: noindex response header overrides everything in your robots.txt and your meta tags, silently — check it with curl -sI before debugging anything else.

JSON-LD entity graphs and @id linking

Structured data is how you state facts about a page in a form that needs no inference. The upgrade most sites have not made is going from a pile of disconnected snippets to a linked graph — every node carrying a stable @id, and other nodes referencing that ID instead of repeating the data.

{ "@context": "https://schema.org", "@graph": [ { "@type": "Person", "@id": "https://site.com/#person", "name": "Your Name", "url": "https://site.com/", "sameAs": ["https://github.com/you", "https://linkedin.com/in/you"] }, { "@type": "TechArticle", "@id": "https://site.com/guides/x.html#article", "headline": "...", "author": {"@id": "https://site.com/#person"}, "publisher": {"@id": "https://site.com/#person"} } ] }

Two things are doing work here. First, author is a reference rather than a duplicated object, so there is exactly one Person entity on the entire site and every page points at it — nothing to drift out of sync, and no ambiguity about whether two similar names are the same person. Second, sameAs is the reconciliation mechanism: it says "the entity on this page is the same entity as these external profiles," which is precisely the link a system needs to merge your site's claims with what it already knows from elsewhere.

Pick the type that matches what the page actually is — TechArticle for a technical guide, SoftwareApplication for a tool, FAQPage for questions and answers — and put a BreadcrumbList on anything nested, since that is how position in a hierarchy is communicated when the URL alone is ambiguous.

The FAQ rule that gets sites penalised

Every question and answer in your FAQPage schema must be visible on the page, in text, matching the markup. Not in a collapsed accordion that only populates on click. Not paraphrased. Not schema-only. Structured data that describes content a visitor cannot find is a guidelines violation and a manual-action risk, and it is a surprisingly common mistake because the schema is so easy to write in isolation.

There is a second, more practical reason to mirror it. Plenty of extractors do not parse JSON-LD at all — they read the visible text. A question in schema and nowhere else is invisible to exactly the consumer you were trying to reach. Mirroring it means both paths find the same answer, which is the entire point.

The reliable way to enforce this is to generate both from one source. On this site, each guide's FAQ entries are a list of question-answer pairs in a data file; the generator emits the FAQPage JSON-LD and the visible FAQ cards from that same list in the same pass. They cannot disagree, because there is no second copy to disagree with.

Write pages that survive being chunked

A retrieval system does not read your page, it reads a piece of it. Whatever the chunking strategy, some paragraph of yours will be evaluated on its own with no surrounding context — the same constraint that governs chunking a corpus for RAG, applied to your own writing.

• Lead with the answer. Put the direct response in the first sentence under a heading, then the explanation. A chunk that opens with three sentences of setup is a chunk whose relevance is not apparent.
• Make headings question-shaped where it is natural. Heading text and query text living in the same space is a real retrieval advantage, not a trick.
• Keep paragraphs self-contained. "As mentioned above" is meaningless in a chunk. Repeat the noun instead of using a pronoun that refers three paragraphs back.
• Put numbers, dates, and units in the text. Specific figures are what get quoted; vague claims are not extractable.
• State scope explicitly. "On Apple Silicon" or "for a five-minute cache window" inside the sentence, not established once at the top of a section and assumed thereafter.

What an answer engine cannot see, ever

• Anything rendered by client-side JavaScript, including content fetched after page load.
• Content behind a cookie wall, an age gate, an email capture, or any interstitial.
• Text baked into images, canvas elements, or video, absent a transcript or alt text.
• Content that only appears after scrolling, if loading is genuinely deferred rather than merely hidden by CSS.
• Anything requiring authentication — obviously, and worth stating because staging environments get indexed and production ones get accidentally gated.
• Iframed content, generally, since the fetcher follows one URL and does not chase embeds.
• Anything your robots.txt disallows for that specific user-agent, which is why the training-versus-retrieval distinction above matters so much.

A verification pass you can run in two minutes

1. curl -sL https://site.com/page and count the words after stripping tags. It should approximate the visible article length.
2. curl -sI https://site.com/page and confirm the content type is text/html and there is no X-Robots-Tag: noindex hiding in the headers.
3. Extract every application/ld+json block and run it through a JSON parser. One trailing comma silently invalidates the entire block and nothing warns you.
4. Confirm every question in your FAQPage schema appears verbatim in the page text.
5. Fetch /robots.txt, /llms.txt, /.well-known/llms.txt, and /sitemap.xml and confirm all four return 200 with the right content type.
6. Diff your sitemap's URL list against the files you actually generated. A regeneration that silently drops a section is the classic failure, and it is invisible until traffic disappears.
7. Confirm every internal link resolves to a file that exists. Dead internal links waste crawl budget and break the graph that connects your pages.

All seven are scriptable in about fifty lines, and running them as a pre-deploy gate is worth more than any individual optimisation, because the failure mode here is silent. Nothing errors when your sitemap loses fourteen URLs or a JSON-LD block stops parsing; the pages simply stop being found, and you notice months later. The same argument applies to the rest of what ships with a site — the security checklist covers the automated gates that catch those failures, and DIRA runs the header and TLS half of this list against a live domain.

The part none of this fixes

Structured data, llms.txt, and semantic markup are parsing aids. They make it easy for a machine to understand what you wrote and impossible for it to misfile you. They do not make thin content worth citing. An answer engine cites a page because it contains a specific, checkable claim that answers a specific question — a number, a formula, a command, a tradeoff stated honestly. Everything in this guide exists to make sure that when you have written something worth citing, nothing in your build pipeline is standing in the way.

Tools referenced in this guide

• AI search optimization — the strategy layer above this build guide — entity consolidation, answer-first writing, IndexNow.
• Embedding models for RAG — how retrieval chunks and ranks pages, which is why self-contained paragraphs matter.
• DIRA — a free scanner that checks TLS, security headers, and what your live domain is exposing.
• Apps — the same structured-data pattern applied to product pages.