Web and SEO terms, defined without the marketing gloss

A canonical tag is an HTML link element that names the URL a site owner prefers a search engine index as the authoritative version among duplicate or near-duplicate pages.

Duplicate content is normal on a real website. The same product page might be reachable through three different URL parameters, a print view, and both an http and https address before anyone does anything wrong. Without guidance, a search engine has to guess which of those near-identical URLs to treat as the real one and how to divide any ranking signal between them. The canonical tag, a <link rel="canonical" href="..."> element in the page head, is the site owner's way of naming a preference.

The tag consolidates signals rather than hiding pages. Links and engagement pointed at the duplicate URLs get folded into the canonical URL's profile instead of being split across near-identical copies competing with each other. That consolidation is the entire value: it stops a site from diluting its own ranking signal against itself.

Self-referencing canonicals, where a page points to itself, are the default correct state for most URLs and are worth setting explicitly rather than leaving to inference. The failure mode worth watching for is a canonical pointing at the wrong URL by accident, often from a templating bug, which can quietly tell a search engine to ignore a page a site actually wants indexed.

ExampleA store with /shoes?color=red, /shoes?color=blue, and /shoes?sort=price all canonicalize to /shoes, so the plain listing page accumulates the ranking signal instead of splitting it three ways.

Common misconceptionA canonical tag is treated as a directive the search engine must obey. It is a hint. Google states plainly that it may choose a different URL as canonical than the one a site names, based on other signals like redirects, internal linking, and a sitemap entry that disagrees with the tag.

See alsoSchema markup, SERP

Schema markup is structured data added to a web page, using the shared vocabulary at schema.org, that describes what a piece of content is in a machine-readable form rather than relying on a search engine to infer it from prose.

A search engine's crawler can read a page's text, but inferring that a block of numbers is a price, that a name is an author, or that a set of steps is a recipe is imprecise work done at enormous scale. Schema markup removes the inference step by labeling the data directly: this string is a Product name, this number is a price, this array is step instructions, using the shared vocabulary maintained at schema.org.

The markup itself does not appear anywhere a visitor sees it. It lives in the page's underlying code, most commonly as a JSON-LD block, and its only audience is machines: search engine crawlers, and increasingly the retrieval systems behind AI answer engines. Its payoff, when it shows up at all, is a richer search result, an eligibility gate for certain result types, or a cleaner extraction for a model summarizing the page.

Coverage varies a lot by content type. Article, Product, FAQPage, HowTo, Organization, and Person schemas are common and well supported. Marking up content that does not match its actual type, or marking up content invisible to a visitor, violates Google's structured data guidelines and can cost a site the very features the markup was meant to earn.

ExampleA recipe page marked up with Recipe schema, listing prep time, ingredients, and yield as structured fields, is what makes a star-rating snippet or a recipe carousel entry possible in the first place.

Common misconceptionAdding schema markup is assumed to directly boost rankings. It does not, on its own; it is an eligibility mechanism for enhanced result formats and a clarity aid for machine readers, not a ranking factor in itself.

Go deeperStructured data for a static site

See alsoJSON-LD, SERP, Canonical tag

JSON-LD is a JSON-based format for encoding schema markup as a self-contained script block in a page's HTML, which is why it is the format search engines and framework tooling generally recommend over inline microdata.

Structured data can be written three ways: microdata attributes scattered through the visible HTML, RDFa attributes doing something similar, or JSON-LD, a single self-contained block of JSON sitting inside a <script type="application/ld+json"> tag. JSON-LD wins on practicality: it does not touch the visible markup at all, so a template can inject or update it without risking the layout, and a developer can validate the whole block in isolation.

The format follows the JSON-LD specification, a general way of adding linked-data semantics to plain JSON using an @context key to point at a vocabulary, most often https://schema.org, and a @type key to name the entity. Everything else is ordinary JSON: strings, numbers, nested objects, and arrays.

Because the block is separate from the rendered page, JSON-LD is also the practical choice for content generated or hydrated client-side, and for any site that wants one canonical source of structured data per page rather than markup interleaved through a dozen HTML elements. Google's own documentation recommends it as the preferred format for exactly this reason.

Example{"@context":"https://schema.org","@type":"Article","headline":"...","author":{"@type":"Person","name":"..."}} dropped into the page head is a complete, valid JSON-LD block.

Common misconceptionJSON-LD is sometimes assumed to be a different vocabulary from microdata's schema.org markup. It is the same schema.org types and properties in a different container format; the vocabulary does not change, only the syntax carrying it.

Go deeperStructured data for a static site

See alsoSchema markup

A SERP, or search engine results page, is the full page a search engine returns for a query, including organic listings, paid ads, and any enhanced features like a knowledge panel, image carousel, or AI-generated summary.

The acronym predates all the features that now compete for space on it. A SERP used to mean, roughly, ten blue links. It now routinely includes paid ads above and around the organic results, a knowledge panel pulled from a structured data source, image or video carousels, a "people also ask" accordion, local map packs for location queries, and, increasingly, an AI-generated summary sitting above everything else.

This matters for anyone doing SEO because "ranking on page one" no longer describes a fixed shape of outcome. A page can rank first among organic results and still sit visually below three ad slots, a featured snippet, and an AI summary, all of which can satisfy the searcher's need without a single click reaching any website. SERP layout is also query-dependent: a navigational query, a local query, and an informational query each pull a different mix of features.

Because the AI summary sits at the top of many SERPs now and answers the query directly, being cited inside that summary has become a competing goal alongside classic ranking, which is the premise behind answer engine optimization as a distinct discipline from traditional SEO.

ExampleSearching a specific product name can return a SERP with a shopping ad row, an AI overview, three organic listings, and a "people also ask" box, all before the fourth actual blue link.

Common misconceptionSERP position is treated as a single, stable number. The same query can return a different SERP layout, and a different rank, based on location, device, search history, and time of day, which is why rank-tracking tools report an approximation rather than a fact.

See alsoCanonical tag, Answer engine optimization (AEO)

Crawl budget is the number of URLs on a site a search engine's crawler is willing and able to fetch within a given time period, determined by the site's perceived crawl demand and the crawler's own rate-limiting decisions.

A crawler cannot fetch every URL on the internet every day, so it allocates its attention. Crawl budget is the combination of crawl rate limit, how fast a crawler is willing to hit a given host without overloading its server, and crawl demand, how much a search engine actually wants to revisit a site's content based on its perceived value and freshness. Together they cap how many pages get crawled, and how often.

The concept is real and Google documents it, but its practical relevance scales with site size. A site with a few dozen or a few hundred pages is not going to exhaust a search engine's willingness to crawl it; Google has said directly that crawl budget is not something most site owners need to worry about. It becomes an operational concern on sites with hundreds of thousands or millions of URLs, where wasted crawl activity on low-value pages, like faceted-navigation duplicates or infinite calendar pages, can measurably delay a crawler reaching the pages that matter.

The honest framing for a small or mid-sized site is that crawl budget is very unlikely to be the reason a page is not indexed. Slow server response times, thin or duplicate content, a robots.txt block, or the page simply not being linked from anywhere are all far more common and far more fixable causes.

ExampleA large e-commerce site generating millions of filter-combination URLs can burn its crawl budget on pages nobody searches for, leaving genuinely new product pages undiscovered for weeks.

Common misconceptionCrawl budget gets cited as an explanation for a small site's indexing problems. For the overwhelming majority of sites it is not the bottleneck at all, and treating it as one distracts from causes like broken internal linking or accidental robots.txt blocks that actually explain the missing page.

Go deeperStructured data for a static site

See alsorobots.txt, IndexNow

robots.txt is a plain-text file at a site's root that tells well-behaved crawlers which URL paths they may or may not fetch, following the Robots Exclusion Protocol, without any power to remove a URL from a search index.

The file sits at a fixed, predictable location, /robots.txt, and uses a small syntax of User-agent, Disallow, and Allow lines to grant or deny specific crawlers access to specific URL paths. It is a voluntary protocol: a crawler has to choose to read and respect it, which the major search engines do, and which plenty of scrapers simply ignore.

What the file actually controls is fetching, not appearing in results. A crawler that respects a Disallow rule will not request that URL's content at all. That is a narrower power than it sounds like, and the gap between "not crawled" and "not indexed" is where the file's most common misuse happens.

robots.txt is also the conventional home of a sitemap pointer (Sitemap: https://example.com/sitemap.xml), which helps a crawler discover URLs faster even though it has nothing to do with the exclusion rules themselves.

ExampleDisallow: /admin/ tells a compliant crawler never to request anything under /admin/, but it does nothing to remove an already-indexed /admin/login page from search results.

Common misconceptionBlocking a URL in robots.txt is assumed to keep it out of search results. It does not. If that URL is linked from somewhere else on the web, a search engine can still index the URL itself, showing it in results with no title or description, precisely because the crawler was blocked from ever fetching the page to see a noindex tag. The correct way to keep a page out of results is a noindex meta tag or header on a page the crawler is allowed to fetch, or password-protecting it outright.

Go deeperStructured data for a static site

See alsoCrawl budget, IndexNow

IndexNow is a protocol that lets a website push an instant notification to participating search engines whenever a URL is added, updated, or deleted, instead of waiting for the engine's crawler to discover the change on its own schedule.

Traditional crawling is pull-based: a search engine decides when to revisit a URL and finds out about changes whenever it happens to crawl again. IndexNow flips that to push-based. A site pings a simple API endpoint with the changed URL and an API key it hosts to prove ownership, and any participating search engine sharing that same IndexNow network learns about the change immediately, without needing its own separate notification.

The protocol was introduced by Microsoft and Yandex, and it is genuinely supported by Bing, Yandex, Seznam, and Naver, which share submissions across the network so a single ping can reach all of them. That is real, functioning infrastructure, not vaporware.

The gap that matters for anyone deciding whether to bother: Google, the search engine most sites care most about, has stated publicly that it does not use IndexNow. A site can implement it correctly and see faster pickup on Bing while seeing no change at all in Google's crawl behavior, because Google relies on its own discovery and crawling systems instead.

ExampleA publisher pings IndexNow after fixing a factual error in a live article; Bing can reflect the correction within hours, while Google discovers it on its own normal recrawl schedule.

Common misconceptionIndexNow support is assumed to mean faster indexing everywhere, including Google. As of writing, Google has said it does not consume IndexNow submissions, so the protocol's benefit is real but confined to the search engines that actually participate in it.

See alsorobots.txt, Crawl budget

Core Web Vitals are a set of three page-experience metrics, currently LCP, CLS, and INP, that Google measures from real-user visits and publishes thresholds for, using them as one of many inputs into search ranking.

Google introduced Core Web Vitals to convert a vague idea, "this page feels fast and stable to use," into a small number of specific, measurable signals rather than the dozens of raw performance metrics available in a tool like Lighthouse. The current set covers three moments in a page's life: how long the biggest visible element takes to render (LCP), how much the layout jumps around unexpectedly while loading (CLS), and how responsive the page feels when a visitor actually interacts with it (INP).

The metrics are field data, not lab data: Google gathers them from real Chrome users through the Chrome User Experience Report rather than from a single simulated test run, and reports them at the 75th percentile of visits so a handful of great or terrible sessions cannot skew the picture. That percentile choice and the specific pass/fail thresholds are Google's own published guidance, and Google has revised both the metric set and the thresholds before, most notably retiring one of the original three vitals entirely.

Core Web Vitals are also a genuinely small ranking factor. Google has described page experience signals as a tiebreaker among pages of otherwise comparable relevance and quality, not something that overrides thin or poorly matched content. The engineering value of chasing good scores, a faster and more stable page for actual visitors, stands on its own regardless of how much weight a ranking algorithm assigns it.

ExampleA page can pass every Core Web Vitals threshold and still rank behind a slower competitor whose content simply answers the query better, because relevance dominates the ranking decision.

Common misconceptionGood Core Web Vitals scores are treated as a major or guaranteed ranking boost. Google frames them as a minor factor that can help break ties between similarly relevant pages, not a lever capable of outranking better content on its own.

See alsoLargest Contentful Paint (LCP), Cumulative Layout Shift (CLS), Interaction to Next Paint (INP)

Largest Contentful Paint is a Core Web Vitals metric that records the render time of the largest text block or image visible within the viewport during a page load, used as a proxy for how quickly the main content feels available.

Earlier load-speed metrics like "time to first byte" or "first paint" tell you when something appeared on screen, not when the thing a visitor actually came for appeared. LCP targets that gap directly: the browser tracks candidate elements as they render, typically a hero image, a large heading, or a big block of text, and reports the point at which the single largest one finished painting inside the visible viewport.

Because LCP looks at the largest element rather than the first one, it is much harder to game with a fast-loading but irrelevant element, like a tiny logo that paints instantly while the actual hero image lags for seconds behind a slow server response or an unoptimized file.

Google publishes a good threshold of 2.5 seconds or less, measured at the 75th percentile of real-user page loads, with anything past 4 seconds classified as poor and the range between as needing improvement. Those specific cutoffs are Google's own published guidance and are subject to revision, so they are worth reconfirming against current documentation rather than treated as fixed forever.

ExampleA blog post's LCP element is usually its featured image; compressing that one image and serving it at the right dimensions often improves LCP more than any other single change on the page.

Common misconceptionLCP is sometimes assumed to measure full page load. It measures one specific render event, the largest element in the viewport, and a page can have a fast LCP while other elements below the fold or off-screen are still loading.

See alsoCore Web Vitals, Cumulative Layout Shift (CLS), Interaction to Next Paint (INP)

Cumulative Layout Shift is a Core Web Vitals metric that scores unexpected movement of visible page elements during load, calculated by multiplying how much of the viewport shifted by how far it moved, summed across every unstable frame.

The problem CLS measures is a familiar one: a page starts loading, text appears, and then an ad or an image loads in above it and shoves everything down right as a visitor was about to tap something. CLS turns that annoyance into a number by multiplying an impact fraction, the proportion of the viewport affected by the shift, by a distance fraction, how far the affected content moved relative to the viewport, for every unexpected layout shift, and summing the results.

CLS is unitless by design, which is part of why it reads as an odd number the first time someone sees it. It is not seconds, pixels, or a percentage; it is a relative score meant to be compared against Google's published thresholds rather than interpreted on its own.

Google publishes a good threshold of 0.1 or less, with anything above 0.25 classified as poor, again measured at the 75th percentile of real-user visits. As with the other vitals, these thresholds are Google's current published guidance and have been revised before, so they should be reconfirmed against current documentation rather than assumed permanent.

ExampleA cookie-consent banner that injects itself above the page content half a second after the initial render is one of the most common real-world causes of a bad CLS score.

Common misconceptionAll layout movement is treated as equally bad for CLS. The metric explicitly excludes shifts triggered within 500ms of a user interaction, like a menu expanding after a tap, since that motion is expected and desired rather than unexpected.

See alsoCore Web Vitals, Largest Contentful Paint (LCP), Interaction to Next Paint (INP)

Interaction to Next Paint is a Core Web Vitals metric that measures the latency between a user interaction and the browser's next visual update, taking a high percentile across all interactions on a page rather than judging just the first one.

INP replaced First Input Delay (FID) as the third official Core Web Vital in March 2024. FID only measured the delay before the browser began processing the very first interaction on a page, which said nothing about whether the tenth click, five minutes into a session, still felt instant. INP was built to close that gap.

The metric observes every click, tap, and key press on a page for as long as the visitor stays on it, measuring the full time from the interaction to the browser's next rendered frame, not just the moment processing starts. Rather than reporting every single interaction, it takes a high percentile across all of them, which surfaces the interactions that actually frustrated a visitor instead of getting averaged away by a page full of fast ones.

Google publishes a good threshold of 200 milliseconds or less, with anything above 500 milliseconds classified as poor, measured, like the other vitals, at the 75th percentile of real-user visits. Those thresholds are Google's own current published guidance and are subject to revision over time, so treat them as the state of play rather than a permanent constant.

ExampleA page that responds instantly to its first button click but stutters for 600 milliseconds on a filter dropdown deep in a session will show that stutter in its INP score, where FID would have missed it entirely.

Common misconceptionINP is assumed to be a renamed version of FID measuring the same thing. It is a different measurement altogether: full interaction-to-paint latency across the entire page lifecycle at a high percentile, not just the delay before the first interaction begins processing.

See alsoCore Web Vitals, Largest Contentful Paint (LCP), Cumulative Layout Shift (CLS)

Answer engine optimization is the practice of structuring content so it is likely to be retrieved, quoted, and cited inside an AI-generated answer, shifting the goal from earning a click on a blue link to earning a mention inside the answer itself.

Classic SEO optimizes for a ranked list of links a person then chooses to click. AI-powered search results and standalone answer engines change the unit of success: a well-optimized page can now satisfy the searcher's need entirely inside a generated summary, with no click to the source at all, or a small citation link buried among several competitors. AEO is the still-forming set of practices aimed at being one of the sources that summary draws from and credits.

The practical overlap with existing technical SEO is large and worth being honest about: clear structure, direct and quotable factual statements, accurate schema markup, and a crawlable, fast, well-linked site all help both a traditional crawler and a retrieval system behind a language model. Much of what gets marketed as a new "AEO strategy" is technical SEO fundamentals, applied with the awareness that a model is now one of the readers.

What is genuinely new and genuinely unproven is the growing list of speculative tactics, specific formatting tricks, particular phrasing patterns, claimed FAQ-schema shortcuts, that circulate without independent verification that they change citation behavior. Because these systems are opaque and each vendor's retrieval process differs, treat any confident claim of a proven AEO tactic with real skepticism until it is verified against the model provider's own documentation or reproducible testing, not a blog post asserting it works.

ExampleA glossary page that states a definition in one clear, self-contained sentence near the top of the entry is easier for a retrieval system to lift directly than the same fact buried in the third paragraph of a narrative explanation.

Common misconceptionAEO gets sold as a distinct, fully-formed discipline with its own settled best practices. As of writing it is closer to an extension of existing content and technical SEO fundamentals applied to a new kind of reader, with a long tail of unverified tactics attached to the name.

Go deeperAI search optimization for a small site

See alsollms.txt, SERP, Schema markup

llms.txt is a proposed convention, not an adopted standard, for a plain-Markdown file at a site's root that lists and links its most important pages in a format meant to be easy for a language model to consume.

The idea, proposed publicly in 2024, borrows the shape of robots.txt and sitemap.xml: one predictable file at /llms.txt, written in Markdown, giving a short site summary followed by a curated list of links to the pages a site owner considers most important for a language model to read. Some proposals also describe an llms-full.txt variant containing the actual page content inline rather than just links.

The honest status of this file, stated plainly: it is a voluntary convention put forward by one party, not a standard ratified by any search engine or model provider, and there is no confirmed public evidence that any major AI provider's crawler or retrieval pipeline actually reads or acts on it. Some tooling and a number of sites have adopted the format anyway, which creates the appearance of momentum without confirming that anything downstream consumes it.

Publishing an llms.txt file is low-cost and does not conflict with anything else on a site, so there is little downside to adding one. The mistake is treating it as a lever with confirmed effect, the way robots.txt or a sitemap demonstrably is. Until a major provider documents that it consumes the file, its value is speculative, and any claim otherwise should be checked against that provider's own current documentation before being repeated.

ExampleA documentation site publishing /llms.txt with a one-paragraph summary and links to its ten most-referenced pages is a real, cheap experiment, not a confirmed indexing channel.

Common misconceptionllms.txt is described as something AI models "read" the way a browser reads robots.txt. As of writing there is no confirmed evidence that any major language model provider's system consumes the file at all; it remains a proposed convention rather than an adopted standard.

Go deeperAI search optimization for a small site

See alsoAnswer engine optimization (AEO), robots.txt

A progressive web app is a website built with a web app manifest and a service worker so it can be installed to a device's home screen, launch in its own window, and continue functioning offline or on a poor connection.

A PWA is not a separate technology from a normal website; it is a normal website that has opted into a small set of browser APIs that unlock app-like behavior. The two required pieces are a web app manifest, a JSON file declaring the app's name, icons, and launch behavior, and a service worker, a background script that intercepts network requests and can serve cached responses when the network is unavailable.

Meeting those requirements lets a browser offer an install prompt that adds the site to a phone or desktop's home screen or app list, launches it in its own window without browser chrome, and lets it keep working, at least partially, without a live connection. None of that requires an app-store submission or review process, since the app is still just a website being served over HTTPS.

The tradeoff against a native app is real and worth naming honestly: a PWA has more limited access to device hardware and OS-level features than a native app built with the platform's own SDK, and on iOS specifically, installation support and available APIs have historically lagged behind Android. For a tool that is mostly UI and local data rather than deep hardware integration, that limitation rarely matters in practice.

ExampleA habit-tracking tool that installs to a home screen, opens in its own window, and still lets a visitor log an entry with no signal is a PWA doing exactly what the manifest and service worker are for.

Common misconceptionA PWA is assumed to require app-store distribution like a native app. It does not; it installs directly from the browser over HTTPS, with no app-store account, review process, or approval needed.

Go deeperBuilding an offline-first PWA

See alsoService worker

A service worker is a JavaScript file that runs on a separate background thread from the page, able to intercept network requests, serve cached responses, and run tasks like push notifications even when no tab for the site is open.

A normal script on a web page runs on the main thread, tied to that one tab, and stops the moment the tab closes. A service worker is registered separately and runs in its own thread outside any specific page, which is what lets it intercept every network request the site makes, decide whether to serve a cached response or go to the network, and keep running certain tasks in the background even after every tab for the site has closed.

That request-interception ability is what makes offline functionality and reliable caching possible at all: the service worker sits between the page and the network as a programmable proxy, deciding per request whether a cached asset is good enough or a fresh network fetch is required. It is also the mechanism behind background sync and push notifications on the web.

Browsers require HTTPS to register a service worker, with an explicit exception for localhost during development, because a script with this much control over network traffic is a serious target if it can be tampered with over an insecure connection.

The classic failure mode is stale content: a service worker's cache is keyed to a cache name a developer controls, and if that name does not change when the underlying assets do, the service worker keeps serving the old cached version indefinitely, even to visitors who reload the page. The fix is a versioned cache name, bumped on every deploy that changes cached assets, so the service worker's install step recognizes the old cache as stale and replaces it.

ExampleA team ships a CSS fix but forgets to bump the cache version string in their service worker, so returning visitors keep seeing the old broken layout until they manually clear site data.

Common misconceptionA service worker is assumed to update itself automatically whenever the site's files change. It does not; the browser only re-fetches the service worker script periodically and only replaces the cache when the developer changes the cache's version identifier, so an unversioned cache can serve stale assets indefinitely.

Go deeperBuilding an offline-first PWA

See alsoProgressive web app (PWA)