How AI Is Changing Search and SEO

The first time I thought seriously about search engines, I was not thinking about search engines at all. I was trying to get a webpage I had written to exist.

I am a computer science undergraduate at San José State University, on track to graduate in May 2028. I tutor other students in CS, I lead curriculum and workshops for a student club that runs coding sessions for high school students, and I am a co-author on a poster presented at the ACM SIGCSE Technical Symposium 2026 on bilingual coding for inclusive computer science learning. What I am not is a person with an institutional platform. I do not have a lab website that inherits a university domain's authority. I came to the United States as a first-generation immigrant, I am here on a dependent visa with no work authorization, and so the ordinary route by which a student builds a professional footprint — internships, an employer bio page, a company blog byline — is closed to me by law rather than by ability. Everything I have published, I published myself, on a static site I host for free.

That constraint turned out to be an unusually good teacher. When you have no borrowed authority, you find out very quickly what search engines actually reward, because nothing is being handed to you by association. And over the past two years, while I was learning those mechanics, the mechanics themselves changed underneath me. That is the honest starting point for the question of where search is going: I have watched a small corner of it move in real time, from the inside, with no budget.

The thing that broke, and the number that describes it

The deal that built the web was simple enough to state in one sentence. You publish something useful, a search engine indexes it, and in exchange for the free indexing it sends you the reader. Ranking was the mechanism, and the click was the payment.

That deal is being renegotiated, and the numbers are not subtle. SparkToro's clickstream analysis of US Google searches from January through April 2026 found that 68.01% of searches ended without a click, up from 60.45% in 2024. A randomized field experiment covered by Search Engine Journal went further and isolated the mechanism rather than just the correlation: assigning users to see or not see AI Overviews, researchers found that AI Overviews reduced organic clicks by 38% on the queries where they appeared, pushing zero-click rates on those queries from 54% to 72% — while self-reported user satisfaction stayed essentially flat when the summaries were removed.

That last detail is the one I keep returning to, because it is the part most commentary skips. Users were not much happier with the AI answer. They just stopped leaving. The synthesis layer captured the attention without having to be dramatically better, simply by being first and being sufficient. Whatever you think about that outcome, it is not a temporary artifact of a product launch. It is a structural change in where the value of an answer gets realized, and it means the currency of search visibility is shifting from the click to the citation.

What actually changed technically: from matching to retrieval

It is easy to describe this as "AI took over search," which is both true and useless as an explanation. The more precise version is that the unit of the system changed.

Classical search is a matching problem. You build an inverted index mapping terms to documents, you score documents against a query using term statistics and a large set of quality and authority signals, and you return a ranked list. The user does the synthesis. Search engine optimization, in that world, was fundamentally the practice of making a document score well: matching the query vocabulary, earning links that carried authority, and keeping the crawl and render path clean enough that the document was seen at all.

Generative search is a retrieval-and-composition problem. A query gets decomposed — often into several sub-queries the user never typed — each sub-query pulls a set of candidate passages, and a language model composes an answer over those passages. The ranked list still exists, but it has been demoted from the product to an intermediate data structure. What the user receives is a paragraph assembled from fragments of several documents.

This changes the optimization target in a way I think is underappreciated. In a matching system, the atomic unit competing for attention is the page. In a retrieval-and-composition system, the atomic unit is the passage. A model does not cite your site; it cites the two hundred words that answered the sub-question it generated. I have seen this from the other side. When I was building parsing tooling during an unpaid engineering stint at a freight company, the job was extracting structured fields out of messages that people had typed into a chat app in whatever format occurred to them — origin, destination, weight, equipment type, buried in run-on sentences. The system that worked was not the one that understood the message holistically. It was the one that could locate a specific, self-contained span of text that answered a specific field, and abstain when no such span existed. Retrieval systems reward content that is chunkable into self-contained, answer-shaped spans, for the mundane reason that a self-contained span survives being lifted out of its context. That is not an SEO trick. It is a property of how the retrieval works.

What I actually implemented, and what it did and did not do

Because I had no platform, I did the technical work by hand, and I want to describe it precisely rather than gesture at it, because the gap between what structured data is sold as and what it does is where most bad SEO advice lives.

On my own site, every article page carries a canonical URL, an accurate title and meta description, Open Graph tags, and a JSON-LD block using schema.org types — Article for the piece itself, Person for me as the author with a stable @id that every page references, and BreadcrumbList for position in the hierarchy. The Person node includes my ORCID identifier and the DOI of the SIGCSE poster in its sameAs array. I maintain a sitemap, a robots.txt, and I have pinged IndexNow on updates.

Here is what that did: it made my identity resolvable. Before, "Yusuf Gadelrab" was an ambiguous string that a machine had no way to connect across a GitHub profile, an ACM Digital Library entry, an institutional repository record, and a personal site. Afterward there was one identifier with edges pointing at each of those. That is what entity-level structured data is genuinely for. It is disambiguation infrastructure, and for someone with an uncommon name and no institutional page carrying his record, it is the difference between being an entity and being a coincidence of characters.

Here is what it did not do: it did not make the content rank, and it did not conjure authority out of markup. Google has been explicit that there is no schema shortcut into generative results, and in 2026 it went further and used its official documentation to knock down a set of claims the optimization industry had been selling. The most instructive case is llms.txt, a proposed root-level Markdown file meant to give language models a curated index of a site. Google's AI-optimization guidance now states plainly that you do not need to create AI text files or special markup to appear in Google Search, including its generative features, because Search does not use them — the file will "neither harm nor help" rankings. Independent measurement backs that up: one analysis of hundreds of millions of LLM bot traffic events found that the share of requests touching an /llms.txt file was statistically negligible.

I find that episode clarifying, and a little funny, because the same field ran the same play twenty years ago with the meta keywords tag: invent a machine-readable file, declare it the new lever, sell the service of installing it, and discover later that no consumer of the file exists at scale. The pattern to notice is that a new interface generates a new mythology faster than it generates evidence, and the mythology is always cheaper to sell than the underlying work.

The part nobody has solved: who pays for the corpus

The genuinely unresolved question about the future of search is not technical. It is economic, and it is being fought at the infrastructure layer rather than in the SERP.

Cloudflare, which sits in front of a large share of web traffic, reported that by June 2026 AI-related crawlers accounted for 52% of total crawler requests, up from 22% in spring 2025. The imbalance matters more than the raw share: training and agent crawlers consume enormously and return no referral traffic, while traditional search crawlers consume less and historically paid the publisher back in visitors. So Cloudflare began charging. Its Pay Per Crawl mechanism returns an HTTP 402 Payment Required to AI crawlers unless they authenticate and agree to pay, with publishers setting a per-retrieval price and choosing Allow, Charge, or Block per crawler. Then, on July 1, 2026, it announced a shift toward "Pay Per Use," compensating publishers when their content creates value rather than merely when it is fetched, and said that from September 15, 2026 its defaults would block mixed-use crawlers — bots that blend search, training, and agent functions — from ad-carrying pages.

I think that is the most consequential development in search right now, and it gets a fraction of the attention that prompt-engineering advice does. It is an attempt to convert an implicit, collapsing exchange into an explicit, priced one. Whether or not that specific implementation survives, the underlying problem does not go away: a synthesis engine is a derivative product. It requires a healthy supply of original, effortful, verifiable primary sources to synthesize from, and it is currently reducing the returns to producing exactly that. You cannot indefinitely strip-mine a commons whose participants were only ever paid in attention, then remove the attention.

There is a version of the future where this resolves badly, and it is not the science-fiction version. It is boring and gradual. Independent publishing thins out. The remaining reliable corpus consolidates into a few licensed sources with the legal departments to negotiate. Answers become fluent and increasingly homogeneous, drawn from a narrowing base, and the long tail of specific, weird, local, non-English, first-hand knowledge — the actual reason the open web was ever worth searching — quietly stops being written down at all.

Why this is an access question, not just a marketing question

I care about that last sentence for a reason that is not abstract to me.

The poster I co-authored at SIGCSE TS 2026, "Exploring Bilingual Coding for Inclusive Computer Science Learning," studied 60 bilingual participants — 40 with little or no prior programming experience and 20 with experience. I am a co-author, listed seventh; my contribution was a share of the work, not the whole of it, and I would rather say that plainly than inflate it. But the premise stayed with me, because it matches what I see every week tutoring and running workshops for high school students: the barrier is frequently not the concept. It is the notation and the vocabulary. A student who fully understands iteration can be stuck because the keywords, the error messages, and every tutorial they can find are in English, and the phrasing they would naturally use to ask for help is not the phrasing the material was written in.

Search is where that gap becomes concrete. A ranked list of ten links is a lossy interface, but it is a transparent and forgiving one: a student who asks a badly-worded question still gets ten chances to recognize the right answer, and can see that sources disagree. A single synthesized paragraph removes that redundancy. It commits to one framing, in one register, at one reading level, with a fluency that makes it sound settled. If the retrieval was thrown off by the way a bilingual student phrased their question, the failure is invisible — there is no visible second-best result to compare against, no signal that the system was uncertain.

This is also personal in a smaller, more specific way. I have ADHD and I am medicated for it. One thing medication does not do is make it easier to detect when a confident paragraph is subtly wrong; if anything, a fluent, authoritative-sounding answer is precisely the kind of input that is hardest to interrupt yourself to verify. A ten-blue-links page is friction, and friction is annoying, but friction is also where verification happens. I do not think this argues for keeping search worse. I think it argues that legible uncertainty is a first-class product requirement — showing when sources conflict, when coverage is thin, when the system is extrapolating — and that it should be engineered with the same seriousness as answer quality, rather than treated as a UX afterthought that hurts engagement metrics.

What I think actually happens next

Four predictions, held with the confidence I think they deserve.

Optimization moves from the page to the passage, and from ranking to inclusion. The practical work becomes making individual claims retrievable, self-contained, attributable, and dated. Not because a model rewards a formatting trick, but because a passage that stands on its own survives extraction and a passage that depends on three paragraphs of setup does not. This is mostly a rediscovery of good technical writing, which is a slightly embarrassing outcome for an industry that spent two decades on link acquisition.

Verifiable identity becomes a ranking-adjacent asset. When a model is deciding which of forty passages saying similar things to cite, provenance is one of the few discriminating signals available. Clear authorship, stable identifiers, dated claims, and a resolvable connection between an author and a real institutional record are cheap to provide and increasingly load-bearing. This is the one piece of the shift that is genuinely fairer to people without budgets: an ORCID and honest markup cost nothing.

Measurement gets much worse before it gets better. The industry's entire feedback loop assumed a click. When the valuable event is an uncredited mention inside a generated answer, on a surface with no referrer, the instrumentation simply does not exist. Expect several years of confident metrics that measure the wrong thing, and treat any tool promising precise "AI visibility share" with the same skepticism you would apply to a keyword-density score in 2011.

The economics get settled by infrastructure and courts, not by search engines. Pay-per-crawl, pay-per-use, licensing deals, and default-block policies at the CDN layer will determine what the corpus of the future contains far more than any ranking algorithm change. The interesting question for the next five years is not which model writes the best paragraph. It is whether anyone is still being paid to write the sentences the paragraph is made of.

The part I keep coming back to

I got into this by accident and from the bottom. I wanted a page about my work to exist in a form a machine could find, because no employer was going to host one for me, and because the legal category I occupy in this country makes the normal paths unavailable. So I learned canonical tags and JSON-LD and sitemaps at one in the morning, the same way I learned Docker and SQL and API error handling — by needing them for something specific.

What that taught me is that search has always been infrastructure for the distribution of knowledge, and infrastructure is never neutral. Whoever designs it decides whose way of asking a question counts as a valid question, whose page is worth the crawl budget, and whose work gets paid for. AI has made the interface enormously better and made those distributional questions enormously more urgent, because a synthesized answer hides its own choices in a way a list of links never could.

The future of search will not be decided by whose model is largest. It will be decided by whether the systems that now sit between people and knowledge choose to show their work — to cite honestly, to admit uncertainty, to keep the door open for the student who asked the question in the wrong language, and to keep paying, in some form, the people who wrote the thing being summarized. That is an engineering problem and a design problem and an economic problem all at once, which is exactly why it is worth working on.

This essay was submitted for the Search Atlas Scholarship. https://searchatlas.com/scholarship