The agentic web is the layer of the internet where AI agents, acting on behalf of humans, discover, read, and transact with websites. It exists alongside the human web and is measured separately.
For most of the internet's history, two kinds of visitor showed up at a website: people, and crawlers building an index. Agents are a new class, and the user-triggered fetcher that sits between crawler and agent is another. An agent is sent by a human with a task, runs autonomously on the user's behalf, and performs multi-step actions. Checking availability. Filling a form. Comparing prices. Completing a purchase. Agents read websites the way a crawler does and act on them the way a user does. That combination is new.
When I want to know what a standard actually does, I go and ask the websites that signed it.
The agentic web is the whole subject of No Hacks. Nobody asked us whether we wanted the internet changed, so I check what AI is actually doing to it and say which parts are good, which are bad, and which are pointless. It is also the single term I find myself explaining most often. So here it is, end to end: what it means, when it arrived, and what it changes.
Updated September 13, 2026: Rebuilt as a reference. Added a dated timeline of the moments that built the agentic web, a glossary, and the visitor-class and category tables. Extended the timeline back to the 1994 Robots Exclusion Standard and the 2023 arrival of named AI fetchers and opt-outs. Cut the rows and figures that were news rather than definition, including the AI-referral traffic numbers, which measure people arriving from AI answers rather than agents acting on their own. Corrected the browser API reference to
document.modelContext.Initial publication: April 19, 2026. Updated as protocols are released, access defaults change, and the courts rule.
GET WEEKLY WEB STRATEGY TIPS FOR THE AI AGE
Practical strategies for making your website work for AI agents and the humans using AI to find you. Once a week you get the new articles, the latest podcast episode, and a few links worth keeping.
Contents
- Agents are a new primary visitor class
- Glossary of agentic web terms
- The agentic web arrived in dated moments, from robots.txt to the browser tool APIs
- Why agent checkout arrived faster than agent-to-agent
- How the agentic web differs from AI search and answer engine optimization
- Machine-First Architecture defines how to build for the agentic web
- Publishers lose referrals, developers get a new API surface, businesses get a second funnel
- The agentic web in one paragraph
Agents are a new primary visitor class
Four visitor classes read websites as of September 2026: humans, crawlers, user-triggered fetchers, and agents. Humans load pages in browsers. Crawlers fetch pages to build an index or a training set. A user-triggered fetcher such as ChatGPT-User or Claude-User fetches one page because a person asked an assistant about it. Agents do both and more, loading pages to extract information and to perform actions on the user's behalf.
| Visitor class | What it does | Executes your JavaScript | Can complete an action |
|---|---|---|---|
| Human | Loads pages in a browser, reads, decides, acts | Yes | Yes |
| Crawler | Fetches pages to build an index or a training set | Usually not | No |
| User-triggered fetcher | Fetches a specific page because a person asked an assistant | Rarely | No |
| Agent | Reads to extract, then acts on the user's behalf | Depends on the agent | Yes |
An agent visiting a retail website might query a product catalog for a user's specification, compare options across listings, authenticate through an OAuth flow, add items to a cart, and complete a checkout. An agent visiting a publication might extract the current article, summarize it alongside other sources, and return a synthesized answer to the user without the user ever loading the page. Each behavior is agentic web traffic. The retail behavior generates revenue. The publication behavior rarely sends referral traffic back. The asymmetry between agent-driven purchases and agent-driven reading is one reason the agentic web's effects are distributed unevenly across sectors.
HUMAN Security's 2026 State of AI Traffic report put the growth of automated traffic during 2025 at roughly eight times the growth of human traffic, as reported by CNBC. For the agent share specifically, the Q2 2026 earnings calls are the clearest public numbers. Etsy's CEO said agentic experiences were still less than 1% of the company's overall traffic. DoorDash's CEO said the volume from the agentic partners it has been testing with is quite low.
Those figures get quoted alongside the much larger growth in traffic arriving from AI answers, and the two get merged into one claim about the agentic web. They are different visitors. A person who clicked through from an AI answer is a person. An agent acting on its own is not, and that is the one still barely arriving.
Glossary of agentic web terms
Agent. Software sent by a person to complete a task, which reads websites and performs multi-step actions on that person's behalf.
Agentic web. The layer of the internet where AI agents, acting on behalf of humans, discover, read, and transact with websites. Comprises the traffic class, the infrastructure serving it, and the protocols governing it.
A2A. A standard for agents belonging to different organisations to find each other and deal across a boundary, as distinct from an agent reaching for its own tools. It complements Model Context Protocol.
Agent card. A small file an agent publishes describing what it is, what it can do, and how to reach it. A2A requires one and names a well-known location on a website as one way to find it.
AI search. Consumer products that retrieve from the web and synthesize an answer. A subset of agentic web activity.
Answer engine optimization. Making content citable and accurately quotable by AI answer systems.
Crawler. Automated software that fetches pages to build an index or a training corpus. Reads, does not act.
User-triggered fetcher. A crawler that fetches one page because a specific person asked an assistant about it. Sits between crawler and agent.
Machine-First Architecture. The framework for building websites agents can read and act on. Four pillars: Identity, Structure, Content, Interaction.
WebMCP. A proposed browser API, incubating in a W3C community group, that lets a website register tools an agent can call directly, as an alternative to the agent parsing the page.
Multi-purpose crawler. A bot that both indexes for search and collects for AI training. Googlebot and Applebot do both, and each offers a separate token so a website can refuse the training half and stay in search. Which crawler does what is recorded in the user-agent landscape.
The agentic web arrived in dated moments, from robots.txt to the browser tool APIs
The agentic web has no single start date. The Robots Exclusion Standard it still runs on dates to 1994, and the agents arrived thirty years later. For most of the three decades in between, the only machine visitor worth designing for was a search crawler, so the interface did not have to change. That is why the arrival felt sudden. The scraping cases that predate it, and the ones that run alongside it, are covered in what courts say about blocking AI bots. What exists is a sequence: the agents arrived, the web reacted, and the courts and the infrastructure are still deciding the terms.
| Date | What happened | Why it mattered |
|---|---|---|
| Jun 1994 | Robot authors agree the Robots Exclusion Standard | The first interface a website has for telling a machine visitor what it may do. Most AI-specific controls since have been new tokens inside that same file, which became RFC 9309 in 2022 |
| Mar 2023 | OpenAI gives ChatGPT a browser and says its user-agent token is ChatGPT-User, "configured to honor websites' robots.txt files" | Live retrieval had already arrived with Bing Chat six weeks earlier. What was new was the name: a website could now see the fetch in its logs and decide about it |
| Aug 2023 | OpenAI documents GPTBot | The token that turned a rumour about AI reading the web into a line in a config file. Earlier AI crawlers existed and were documented, but this is the one website owners started writing rules for |
| Sep 2023 | Microsoft gives Bing Chat its own opt-out | Content tagged NOARCHIVE stays out of Bing Chat answers and "will still appear in our search results". The first time a search company separated the AI answer from the search result |
| Jul 2024 | Cloudflare launches one-click AI bot blocking | The first mass expression of "no" from the website side, and the first time refusing AI crawlers took one click rather than a config file |
| Oct 2024 | Anthropic launches Computer Use in public beta | Anthropic described it as the first frontier model to offer computer use in public beta. A model that could drive a browser rather than describe one |
| Nov 2024 | Anthropic releases Model Context Protocol | The first broadly adopted way to give a model tools. Model Context Protocol is the one of these protocols a website owner is most likely to meet |
| Jan 2025 | OpenAI launches Operator | The first consumer agent aimed squarely at completing web tasks. It was folded into a successor and withdrawn seven months later, which is the pattern below |
| Jul 2025 | Perplexity launches Comet | An agentic browser as a consumer product, not a research preview |
| Nov 2025 | Amazon sues Perplexity under the CFAA | The case asking whether an agent visiting a website is the user or the vendor. The Ninth Circuit answered in August 2026: it is the user who accesses the website |
| Jan 2026 | Google announces Universal Commerce Protocol with Shopify, Etsy, Wayfair, Target and Walmart | Named merchants and one platform committed to a shared agent-checkout standard. Checkout went live the following month with two of them, Etsy and Wayfair |
| Feb 2026 | WebMCP appears in Chrome Canary | Websites can register tools an agent calls directly. A second interface to the same page |
| Jun 2026 | WebMCP enters origin trial in Chrome 149 | The proposal leaves the flag. Edge 150 follows, and in both browsers it stays behind a trial |
Operator was announced, launched and withdrawn inside a year, the withdrawal arriving after the capability had been described as working, and it is not the only one. A claim about what agents can do should be read against that record rather than against the announcement.
Why agent checkout arrived faster than agent-to-agent
Universal Commerce Protocol went from announcement to live checkout in one month. A2A was published nine months earlier and mostly has names rather than traffic. Both needed parties who do not control each other, so the difference is not how many had to agree.
Coordination has to happen somewhere, and who does it decides the speed. That also explains the wider gap: platforms keep adding agent-facing surfaces by default, browsers keep theirs behind trials, and Etsy told investors agentic experiences were still under 1% of its overall traffic.
Model Context Protocol needed nobody's permission. You install a server, your own assistant uses it, and the value arrives that afternoon whether or not anyone else has bothered. One party, an immediate payoff, and adoption follows.
Universal Commerce Protocol needed a merchant, a platform and a payment network to move together, which sounds slower and was not. Google did the assembling privately and announced the standard with Shopify, Etsy, Wayfair, Target and Walmart already named. Checkout was live the following month with two of the five, Etsy and Wayfair. The coordination happened before the announcement rather than after it.
A2A is the case where nobody did the assembling. It is the standard for one organisation's agent to deal with another's, and it pays nothing until a counterparty you do not control has also done the work. A long list of companies put their names to it. In September 2026 I took that partner list from the project's own website, went to the address each company publishes, and asked each one for /.well-known/agent-card.json, the location the specification names. Almost none answered. That is not proof they have not implemented it, since a card can sit behind authentication or on a per-customer instance rather than at a public address. What it does show is that signing a standard and running it are different things.
So what is reaching websites today is single-player. An agent goes out on one person's behalf, and the things it meets answer without wanting anything back. That is a web of tools and pages rather than a market of counterparties.
So the question when a new protocol is announced is whether anyone has already got the counterparties in a room. A standard published for the industry to adopt, like A2A, waits. A standard published with its counterparties already signed, like Universal Commerce Protocol, does not. And a browser API that every website has to implement one at a time, like WebMCP, moves at the speed of websites, which is slow whatever the specification says.
None of that means the demand side stays small. It means the order is predictable, and that a website owner has time to do the work while the volume is low.
How the agentic web differs from AI search and answer engine optimization
The agentic web, AI search, answer engine optimization and Machine-First Architecture are four different things: the territory, one subset of it, one discipline inside that subset, and the build-side response.
| Term | What it covers | Relationship to the agentic web |
|---|---|---|
| Agentic web | The traffic, infrastructure and protocols for agents acting on websites for humans | The whole territory |
| AI search | Consumer products that retrieve and synthesize: ChatGPT search, Perplexity, Google AI Mode | One subset of agentic web activity |
| Answer engine optimization | The practice of making content citable and accurate in AI answers | One discipline inside the citation behavior |
| Machine-First Architecture | The framework for building a website agents can read and act on | The build-side response |
AI search refers to search products powered by large language models. AI search is a consumer product that retrieves and synthesizes. The agentic web is broader. It includes AI search agents visiting websites, and it also includes transactional agents, booking agents, research agents, and custom agents built on top of APIs and browser runtimes. Other agent categories operate entirely outside search.
Answer engine optimization is the SEO-adjacent discipline of optimizing content so AI search systems cite it accurately. It is a specific practice within the broader context of the agentic web, and it is the front door rather than the whole building: the same machine that cites you will also buy from you, get charged to read you, and create legal exposure. The No Hacks guide to Answer Engine Optimization and the No Hacks primer on moving from SEO to answer engine optimization cover the practical side.
Machine-First Architecture is the specific framework that structures the build-side work.
Machine-First Architecture defines how to build for the agentic web
Machine-First Architecture has four pillars: Identity, Structure, Content, and Interaction. I introduced it because the existing frameworks for making websites work for AI agents were either too general (SEO) or too narrow (schema.org). The pillars are what I test every website against. Episode 221 of the No Hacks podcast introduces them in detail.
Identity
A website in the agentic web needs unambiguous machine-readable identity. An agent has to resolve who the website is, what it sells or publishes, and which authoritative source it represents. In practice that means canonical URLs, consistent entity naming across pages and off-website, verified presence on the platforms agents query, and cryptographic signals where applicable. An agent that cannot resolve a website's identity confidently falls back to pattern-matching, and pattern-matching loses to competitors with clearer identity signals.
Structure
Critical content must not depend on client-side JavaScript execution to become visible. Whether an agent executes JavaScript depends on the agent, and the ones that do not are the reason this pillar exists at all. Structured data, server-side rendering, and semantic HTML all fall under this pillar. The lesson from mobile-first indexing applies: infrastructure that depends on fragile rendering is the first thing to fail when a new visitor class arrives.
Content
An agent consumes content as answer-units. An agent extracts the sentence or paragraph that answers the user's question, frequently without surrounding context. This pillar covers answer-first architecture, citable specificity, provenance signals, and temporal markers. The working rule: any sentence should survive extraction standalone. An agent quoting it should not need the surrounding paragraphs to make the quote accurate. The No Hacks guide to how AI agents see your website walks through this.
Interaction
Agents do not only read, they act. This pillar defines how an agent completes a task: what actions the website exposes, how workflows recover from errors, and how an agent's identity and permissions are verified. Interaction is the fastest-moving of the four pillars. WebMCP lets websites register structured tools an agent can call directly. Universal Commerce Protocol standardizes agent checkout. MCP, A2A, NLWeb, and AGENTS.md cover the other protocols in this layer.
Publishers lose referrals, developers get a new API surface, businesses get a second funnel
Google search referrals to publishers fell roughly a third globally in the year to November 2025, while merchants gained an agent-facing interface they did not ask for and mostly did not build. One shift produces three different problems.
Publishers. That decline is the publisher half of the shift, and in the US organic Google search referrals fell 38% year over year (Press Gazette, reporting Chartbeat data from the Reuters Institute). The agent layer reads publisher content and synthesizes it directly, often without returning a user to the source page. Display-ad, affiliate, and page-view monetization compress in parallel. Page-view economics are thinning structurally rather than temporarily, so the revenue mix has to move toward subscriptions, licensing deals with AI labs, and direct audience relationships.
Developers. A new API surface is active. The browser API for registering agent-callable tools now lives at document.modelContext. Chrome's own documentation uses document.modelContext. Earlier material used navigator.modelContext, so feature-detect rather than assuming either. (For the broader inventory of agentic browsers, automation frameworks, and enterprise APIs, see The Agentic Browser Landscape in 2026.) Model Context Protocol servers, OAuth flows for agents, and agent identity verification layers are live infrastructure now. The primitives are cheap to learn now and expensive to retrofit later. Three costs surface in practice: inference cost per agent task, authentication flows, and error recovery for multi-step actions.
Businesses with transactional websites. The agent share of traffic is small today and the interface to serve it is arriving by default on the platforms most merchants already sell through. That combination is the argument for preparing early: the work is cheap while the volume is low, and the volume is not under your control. Small agent volume and a default agent interface belong in one plan: the referral channel deserves attention now, the agent channel deserves preparation. Check what machines actually receive from your pages, fix the signals that work against real agent runtimes, and run the agent funnel as a second funnel alongside the human one. Which bots reach your website at all is covered in The AI User-Agent Landscape. The buying-flow protocols are in the No Hacks guide to agentic commerce.
The agentic web in one paragraph
The agentic web is the layer of the internet where AI agents, acting on behalf of humans, discover, read, and transact with websites. Its infrastructure is being built faster than most websites are adapting to it, and faster than agents are actually using it. Machine-First Architecture is the framework for building for it, with four pillars: Identity, Structure, Content, Interaction.
I shifted the whole focus of No Hacks in 2025 because the gap between what is being built and what most builders know is wider than it has been at any point since mobile. The agentic web is the biggest piece of that gap. The supply side is being built at speed. The demand side is barely here. Both of those are true, and a website owner has to prepare for agent traffic arriving without betting the year on when it does.