Why can't ChatGPT see my website even though it ranks #1 on Google?

ChatGPT's crawler (GPTBot) does not execute JavaScript. If your site loads content via JavaScript frameworks like React, Vue, or Angular, the crawler sees a blank page. Google Bot uses headless Chrome to render JavaScript, but most AI crawlers skip this step to conserve resources and maintain speed. Check your page source before JavaScript runs to see what AI systems actually receive.

How do I check if my website is blocking AI crawlers?

Review your CDN settings in Cloudflare, AWS WAF, or similar services. Cloudflare defaulted to blocking AI bots in July 2025, meaning many sites block GPTBot, ClaudeBot, and PerplexityBot without the owner knowing. Check your server logs for 403 or 429 responses to these user agents. You can also use tools like Glimpse at glimpse.webperformancetools.com to test how AI sees your site.

What is the difference between Google Bot and AI crawlers like GPTBot?

Google Bot uses headless Chrome to fully render pages including JavaScript execution, CSS styling, and dynamic content. AI crawlers like GPTBot and ClaudeBot only read raw HTML without executing any JavaScript. This means content loaded by client-side scripts is visible to Google but invisible to AI systems. The technical requirements are fundamentally different.

How fast does my website need to be for AI crawlers?

Server response time should be under 200 milliseconds. Research shows sites with sub-one-second load times receive three times more crawler requests than those over three seconds. AI systems operate under real-time constraints when answering user queries, so slow sites simply get skipped rather than retried. Aim for First Contentful Paint under one second and Largest Contentful Paint under 2.5 seconds.

Does infinite scroll affect AI visibility?

Yes, significantly. Google's Martin Split confirmed that Google Bot doesn't scroll, and AI crawlers definitely won't either. Content loaded dynamically as users scroll down is completely invisible to AI systems. If you use infinite scroll for product listings, articles, or galleries, only the items in the initial HTML response will be seen by AI crawlers. Consider paginated URLs with standard HTML links instead.

What percentage of search traffic now comes from AI agents?

AI agents account for approximately 33% of organic search activity according to recent analysis. GPTBot traffic grew 305% from May 2024 to May 2025, while PerplexityBot saw 157,000% growth in raw requests. On Vercel's infrastructure alone, GPTBot and ClaudeBot combined generate roughly 20% of Google Bot's total traffic volume. This represents a fundamental shift in how people discover web content.

All Episodes

January 28, 202629 minEpisode 215

215: THE AGENT-BROKEN WEB - WHY AI CAN'T SEE YOUR WEBSITE

Spotify Apple

Something strange is happening to the web. Websites ranking on the first page of Google are invisible to ChatGPT, Claude, and Perplexity. Not because of bad content or poor SEO, but because of technical decisions that made perfect sense for human visitors: JavaScript rendering, infinite scroll, aggressive bot protection. These choices now render sites invisible to the AI systems rapidly becoming the primary way people discover the internet.

The numbers make this impossible to ignore. GPTBot traffic grew 305% in a single year. AI agents now drive 33% of organic search activity. Yet analysis of AI crawler behavior reveals they execute zero JavaScript, never scroll, and get blocked by default on most CDNs. The web built for human eyeballs and mouse clicks is fundamentally incompatible with how AI systems consume information. Fixing this requires rethinking site architecture from the ground up.

AI Crawler Traffic ExplosionThe JavaScript Invisibility ProblemAccidental AI BlockingSpeed as a Ranking FactorInfinite Scroll and Lazy LoadingThe Six Critical Fixes

KEY TAKEAWAYS

View your page source before JavaScript executes. If your content isn't in that raw HTML, AI systems can't see it.
Check your CDN's bot management settings immediately. Cloudflare defaults to blocking AI bots, and most site owners don't realize they've cut themselves off from 33% of organic search traffic.
Sites with server response times under 200ms receive three times more crawler visits. Speed isn't just about user experience anymore, it determines AI visibility.
If your site uses infinite scroll, AI crawlers see only what loads in the initial HTML. Everything below the fold might as well not exist.
These fixes work triple duty: they improve traditional SEO, user experience, and accessibility while making your site visible to AI systems.

SHOW NOTES

The Numbers Nobody Expected

Vercel's research shows GPTBot generated 569 million requests monthly across their network. ClaudeBot added another 370 million. Combined, AI crawler traffic now represents roughly 20% of Google Bot activity on major infrastructure. PerplexityBot grew 157,000% in raw requests over a single year.

These aren't bots scraping content for training data. That era is ending. Cloudflare's 2025 analysis found the shift moving from training crawlers to user-action agents: AI systems fetching content in real time to answer specific queries. When someone asks ChatGPT about a product, service, or topic, these bots hit websites to retrieve current information. If they can't retrieve it, that website doesn't exist in the AI's response.

Why JavaScript Breaks Everything

Pre-render.io analyzed over 500 million GPTBot requests and found zero evidence of JavaScript execution. None. AI crawlers consume raw HTML to conserve compute resources and maintain the low latency their systems require.

This creates an absurd situation. A site built with React, Vue, or Angular that loads all content via JavaScript appears completely blank to ChatGPT. Product details, pricing, navigation, reviews: invisible. The only content AI systems see is whatever exists in the initial HTML response before any scripts run.

Google Bot renders JavaScript using headless Chrome. Most site owners assume all crawlers work this way. They don't. Among major AI players, only Google's Gemini and Apple's bot execute JavaScript. The rest see your site the way it looked in 1999.

The Accidental Blocking Epidemic

Cloudflare defaulted to blocking AI bots in July 2025. This single configuration change made millions of websites invisible to AI search overnight, with most site owners completely unaware.

Analysis of 3,900 top domains found AI crawlers were the most frequently blocked user agents. Some blocking is intentional: Amazon sent cease-and-desist letters to Perplexity over its shopping agent. But most is accidental, the result of accepting default security settings without understanding the implications.

Speed Determines Visibility

Kevin Indig's research established strict latency requirements for AI crawler access. Server response must come in under 200 milliseconds. Sites achieving sub-one-second load times receive three times more crawler requests than those taking over three seconds.

Why does this matter? AI systems operate under real-time constraints. When answering a user query, they have milliseconds to fetch supporting content. Slow sites don't get retried. They simply get skipped. The content might be excellent, but if it takes two seconds to arrive, it never enters the candidate pool.

Infinite Scroll Is a Dead End

Google's Martin Split confirmed that Google Bot doesn't scroll. AI crawlers certainly won't. Content loaded dynamically as users scroll down the page is completely invisible to AI systems.

This affects product listings, article archives, image galleries, and anything using lazy loading for performance optimization. The irony is sharp: lazy loading exists to improve user experience by deferring content until needed. For AI systems, "deferring" means "never seeing."

What Actually Needs to Change

Server-side rendering isn't optional anymore. Sites using client-side JavaScript frameworks need to migrate to Next.js, Nuxt, or similar solutions that pre-render content. This is the most significant architectural change, and there's no shortcut.

Structured data using JSON-LD matters more than ever. Microsoft confirmed Bing uses schema.org markup for Copilot integration. Perplexity does the same. The markup must match visible HTML content exactly, since mismatches trigger deception flags.

Bot management requires deliberate configuration. Don't accept CDN defaults. Make explicit decisions about GPTBot, ClaudeBot, and PerplexityBot access based on business strategy, not security theater.

Want to see how AI crawlers view your site? Glimpse is a free tool that shows exactly what AI systems can and can't see. The good news: these changes improve everything simultaneously. Traditional SEO benefits. Page speed improves. Accessibility gets better. The web built for AI agents happens to also be better for humans.

QUESTIONS ANSWERED

LATEST EPISODES

EP 214January 14, 202629 min

214: How Google Just Took Control of AI Commerce (UCP + Apple Deal Explained)

Google just made two massive moves in 48 hours, and together, they could reshape how AI interacts with commerce forever.

EP 213January 7, 202641 min

213: Why Google & ChatGPT Are Ignoring Your "Dead" Content with Jono Alderson

For the last 20 years, digital marketing had one goal: drag a human across a "threshold", onto your website, so you could control the message and sell the product.

EP 212December 17, 202511 min

212: Goodbye 2025 - Digital Tinnitus and the Non-Human Future

If 2025 felt like a constant, high-pitched ringing in your ears, you aren't alone.

EP 211December 10, 202537 min

211: Why AI is Killing Your Clicks: The New Metrics for a Zero-Click World with Joe Doveton

The ground beneath the digital marketing industry is shifting. For decades, the mantra was simple: optimize for traffic, measure clicks, and track conversions. But with the rise of Generative AI, Large Language Models (LLMs), and Answer Engines, that rulebook is obsolete. In this powerful episode, I sit down with Joe Doveton to discuss the urgent reality facing every brand that relies on web traffic.

EP 210November 26, 202537 min

210: AI Agents Are Here (And They Hate Your Website) with Jes Scholz

There is a new user on the internet. It doesn't have eyes, it reads code, and it has zero tolerance for bad UX.

ENJOYING THIS EPISODE?

Subscribe to get notified when new episodes drop. Available on all major podcast platforms.

Subscribe Now

Browse All Episodes