Our Own Site Was Invisible to AI Crawlers: Anatomy of a Prerender/Deploy Failure
GEON sells AI search visibility. Our own site shipped in a state AI crawlers couldn't read, and nothing about it looked broken — every request returned HTTP 200. The measurements, the root cause, and the deploy gate we built so it can't be skipped again.
I'm Deniz, I work at GEON — we measure how visible brands are in ChatGPT, Perplexity, Claude, Gemini, Grok, and DeepSeek. Our own site was, to an AI crawler, close to a blank page. There was no error, no warning, no broken link. Every request came back HTTP 200. The content just wasn't there. This is what the failure was, why it survived unnoticed, and why the fix isn't a better build script — it's an enforcement point that cannot be bypassed.
What we measured
On 2026-08-04, we requested https://usegeon.com with the header User-Agent: GPTBot/1.0 — the same response OpenAI's crawler would get. Results:
| Route | Result |
|---|---|
/ |
HTTP 200, text/html, 3,359 bytes, no <h1>, 6 visible words |
/blog |
Byte-for-byte the same 3,359-byte response |
/docs |
Byte-for-byte the same 3,359-byte response |
/sitemap.xml |
HTTP 200 but text/html — the sitemap named in robots.txt resolved to a web page, not XML |
The 6 words are literal: to a client that doesn't execute JavaScript, the entire page was the title tag, GEON - AI Platform Visibility Monitoring. <div id="root"> was empty, because the React code that fills it in never ran. We have no vendor commitment to rely on that GPTBot, ClaudeBot, or PerplexityBot executes JavaScript — and to any client that doesn't render, there was no product page, no blog, no docs. There was just that title.
The <head> was also entirely correct: a real <title>, an accurate meta description, complete Open Graph and Twitter Card tags, a canonical link, favicon and manifest references — all present and well-formed. Anyone glancing at view-source would have every reason to think the page was fine. The problem was in <body>: the <div id="root"> that React was supposed to fill in never got filled.
Meanwhile robots.txt was correct and pointed at https://usegeon.com/sitemap.xml, and llms.txt was correct too, sitting at a real 6,549 bytes. So the crawler-facing surface was half right — and that's exactly why it survived unnoticed. Check robots.txt, it looks fine. Check llms.txt, it looks fine. Check <head>, it looks fine. Unless you check what's actually in the body, you don't see the gap.
Why it happened
The cause is mundane and irritating: the prerender chain — the scripts that generate static HTML, the sitemap generator, the steps that prerender blog and docs pages — lives inside the root-level yarn build command. But firebase deploy --only hosting, and a plain yarn build run from inside web/, never trigger that chain. They run vite build alone, which on its own produces an empty SPA shell: a <div id="root"> waiting for React to fill it in. Whatever ends up in dist gets published. Nothing enforced using the correct chain.
This wasn't broken code. It was the right script, called from the wrong place — or not called at all. The build succeeded. The deploy succeeded. No CI step turned red. The result was missing, not broken, and those two failure modes get caught very differently.
The fix: not a build script, a gate that can't be walked around
Reminding people to run the right script wasn't going to hold — nobody was forgetting on purpose, they were just running a different, equally reasonable-looking command. So we wrote a read-only verification script, web/scripts/verify-prerender.ts, and wired it into hosting.predeploy in firebase.json. That means however the deploy is invoked — firebase deploy, CI, someone typing the command by hand — this script runs first, and nothing reaches hosting unless it passes.
The script opens the static HTML generated for /, /docs, /blog, and a Turkish comparison page (/karsilastirma/peec-ai — a canary added to catch Turkish prerendering breaking silently while the English pages keep generating fine), and for each one asserts a real <h1> tag is present and that, after stripping <script> and <style> blocks, at least 20 visible words remain — that's what a crawler with no JavaScript would actually see. It also checks that sitemap.xml is real XML containing <urlset> or <sitemapindex>, not the SPA's HTML fallback. If any page is missing or under the threshold, it names the specific page and exits with code 1, stopping the deploy. It writes nothing to disk, and it runs in about 0.2 seconds.
On a correctly built dist, it reports:
✓ Pre-deploy verification passed
/: 47 words
/docs: 46 words
/blog: 1117 words
/karsilastirma/peec-ai: 641 words
sitemap.xml: 94 URLs
On a build produced by the skipping path, it reports:
✖ Pre-deploy verification FAILED — refusing to publish an unprerendered build.
• / — no <h1> in static HTML (SPA shell was deployed)
• / — only 6 visible words without JS (min 20)
• /docs — dist/docs/index.html is missing
• sitemap.xml exists but is not valid XML (likely the SPA fallback)
Where production stands right now
That fix is merged. But when I checked production again while writing this, the redeploy hadn't happened yet: https://usegeon.com/ with a GPTBot User-Agent still returns the same 3,359-byte, no-<h1>, 6-word response, and /sitemap.xml still returns text/html. The gate exists in code; it hasn't been closed in production yet. I'm saying this plainly because claiming production was fixed when it wasn't would be repeating exactly the mistake this post is about.
The lesson isn't "SPAs are bad"
The framework wasn't the problem — the correct build chain already existed. The problem was that nothing forced anyone to use it. An unenforced rule is a rule that eventually gets skipped; that's not a statement about anyone's diligence, it's a statement about probability over enough deploys. The second, more important part: this failure class is silent. No error code, no alert, no broken build — only absence. HTTP 200 tells you the server answered. It tells you nothing about whether the content it sent was there.
Check your own site
Two commands, about a minute:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" \
-A "GPTBot/1.0" https://yoursite.com/
curl -s -A "GPTBot/1.0" https://yoursite.com/sitemap.xml | head -c 200
The first tells you your homepage's byte size — if several routes (homepage, blog, docs) come back with the exact same byte count, they're likely all serving the same empty shell. The second shows whether your sitemap is actually XML; if the output starts with <!doctype html>, your sitemap is serving the shell too. For a sharper check: curl -A "GPTBot/1.0" https://yoursite.com/ | grep -c "<h1" — zero means a crawler with no JavaScript can't see a heading on your page at all.
What to do about it
If you have a build step that generates static HTML, make sure there's no deploy path left that can skip it — wire it into hosting.predeploy or your platform's equivalent, the way we did, rather than trusting people to remember the right command. After every deploy, check your homepage and sitemap by hand with a GPTBot user agent; automating this is better, but the manual check costs nothing. And stop reading HTTP 200 as "it works" — whether the content actually arrived is a separate question, and it's the one that matters.
Deniz
Content & GEO Strategy