A client migrated their e-commerce platform in March. Two weeks later, Search Console showed 40% of their previously indexed pages sitting in "Discovered — currently not indexed." Nothing about the migration touched content quality. The product copy was untouched. The problem, once we pulled server logs, turned out to be embarrassingly simple: Googlebot was spending nearly all its time fetching sort variants, filter combinations, and session-tagged URLs, while the actual product pages sat further down the queue than they used to.

Meanwhile the origin server was running at 95% CPU because every one of those GET requests triggered an uncached database query. Rich Results Test passed cleanly on the individual URLs we checked. The sitemap was fine. None of that mattered, because the request pipeline underneath it all was the actual bottleneck.

This is a more common failure mode than people expect, and it has nothing to do with content or sitemaps. It's about how your application handles the humble HTTP GET request — the one method every crawler, and increasingly every LLM retrieval bot, relies on exclusively.

GET Is Supposed to Be Boring. Treat It That Way.

RFC 9110 defines GET as safe and idempotent — a method whose semantics are essentially read-only. Googlebot and Bingbot lean on that guarantee for indexing. The newer generation of AI bots — GPTBot, ClaudeBot, and similar — rely on it too, though it's worth being precise about what they're actually doing: some of that traffic is training-data collection, and some (increasingly) is live retrieval for answering a user's query in real time, similar in spirit to what a search crawler does. Either way, none of them submit forms or issue POST requests to discover content. Every fetch, every image download, every JSON-LD extraction starts as a GET.

The trouble starts when your application quietly breaks that contract. I've seen GET requests that write a row to a page_views table on every hit, or that generate a fresh session token each time. On a normal day that's invisible. The moment Googlebot fires off a burst of concurrent fetches across thousands of URLs on your site, you've built yourself a self-inflicted database lock.

Response latency matters here too, though I'd be careful about pinning an exact millisecond number to it — Google hasn't published one, and I haven't seen a reliable, consistent threshold across the sites I've audited. What is consistent, and what Google's own crawl budget documentation confirms, is that Googlebot scales back its crawl rate when it detects your server struggling to keep up. Slow, unpredictable GET responses cost you crawl volume even without a specific number to point to.

When the HTML Shell Is Empty

Modern frameworks built around client-side rendering hand Googlebot a nearly empty HTML shell on the first GET request:

That initial payload goes through a first-pass indexing check. If your canonical tag, your title, and your JSON-LD aren't in that first response, the page gets pushed to Google's rendering queue, which then has to issue secondary GET requests for your JS bundle and any API calls needed to fill the page. If any of those secondary fetches time out or return a 5xx, the render fails silently, and what gets indexed is whatever was left in that near-empty shell.

Server-side rendering or static generation sidesteps this entirely by putting the complete document — canonical tag, structured data, visible content — into the very first response. That's not a controversial recommendation; it's the same guidance you'll find across Google's own material on JavaScript and SEO.

The Parameter Explosion Nobody Notices Until It's Too Late

This is the part that actually killed my client's crawl budget. A category page with five filterable attributes — category, color, size, sort, view — generates enormous numbers of technically distinct URLs the moment those parameters get appended without any ordering discipline:

/category/shoes?color=red&size=10&sort=price_asc
/category/shoes?size=10&color=red&sort=price_asc
/category/shoes?sort=price_asc&color=red&size=10&utm_source=newsletter
/category/shoes?color=red&size=10&sort=price_asc&session_id=987654

To a crawler, every one of those strings is a separate URL. If your server returns 200 OK for all of them, you're handing Googlebot an effectively infinite parameter space to explore, at the direct expense of crawling anything new. Google's own documentation on managing crawl budget for large sites specifically calls out consolidating duplicate URLs as one of the few real levers site owners have — everything else about crawl budget is determined by Google, but this part is on you.

The fix is enforcing parameter discipline at the routing layer: sort query keys alphabetically, strip tracking parameters (utm_*, gclid, fbclid) before your application logic even runs, and 301 non-canonical variants to the deterministic base URL.

Don't Reach for robots.txt Disallow as Your First Move

A mistake I still see constantly: blocking parameterized URLs with Disallow: /*?* in robots.txt, sometimes alongside references to Search Console's old URL Parameters tool — which Google retired years ago and no longer offers as an option. Disallow feels like the obvious fix, but it creates two problems that are easy to miss:

1. You block link equity, not just crawling. If external or internal links point to those parameterized URLs, Google can't process them to see outbound links or canonical signals, because it's forbidden from fetching the page at all.
2. You can't un-index what you can't crawl. If a parameterized URL is already indexed, adding a Disallow rule prevents Googlebot from ever discovering a noindex tag or a redirect on that URL. It stays stuck in the index — a disallowed page can still surface in results if it's linked to from elsewhere, since robots.txt controls crawling, not indexing.

The better fix lives at the edge: normalize the URL, strip junk parameters, and 301 to canonical form before the request ever touches your application server.

// Cloudflare Worker: normalize GET requests before they hit the origin
addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(request) {
  if (request.method !== 'GET') return fetch(request);

  const url = new URL(request.url);
  const params = url.searchParams;
  const blocked = ['utm_source', 'utm_medium', 'utm_campaign', 'gclid', 'fbclid', 'session_id'];

  let changed = false;
  for (const key of blocked) {
    if (params.has(key)) { params.delete(key); changed = true; }
  }

  const sorted = new URLSearchParams();
  for (const key of Array.from(params.keys()).sort()) {
    for (const val of params.getAll(key).sort()) sorted.append(key, val);
  }
  if (params.toString() !== sorted.toString()) changed = true;

  if (changed) {
    url.search = sorted.toString();
    return Response.redirect(url.toString(), 301);
  }
  return fetch(request);
}

Conditional Requests: Let the Server Say "Nothing Changed"

When Googlebot recrawls a known URL, a well-configured server should let it validate rather than re-download. The first response carries validator headers:

HTTP/1.1 200 OK
Last-Modified: Sun, 19 Jan 2025 18:30:00 GMT
ETag: "33a6409618408200dd3f382d370f47bc"
Cache-Control: public, max-age=3600, s-maxage=86400

On the next pass, Googlebot sends those tokens back:

GET /category/shoes HTTP/1.1
If-Modified-Since: Sun, 19 Jan 2025 18:30:00 GMT
If-None-Match: "33a6409618408200dd3f382d370f47bc"

If nothing changed, the server returns an empty-body 304 instead of the full page. That single response can drop from a few hundred kilobytes to under 300 bytes — real savings on server CPU and bandwidth, and one less reason for Googlebot to throttle back on your domain.

A few header details worth getting right on the CDN layer:

• s-maxage tells shared caches how long a response stays fresh, independent of browser cache lifetimes.
• stale-while-revalidate lets the edge serve a slightly stale response instantly while it quietly refreshes in the background.
• Be careful with Vary: User-Agent. It's occasionally still used for serving different HTML to mobile versus desktop, but it fragments your cache badly — every distinct Googlebot variant (desktop, mobile, image) bypasses the cache and forces a fresh origin fetch. A responsive layout avoids the problem entirely, which is the direction Google has been steering site owners for years now.

Three Failures I've Actually Debugged

The Infinite Facet Loop

An e-commerce catalog let facet filters append to the URL with no ordering rules or path limits. Every facet click exposed ten new sub-facet combinations, and within weeks Googlebot was spending most of its visits on parameter loops instead of new product lines.

The broken version accepts anything:

app.get('/shop/category', async (req, res) => {
  const products = await db.query('SELECT * FROM products WHERE category = ?', [req.query.cat]);
  res.status(200).send(renderTemplate('catalog', { products }));
});

The fix restricts valid parameters and redirects everything else:

const VALID_PARAMS = new Set(['brand', 'color', 'page']);

app.get('/shop/category', async (req, res) => {
  const keys = Object.keys(req.query);
  const invalid = keys.filter(k => !VALID_PARAMS.has(k));

  if (invalid.length > 0) {
    const clean = new URLSearchParams();
    for (const k of keys) if (VALID_PARAMS.has(k)) clean.set(k, req.query[k]);
    const qs = clean.toString();
    return res.redirect(301, qs ? `${req.path}?${qs}` : req.path);
  }

  const products = await db.query('SELECT * FROM products WHERE category = ?', [req.query.brand]);
  res.setHeader('Link', `; rel="canonical"`);
  res.status(200).send(renderTemplate('catalog', { products }));
});

Verify it with a direct request against a junk parameter — it should come back as a 301, not a 200:

curl -I -A "Googlebot" "https://example.com/shop/category?color=red&junk_id=999"

Cache Poisoning Across Device Types

A site serves different HTML for mobile and desktop but omits Vary: User-Agent, or sets it too broadly alongside Cookie. The CDN caches whichever version arrives first and serves it to every subsequent request regardless of device — desktop crawlers end up seeing the mobile shell, which shows up in Search Console as a mismatch between what was crawled and what a real visitor sees.

# Missing Vary header - CDN serves one cached version to everyone
location / {
    proxy_pass http://node_upstream;
    add_header Cache-Control "public, max-age=86400";
}

# Correct: separates cache buckets by device
location / {
    proxy_pass http://node_upstream;
    add_header Cache-Control "public, max-age=86400, s-maxage=604800";
    add_header Vary "User-Agent, Accept-Encoding";
}

Test it by sending both a desktop and a mobile Googlebot user agent against the same URL and comparing the response headers directly.

The Loading Spinner Google Indexes

A React app fetches product data client-side, after the initial HTML has already been sent:

export default function ProductPage({ productId }) {
  const [product, setProduct] = useState(null);

  useEffect(() => {
    fetch(`/api/v1/products/${productId}`)
      .then(res => res.json())
      .then(setProduct);
  }, [productId]);

  if (!product) return 

; // this is what gets indexed if the fetch is slow
  return 

{product.title}

{product.description}

;
}

If that fetch call is slow, blocked by CORS, or times out during Google's rendering pass, there's a real chance "Loading..." is what gets captured and indexed instead of the actual product data. Google's Web Rendering Service does retry failed renders, so it isn't guaranteed to happen on every page every time — but on a large catalog, even an occasional failure rate compounds into a meaningful chunk of pages indexed with no real content. Fetching the data server-side before the response is sent removes the dependency on a secondary request succeeding at exactly the right moment:

export async function getServerSideProps({ params }) {
  const res = await fetch(`https://api.internal/v1/products/${params.productId}`);
  const product = await res.json();
  if (!product) return { notFound: true };
  return { props: { product } };
}

export default function ProductPage({ product }) {
  return (
    

{product.title}

{product.description}

  );
}

If you want to check where your own site is losing crawl efficiency, our free technical SEO tools can help you spot parameter bloat and rendering gaps before they compound into an indexing problem. (Disclosure: I built this toolkit — the audit patterns above come from real client work, not from testing our own product.)