Unverified Commit fbe0d562 authored by Joseph's avatar Joseph Committed by GitHub

docs: rewrite the bots and crawlers section in the caching guide (#98782)

## Summary

The Bots and crawlers section in the caching guide said that all bots
and crawlers skip the static shell because "they need a complete
document". Only HTML-limited bots do, and the reason is metadata: with
Partial Prerendering, `generateMetadata` can resolve dynamically and
stream in after the shell, which puts the tags in the body rather than
the `<head>`. Bots that need them in the `<head>` get a dynamic render
instead.

The second paragraph now states the consequence directly: if the
build-time prerender read something unreachable at request time, the
render throws and those bots get a 500. Which user agents this covers,
and the `htmlLimitedBots` option for changing the list, stay owned by
the Streaming guide, which the section links to.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
parent 7395b29c
...@@ -604,6 +604,6 @@ See [ISR with Cache Components](/docs/app/guides/incremental-static-regeneration ...@@ -604,6 +604,6 @@ See [ISR with Cache Components](/docs/app/guides/incremental-static-regeneration
## Bots and crawlers ## Bots and crawlers
Browsers receive the static shell instantly. Bots and crawlers are detected by their user agent and handled differently: because they need a complete document, Next.js skips the shell and renders the entire page dynamically at request time, then sends the finished HTML once the render completes. With [Partial Prerendering](#prerendering), metadata can be dynamic while the rest of the page is prerendered into a static shell. The shell is served immediately and the metadata streams in after. Some bots and crawlers cannot handle this, and need to see the metadata included in the `<head>` tag of the document. Next.js detects them by their user agent, skips the static shell, and renders the page dynamically at request time.
Because the shell is re-rendered instead of reused, work that completed during prerendering now runs at request time for a bot. If part of your shell depends on inputs that only exist while prerendering, such as build-time data or values that are not reachable in the request-time environment, a page that loads for a person can fail to render for a crawler. Make sure the data your shell relies on is also available at request time. See [Bots and crawlers](/docs/app/guides/streaming#bots-and-crawlers) in the Streaming guide for more details. Data sources available during the build-time prerender have to be available at request time too. Otherwise the render may throw, and these bots receive a 500 error. See [Bots and crawlers](/docs/app/guides/streaming#bots-and-crawlers) in the Streaming guide for which user agents this covers and how to change the list.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment