Unverified Commit 8986037d authored by Delba de Oliveira's avatar Delba de Oliveira Committed by GitHub

15.2 docs: document missing `htmlLimitedBots` option (#76616)

- Minor updates to the new Streaming Metadata section
- Document new next.config.ts option: `htmlLimitedBots`
parent 8df92f1d
...@@ -1197,31 +1197,38 @@ export const metadata = { ...@@ -1197,31 +1197,38 @@ export const metadata = {
} }
``` ```
#### Streaming Metadata ## Streaming metadata
Starting from v15.2, metadata returned by `generateMetadata` will be streamed to the client. This allows Next.js to inject metadata into the HTML as soon as it's resolved. Metadata returned by `generateMetadata` is streamed to the client. This allows Next.js to inject metadata into the HTML as soon as it's resolved.
Since page metadata often primarily targets bots & crawlers, Next.js will continue to block the render until the metadata is resolved for **HTML-limited bots**. Since page metadata primarily targets bots and crawlers, Next.js will stream metadata for bots that can execute JavaScript and inspect the full page DOM (e.g. `Googlebot`). However, metadata will continue blocking the render of the page for **HTML-limited** bots (e.g. `Twitterbot`) as these cannot execute JavaScript while crawling.
Some bots, like `Googlebot`, can execute JavaScript and are able to inspect the full page DOM, meaning they **don't** require blocking metadata. However, bots like `Twitterbot` **cannot** execute JavaScript while crawling a page—they fall into the **HTML-limited** category.
Next.js automatically detects the user agent of incoming requests to determine whether to serve streaming metadata or fallback to blocking metadata. Next.js automatically detects the user agent of incoming requests to determine whether to serve streaming metadata or fallback to blocking metadata.
If you need to customize this list, you can define them manually using the `htmlLimitedBots` option in `next.config.js`. Next.js will ensure user agents matching this regex receive blocking metadata when requesting your web page. If you need to customize this list, you can define them manually using the `htmlLimitedBots` option in `next.config.js`. Next.js will ensure user agents matching this regex receive blocking metadata when requesting your web page.
Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. This is advanced behavior, and the default should be sufficient for most cases.
```js filename="next.config.js" ```ts filename="next.config.ts" switcher
import type { NextConfig } from 'next'
const config: NextConfig = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}
export default config
```
```js filename="next.config.js" switcher
module.exports = { module.exports = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler', htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
} }
``` ```
> **Note:** Next.js includes [a default list of HTML limited bots](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts) Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. This is advanced behavior, and the default should be sufficient for most cases.
## Version History ## Version History
| Version | Changes | | Version | Changes |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `v15.2.0` | introduced streaming support to `generateMetadata`. | | `v15.2.0` | Introduced streaming support to `generateMetadata`. |
| `v13.2.0` | `viewport`, `themeColor`, and `colorScheme` deprecated in favor of the [`viewport` configuration](/docs/app/api-reference/functions/generate-viewport). | | `v13.2.0` | `viewport`, `themeColor`, and `colorScheme` deprecated in favor of the [`viewport` configuration](/docs/app/api-reference/functions/generate-viewport). |
| `v13.2.0` | `metadata` and `generateMetadata` introduced. | | `v13.2.0` | `metadata` and `generateMetadata` introduced. |
---
title: htmlLimitedBots
description: Specify a list of user agents that should receive blocking metadata.
---
The `htmlLimitedBots` config allows you to specify a list of user agents that should receive blocking metadata instead of [streaming metadata](/docs/app/api-reference/functions/generate-metadata#streaming-metadata).
```ts filename="next.config.ts" switcher
import type { NextConfig } from 'next'
const config: NextConfig = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}
export default config
```
```js filename="next.config.js" switcher
module.exports = {
htmlLimitedBots: 'MySpecialBot|MyAnotherSpecialBot|SimpleCrawler',
}
```
## Default list
Next.js includes [a default list of HTML limited bots](https://github.com/vercel/next.js/blob/canary/packages/next/src/shared/lib/router/utils/html-bots.ts).
Specifying a `htmlLimitedBots` config will override the Next.js' default list, allowing you full control over what user agents should opt into this behavior. However, this is advanced behavior, and the default should be sufficient for most cases.
## Version History
| Version | Changes |
| ------- | ------------------------------------ |
| 15.2.0 | `htmlLimitedBots` option introduced. |
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment