Unverified Commit 9dbb357d authored by Hendrik Liebau's avatar Hendrik Liebau Committed by GitHub

[backport] Encode the cache item name built by `unstable_cache` (#97313)

Backports #96936 ("[refactor] Rename `encodeCacheTag` to
`encodeHeaderSafe`") and #96937 ("Encode the cache item name built by
`unstable_cache`") to `next-16-3`.

Fixes #76286. A cache implementation may serialize cache metadata into
HTTP request headers, whose values are limited to Latin-1.
`unstable_cache` assembles a cache item name from the request URL and
the name of the cached callback, and neither part was encoded. When that
name holds a character above U+00FF the conversion throws before the
request is dispatched, so the read never reaches the cache and the write
that follows it fails the same way. Nothing is stored, nothing is found,
and the entry falls back to the origin on every render.

The reachable case is a non-ASCII query parameter on any dynamic route
that calls `unstable_cache`, whether or not the route reads
`searchParams`, because `URLSearchParams` returns decoded keys and
values while the pathname stays percent-encoded.

The rename is folded in because it is the prerequisite: #96937 imports
`encodeHeaderSafe` from `server/lib/encode-header-safe`, which the
rename creates. It carries no behavior change, the function body is
identical apart from the parameter name, and the two were stacked in
this same order on canary.

### Verification on this branch

Cherry-picked from `1f0cd93b` and `4286a42e` in that order, with no
conflicts.

`packages/next/src/server/web/spec-extension/unstable-cache.ts` is not
byte-identical to canary, because the release branch predates the
unrelated `willConsumerServerCache` change and keeps
`workStore.isStaticGeneration` at those two call sites. The diff this
backport applies to that file is exactly the encoding change and nothing
else. Every other source file is byte-identical to canary, and no
reference to the old `encodeCacheTag` name remains.

Behavioral verification is left to CI on this branch; the change carries
the `non-ascii-cache-item-name` end-to-end suite, which exercises a real
cache handler.

<!-- NEXT_JS_LLM -->
parent 323a17c1
/**
* Percent-encode every character outside printable ASCII so a tag value can be
* safely serialized as part of the `x-next-cache-tags` HTTP header.
*
* Node's `validateHeaderValue` rejects any code unit outside `\t\x20-\x7e`, so
* a matched route path or user-supplied tag containing a non-ASCII character
* (Hebrew, Arabic, Chinese, emoji, …) would otherwise throw `ERR_INVALID_CHAR`
* and crash ISR on every affected request.
*
* This is applied at the public boundaries — tag construction
* (`getImplicitTags`, `validateTags`) and invalidation input (`revalidatePath`,
* `revalidateTag`, `updateTag`) — so storage, comparison, and the wire all see
* the same canonical ASCII-safe form.
*
* The character class `[\t\x20-\x7e]` mirrors Node's `validHdrChars` table —
* `\t` plus printable ASCII through `~`. Anything outside that is rejected
* by `validateHeaderValue`, so we encode runs of those characters and leave
* everything else (`,`, `/`, `%`, `[`, `]`, `_`, `-`, `\t`, …) byte-for-byte
* unchanged. This preserves the comma-separated header format and the
* dynamic-segment markers in derived tags (`_N_T_/[slug]/page`).
*
* Properties:
* - Fast-path: input that already fits the validation class is returned
* unchanged. This makes the encoder idempotent on already-encoded `%xx`
* sequences.
* - Matches *runs* of out-of-class code units so surrogate pairs (e.g. an
* emoji) are handed to `encodeURIComponent` as a complete code point — a
* per-code-unit regex would split the pair and throw `URIError`.
*/
const OUT_OF_CLASS_CHAR = /[^\t\x20-\x7e]/
const OUT_OF_CLASS_RUN = /[^\t\x20-\x7e]+/g
export function encodeCacheTag(tag: string): string {
return OUT_OF_CLASS_CHAR.test(tag)
? tag.replace(OUT_OF_CLASS_RUN, (run) => encodeURIComponent(run))
: tag
}
/**
* Percent-encode every character other than horizontal tab and printable ASCII
* so a value can be safely serialized into an HTTP header.
*
* Node's `validateHeaderValue` and `fetch`'s ByteString conversion accept
* different character ranges. Restricting values to `\t\x20-\x7e` produces a
* conservative representation that both can serialize. Characters outside a
* transport's accepted range (Hebrew, Arabic, Chinese, emoji, …) would
* otherwise throw `ERR_INVALID_CHAR` or a `TypeError`, which crashes ISR or
* fails a cache read on every affected request.
*
* Cache tags are encoded at the public boundaries — tag construction
* (`getImplicitTags`, `validateTags`) and invalidation input
* (`revalidatePath`, `revalidateTag`, `updateTag`) — so storage, comparison,
* and the wire all see the same canonical form.
*
* The class is narrower than either transport accepts. Everything inside it
* passes through byte-for-byte, including `,`, `/`, `%`, `[`, `]`, `_`, `-` and
* `\t`, which preserves the comma-separated header format and the
* dynamic-segment markers in derived tags (`_N_T_/[slug]/page`).
*
* Properties:
* - Fast-path: input that already fits the class is returned unchanged. This
* makes the encoder idempotent on already-encoded `%xx` sequences.
* - Matches *runs* of out-of-class code units so surrogate pairs (e.g. an
* emoji) are handed to `encodeURIComponent` as a complete code point — a
* per-code-unit regex would split the pair and throw `URIError`.
*/
const OUT_OF_CLASS_CHAR = /[^\t\x20-\x7e]/
const OUT_OF_CLASS_RUN = /[^\t\x20-\x7e]+/g
export function encodeHeaderSafe(value: string): string {
return OUT_OF_CLASS_CHAR.test(value)
? value.replace(OUT_OF_CLASS_RUN, (run) => encodeURIComponent(run))
: value
}
import { NEXT_CACHE_IMPLICIT_TAG_ID } from '../../lib/constants'
import type { OpaqueFallbackRouteParams } from '../request/fallback-params'
import { getCacheHandlerEntries } from '../use-cache/handlers'
import { encodeCacheTag } from './encode-cache-tag'
import { encodeHeaderSafe } from './encode-header-safe'
import { createLazyResult, type LazyResult } from './lazy-result'
export interface ImplicitTags {
......@@ -92,14 +92,14 @@ export async function getImplicitTags(
// `x-next-cache-tags`. Idempotent on already-ASCII input.
const derivedTags = getDerivedTags(page)
for (let tag of derivedTags) {
tag = encodeCacheTag(`${NEXT_CACHE_IMPLICIT_TAG_ID}${tag}`)
tag = encodeHeaderSafe(`${NEXT_CACHE_IMPLICIT_TAG_ID}${tag}`)
tags.add(tag)
}
// Add the tags from the pathname. If the route has unknown params, we don't
// want to add the pathname as a tag, as it will be invalid.
if (pathname && (!fallbackRouteParams || fallbackRouteParams.size === 0)) {
const tag = encodeCacheTag(`${NEXT_CACHE_IMPLICIT_TAG_ID}${pathname}`)
const tag = encodeHeaderSafe(`${NEXT_CACHE_IMPLICIT_TAG_ID}${pathname}`)
tags.add(tag)
}
......
......@@ -35,7 +35,7 @@ import {
import { cloneResponse } from './clone-response'
import type { IncrementalCache } from './incremental-cache'
import { RenderStage } from '../app-render/staged-rendering'
import { encodeCacheTag } from './encode-cache-tag'
import { encodeHeaderSafe } from './encode-header-safe'
import type { Span } from './trace/tracer'
const isEdgeRuntime = process.env.NEXT_RUNTIME === 'edge'
......@@ -107,7 +107,7 @@ export function validateTags(tags: any[], description: string) {
// Encode so a non-ASCII tag can be safely serialized into the
// `x-next-cache-tags` HTTP header without tripping Node's header
// validation. Length is checked on the raw input above.
validTags.push(encodeCacheTag(tag))
validTags.push(encodeHeaderSafe(tag))
}
if (validTags.length > NEXT_CACHE_TAG_MAX_ITEMS) {
......
......@@ -16,7 +16,7 @@ import {
ActionDidRevalidateStaticAndDynamic as ActionDidRevalidate,
} from '../../../shared/lib/action-revalidation-kind'
import { removeTrailingSlash } from '../../../shared/lib/router/utils/remove-trailing-slash'
import { encodeCacheTag } from '../../lib/encode-cache-tag'
import { encodeHeaderSafe } from '../../lib/encode-header-safe'
import { validateAndNormalizeCacheLifeProfile } from '../../use-cache/cache-life-profile'
type CacheLifeConfig = {
......@@ -40,7 +40,7 @@ export function revalidateTag(tag: string, profile: string | CacheLifeConfig) {
} else if (typeof profile === 'object') {
profile = validateAndNormalizeCacheLifeProfile(profile, { kind: 'inline' })
}
return revalidate([encodeCacheTag(tag)], `revalidateTag ${tag}`, profile)
return revalidate([encodeHeaderSafe(tag)], `revalidateTag ${tag}`, profile)
}
/**
......@@ -62,7 +62,7 @@ export function updateTag(tag: string) {
)
}
// updateTag uses immediate expiration (no profile) without deprecation warning
return revalidate([encodeCacheTag(tag)], `updateTag ${tag}`, undefined)
return revalidate([encodeHeaderSafe(tag)], `updateTag ${tag}`, undefined)
}
/**
......@@ -105,7 +105,7 @@ export function revalidatePath(originalPath: string, type?: 'layout' | 'page') {
return
}
let normalizedPath = `${NEXT_CACHE_IMPLICIT_TAG_ID}${encodeCacheTag(removeTrailingSlash(originalPath))}`
let normalizedPath = `${NEXT_CACHE_IMPLICIT_TAG_ID}${encodeHeaderSafe(removeTrailingSlash(originalPath))}`
if (type) {
normalizedPath += `${normalizedPath.endsWith('/') ? '' : '/'}${type}`
......
......@@ -2,6 +2,7 @@ import type { IncrementalCache } from '../../lib/incremental-cache'
import { CACHE_ONE_YEAR_SECONDS } from '../../../lib/constants'
import { validateRevalidate, validateTags } from '../../lib/patch-fetch'
import { encodeHeaderSafe } from '../../lib/encode-header-safe'
import {
workAsyncStorage,
type WorkStore,
......@@ -137,7 +138,20 @@ export function unstable_cache<T extends Callback>(
const cacheKey =
await incrementalCache.generateSimpleCacheKey(invocationKey)
// $urlWithPath,$sortedQueryStringKeys,$hashOfEveryThingElse
const fetchUrl = `unstable_cache ${fetchUrlPrefix} ${cb.name ? ` ${cb.name}` : cacheKey}`
//
// A cache implementation may serialize this name into an HTTP request
// header, so it is encoded here. Both parts can carry a character above
// U+00FF: the search parameters are decoded, and a JavaScript identifier
// may hold one. The character class leaves the separating spaces and the
// URL punctuation untouched, so the shape above is preserved.
//
// `toWellFormed` replaces lone surrogates, which `cb.name` can hold and
// which `encodeURIComponent` rejects. The name identifies the call for
// debug metrics, so a replacement character is an acceptable trade for
// not failing the render.
const fetchUrl = encodeHeaderSafe(
`unstable_cache ${fetchUrlPrefix} ${cb.name ? ` ${cb.name}` : cacheKey}`.toWellFormed()
)
const fetchIdx =
(workStore ? workStore.nextFetchId : noStoreFetchIdx) ?? 1
......
import { unstable_cache } from 'next/cache'
import { connection } from 'next/server'
// Returns a different value on every call, so a cache that is not working is
// visible in the rendered output.
const RANDOM_ENDPOINT = 'https://next-data-api-endpoint.vercel.app/api/random'
// The callback is anonymous, so the item name is the request URL plus a hash.
// This route covers the part of the name that comes from the URL.
const getConfiguration = unstable_cache(
async () => {
const res = await fetch(`${RANDOM_ENDPOINT}?unstable-cache`)
return (await res.text()).trim()
},
['configuration'],
{ tags: ['ascii-tag'], revalidate: 3600 }
)
// The fetch cache reaches the same cache layer, but derives its item name from
// the request URL, which is already percent-encoded. It is a control: it has to
// keep caching for the very request that breaks the entry above.
async function getFetchedValue() {
const res = await fetch(`${RANDOM_ENDPOINT}?fetch-cache`, {
next: { revalidate: 3600, tags: ['ascii-fetch-tag'] },
})
return (await res.text()).trim()
}
export default async function Page() {
// The item name only carries the request URL for a dynamic render. A
// prerender uses the route pattern instead, which is always ASCII.
//
// The page never reads `searchParams`. The query string reaches the item name
// through the request URL either way, so a dynamic route is affected by a
// query it does not look at, and by one a caller appends to it.
await connection()
return (
<>
<p id="cached">{await getConfiguration()}</p>
<p id="fetched">{await getFetchedValue()}</p>
</>
)
}
import { ReactNode, Suspense } from 'react'
export default function Root({ children }: { children: ReactNode }) {
return (
<html>
<body>
<Suspense fallback={<p>Loading...</p>}>{children}</Suspense>
</body>
</html>
)
}
import { unstable_cache } from 'next/cache'
import { connection } from 'next/server'
const RANDOM_ENDPOINT = 'https://next-data-api-endpoint.vercel.app/api/random'
// `encodeURIComponent` rejects a lone surrogate outright, so an item name
// holding one fails the render rather than the cache read. That makes this a
// stricter case than a name that is merely unrepresentable in a header.
//
// A lone surrogate cannot be written as an identifier, so the name is attached
// through a computed key. The key is a string literal, which a bundler cannot
// rename, so this route carries the constraint in every mode.
const LONE_SURROGATE = '\uD800'
const getConfiguration = unstable_cache(
{
[LONE_SURROGATE]: async () => {
const res = await fetch(`${RANDOM_ENDPOINT}?lone-surrogate`)
return (await res.text()).trim()
},
}[LONE_SURROGATE],
['lone-surrogate'],
{ revalidate: 3600 }
)
export default async function Page() {
await connection()
return <p id="cached">{await getConfiguration()}</p>
}
import { unstable_cache } from 'next/cache'
import { connection } from 'next/server'
const RANDOM_ENDPOINT = 'https://next-data-api-endpoint.vercel.app/api/random'
// A JavaScript identifier may hold characters above U+00FF, and a named
// callback puts its name into the cache item name. This route is requested
// under a pure ASCII URL, so the name is the only part of the item name that
// can be unrepresentable, and a failure here points at nothing else.
//
// A production build renames the binding, so this route only carries the
// constraint in development.
const pobierzKonfigurację = async () => {
const res = await fetch(`${RANDOM_ENDPOINT}?named-callback`)
return (await res.text()).trim()
}
const getConfiguration = unstable_cache(
pobierzKonfigurację,
['named-configuration'],
{ revalidate: 3600 }
)
export default async function Page() {
await connection()
return <p id="cached">{await getConfiguration()}</p>
}
// Reports any value that Next.js hands to the cache layer and that cannot be
// carried in an HTTP header.
//
// A cache implementation may serialize the tags, the soft tags, and the cache
// item name into request headers. Header values are limited to Latin-1, so a
// character above U+00FF cannot be represented and the conversion throws. A
// value that fails it never reaches the cache, and the read is
// indistinguishable from a miss.
//
// Only the value is constrained, not the header it ends up in, so the probe
// uses an arbitrary header name.
const CHECKED_FIELDS = ['tags', 'softTags', 'fetchUrl']
const PROBE_HEADER = 'x-cache-probe'
function reportUnrepresentableValues(operation, ctx) {
for (const field of CHECKED_FIELDS) {
const raw = ctx[field]
const value = Array.isArray(raw) ? raw.join(',') : raw
if (!value) {
continue
}
try {
new Headers({ [PROBE_HEADER]: value })
} catch {
console.error(
`CACHE_PROBE unsafe ${operation} ${field} ${JSON.stringify(value)}`
)
}
}
}
module.exports = class CacheHandler {
async get(key, ctx = {}) {
// Logged on every read so a test can tell "nothing unrepresentable" apart
// from "the handler was never consulted".
console.error(`CACHE_PROBE get`)
reportUnrepresentableValues('get', ctx)
return null
}
async set(key, data, ctx = {}) {
reportUnrepresentableValues('set', ctx)
}
async revalidateTag() {}
resetRequestCache() {}
}
/**
* @type {import('next').NextConfig}
*/
// A deployment checks the constraint through the real cache implementation,
// which serializes cache metadata into request headers itself. Everywhere else
// the fixture supplies a handler that applies the same conversion.
const nextConfig = process.env.VERCEL
? {}
: {
cacheHandler: require.resolve('./cache-handler.js'),
// The in-memory cache would answer repeat lookups without consulting the
// handler, which is where the local assertions read from.
cacheMaxMemorySize: 0,
}
module.exports = nextConfig
import { nextTestSetup } from 'e2e-utils'
import { retry } from 'next-test-utils'
// Cache metadata reaches a cache implementation through HTTP request headers,
// whose values are limited to Latin-1. Every tag, soft tag, and cache item name
// therefore has to be representable there. A value that is not gets lost before
// it reaches the cache, so the entry is never found and never stored, and the
// route calls the origin on every render without reporting an error.
//
// `unstable_cache` assembles its item name from the request URL and the name of
// the cached callback, so the fixture covers those two parts on separate
// routes. `/[slug]` is requested with a non-ASCII segment and a non-ASCII query
// parameter and holds an anonymous callback. `/named-callback` is requested
// under a pure ASCII URL and holds a callback whose name is non-ASCII. A
// failure therefore names the part of the item name it comes from.
describe('non-ASCII cache item name', () => {
const { next, isNextDeploy, isNextDev } = nextTestSetup({
files: __dirname,
})
const urlPath = `/${encodeURIComponent('odzież')}?q=${encodeURIComponent('模型')}`
const namedCallbackPath = '/named-callback'
const loneSurrogatePath = '/lone-surrogate'
async function readValues() {
const $ = await next.render$(urlPath)
const values = {
cached: $('#cached').text(),
fetched: $('#fetched').text(),
}
// Both entries hold a random number. Asserting the shape keeps a render
// that produced neither from reading as two equal results.
for (const value of Object.values(values)) {
expect(value).toMatch(/^0\.\d+$/)
}
return values
}
async function expectEveryValueRepresentable(requestPath: string) {
const outputIndex = next.cliOutput.length
const res = await next.fetch(requestPath)
expect(res.status).toBe(200)
await retry(async () => {
// Without this the assertion would also hold when the handler is never
// consulted at all.
expect(next.cliOutput.slice(outputIndex)).toContain('CACHE_PROBE get')
})
expect(next.cliOutput.slice(outputIndex)).not.toContain(
'CACHE_PROBE unsafe'
)
}
if (isNextDeploy) {
it('keeps caching a request with a non-ASCII query parameter', async () => {
// The real cache implementation performs the conversion, so an item name
// it cannot carry surfaces as a cache that never returns anything: the
// entry is recomputed and the rendered value changes on every request.
// The fetched value has to stay put throughout, which separates a broken
// item name from a cache that is down.
//
// Only the URL route is covered here. A production build renames the
// callback on the other one, which leaves nothing unrepresentable in its
// item name.
await readValues()
await retry(async () => {
const first = await readValues()
const second = await readValues()
expect(second).toEqual(first)
})
})
} else {
it('encodes a non-ASCII query parameter in the item name', async () => {
await expectEveryValueRepresentable(urlPath)
})
it('encodes a lone surrogate in the callback name', async () => {
// `encodeURIComponent` rejects a lone surrogate, which fails the render
// instead of the cache read. The response is still a 200 that carries an
// error, so the assertion has to be on the rendered output.
const $ = await next.render$(loneSurrogatePath)
expect($('#cached').text()).toMatch(/^0\.\d+$/)
})
// A production build renames the callback, which leaves nothing
// unrepresentable in its item name, so the constraint is only observable in
// development.
if (isNextDev) {
it('encodes a non-ASCII callback name in the item name', async () => {
await expectEveryValueRepresentable(namedCallbackPath)
})
}
}
})
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment