Unverified Commit 1ff6a803 authored by Hendrik Liebau's avatar Hendrik Liebau Committed by GitHub

Reuse completed cache entries for the rest of a request (#96727)

Calling the same `'use cache: private'` function twice in one request
executed its body twice in production. Preloading at the top of a
segment and reading the same function again lower down for
composability, which is the shape that motivates a preload in the first
place, therefore did the work twice instead of once.

The intra-request dedupe map dropped an entry as soon as its fill
completed, so it only ever covered concurrent invocations. A later
invocation fell through to the cache handler, and the `React.cache` memo
wrapping every cache function missed whenever the arguments were not
reference-equal. Public caches got a handler hit out of that, but
private caches have no handler in production and their entries are
excluded from the immutable Resume Data Cache of a dynamic request, so
nothing had stored the entry.

Completed invocations now move into `completedCacheInvocations` on the
work store instead of being dropped, and a later invocation joins that
entry. The pending map keeps its previous semantics untouched, because a
concurrent joiner shares a fill that is genuinely in flight and must not
re-run the discard checks against it, whereas a completed entry is a
stored value and is only reused when the caller has not asked to bypass
caches and nothing has invalidated it since. Private caches still get no
cache handler, so the map is what backs them, and it lives on the work
store and so cannot carry request-derived data beyond the request that
produced it.

Retention is limited to the kinds where it saves real work: private
caches, and kinds whose handler came from the platform or from
`cacheHandlers` config, where a read can be a network round trip.
`isBuiltInCacheHandler` answers that by tracking the handler instances
`initializeCacheHandlers` constructs itself. Recording instances rather
than kinds is what keeps aliasing right, since a self-hosted `remote`
resolves to the very same in-memory handler as `default` and retaining
its entries would duplicate that cache for nothing, while a
platform-supplied `remote` does not.

The `use-cache-custom-handler` suite covers the mechanism rather than
the observable value, counting `::get` calls to assert that two
sequential reads reach the handler once. Value equality alone cannot
distinguish a retained entry from a handler hit, which is also why the
new `dedup-sequential` fixture passes without this change and serves
only as a regression pin next to `private-dedup-sequential`.
parent 5da1c1ae
...@@ -111,6 +111,21 @@ export interface WorkStore { ...@@ -111,6 +111,21 @@ export interface WorkStore {
*/ */
pendingCacheInvocations?: Map<string, Promise<SharedCacheResult>> pendingCacheInvocations?: Map<string, Promise<SharedCacheResult>>
/**
* Invocations from this request that have already completed, keyed the same
* way as `pendingCacheInvocations`. Entries move here when their fill
* finishes rather than being dropped, so a later invocation of the same cache
* function reuses the entry instead of repeating the cache handler lookup
* and, on a miss, the work.
*
* Only populated for kinds where that saves something real: private caches,
* which have no cache handler in production, and kinds whose handler was
* supplied by the platform or by `cacheHandlers` config, whose reads may be
* remote. A built-in handler read is a map lookup, so retaining its entries
* would cost memory for nothing.
*/
completedCacheInvocations?: Map<string, Promise<SharedCacheResult>>
/** /**
* Set by the dev-server's hang-detection probe worker (see * Set by the dev-server's hang-detection probe worker (see
* `use-cache-probe-worker.ts`) to switch `cache()` into a one-shot fill * `use-cache-probe-worker.ts`) to switch `cache()` into a one-shot fill
......
...@@ -15,6 +15,7 @@ const handlersSymbol = Symbol.for('@next/cache-handlers') ...@@ -15,6 +15,7 @@ const handlersSymbol = Symbol.for('@next/cache-handlers')
const handlersMapSymbol = Symbol.for('@next/cache-handlers-map') const handlersMapSymbol = Symbol.for('@next/cache-handlers-map')
const handlersSetSymbol = Symbol.for('@next/cache-handlers-set') const handlersSetSymbol = Symbol.for('@next/cache-handlers-set')
const privateHandlerSymbol = Symbol.for('@next/cache-handlers-private') const privateHandlerSymbol = Symbol.for('@next/cache-handlers-private')
const builtInHandlersSymbol = Symbol.for('@next/cache-handlers-built-in')
const devFrontHandlersSymbol = Symbol.for('@next/cache-handlers-dev-fronts') const devFrontHandlersSymbol = Symbol.for('@next/cache-handlers-dev-fronts')
const devTieredHandlersSymbol = Symbol.for('@next/cache-handlers-dev-tiered') const devTieredHandlersSymbol = Symbol.for('@next/cache-handlers-dev-tiered')
const memoryCacheDisabledSymbol = Symbol.for( const memoryCacheDisabledSymbol = Symbol.for(
...@@ -41,6 +42,7 @@ const reference: typeof globalThis & { ...@@ -41,6 +42,7 @@ const reference: typeof globalThis & {
} }
[handlersMapSymbol]?: Map<string, CacheHandler> [handlersMapSymbol]?: Map<string, CacheHandler>
[handlersSetSymbol]?: Set<CacheHandler> [handlersSetSymbol]?: Set<CacheHandler>
[builtInHandlersSymbol]?: Set<CacheHandler>
// DEV-only // DEV-only
[privateHandlerSymbol]?: CacheHandler [privateHandlerSymbol]?: CacheHandler
[devFrontHandlersSymbol]?: Map<string, CacheHandler> [devFrontHandlersSymbol]?: Map<string, CacheHandler>
...@@ -64,6 +66,8 @@ export function initializeCacheHandlers(cacheMaxMemorySize: number): boolean { ...@@ -64,6 +66,8 @@ export function initializeCacheHandlers(cacheMaxMemorySize: number): boolean {
debug?.('initializing cache handlers') debug?.('initializing cache handlers')
const handlersMap = new Map<string, CacheHandler>() const handlersMap = new Map<string, CacheHandler>()
reference[handlersMapSymbol] = handlersMap reference[handlersMapSymbol] = handlersMap
const builtInHandlers = new Set<CacheHandler>()
reference[builtInHandlersSymbol] = builtInHandlers
// In development, `cacheMaxMemorySize: 0` would make the built-in default // In development, `cacheMaxMemorySize: 0` would make the built-in default
// handler a no-op, so every reload would miss. Use a real in-memory size // handler a no-op, so every reload would miss. Use a real in-memory size
...@@ -84,6 +88,7 @@ export function initializeCacheHandlers(cacheMaxMemorySize: number): boolean { ...@@ -84,6 +88,7 @@ export function initializeCacheHandlers(cacheMaxMemorySize: number): boolean {
} else { } else {
debug?.('setting "default" cache handler from default') debug?.('setting "default" cache handler from default')
fallback = createDefaultCacheHandler(builtInSize) fallback = createDefaultCacheHandler(builtInSize)
builtInHandlers.add(fallback)
} }
handlersMap.set('default', fallback) handlersMap.set('default', fallback)
...@@ -97,6 +102,7 @@ export function initializeCacheHandlers(cacheMaxMemorySize: number): boolean { ...@@ -97,6 +102,7 @@ export function initializeCacheHandlers(cacheMaxMemorySize: number): boolean {
} }
} else { } else {
const handler = createDefaultCacheHandler(builtInSize) const handler = createDefaultCacheHandler(builtInSize)
builtInHandlers.add(handler)
debug?.('setting "default" cache handler from default') debug?.('setting "default" cache handler from default')
handlersMap.set('default', handler) handlersMap.set('default', handler)
...@@ -189,6 +195,27 @@ export function isCustomCacheHandler(kind: string): boolean { ...@@ -189,6 +195,27 @@ export function isCustomCacheHandler(kind: string): boolean {
return reference[devFrontHandlersSymbol]?.has(kind) ?? false return reference[devFrontHandlersSymbol]?.has(kind) ?? false
} }
/**
* Whether `kind` resolves to one of the in-memory handlers that
* `initializeCacheHandlers` constructs itself, rather than one supplied by the
* platform through the `@next/cache-handlers` global or by `cacheHandlers`
* config. Reads from a built-in handler are map lookups, so repeating one
* within a request costs nothing; reads from a supplied handler may be a
* network round trip. Unlike `isCustomCacheHandler`, this holds in production.
*
* Tracking the instances rather than the kinds is what keeps aliasing correct:
* self-hosted, `remote` resolves to the same handler as `default` and so
* reports as built-in, while a platform-supplied `remote` does not.
*/
export function isBuiltInCacheHandler(kind: string): boolean {
const handler = reference[handlersMapSymbol]?.get(kind)
return (
handler !== undefined &&
Boolean(reference[builtInHandlersSymbol]?.has(handler))
)
}
/** /**
* Get the dev-only tiered cache handler for a custom `kind`: a fast built-in * Get the dev-only tiered cache handler for a custom `kind`: a fast built-in
* in-memory front handler in front of the user-configured backing handler, so * in-memory front handler in front of the user-configured backing handler, so
......
...@@ -73,6 +73,7 @@ import { ...@@ -73,6 +73,7 @@ import {
getCacheHandler, getCacheHandler,
getDevTieredCacheHandler, getDevTieredCacheHandler,
getPrivateCacheHandler, getPrivateCacheHandler,
isBuiltInCacheHandler,
isCustomCacheHandler, isCustomCacheHandler,
isMemoryCacheDisabled, isMemoryCacheDisabled,
} from './handlers' } from './handlers'
...@@ -257,17 +258,35 @@ class ResolvableSharedCacheResult { ...@@ -257,17 +258,35 @@ class ResolvableSharedCacheResult {
private readonly registrations: Array<{ private readonly registrations: Array<{
map: Map<string, Promise<SharedCacheResult>> map: Map<string, Promise<SharedCacheResult>>
key: string key: string
/**
* Where to move the entry once the invocation completes, instead of
* dropping it. Only set for the request-scoped map, and only for kinds
* whose entries are worth retaining; the module-scope cross-request map
* must never retain, since its entries would outlive the request that
* produced them.
*/
retentionMap?: Map<string, Promise<SharedCacheResult>>
}> = [] }> = []
registerIn(map: Map<string, Promise<SharedCacheResult>>, key: string): void { registerIn(
map: Map<string, Promise<SharedCacheResult>>,
key: string,
retentionMap?: Map<string, Promise<SharedCacheResult>>
): void {
map.set(key, this.deferred.promise) map.set(key, this.deferred.promise)
this.registrations.push({ map, key }) this.registrations.push({ map, key, retentionMap })
} }
resolve(result: SharedCacheResult): void { resolve(result: SharedCacheResult): void {
this.deferred.resolve(result) this.deferred.resolve(result)
if (result.type === 'cached') { if (result.type === 'cached') {
result.entry.pendingMetadata.finally(this.cleanup.bind(this)) // Retain only an entry that collected successfully. A failed collection
// leaves metadata a later invocation cannot read, so it is dropped like
// any other failure and the next invocation regenerates.
result.entry.pendingMetadata.then(
() => this.cleanupAndRetain(),
() => this.cleanup()
)
} else { } else {
this.cleanup() this.cleanup()
} }
...@@ -283,11 +302,29 @@ class ResolvableSharedCacheResult { ...@@ -283,11 +302,29 @@ class ResolvableSharedCacheResult {
this.cleanup() this.cleanup()
} }
/**
* Drops the pending registrations, leaving nothing for a later invocation to
* join. Used when the invocation produced no entry that a later one could
* serve: an error has no value, and a 'prerender-dynamic' result is a hanging
* promise bound to the leader's render signal.
*/
private cleanup(): void { private cleanup(): void {
for (const { map, key } of this.registrations) { for (const { map, key } of this.registrations) {
map.delete(key) map.delete(key)
} }
} }
/**
* Drops the pending registrations and moves the entry into the retention map
* where one was given, so a later invocation in the same request can reuse it
* instead of repeating the lookup and, on a miss, the work.
*/
private cleanupAndRetain(): void {
for (const { map, key, retentionMap } of this.registrations) {
map.delete(key)
retentionMap?.set(key, this.deferred.promise)
}
}
} }
/** /**
...@@ -521,6 +558,51 @@ function saveSharedCacheEntryToResumeDataCache( ...@@ -521,6 +558,51 @@ function saveSharedCacheEntryToResumeDataCache(
debug?.('Resume Data Cache entry saved by joiner', serializedCacheKey) debug?.('Resume Data Cache entry saved by joiner', serializedCacheKey)
} }
/**
* Completes a join onto another invocation's entry, whether that invocation is
* still filling or has already finished: forks the stream, saves to this
* invocation's RDC if the leader had none, and balances the cache signal read
* once the entry is fully collected.
*/
function serveJoinedCacheEntry(
sharedCacheEntry: SharedCacheEntry,
serializedCacheKey: string,
resumeDataCache: ResumeDataCache | null,
cacheContext: CacheContext,
cacheSignal: CacheSignal | null
): ReadableStream<Uint8Array> {
const stream = sharedCacheEntry.fork()
// If the leader was nested inside another cache (no accessible RDC), it
// couldn't save to the RDC. This joiner may be top-level with an RDC, in
// which case it must save here; otherwise the RDC lookup during the final
// prerender will miss.
saveSharedCacheEntryToResumeDataCache(
serializedCacheKey,
sharedCacheEntry,
resumeDataCache
)
// End the cache signal read when the result is fully collected, not when the
// stream is available. A failed collection has no metadata to propagate but
// must still balance the read, so both settlements end it; leaving it open
// would stall a prerender waiting for its cache reads. The trailing .catch()
// covers propagation itself throwing after the rendering stream resolved.
sharedCacheEntry.pendingMetadata
.then(
(metadata) => {
cacheSignal?.endRead()
maybePropagateCacheEntryMetadata(cacheContext, metadata)
},
() => {
cacheSignal?.endRead()
}
)
.catch(() => {})
return stream
}
function saveToCacheHandler( function saveToCacheHandler(
cacheHandler: CacheReadWriteHandler, cacheHandler: CacheReadWriteHandler,
workStore: WorkStore, workStore: WorkStore,
...@@ -2702,15 +2784,27 @@ export async function cache( ...@@ -2702,15 +2784,27 @@ export async function cache(
// doing redundant work. This also saves cache handler `get` calls which may // doing redundant work. This also saves cache handler `get` calls which may
// be HTTP round-trips for remote handlers. // be HTTP round-trips for remote handlers.
if (stream === undefined) { if (stream === undefined) {
const intraRequestPendingCacheInvocation = const pendingInvocation =
workStore.pendingCacheInvocations?.get(serializedCacheKey) workStore.pendingCacheInvocations?.get(serializedCacheKey)
if (intraRequestPendingCacheInvocation) { // A pending invocation is joined unconditionally: its fill is shared, so
// every joiner receives whatever that fill produces. A completed one is a
// stored entry instead, so it is only reused when the caller hasn't asked
// to bypass caches, and only if nothing has invalidated it since.
const completedInvocation =
pendingInvocation === undefined &&
!shouldForceRevalidate(workStore, workUnitStore)
? workStore.completedCacheInvocations?.get(serializedCacheKey)
: undefined
const joinedInvocation = pendingInvocation ?? completedInvocation
if (joinedInvocation) {
const cacheSignal = getCacheSignal(workUnitStore) const cacheSignal = getCacheSignal(workUnitStore)
cacheSignal?.beginRead() cacheSignal?.beginRead()
debug?.('joining pending intra-request invocation', serializedCacheKey) debug?.('joining intra-request invocation', serializedCacheKey)
const sharedCacheResult = await intraRequestPendingCacheInvocation const sharedCacheResult = await joinedInvocation
if (sharedCacheResult.type === 'prerender-dynamic') { if (sharedCacheResult.type === 'prerender-dynamic') {
debug?.('joined invocation is prerender-dynamic', serializedCacheKey) debug?.('joined invocation is prerender-dynamic', serializedCacheKey)
...@@ -2718,33 +2812,49 @@ export async function cache( ...@@ -2718,33 +2812,49 @@ export async function cache(
return sharedCacheResult.hangingPromise return sharedCacheResult.hangingPromise
} }
// A completed entry may have been invalidated since it was produced, by
// an `updateTag()` in a server action earlier in this request. The
// implicit tags expiration is passed as 0 because it cannot apply to a
// retained entry: it is memoized for the request, so re-checking it
// against an unchanged entry timestamp only repeats the answer the leader
// already got.
const metadata =
completedInvocation === undefined
? undefined
: await sharedCacheResult.entry.pendingMetadata
if (
metadata !== undefined &&
shouldDiscardCacheEntry(
metadata,
workStore,
workUnitStore,
implicitTags,
0
)
) {
debug?.('discarding completed invocation', serializedCacheKey)
workStore.completedCacheInvocations?.delete(serializedCacheKey)
// This can take the signal's count to zero. The leader path below opens
// a read again synchronously, before it awaits anything, which is what
// stops a prerender from concluding that every cache read has settled
// while a fill is still about to start.
cacheSignal?.endRead()
} else {
debug?.( debug?.(
'joined invocation resolved with cached entry', 'joined invocation resolved with cached entry',
serializedCacheKey serializedCacheKey
) )
stream = sharedCacheResult.entry.fork() stream = serveJoinedCacheEntry(
// If the leader was nested inside another cache (no accessible RDC), it
// couldn't save to the RDC. This joiner may be top-level with an RDC, in
// which case it must save here; otherwise the RDC lookup during the
// final prerender will miss.
saveSharedCacheEntryToResumeDataCache(
serializedCacheKey,
sharedCacheResult.entry, sharedCacheResult.entry,
resumeDataCache serializedCacheKey,
resumeDataCache,
cacheContext,
cacheSignal
) )
}
// End the cache signal read when the result is fully collected, not when
// the stream is available. Fire-and-forget propagation runs in the same
// .then() callback. .catch() prevents unhandled rejection if collection
// fails after the rendering stream was already resolved.
sharedCacheResult.entry.pendingMetadata
.then((metadata) => {
cacheSignal?.endRead()
maybePropagateCacheEntryMetadata(cacheContext, metadata)
})
.catch(() => {})
} }
} }
...@@ -2762,9 +2872,25 @@ export async function cache( ...@@ -2762,9 +2872,25 @@ export async function cache(
string, string,
Promise<SharedCacheResult> Promise<SharedCacheResult>
>()) >())
// Retain the completed entry for the rest of the request where a later
// invocation would otherwise repeat real work: private caches have no cache
// handler to fall back on in production, and a platform- or config-supplied
// handler may be remote, so a second `get` can be a round trip. A built-in
// handler read is a map lookup, so retaining its entries would hold a
// forked stream buffer for nothing.
const completedCacheInvocations =
isPrivate || !isBuiltInCacheHandler(kind)
? (workStore.completedCacheInvocations ??= new Map<
string,
Promise<SharedCacheResult>
>())
: undefined
resolvableSharedCacheResult.registerIn( resolvableSharedCacheResult.registerIn(
intraRequestPendingCacheInvocations, intraRequestPendingCacheInvocations,
serializedCacheKey serializedCacheKey,
completedCacheInvocations
) )
// Cross-request deduplication lets concurrent requests for the same key // Cross-request deduplication lets concurrent requests for the same key
...@@ -3510,7 +3636,7 @@ function shouldForceRevalidate( ...@@ -3510,7 +3636,7 @@ function shouldForceRevalidate(
} }
function shouldDiscardCacheEntry( function shouldDiscardCacheEntry(
entry: CacheEntry, entry: Pick<CacheEntry, 'tags' | 'timestamp'>,
workStore: WorkStore, workStore: WorkStore,
workUnitStore: WorkUnitStore, workUnitStore: WorkUnitStore,
implicitTags: string[], implicitTags: string[],
......
import { connection } from 'next/server'
import { setTimeout } from 'timers/promises'
async function getData(params: { id: number }) {
'use cache'
return `${params.id}:${new Date().toISOString()}`
}
export default async function Page() {
await connection()
// A fresh object literal per call, so the React.cache memo misses on
// reference equality, and a delay between them so the second call is outside
// the in-flight dedupe window. Reuse therefore has to come from a stored
// entry.
const first = await getData({ id: 1 })
await setTimeout(100)
const second = await getData({ id: 1 })
return (
<div>
<p id="first">{first}</p>
<p id="second">{second}</p>
</div>
)
}
...@@ -156,4 +156,27 @@ describe('use-cache-custom-handler', () => { ...@@ -156,4 +156,27 @@ describe('use-cache-custom-handler', () => {
) )
}) })
}) })
it('should reach the cache handler only once for sequential calls in a request', async () => {
const $ = await next.render$('/sequential-dedupe')
expect($('#first').text()).toBe($('#second').text())
await retry(async () => {
const cliOutput = next.cliOutput.slice(outputIndex)
// The first call misses and fills. The second is served from the entry
// retained for this request, so it never reaches the handler, which for a
// registered handler can be a remote round trip.
expect(cliOutput).toIncludeRepeated(`ModernCustomCacheHandler::get`, 1)
// Matched up to the escaped opening bracket of the logged cache key,
// since `::set-resolved-entry` would otherwise count as another `::set`.
// The matcher compiles its argument as a regular expression.
expect(cliOutput).toIncludeRepeated(
`ModernCustomCacheHandler::set \\[`,
1
)
})
})
}) })
import { connection } from 'next/server'
import { setTimeout } from 'timers/promises'
let invocations = 0
async function getData(params: { id: number }) {
'use cache'
return `${params.id}:${++invocations}`
}
export default async function Page() {
await connection()
// The public counterpart of `/private-dedup-sequential`, with the same call
// shape: a fresh object literal per call so the React.cache memo misses, and
// a delay that puts the second call outside the in-flight dedupe window. Here
// the cache handler serves the second call.
const first = await getData({ id: 1 })
await setTimeout(100)
const second = await getData({ id: 1 })
return (
<div>
<p className="first">{first}</p>
<p className="second">{second}</p>
</div>
)
}
import { connection } from 'next/server'
import { setTimeout } from 'timers/promises'
let invocations = 0
async function getData(params: { id: number }) {
'use cache: private'
return `${params.id}:${++invocations}`
}
export default async function Page() {
await connection()
// Each call passes a fresh object literal, so the React.cache memo that wraps
// every cache function misses on reference equality and the lookup falls
// through to the serialized cache key.
const first = await getData({ id: 1 })
// The intra-request dedupe map drops an entry once its fill completes, so
// this delay puts the second call outside the in-flight window. Reuse then
// depends on the cache store rather than on joining a pending fill.
await setTimeout(100)
const second = await getData({ id: 1 })
return (
<div>
<p className="first">{first}</p>
<p className="second">{second}</p>
</div>
)
}
...@@ -1668,13 +1668,23 @@ describe('use-cache', () => { ...@@ -1668,13 +1668,23 @@ describe('use-cache', () => {
expect(randA).toBe(randB) expect(randA).toBe(randB)
}) })
it('should dedupe private caches within a single request', async () => { it('should dedupe private caches for concurrent calls within a single request', async () => {
const browser = await next.browser('/private-dedup') const browser = await next.browser('/private-dedup')
const first = await browser.elementByCss('.rand:nth-of-type(1)').text() const first = await browser.elementByCss('.rand:nth-of-type(1)').text()
const second = await browser.elementByCss('.rand:nth-of-type(2)').text() const second = await browser.elementByCss('.rand:nth-of-type(2)').text()
expect(first).toBe(second) expect(first).toBe(second)
}) })
it('should dedupe private caches for sequential calls within a single request', async () => {
const $ = await next.render$('/private-dedup-sequential')
expect($('.first').text()).toBe($('.second').text())
})
it('should dedupe caches for sequential calls within a single request', async () => {
const $ = await next.render$('/dedup-sequential')
expect($('.first').text()).toBe($('.second').text())
})
it('should dedupe sequential calls in a server action and its subsequent render', async () => { it('should dedupe sequential calls in a server action and its subsequent render', async () => {
const browser = await next.browser('/action-dedupe') const browser = await next.browser('/action-dedupe')
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment