Unverified Commit 9a31a176 authored by dan's avatar dan Committed by GitHub

Count URL key length in the filesystem route cache LRU (#96229)

Fixes #94890

Adopted from #96187 and #94699, with both authors credited on the
commit.

The router server keeps an in-memory cache that maps request URL paths
to what they resolve to: a page, a static file, or nothing.

It's bounded by a size budget, but the size of each entry was computed
from the cached value only. The URL string used as the key was free, and
"resolves to nothing" entries counted as size 1. So a long-lived server
receiving many unique URLs — bot crawls, long slugs, `/order/:id`-style
traffic — could accumulate up to ~1M URL strings while the cache
believed it was within budget.

That looks like a slow memory leak that tracks how many distinct URLs
you've served rather than how much traffic you get. #94890 has several
production reports, including two deployments that patched key
accounting in and saw memory go flat.

The idea of the fix is to make the cache's bookkeeping match what an
entry actually keeps alive, so the budget becomes a real memory bound.
That takes three things:

1. Count the key string, not just the value.
2. Count the fixed cost an entry has regardless of string lengths — the
per-entry bookkeeping objects, measured at ~120 bytes. Without this, a
flood of *short* unique URLs stays "cheap" on paper while the
bookkeeping itself grows unbounded.
3. Copy strings before caching them. When V8 takes a substring of a long
string (which is how the pathname is extracted from the request URL), it
doesn't copy the characters — it creates a small object that points into
the original string, and the original can't be garbage-collected while
the substring is alive. So caching a 30-char pathname could actually
hold the entire URL in memory, kilobyte query string and all, and the
bookkeeping only sees the 30 chars. Copying the string before caching
makes it hold only its own characters.

With the bookkeeping honest, the budget is set to 8 MiB, roughly the
real memory the cache could already use for legitimate entries before
this change, so capacity isn't reduced — the previous attempt at this
fix (#94942) stalled on that concern.


## Test Plan

A/B on a production `next start` serving 60k requests with unique long
URLs (the reported traffic shape): canary grows +79 MB and keeps
growing; this branch grows +9 MB and plateaus — the budget working as
intended. The per-entry costs above were measured directly on the cache
with forced GC (they fit `118 + key length` bytes within a few percent,
which is also what predicts the A/B numbers). `lru-cache` unit tests
(26/26) and `pnpm --filter=next types` pass.

---------
Co-authored-by: 's avatarMarcos Hernanz <96699542+marcoshernanz@users.noreply.github.com>
Co-authored-by: 's avatardmchoi77 <dm.choi.dev@gmail.com>
parent 606b8e88
...@@ -105,6 +105,20 @@ describe('LRUCache', () => { ...@@ -105,6 +105,20 @@ describe('LRUCache', () => {
expect(cache.currentSize).toBe(8) // 5 + 2 + 1 expect(cache.currentSize).toBe(8) // 5 + 2 + 1
}) })
it('should pass the cache key to custom size calculation', () => {
const cache = new LRUCache<string>(
10,
(value, key) => value.length + key.length
)
cache.set('long-key', 'ab') // size 10
cache.set('b', 'c') // total size 12, should evict long-key
expect(cache.has('long-key')).toBe(false)
expect(cache.has('b')).toBe(true)
expect(cache.currentSize).toBe(2)
})
it('should prevent adding item larger than max size when lru is empty', () => { it('should prevent adding item larger than max size when lru is empty', () => {
const consoleSpy = jest.spyOn(console, 'warn').mockImplementation() const consoleSpy = jest.spyOn(console, 'warn').mockImplementation()
const cache = new LRUCache<string>(5, (value) => value.length) const cache = new LRUCache<string>(5, (value) => value.length)
......
...@@ -49,12 +49,14 @@ export class LRUCache<T> { ...@@ -49,12 +49,14 @@ export class LRUCache<T> {
private readonly tail: SentinelNode<T> private readonly tail: SentinelNode<T>
private totalSize: number = 0 private totalSize: number = 0
private readonly maxSize: number private readonly maxSize: number
private readonly calculateSize: ((value: T) => number) | undefined private readonly calculateSize:
| ((value: T, key: string) => number)
| undefined
private readonly onEvict: ((key: string, value: T) => void) | undefined private readonly onEvict: ((key: string, value: T) => void) | undefined
constructor( constructor(
maxSize: number, maxSize: number,
calculateSize?: (value: T) => number, calculateSize?: (value: T, key: string) => number,
onEvict?: (key: string, value: T) => void onEvict?: (key: string, value: T) => void
) { ) {
this.maxSize = maxSize this.maxSize = maxSize
...@@ -124,7 +126,7 @@ export class LRUCache<T> { ...@@ -124,7 +126,7 @@ export class LRUCache<T> {
* - O(k) where k is the number of items evicted (can be O(N) for variable sizes) * - O(k) where k is the number of items evicted (can be O(N) for variable sizes)
*/ */
public set(key: string, value: T): boolean { public set(key: string, value: T): boolean {
const size = this.calculateSize?.(value) ?? 1 const size = this.calculateSize?.(value, key) ?? 1
if (size <= 0) { if (size <= 0) {
throw new Error( throw new Error(
`LRUCache: calculateSize returned ${size}, but size must be > 0. ` + `LRUCache: calculateSize returned ${size}, but size must be > 0. ` +
......
...@@ -106,6 +106,23 @@ export const buildCustomRoute = <T>( ...@@ -106,6 +106,23 @@ export const buildCustomRoute = <T>(
} }
} }
// Measured retained cost of a cache entry beyond its strings (LRUNode,
// Map slot, string header): ~120 bytes. Counting it keeps the entry count
// bounded even when keys are short, so the budget approximates retained
// bytes.
const FS_LRU_ENTRY_OVERHEAD = 128
const FS_LRU_MAX_SIZE = 8 * 1024 * 1024
// The pathname passed to getItem is usually a V8 slice of the full request
// URL, and a sliced string retains its parent — including the query string —
// for as long as the cache holds the key. Store a flat copy instead.
// The JSON round-trip returns an equal string for every input (unlike a
// Buffer round-trip, which replaces lone surrogates), so distinct keys can
// never collide on the stored copy.
function flatKeyCopy(key: string): string {
return JSON.parse(JSON.stringify(key))
}
export async function setupFsCheck(opts: { export async function setupFsCheck(opts: {
dir: string dir: string
dev: boolean dev: boolean
...@@ -113,12 +130,17 @@ export async function setupFsCheck(opts: { ...@@ -113,12 +130,17 @@ export async function setupFsCheck(opts: {
config: NextConfigRuntime config: NextConfigRuntime
}) { }) {
const getItemsLru = !opts.dev const getItemsLru = !opts.dev
? new LRUCache<FsOutput | null>(1024 * 1024, function length(value) { ? new LRUCache<FsOutput | null>(FS_LRU_MAX_SIZE, function length(
value,
key
) {
const size = FS_LRU_ENTRY_OVERHEAD + key.length
if (!value) { if (!value) {
// Null entries (negative cache) still need a non-zero size for LRU eviction // Negative cache entries only retain their key.
return 1 return size
} }
return ( return (
size +
(value.fsPath || '').length + (value.fsPath || '').length +
value.itemPath.length + value.itemPath.length +
value.type.length value.type.length
...@@ -723,15 +745,17 @@ export async function setupFsCheck(opts: { ...@@ -723,15 +745,17 @@ export async function setupFsCheck(opts: {
fsPath, fsPath,
locale, locale,
itemsRoot, itemsRoot,
itemPath: curItemPath, // itemPath is usually a slice of the request URL too; keep a
// flat copy so the cached value doesn't retain the full URL.
itemPath: flatKeyCopy(curItemPath),
} }
getItemsLru?.set(itemKey, itemResult) getItemsLru?.set(flatKeyCopy(itemKey), itemResult)
return itemResult return itemResult
} }
} }
getItemsLru?.set(itemKey, null) getItemsLru?.set(flatKeyCopy(itemKey), null)
return null return null
}, },
getDynamicRoutes() { getDynamicRoutes() {
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment