Unverified Commit 770f1577 authored by Pete Hunt's avatar Pete Hunt Committed by GitHub

fix: release compression stream when client disconnects mid-response (#96173)

## Summary

Fixes #94919.

When a client disconnects mid-response, the compression middleware's
zlib stream is never released, and every aborted compressed response
permanently retains its deflate state (~256 KiB). On a server taking
traffic that aborts mid-stream (bots, CDNs, users navigating away) RSS
climbs until the process is OOM-killed.

`router-server.ts` applies the vendored `compression` middleware, which
ends its zlib stream *only* from inside its own `res.end()` wrapper. On
a client disconnect Next destroys the response instead, so that wrapper
never runs and the stream stays open. An open zlib stream is pinned by
its native handle, so it is not merely uncollected-for-now — it survives
GC entirely.

This is why the reported symptom looks unlike a normal JS leak: the JS
heap stays nearly flat while RSS grows without bound, because the
retained deflate state is native memory. It affects `next start`, `next
dev`, and custom servers — anything going through the router server with
the default `compress: true`. Deployments that let a proxy handle
compression never see it.

Note that ending the stream is not sufficient to recover it: its `data`
handler writes to the already-destroyed response, `res.write()` returns
`false`, the middleware pauses the stream, and it never reaches `end`.
It has to be destroyed. `releaseCompressionStream` carries a comment
explaining this so the cleanup isn't "simplified" back into a no-op
later.

The stream is reached through the one handle the middleware exposes: it
forwards `res.on('drain', …)` to the zlib stream once that stream
exists, and `EventEmitter#on` returns the emitter it was called on.
Upstream `compression@1.8.1` still has no premature-close handling, so
this can't be fixed by bumping the vendored copy. The added test asserts
that assumption and will fail if a future vendored version handles this
itself, signalling that the workaround can be dropped.

### Before / after

Measured with a minimal custom Node server (no Express/axios) rendering
a 1.25 MB dynamic App Router page, 8 rounds × 300 requests cancelled
after the first chunk, `heapUsed`/`rss` sampled **after forced GC** each
round. Same build, cleanup toggled behind a temporary env var:

| | without cleanup | with cleanup |
| --- | --- | --- |
| zlib contexts never released | 2400 of 2400 (100%) | **0** of 2600 |
| RSS | 98.2 → 1069.6 MiB, linear | 100 → 401.5 MiB, plateaus |
| `external` | 3.6 → 76.5 MiB | 3.6 → 4.5 MiB |
| `heapUsed` | 21.2 → 79.1 MiB | 21.2 → 27.4 MiB |

Without the cleanup, RSS grows by a steady ~93 MiB per 300 aborted
requests across rounds 2–8 (≈310 KiB per request, consistent with zlib's
default deflate state plus per-stream buffers), and every unreleased
stream is *still reachable after repeated forced GC* — permanent
retention rather than GC lag. With the cleanup, all 2600 contexts are
closed and RSS plateaus, with the residual difference from baseline
being allocator retention rather than per-request growth.

As a control, 200 fully-read responses issued into the same un-fixed
process all closed normally, isolating mid-stream disconnects as the
trigger.

The issue also attributes the leak to a retained RSC element tree held
via the `WeakMap` in `use-flight-response.tsx`. That part did not
reproduce on canary: across 8000 aborted requests spanning dynamic
renders, `redirect()`, `notFound()` and render throws, and 3000 full
reads of the same 1.25 MB payload, post-GC heap stayed flat at ~29 MiB.
The native `zlib_memory` line in the reporter's heap-snapshot diff was
the actual cause.

## Verification

- `pnpm test-start-turbo test/e2e/custom-server/custom-server.test.ts`
and `pnpm test-dev-turbo test/e2e/custom-server/custom-server.test.ts` —
covers `Content-Encoding: gzip` through a custom server, i.e. the code
path being changed (46 and 45 passing)
- `npx jest packages/next/src/server/lib` — 188 passing, including the 4
new cases
- Manual: verified a gzipped response still decompresses byte-identical
to the same response with `accept-encoding: identity` (1,250,721 bytes),
with no errors or `MaxListenersExceededWarning` under sustained load
- Manual: the before/after table above, re-measured against this
branch's HEAD

<!-- NEXT_JS_LLM -->
Co-authored-by: 's avatarPete Hunt <239742+petehunt@users.noreply.github.com>
parent 3c4afc1a
/* eslint-env jest */
import type { AddressInfo } from 'node:net'
import http from 'node:http'
import zlib from 'node:zlib'
import setupCompression from 'next/dist/compiled/compression'
import { releaseCompressionStream } from './release-compression-stream'
type StreamRecord = { closed: boolean }
/** Records every zlib stream the middleware creates, and whether it closed. */
function trackCompressionStreams(): {
records: StreamRecord[]
restore: () => void
} {
const records: StreamRecord[] = []
const originals: Array<[string, unknown]> = []
for (const name of ['createGzip', 'createDeflate'] as const) {
const original = zlib[name]
originals.push([name, original])
// zlib's exports are non-writable, so they have to be redefined.
Object.defineProperty(zlib, name, {
configurable: true,
writable: true,
value: (...args: Parameters<typeof original>) => {
const stream = original(...args)
const record: StreamRecord = { closed: false }
stream.once('close', () => {
record.closed = true
})
records.push(record)
return stream
},
})
}
return {
records,
restore: () => {
for (const [name, original] of originals) {
Object.defineProperty(zlib, name, {
configurable: true,
writable: true,
value: original,
})
}
},
}
}
function createServer({ release }: { release: boolean }) {
const compress = setupCompression()
return http.createServer((req, res) => {
// @ts-expect-error not express req/res
compress(req, res, () => {})
if (release) {
res.once('close', () => {
if (res.writableFinished) return
releaseCompressionStream(res)
})
}
if (req.url === '/complete') {
res.setHeader('content-type', 'text/html')
res.end(`<html><body>${'x'.repeat(4096)}</body></html>`)
return
}
res.setHeader('content-type', 'text/html')
res.write(`<html><body>${'x'.repeat(4096)}`)
// Flush so the client receives a chunk, then leave the response open.
;(res as any).flush()
})
}
async function listen(server: http.Server): Promise<string> {
await new Promise<void>((resolve) => server.listen(0, resolve))
const { port } = server.address() as AddressInfo
return `http://127.0.0.1:${port}`
}
/** Reads the first chunk, then disconnects mid-response. */
async function requestThenDisconnect(url: string) {
const res = await fetch(url, {
headers: { 'accept-encoding': 'gzip' },
signal: AbortSignal.timeout(10_000),
})
const reader = res.body!.getReader()
await reader.read()
await reader.cancel()
}
describe('releaseCompressionStream', () => {
let tracker: ReturnType<typeof trackCompressionStreams>
let server: http.Server | undefined
beforeEach(() => {
tracker = trackCompressionStreams()
})
afterEach(async () => {
tracker.restore()
if (server) {
await new Promise<void>((resolve) => server!.close(() => resolve()))
server = undefined
}
})
it('releases the zlib stream when the client disconnects mid-response', async () => {
server = createServer({ release: true })
const url = await listen(server)
await requestThenDisconnect(url)
// Let the response's `close` event run.
await new Promise((resolve) => setTimeout(resolve, 100))
expect(tracker.records).toHaveLength(1)
expect(tracker.records[0].closed).toBe(true)
})
it('is still needed by the vendored compression middleware', async () => {
// If this starts failing, the vendored `compression` handles premature close
// itself and `releaseCompressionStream` can be removed.
server = createServer({ release: false })
const url = await listen(server)
await requestThenDisconnect(url)
await new Promise((resolve) => setTimeout(resolve, 100))
expect(tracker.records).toHaveLength(1)
expect(tracker.records[0].closed).toBe(false)
})
it('does not affect responses that complete normally', async () => {
server = createServer({ release: true })
const url = await listen(server)
const res = await fetch(`${url}/complete`, {
headers: { 'accept-encoding': 'gzip' },
signal: AbortSignal.timeout(10_000),
})
expect(res.headers.get('content-encoding')).toBe('gzip')
// `fetch` decompresses, so this verifies the payload survived intact.
expect(await res.text()).toBe(
`<html><body>${'x'.repeat(4096)}</body></html>`
)
await new Promise((resolve) => setTimeout(resolve, 100))
expect(tracker.records).toHaveLength(1)
expect(tracker.records[0].closed).toBe(true)
})
it('is a no-op when compression is not active for the response', async () => {
const compress = setupCompression()
// Asserted in the test body so a failure isn't swallowed by the callback.
const outcome: { threw: unknown; drainListeners: number } = {
threw: null,
drainListeners: -1,
}
server = http.createServer((req, res) => {
// @ts-expect-error not express req/res
compress(req, res, () => {})
// `application/octet-stream` is not compressible, so no zlib stream is
// created and there is nothing to release.
res.setHeader('content-type', 'application/octet-stream')
res.write('x'.repeat(4096))
res.once('close', () => {
try {
releaseCompressionStream(res)
} catch (err) {
outcome.threw = err
}
outcome.drainListeners = res.listenerCount('drain')
})
})
const url = await listen(server)
await requestThenDisconnect(url)
await new Promise((resolve) => setTimeout(resolve, 100))
expect(tracker.records).toHaveLength(0)
expect(outcome.threw).toBeNull()
// Cleanup must not leave a stray listener behind on the response.
expect(outcome.drainListeners).toBe(0)
})
})
import type { ServerResponse } from 'node:http'
import { Duplex } from 'node:stream'
const noop = () => {}
/**
* Destroys the zlib stream that the `compression` middleware left open on a
* response that never finished.
*
* The middleware ends its stream only from its own `res.end()` wrapper, so a
* client disconnect leaves it open, and an open zlib stream is pinned by its
* native handle: it survives GC, permanently retaining ~256 KiB of deflate
* state. Ending it rather than destroying it does not work -- its `data` handler
* writes to the dead response, `res.write()` returns `false`, and the middleware
* pauses the stream, so it never reaches `end`.
*
* `res.on('drain', …)` is forwarded to that stream and `EventEmitter#on` returns
* the emitter, which is the only handle to it from outside. When compression is
* inactive the call returns `res` itself.
*/
export function releaseCompressionStream(res: ServerResponse): void {
const maybeStream: unknown = res.on('drain', noop)
// The `Duplex` check keeps a future middleware change from turning this into a
// `destroy()` on a non-stream, which would throw from a `close` handler.
if (maybeStream === res || !(maybeStream instanceof Duplex)) {
res.off('drain', noop)
return
}
maybeStream.destroy()
}
...@@ -24,6 +24,7 @@ import { addRequestMeta, getRequestMeta } from '../request-meta' ...@@ -24,6 +24,7 @@ import { addRequestMeta, getRequestMeta } from '../request-meta'
import { pathHasPrefix } from '../../shared/lib/router/utils/path-has-prefix' import { pathHasPrefix } from '../../shared/lib/router/utils/path-has-prefix'
import { removePathPrefix } from '../../shared/lib/router/utils/remove-path-prefix' import { removePathPrefix } from '../../shared/lib/router/utils/remove-path-prefix'
import setupCompression from 'next/dist/compiled/compression' import setupCompression from 'next/dist/compiled/compression'
import { releaseCompressionStream } from './release-compression-stream'
import { signalFromNodeResponse } from '../web/spec-extension/adapters/next-request' import { signalFromNodeResponse } from '../web/spec-extension/adapters/next-request'
import { isPostpone } from './router-utils/is-postpone' import { isPostpone } from './router-utils/is-postpone'
import { isNonHtmlSecFetchDest } from './is-non-html-sec-fetch-dest' import { isNonHtmlSecFetchDest } from './is-non-html-sec-fetch-dest'
...@@ -342,6 +343,14 @@ export async function initialize(opts: { ...@@ -342,6 +343,14 @@ export async function initialize(opts: {
if (compress) { if (compress) {
// @ts-expect-error not express req/res // @ts-expect-error not express req/res
compress(req, res, () => {}) compress(req, res, () => {})
// On client disconnect the middleware never ends its zlib stream, which
// then leaks past GC. See `releaseCompressionStream`.
res.once('close', () => {
if (res.writableFinished) return
releaseCompressionStream(res)
})
} }
req.on('error', (_err) => { req.on('error', (_err) => {
// TODO: log socket errors? // TODO: log socket errors?
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment