Unverified Commit 37d603bf authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

[lts-active] Keep OpenTelemetry request traces together (#100349, #100294,…

[lts-active] Keep OpenTelemetry request traces together (#100349, #100294, #100324, #100350, #100552, #100583) (#100542)

Backports the OpenTelemetry request-trace stack to
`releases/lts/active`:

- #100349 Start the request span before route module preparation
- #100294 Keep middleware and render in one OpenTelemetry trace as
siblings (re-lands #100136, whose LTS backport #100226 was reverted in
#100269)
- #100324 docs: note that instrumentation-time spans don't join request
traces
- #100350 Emit instrumentation lifecycle spans in the request that
registered it
- #100552 Preserve deferred middleware instrumentation spans when the
adapter throws
- #100583 test: add non-adapter deploy snapshots to
otel-middleware-trace. Without it, `otel-middleware-trace` fails the
release deploy tests that deploy with `NEXT_ENABLE_ADAPTER=0`, as on
v16.5.0-canary.23.

The six canary merge commits are cherry-picked in order with `-x`, and
all applied cleanly. The patch content is identical to canary's; only
blob IDs and line offsets differ.

## Verification

- `test/e2e/opentelemetry/instrumentation/`: `test-start-turbo` (122
passed) and `test-dev-turbo` (67 passed). The #100552 regression test
fails without its `middleware.ts` change.
- `test/e2e/app-dir/otel-middleware-trace`: `test-start-turbo`,
`test-dev-turbo` and `test-start-webpack` (30 passed each, inline
snapshots unchanged). The #100583 commit leaves the test file
byte-identical to canary's.
- Unit tests for `instrumentation-globals`, `tracer` and
`forward-trace-context` (40 passed), plus `pnpm --filter=next types`
- Built with `CI=true pnpm build-all` against the published `@next/swc`
16.4.2 binary. The backport touches no Rust.
- PR CI deploys only through the adapter, so the non-adapter snapshots
from #100583 are confirmed by the next release deploy run.

<!-- NEXT_JS_LLM -->


<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
parent cf9d27d9
...@@ -248,7 +248,9 @@ export async function fetchGithubStars() { ...@@ -248,7 +248,9 @@ export async function fetchGithubStars() {
``` ```
The `register` function will execute before your code runs in a new environment. The `register` function will execute before your code runs in a new environment.
You can start creating new spans, and they should be correctly added to the exported trace. After that, spans you create while handling a request are added to that request's trace.
> **Good to know**: Spans created while the `instrumentation` file is loaded, or inside `register`, don't automatically join a request trace. Spans started before the tracer provider is installed (for example, at the top level of the file before `registerOTel` runs) are not recorded, and spans started inside `register` after `registerOTel` (or your OpenTelemetry SDK) has started typically become separate root traces. To trace work as part of a request, create the span in code that runs while handling the request, such as when rendering a page or in a Route Handler or API Route.
## Default Spans in Next.js ## Default Spans in Next.js
......
...@@ -15,7 +15,6 @@ import { RouteKind } from '../../server/route-kind' with { 'turbopack-transition ...@@ -15,7 +15,6 @@ import { RouteKind } from '../../server/route-kind' with { 'turbopack-transition
import { getRevalidateReason } from '../../server/instrumentation/utils' with { 'turbopack-transition': 'next-server-utility' } import { getRevalidateReason } from '../../server/instrumentation/utils' with { 'turbopack-transition': 'next-server-utility' }
import { import {
getTracer, getTracer,
SpanKind,
SpanStatusCode, SpanStatusCode,
type Span, type Span,
} from '../../server/lib/trace/tracer' with { 'turbopack-transition': 'next-server-utility' } } from '../../server/lib/trace/tracer' with { 'turbopack-transition': 'next-server-utility' }
...@@ -242,6 +241,47 @@ export function createAppPageEntrypoint({ ...@@ -242,6 +241,47 @@ export function createAppPageEntrypoint({
const multiZoneDraftMode = process.env const multiZoneDraftMode = process.env
.__NEXT_MULTI_ZONE_DRAFT_MODE as any as boolean .__NEXT_MULTI_ZONE_DRAFT_MODE as any as boolean
// Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace.
// When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest(
req,
{
tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget,
instrumentation,
},
(requestSpan, requestRootSpan) =>
handleRequest(
req,
res,
ctx,
isMinimalMode,
multiZoneDraftMode,
requestSpan,
requestRootSpan
)
)
}
async function handleRequest(
req: IncomingMessage,
res: ServerResponse,
ctx: {
waitUntil?: (prom: Promise<void>) => void
requestMeta?: RequestMeta
},
isMinimalMode: boolean,
multiZoneDraftMode: boolean,
requestSpan: Span | undefined,
requestRootSpan: Span | undefined
) {
const prepareResult = await routeModule.prepare(req, res, { const prepareResult = await routeModule.prepare(req, res, {
srcPage, srcPage,
multiZoneDraftMode, multiZoneDraftMode,
...@@ -752,10 +792,6 @@ export function createAppPageEntrypoint({ ...@@ -752,10 +792,6 @@ export function createAppPageEntrypoint({
const method = req.method || 'GET' const method = req.method || 'GET'
const tracer = getTracer() const tracer = getTracer()
const activeSpan = tracer.getActiveScopeSpan()
const isWrappedByNextServer = Boolean(
routerServerContext?.isWrappedByNextServer
)
const render404 = async () => { const render404 = async () => {
// TODO: should route-module itself handle rendering the 404 // TODO: should route-module itself handle rendering the 404
if (routerServerContext?.render404) { if (routerServerContext?.render404) {
...@@ -773,7 +809,6 @@ export function createAppPageEntrypoint({ ...@@ -773,7 +809,6 @@ export function createAppPageEntrypoint({
interceptionRoutePatterns interceptionRoutePatterns
) )
res.setHeader('Vary', varyHeader) res.setHeader('Vary', varyHeader)
let parentSpan: Span | undefined
const invokeRouteModule = async ( const invokeRouteModule = async (
span: Span | undefined, span: Span | undefined,
context: AppPageRouteHandlerContext, context: AppPageRouteHandlerContext,
...@@ -836,11 +871,12 @@ export function createAppPageEntrypoint({ ...@@ -836,11 +871,12 @@ export function createAppPageEntrypoint({
}) })
span.updateName(name) span.updateName(name)
// Propagate http.route to the parent span if one exists (e.g. // Propagate http.route to the request's root span if it isn't the
// a platform-created HTTP span in adapter deployments). // request span (e.g. a platform-created HTTP span in adapter
if (parentSpan && parentSpan !== span) { // deployments).
parentSpan.setAttribute('http.route', route) if (requestRootSpan && requestRootSpan !== span) {
parentSpan.updateName(name) requestRootSpan.setAttribute('http.route', route)
requestRootSpan.updateName(name)
} }
}) })
} }
...@@ -2389,31 +2425,7 @@ export function createAppPageEntrypoint({ ...@@ -2389,31 +2425,7 @@ export function createAppPageEntrypoint({
}) })
} }
// TODO: activeSpan code path is for when wrapped by return await handleResponse(requestSpan)
// next-server can be removed when this is no longer used
if (isWrappedByNextServer && activeSpan) {
await handleResponse(activeSpan)
} else {
parentSpan = tracer.getActiveScopeSpan()
return await tracer.withPropagatedContext(
req.headers,
() =>
tracer.trace(
BaseServerSpan.handleRequest,
{
spanName: `${method} ${srcPage}`,
kind: SpanKind.SERVER,
attributes: {
'http.method': method,
'http.target': httpTarget,
},
},
handleResponse
),
undefined,
!isWrappedByNextServer
)
}
} catch (err) { } catch (err) {
// If an Instant Navigation Testing document render fails (e.g. the page // If an Instant Navigation Testing document render fails (e.g. the page
// blocks at the root with no Suspense boundary above it, producing an empty // blocks at the root with no Suspense boundary above it, producing an empty
......
...@@ -15,7 +15,6 @@ import { ...@@ -15,7 +15,6 @@ import {
import { import {
getTracer, getTracer,
type Span, type Span,
SpanKind,
SpanStatusCode, SpanStatusCode,
} from '../../server/lib/trace/tracer' } from '../../server/lib/trace/tracer'
import { setManifestsSingleton } from '../../server/app-render/manifests-singleton' import { setManifestsSingleton } from '../../server/app-render/manifests-singleton'
...@@ -131,6 +130,47 @@ export async function handler( ...@@ -131,6 +130,47 @@ export async function handler(
const multiZoneDraftMode = process.env const multiZoneDraftMode = process.env
.__NEXT_MULTI_ZONE_DRAFT_MODE as any as boolean .__NEXT_MULTI_ZONE_DRAFT_MODE as any as boolean
// Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace.
// When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest(
req,
{
tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget: req.url,
instrumentation,
},
(requestSpan, requestRootSpan) =>
handleRequest(
req,
res,
ctx,
srcPage,
multiZoneDraftMode,
requestSpan,
requestRootSpan
)
)
}
async function handleRequest(
req: IncomingMessage,
res: ServerResponse,
ctx: {
waitUntil?: (prom: Promise<void>) => void
requestMeta?: RequestMeta
},
srcPage: string,
multiZoneDraftMode: boolean,
requestSpan: Span | undefined,
requestRootSpan: Span | undefined
) {
const prepareResult = await routeModule.prepare(req, res, { const prepareResult = await routeModule.prepare(req, res, {
srcPage, srcPage,
multiZoneDraftMode, multiZoneDraftMode,
...@@ -220,7 +260,6 @@ export async function handler( ...@@ -220,7 +260,6 @@ export async function handler(
const method = req.method || 'GET' const method = req.method || 'GET'
const tracer = getTracer() const tracer = getTracer()
const activeSpan = tracer.getActiveScopeSpan()
const isWrappedByNextServer = Boolean( const isWrappedByNextServer = Boolean(
routerServerContext?.isWrappedByNextServer routerServerContext?.isWrappedByNextServer
) )
...@@ -391,10 +430,7 @@ export async function handler( ...@@ -391,10 +430,7 @@ export async function handler(
} }
} }
const handleResponse = async ( const handleResponse = async (currentSpan: Span | undefined) => {
currentSpan: Span | undefined,
parentSpan: Span | undefined
) => {
try { try {
const cacheEntry = await routeModule.handleResponse({ const cacheEntry = await routeModule.handleResponse({
req, req,
...@@ -579,39 +615,22 @@ export async function handler( ...@@ -579,39 +615,22 @@ export async function handler(
}) })
currentSpan.updateName(name) currentSpan.updateName(name)
// Propagate http.route to the parent span if one exists (e.g. // Propagate http.route to the request's root span if it isn't the
// a platform-created HTTP span in adapter deployments). // request span (e.g. a platform-created HTTP span in adapter
if (parentSpan && parentSpan !== currentSpan) { // deployments).
parentSpan.setAttribute('http.route', route) if (requestRootSpan && requestRootSpan !== currentSpan) {
parentSpan.updateName(name) requestRootSpan.setAttribute('http.route', route)
requestRootSpan.updateName(name)
} }
})() })()
} }
} }
// TODO: activeSpan code path is for when wrapped by // When invoked directly, the request span was started before `prepare`;
// next-server can be removed when this is no longer used // record the request target as normalized by it. next-server's request span
if (isWrappedByNextServer && activeSpan) { // already has it.
await handleResponse(activeSpan, undefined) if (!isWrappedByNextServer && req.url !== undefined) {
} else { requestSpan?.setAttribute('http.target', req.url)
let parentSpan = tracer.getActiveScopeSpan()
await tracer.withPropagatedContext(
req.headers,
() =>
tracer.trace(
BaseServerSpan.handleRequest,
{
spanName: `${method} ${srcPage}`,
kind: SpanKind.SERVER,
attributes: {
'http.method': method,
'http.target': req.url,
},
},
(span) => handleResponse(span, parentSpan)
),
undefined,
!isWrappedByNextServer
)
} }
await handleResponse(requestSpan)
} }
...@@ -75,12 +75,15 @@ function errorHandledHandler(fn: AdapterOptions['handler']) { ...@@ -75,12 +75,15 @@ function errorHandledHandler(fn: AdapterOptions['handler']) {
} }
const internalHandler: EdgeHandler = async (opts) => { const internalHandler: EdgeHandler = async (opts) => {
let emitInstrumentationSpans: (() => void) | undefined
let requestStartTime: number | undefined
if (process.env.NEXT_RUNTIME !== 'edge') { if (process.env.NEXT_RUNTIME !== 'edge') {
// This mirrors what `RouteModule#prepare` does for routes // This mirrors what `RouteModule#prepare` does for routes
// edge runtime handles loading instrumentation at the edge adapter level // edge runtime handles loading instrumentation at the edge adapter level
requestStartTime = performance.timeOrigin + performance.now()
const { join, relative } = const { join, relative } =
require('node:path') as typeof import('node:path') require('node:path') as typeof import('node:path')
const { ensureInstrumentationRegistered } = const { ensureInstrumentationRegisteredWithDeferredSpans } =
require('../../server/lib/router-utils/instrumentation-globals.external') as typeof import('../../server/lib/router-utils/instrumentation-globals.external') require('../../server/lib/router-utils/instrumentation-globals.external') as typeof import('../../server/lib/router-utils/instrumentation-globals.external')
const absoluteProjectDir = join( const absoluteProjectDir = join(
/* turbopackIgnore: true */ /* turbopackIgnore: true */
...@@ -92,16 +95,35 @@ const internalHandler: EdgeHandler = async (opts) => { ...@@ -92,16 +95,35 @@ const internalHandler: EdgeHandler = async (opts) => {
? relative(absoluteProjectDir, absoluteDistDir) ? relative(absoluteProjectDir, absoluteDistDir)
: '.next' : '.next'
await ensureInstrumentationRegistered(absoluteProjectDir, distDir) // When this request registers `instrumentation.ts`, the adapter emits its
// lifecycle spans inside the middleware span.
emitInstrumentationSpans =
await ensureInstrumentationRegisteredWithDeferredSpans(
absoluteProjectDir,
distDir
)
} }
return adapter({ try {
...opts, return await adapter({
IncrementalCache, ...opts,
incrementalCacheHandler, IncrementalCache,
page, incrementalCacheHandler,
handler: errorHandledHandler(handlerUserland), page,
}) handler: errorHandledHandler(handlerUserland),
emitInstrumentationSpans,
requestStartTime,
})
} catch (err) {
// The adapter can throw before it runs the middleware and emits the
// lifecycle spans (e.g. when the configured cache handler throws). Later
// requests won't emit them, so emit them here instead (a no-op if they were
// emitted). Tracing must not replace the adapter's error.
try {
emitInstrumentationSpans?.()
} catch {}
throw err
}
} }
export async function handler( export async function handler(
......
...@@ -12,7 +12,7 @@ import { hoist } from './helpers' ...@@ -12,7 +12,7 @@ import { hoist } from './helpers'
import * as userland from 'VAR_USERLAND' import * as userland from 'VAR_USERLAND'
import { import {
getTracer, getTracer,
SpanKind, recordSpanError,
SpanStatusCode, SpanStatusCode,
} from '../../server/lib/trace/tracer' } from '../../server/lib/trace/tracer'
import { BaseServerSpan } from '../../server/lib/trace/constants' import { BaseServerSpan } from '../../server/lib/trace/constants'
...@@ -67,6 +67,38 @@ export async function handler( ...@@ -67,6 +67,38 @@ export async function handler(
srcPage = srcPage.replace(/\/index$/, '') || '/' srcPage = srcPage.replace(/\/index$/, '') || '/'
} }
// Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace.
// When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest(
req,
{
tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget: req.url,
instrumentation,
},
(requestSpan, requestRootSpan) =>
handleRequest(req, res, ctx, srcPage, requestSpan, requestRootSpan)
)
}
async function handleRequest(
req: IncomingMessage,
res: ServerResponse,
ctx: {
waitUntil?: (prom: Promise<void>) => void
requestMeta?: RequestMeta
},
srcPage: string,
requestSpan: Span | undefined,
requestRootSpan: Span | undefined
) {
const prepareResult = await routeModule.prepare(req, res, { srcPage }) const prepareResult = await routeModule.prepare(req, res, { srcPage })
if (!prepareResult) { if (!prepareResult) {
...@@ -82,14 +114,12 @@ export async function handler( ...@@ -82,14 +114,12 @@ export async function handler(
const method = req.method || 'GET' const method = req.method || 'GET'
const tracer = getTracer() const tracer = getTracer()
const activeSpan = tracer.getActiveScopeSpan()
const isWrappedByNextServer = Boolean( const isWrappedByNextServer = Boolean(
routerServerContext?.isWrappedByNextServer routerServerContext?.isWrappedByNextServer
) )
const onRequestError = const onRequestError =
routeModule.instrumentationOnRequestError.bind(routeModule) routeModule.instrumentationOnRequestError.bind(routeModule)
let parentSpan: Span | undefined
const invokeRouteModule = async (span?: Span) => const invokeRouteModule = async (span?: Span) =>
routeModule routeModule
.render(req, res, { .render(req, res, {
...@@ -161,38 +191,31 @@ export async function handler( ...@@ -161,38 +191,31 @@ export async function handler(
}) })
span.updateName(name) span.updateName(name)
// Propagate http.route to the parent span if one exists (e.g. // Propagate http.route to the request's root span if it isn't the
// a platform-created HTTP span in adapter deployments). // request span (e.g. a platform-created HTTP span in adapter
if (parentSpan && parentSpan !== span) { // deployments).
parentSpan.setAttribute('http.route', route) if (requestRootSpan && requestRootSpan !== span) {
parentSpan.updateName(name) requestRootSpan.setAttribute('http.route', route)
requestRootSpan.updateName(name)
} }
}) })
// TODO: activeSpan code path is for when wrapped by // When invoked directly, the request span was started before `prepare`;
// next-server can be removed when this is no longer used // record the request target as normalized by it. next-server's request
if (isWrappedByNextServer && activeSpan) { // span already has it.
await invokeRouteModule(activeSpan) if (!isWrappedByNextServer && req.url !== undefined) {
} else { requestSpan?.setAttribute('http.target', req.url)
parentSpan = tracer.getActiveScopeSpan() }
await tracer.withPropagatedContext( try {
req.headers, await invokeRouteModule(requestSpan)
() => } catch (err) {
tracer.trace( // A request span started for a direct invocation outlives the error
BaseServerSpan.handleRequest, // handling below, which only rethrows in dev: record the error on the
{ // span here otherwise.
spanName: `${method} ${srcPage}`, if (!isWrappedByNextServer && requestSpan && !routeModule.isDev) {
kind: SpanKind.SERVER, recordSpanError(requestSpan, err as Error)
attributes: { }
'http.method': method, throw err
'http.target': req.url,
},
},
invokeRouteModule
),
undefined,
!isWrappedByNextServer
)
} }
} catch (err) { } catch (err) {
// we re-throw in dev to show the error overlay // we re-throw in dev to show the error overlay
......
...@@ -884,87 +884,137 @@ export default abstract class Server< ...@@ -884,87 +884,137 @@ export default abstract class Server<
const method = req.method.toUpperCase() const method = req.method.toUpperCase()
const tracer = getTracer() const tracer = getTracer()
const handleRequest = () => const handleRequest = () => {
tracer.withPropagatedContext(req.headers, () => { // The render pass after a middleware invocation continues the trace
// Capture the parent span before creating the handleRequest span. // context on the request, even when another span is active (e.g. a
// When deployed with an adapter, the platform's runtime may create its // custom server or APM HTTP span). That is the context the
// own OTEL HTTP server span before Next.js runs. We propagate http.route // middleware-invoke pass wrote onto the request, so the middleware span
// to this parent span so APM tools (e.g. Datadog) can derive the // and the render are siblings under that pass's span, or a `traceparent`
// resource name correctly. // the middleware explicitly set as a request-header override.
const parentSpan = tracer.getActiveScopeSpan() const forcePropagation =
!getRequestMeta(req, 'middlewareInvoke') &&
return tracer.trace( getRequestMeta(req, 'middlewareInvokeTraceparent') !== undefined
BaseServerSpan.handleRequest, // With forced propagation the active (host) span isn't the parent, so
{ // capture it before propagating to keep propagating `http.route` to it.
spanName: `${method}`, const hostSpan = forcePropagation
kind: SpanKind.SERVER, ? tracer.getActiveScopeSpan()
attributes: { : undefined
'http.method': method,
'http.target': req.url, return tracer.withPropagatedContext(
req.headers,
() => {
// Capture the parent span before creating the handleRequest span.
// When deployed with an adapter, the platform's runtime may create its
// own OTEL HTTP server span before Next.js runs. We propagate http.route
// to this parent span so APM tools (e.g. Datadog) can derive the
// resource name correctly.
const parentSpan = forcePropagation
? hostSpan
: tracer.getActiveScopeSpan()
return tracer.trace(
BaseServerSpan.handleRequest,
{
spanName: `${method}`,
kind: SpanKind.SERVER,
attributes: {
'http.method': method,
'http.target': req.url,
},
}, },
}, async (span) => {
async (span) => if (getRequestMeta(req, 'middlewareInvoke')) {
this.handleRequestImpl(req, res, parsedUrl).finally(() => { // When middleware lets the request continue, the router server
if (!span) return // calls `handleRequest` twice: once to run middleware (this pass)
// and once to render. Publish this span's W3C trace context on
const isRSCRequest = getRequestMeta(req, 'isRSCRequest') ?? false // the request, so the middleware span (including the Edge
span.setAttributes({ // sandbox, which extracts it from the request headers) and the
'http.status_code': res.statusCode, // render pass are both children of this span (siblings) instead
'next.rsc': isRSCRequest, // of starting new traces.
}) const traceContext = tracer.getTraceContextHeaders()
if (traceContext) {
if (res.statusCode && res.statusCode >= 500) { Object.assign(req.headers, traceContext)
// For 5xx status codes: SHOULD be set to 'Error' span status. if (!traceContext.tracestate) {
// x-ref: https://opentelemetry.io/docs/specs/semconv/http/http-spans/#status // Don't pair this span's `traceparent` with a
span.setStatus({ // `tracestate` of another trace context (e.g. an incoming
code: SpanStatusCode.ERROR, // one, when a host span is this span's parent).
}) delete req.headers.tracestate
// For span status 'Error', SHOULD set 'error.type' attribute. }
span.setAttribute('error.type', res.statusCode.toString()) addRequestMeta(
} req,
'middlewareInvokeTraceparent',
const rootSpanAttributes = tracer.getRootSpanAttributes() traceContext.traceparent
// We were unable to get attributes, probably OTEL is not enabled )
if (!rootSpanAttributes) return }
if (
rootSpanAttributes.get('next.span_type') !==
BaseServerSpan.handleRequest
) {
console.warn(
`Unexpected root span type '${rootSpanAttributes.get(
'next.span_type'
)}'. Please report this Next.js issue https://github.com/vercel/next.js`
)
return
} }
const route = rootSpanAttributes.get('next.route') return this.handleRequestImpl(req, res, parsedUrl).finally(() => {
if (route) { if (!span) return
const name = isRSCRequest
? `RSC ${method} ${route}`
: `${method} ${route}`
const isRSCRequest =
getRequestMeta(req, 'isRSCRequest') ?? false
span.setAttributes({ span.setAttributes({
'next.route': route, 'http.status_code': res.statusCode,
'http.route': route, 'next.rsc': isRSCRequest,
'next.span_name': name,
}) })
span.updateName(name)
if (res.statusCode && res.statusCode >= 500) {
// Propagate http.route to the parent span if one exists and // For 5xx status codes: SHOULD be set to 'Error' span status.
// is different from the handleRequest span. This ensures APM // x-ref: https://opentelemetry.io/docs/specs/semconv/http/http-spans/#status
// tools that read attributes from the outermost span (e.g. span.setStatus({
// a platform-created HTTP span) can derive the resource name. code: SpanStatusCode.ERROR,
if (parentSpan && parentSpan !== span) { })
parentSpan.setAttribute('http.route', route) // For span status 'Error', SHOULD set 'error.type' attribute.
span.setAttribute('error.type', res.statusCode.toString())
} }
} else {
span.updateName(isRSCRequest ? `RSC ${method}` : `${method}`) const rootSpanAttributes = tracer.getRootSpanAttributes()
} // We were unable to get attributes, probably OTEL is not enabled
}) if (!rootSpanAttributes) return
)
}) if (
rootSpanAttributes.get('next.span_type') !==
BaseServerSpan.handleRequest
) {
console.warn(
`Unexpected root span type '${rootSpanAttributes.get(
'next.span_type'
)}'. Please report this Next.js issue https://github.com/vercel/next.js`
)
return
}
const route = rootSpanAttributes.get('next.route')
if (route) {
const name = isRSCRequest
? `RSC ${method} ${route}`
: `${method} ${route}`
span.setAttributes({
'next.route': route,
'http.route': route,
'next.span_name': name,
})
span.updateName(name)
// Propagate http.route to the parent span if one exists and
// is different from the handleRequest span. This ensures APM
// tools that read attributes from the outermost span (e.g.
// a platform-created HTTP span) can derive the resource name.
if (parentSpan && parentSpan !== span) {
parentSpan.setAttribute('http.route', route)
}
} else {
span.updateName(isRSCRequest ? `RSC ${method}` : `${method}`)
}
})
}
)
},
undefined,
forcePropagation
)
}
if (!isRequestInsightsEnabled()) { if (!isRequestInsightsEnabled()) {
return handleRequest() return handleRequest()
......
...@@ -19,16 +19,23 @@ type InstrumentationSpanTiming = { ...@@ -19,16 +19,23 @@ type InstrumentationSpanTiming = {
error?: unknown error?: unknown
} }
/** A lifecycle span of `instrumentation.ts`, recorded to be emitted later. */
type InstrumentationLifecycleSpan = {
type: InstrumentationSpan
spanName: string
timing: InstrumentationSpanTiming
}
function getInstrumentationTimestamp() { function getInstrumentationTimestamp() {
// Date.now() is treated as synchronous I/O while Cache Components render. // Date.now() is treated as synchronous I/O while Cache Components render.
return performance.timeOrigin + performance.now() return performance.timeOrigin + performance.now()
} }
function traceBackdatedInstrumentationSpan( function traceBackdatedInstrumentationSpan({
type: InstrumentationSpan, type,
spanName: string, spanName,
{ startTime, endTime, error }: InstrumentationSpanTiming timing: { startTime, endTime, error },
) { }: InstrumentationLifecycleSpan) {
getTracer().trace( getTracer().trace(
type, type,
{ spanName, startTime, endTime }, { spanName, startTime, endTime },
...@@ -47,16 +54,29 @@ function traceBackdatedInstrumentationSpan( ...@@ -47,16 +54,29 @@ function traceBackdatedInstrumentationSpan(
) )
} }
function traceInstrumentationModuleLoad() { /**
if (!instrumentationModuleLoadTiming) return * Takes the recorded timing of loading the instrumentation module, if not
* emitted yet, as a lifecycle span.
*/
function takeInstrumentationModuleLoadSpan():
| InstrumentationLifecycleSpan
| undefined {
if (!instrumentationModuleLoadTiming) return undefined
const timing = instrumentationModuleLoadTiming const timing = instrumentationModuleLoadTiming
instrumentationModuleLoadTiming = undefined instrumentationModuleLoadTiming = undefined
traceBackdatedInstrumentationSpan( return {
InstrumentationSpan.loadModule, type: InstrumentationSpan.loadModule,
'load instrumentation module', spanName: 'load instrumentation module',
timing timing,
) }
}
function traceInstrumentationModuleLoad() {
const span = takeInstrumentationModuleLoadSpan()
if (span) {
traceBackdatedInstrumentationSpan(span)
}
} }
export async function getInstrumentationModule( export async function getInstrumentationModule(
...@@ -102,7 +122,22 @@ export async function getInstrumentationModule( ...@@ -102,7 +122,22 @@ export async function getInstrumentationModule(
let instrumentationModulePromise: Promise<any> | null = null let instrumentationModulePromise: Promise<any> | null = null
async function registerInstrumentation(projectDir: string, distDir: string) { /**
* Loads `instrumentation.ts` and calls its `register()`.
*
* Its lifecycle spans (`load instrumentation module`, `register
* instrumentation`) are backdated and emitted afterwards, since `register()`
* commonly installs the tracer provider. They are emitted right away in the
* active context, unless `deferredSpans` is given: then successfully completed
* steps are added to it instead, for the caller to emit in another context
* (the request that triggered registration). Failed steps are always emitted
* right away.
*/
async function registerInstrumentation(
projectDir: string,
distDir: string,
deferredSpans?: InstrumentationLifecycleSpan[]
) {
// Ensure registerInstrumentation is not called in production build // Ensure registerInstrumentation is not called in production build
if (process.env.NEXT_PHASE === 'phase-production-build') { if (process.env.NEXT_PHASE === 'phase-production-build') {
return return
...@@ -121,9 +156,11 @@ async function registerInstrumentation(projectDir: string, distDir: string) { ...@@ -121,9 +156,11 @@ async function registerInstrumentation(projectDir: string, distDir: string) {
if (instrumentation?.register) { if (instrumentation?.register) {
const startTime = getInstrumentationTimestamp() const startTime = getInstrumentationTimestamp()
let error: unknown let error: unknown
let registered = false
try { try {
await instrumentation.register() await instrumentation.register()
extendInstrumentationAfterRegistration() extendInstrumentationAfterRegistration()
registered = true
} catch (err: any) { } catch (err: any) {
error = err error = err
err.message = `An error occurred while loading instrumentation hook: ${err.message}` err.message = `An error occurred while loading instrumentation hook: ${err.message}`
...@@ -133,15 +170,31 @@ async function registerInstrumentation(projectDir: string, distDir: string) { ...@@ -133,15 +170,31 @@ async function registerInstrumentation(projectDir: string, distDir: string) {
// register() commonly installs the provider, so emit both lifecycle // register() commonly installs the provider, so emit both lifecycle
// spans afterward with their original timestamps. // spans afterward with their original timestamps.
traceInstrumentationModuleLoad() const spans = [
traceBackdatedInstrumentationSpan( takeInstrumentationModuleLoadSpan(),
InstrumentationSpan.register, {
'register instrumentation', type: InstrumentationSpan.register,
{ startTime, endTime, error } spanName: 'register instrumentation',
) timing: { startTime, endTime, error },
},
].filter((span) => span !== undefined)
// Only defer after a successful registration: a failing request has no
// request span to emit them in.
if (deferredSpans && registered) {
deferredSpans.push(...spans)
} else {
spans.forEach(traceBackdatedInstrumentationSpan)
}
} }
} else { } else {
traceInstrumentationModuleLoad() const span = takeInstrumentationModuleLoadSpan()
if (span) {
if (deferredSpans) {
deferredSpans.push(span)
} else {
traceBackdatedInstrumentationSpan(span)
}
}
} }
} }
...@@ -172,3 +225,36 @@ export function ensureInstrumentationRegistered( ...@@ -172,3 +225,36 @@ export function ensureInstrumentationRegistered(
} }
return registerInstrumentationPromise return registerInstrumentationPromise
} }
/**
* Like `ensureInstrumentationRegistered`, for a request that may trigger
* registration (direct entrypoints, without next-server). When this call
* starts registration and it succeeds, the lifecycle spans aren't emitted yet:
* the returned function emits them in the context it's called in, so they can
* become children of the request's span. Returns `undefined` when
* registration was already started by another caller.
*/
export async function ensureInstrumentationRegisteredWithDeferredSpans(
projectDir: string,
distDir: string
): Promise<(() => void) | undefined> {
if (registerInstrumentationPromise) {
await registerInstrumentationPromise
return undefined
}
const deferredSpans: InstrumentationLifecycleSpan[] = []
registerInstrumentationPromise = registerInstrumentation(
projectDir,
distDir,
deferredSpans
)
await registerInstrumentationPromise
if (deferredSpans.length === 0) {
return undefined
}
return () => {
// Emit at most once.
deferredSpans.splice(0).forEach(traceBackdatedInstrumentationSpan)
}
}
import fs from 'node:fs'
import os from 'node:os'
import path from 'node:path'
const mockTraced: Array<{ type: string; spanName: string; error: boolean }> = []
jest.mock('../trace/tracer', () => ({
SpanStatusCode: { ERROR: 2 },
getTracer: () => ({
trace: (
type: string,
options: { spanName: string },
fn: (span: unknown) => void
) => {
let error = false
fn({
recordException: () => {},
setAttribute: () => {},
setStatus: () => {
error = true
},
})
mockTraced.push({ type, spanName: options.spanName, error })
},
}),
}))
jest.mock('./instrumentation-node-extensions', () => ({
afterRegistration: () => {},
}))
type InstrumentationGlobals =
typeof import('./instrumentation-globals.external')
const LOAD = 'Instrumentation.loadModule'
const REGISTER = 'Instrumentation.register'
let projectDir: string
function writeInstrumentation(source: string) {
const serverDir = path.join(projectDir, '.next', 'server')
fs.mkdirSync(serverDir, { recursive: true })
fs.writeFileSync(path.join(serverDir, 'instrumentation.js'), source)
}
function loadGlobals(): InstrumentationGlobals {
let globals: InstrumentationGlobals | undefined
jest.isolateModules(() => {
globals =
require('./instrumentation-globals.external') as typeof import('./instrumentation-globals.external')
})
return globals!
}
describe('instrumentation lifecycle spans', () => {
beforeEach(() => {
mockTraced.length = 0
projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'next-instrumentation-'))
})
afterEach(() => {
fs.rmSync(projectDir, { recursive: true, force: true })
})
it('emits them right away for startup registration', async () => {
writeInstrumentation('exports.register = () => {}')
const globals = loadGlobals()
await globals.ensureInstrumentationRegistered(projectDir, '.next')
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD, REGISTER])
})
it('defers them to the request that triggered registration, once', async () => {
writeInstrumentation('exports.register = () => {}')
const globals = loadGlobals()
const [emit, concurrentEmit] = await Promise.all([
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
),
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
),
])
// Only the initiating request gets them, and nothing was emitted yet.
expect(concurrentEmit).toBeUndefined()
expect(mockTraced).toEqual([])
emit!()
emit!()
expect(mockTraced).toEqual([
{
type: LOAD,
spanName: 'load instrumentation module',
error: false,
},
{ type: REGISTER, spanName: 'register instrumentation', error: false },
])
// Later requests don't emit them again.
expect(
await globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).toBeUndefined()
})
it('does not defer them when a request awaits startup registration', async () => {
writeInstrumentation('exports.register = () => {}')
const globals = loadGlobals()
const startup = globals.ensureInstrumentationRegistered(projectDir, '.next')
const emit = await globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
await startup
expect(emit).toBeUndefined()
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD, REGISTER])
})
it('defers the module load span without a register export', async () => {
writeInstrumentation('exports.onRequestError = () => {}')
const globals = loadGlobals()
const emit = await globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
expect(mockTraced).toEqual([])
emit!()
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD])
})
it('emits them right away when register() throws', async () => {
writeInstrumentation(
'exports.register = () => { throw new Error("register failed") }'
)
const globals = loadGlobals()
await expect(
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).rejects.toThrow(
'An error occurred while loading instrumentation hook: register failed'
)
// The load succeeded; only the register span has an error status.
expect(mockTraced).toEqual([
{
type: LOAD,
spanName: 'load instrumentation module',
error: false,
},
{ type: REGISTER, spanName: 'register instrumentation', error: true },
])
})
it('emits them right away when register() throws a non-error value', async () => {
writeInstrumentation('exports.register = () => { throw undefined }')
const globals = loadGlobals()
await expect(
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).rejects.toThrow()
// Nothing is left waiting for a request span that won't be created.
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD, REGISTER])
})
it('emits the module load span right away when loading fails', async () => {
writeInstrumentation('throw new Error("load failed")')
const globals = loadGlobals()
await expect(
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).rejects.toThrow('load failed')
expect(mockTraced).toEqual([
{
type: LOAD,
spanName: 'load instrumentation module',
error: true,
},
])
})
})
...@@ -133,6 +133,13 @@ enum MiddlewareSpan { ...@@ -133,6 +133,13 @@ enum MiddlewareSpan {
execute = 'Middleware.execute', execute = 'Middleware.execute',
} }
enum NextSpan {
// Root span inserted by the middleware adapter when the request has no
// trace context (separately invoked middleware without a parent), so the
// middleware and the render are siblings in one trace.
request = 'Next.request',
}
type SpanTypes = type SpanTypes =
| `${BaseServerSpan}` | `${BaseServerSpan}`
| `${LoadComponentsSpan}` | `${LoadComponentsSpan}`
...@@ -150,10 +157,12 @@ type SpanTypes = ...@@ -150,10 +157,12 @@ type SpanTypes =
| `${RouteModuleSpan}` | `${RouteModuleSpan}`
| `${ResolveMetadataSpan}` | `${ResolveMetadataSpan}`
| `${MiddlewareSpan}` | `${MiddlewareSpan}`
| `${NextSpan}`
// This list is used to filter out spans that are not relevant to the user // This list is used to filter out spans that are not relevant to the user
export const NextVanillaSpanAllowlist = new Set([ export const NextVanillaSpanAllowlist = new Set([
MiddlewareSpan.execute, MiddlewareSpan.execute,
NextSpan.request,
BaseServerSpan.handleRequest, BaseServerSpan.handleRequest,
RenderSpan.getServerSideProps, RenderSpan.getServerSideProps,
RenderSpan.getStaticProps, RenderSpan.getStaticProps,
...@@ -201,6 +210,7 @@ export { ...@@ -201,6 +210,7 @@ export {
RouteModuleSpan, RouteModuleSpan,
ResolveMetadataSpan, ResolveMetadataSpan,
MiddlewareSpan, MiddlewareSpan,
NextSpan,
} }
export type { SpanTypes } export type { SpanTypes }
...@@ -7,6 +7,7 @@ import type { ...@@ -7,6 +7,7 @@ import type {
ContextManager, ContextManager,
TextMapGetter, TextMapGetter,
TextMapPropagator, TextMapPropagator,
TextMapSetter,
} from '@opentelemetry/api' } from '@opentelemetry/api'
import type { WorkStore } from '../../app-render/work-async-storage.external' import type { WorkStore } from '../../app-render/work-async-storage.external'
import type { WorkUnitStore } from '../../app-render/work-unit-async-storage.external' import type { WorkUnitStore } from '../../app-render/work-unit-async-storage.external'
...@@ -33,7 +34,14 @@ import { ...@@ -33,7 +34,14 @@ import {
NodeSpan, NodeSpan,
RouteModuleSpan, RouteModuleSpan,
} from './constants' } from './constants'
import { SpanKind, SpanStatusCode, getTracer } from './tracer' import {
BubbledError,
recordSpanError,
SpanKind,
SpanStatusCode,
getTracer,
type Span,
} from './tracer'
const customContextKey = createContextKey('next.tracer.test.custom-context') const customContextKey = createContextKey('next.tracer.test.custom-context')
const originalRequestInsights = process.env.__NEXT_REQUEST_INSIGHTS const originalRequestInsights = process.env.__NEXT_REQUEST_INSIGHTS
...@@ -120,6 +128,48 @@ class CustomPropagator implements TextMapPropagator { ...@@ -120,6 +128,48 @@ class CustomPropagator implements TextMapPropagator {
} }
} }
// A minimal W3C `traceparent` propagator (`@opentelemetry/core` isn't a
// dependency here).
class TraceparentPropagator implements TextMapPropagator {
fields(): string[] {
return ['traceparent']
}
inject(
injectedContext: Context,
carrier: unknown,
setter: TextMapSetter<unknown>
): void {
const spanContext = trace.getSpanContext(injectedContext)
if (!spanContext) return
const flags = spanContext.traceFlags.toString(16).padStart(2, '0')
setter.set(
carrier,
'traceparent',
`00-${spanContext.traceId}-${spanContext.spanId}-${flags}`
)
}
extract(
extractedContext: Context,
carrier: Record<string, string | undefined>,
mapGetter: TextMapGetter<Record<string, string | undefined>>
): Context {
const value = mapGetter.get(carrier, 'traceparent')
const [, traceId, spanId, flags] =
(typeof value === 'string' && value.split('-')) || []
if (!traceId || !spanId || !flags) {
return extractedContext
}
return trace.setSpanContext(extractedContext, {
traceId,
spanId,
traceFlags: parseInt(flags, 16),
isRemote: true,
})
}
}
describe('withPropagatedContext', () => { describe('withPropagatedContext', () => {
beforeEach(() => { beforeEach(() => {
context.disable() context.disable()
...@@ -164,6 +214,74 @@ describe('withPropagatedContext', () => { ...@@ -164,6 +214,74 @@ describe('withPropagatedContext', () => {
}) })
}) })
describe('trace context with the W3C propagator', () => {
const TRACE_ID = '0af7651916cd43dd8448eb211c80319c'
const SPAN_ID = 'b7ad6b7169203331'
beforeEach(() => {
context.disable()
propagation.disable()
context.setGlobalContextManager(new TestContextManager())
propagation.setGlobalPropagator(new TraceparentPropagator())
})
afterEach(() => {
propagation.disable()
context.disable()
})
const withSpanContext = <T>(
spanContext: Parameters<typeof trace.wrapSpanContext>[0],
fn: () => T
) =>
context.with(
trace.setSpan(ROOT_CONTEXT, trace.wrapSpanContext(spanContext)),
fn
)
it('prefers the carrier traceparent over the active span in force mode', () => {
const result = withSpanContext(
{
traceId: '0123456789abcdef0123456789abcdef',
spanId: '0123456789abcdef',
traceFlags: 1,
isRemote: false,
},
() =>
getTracer().withPropagatedContext(
{ traceparent: `00-${TRACE_ID}-${SPAN_ID}-01` },
() => trace.getSpanContext(context.active()),
getter,
true
)
)
expect(result).toMatchObject({ traceId: TRACE_ID, spanId: SPAN_ID })
})
it('always includes the tracestate key in the trace context headers', () => {
const spanContext = {
traceId: TRACE_ID,
spanId: SPAN_ID,
traceFlags: 1,
isRemote: false,
}
// A recording span, so OpenTelemetry counts as enabled.
const span = Object.assign(trace.wrapSpanContext(spanContext), {
isRecording: () => true,
})
const result = context.with(trace.setSpan(ROOT_CONTEXT, span), () =>
getTracer().getTraceContextHeaders()
)
expect(result).toStrictEqual({
traceparent: `00-${TRACE_ID}-${SPAN_ID}-01`,
tracestate: undefined,
})
})
})
describe('local span recording', () => { describe('local span recording', () => {
beforeEach(() => { beforeEach(() => {
process.env.__NEXT_DEV_SERVER = '1' process.env.__NEXT_DEV_SERVER = '1'
...@@ -827,3 +945,40 @@ describe('local span recording', () => { ...@@ -827,3 +945,40 @@ describe('local span recording', () => {
}) })
}) })
}) })
describe('recordSpanError', () => {
function createSpan() {
return {
recordException: jest.fn(),
setAttribute: jest.fn(),
setStatus: jest.fn(),
end: jest.fn(),
}
}
it('records the error on the span without ending it', () => {
const span = createSpan()
const error = new TypeError('boom')
recordSpanError(span as unknown as Span, error)
expect(span.recordException).toHaveBeenCalledWith(error)
expect(span.setAttribute).toHaveBeenCalledWith('error.type', 'TypeError')
expect(span.setStatus).toHaveBeenCalledWith({
code: SpanStatusCode.ERROR,
message: 'boom',
})
expect(span.end).not.toHaveBeenCalled()
})
it('only marks bubbled errors', () => {
const span = createSpan()
recordSpanError(span as unknown as Span, new BubbledError(true))
expect(span.setAttribute).toHaveBeenCalledWith('next.bubble', true)
expect(span.recordException).not.toHaveBeenCalled()
expect(span.setStatus).not.toHaveBeenCalled()
expect(span.end).not.toHaveBeenCalled()
})
})
...@@ -67,11 +67,11 @@ export function isBubbledError(error: unknown): error is BubbledError { ...@@ -67,11 +67,11 @@ export function isBubbledError(error: unknown): error is BubbledError {
return error instanceof BubbledError return error instanceof BubbledError
} }
const closeSpanWithError = ( /**
span: Span, * Records `error` on `span` the way a span traced with `NextTracer.trace` that
error?: Error, * throws it does, without ending the span.
endTime?: Parameters<Span['end']>[0] */
) => { export const recordSpanError = (span: Span, error?: Error) => {
if (isBubbledError(error) && error.bubble) { if (isBubbledError(error) && error.bubble) {
span.setAttribute('next.bubble', true) span.setAttribute('next.bubble', true)
} else { } else {
...@@ -81,6 +81,14 @@ const closeSpanWithError = ( ...@@ -81,6 +81,14 @@ const closeSpanWithError = (
} }
span.setStatus({ code: SpanStatusCode.ERROR, message: error?.message }) span.setStatus({ code: SpanStatusCode.ERROR, message: error?.message })
} }
}
const closeSpanWithError = (
span: Span,
error?: Error,
endTime?: Parameters<Span['end']>[0]
) => {
recordSpanError(span, error)
span.end(endTime) span.end(endTime)
} }
...@@ -178,6 +186,26 @@ interface NextTracer { ...@@ -178,6 +186,26 @@ interface NextTracer {
*/ */
getTracePropagationData(): ClientTraceDataEntry[] getTracePropagationData(): ClientTraceDataEntry[]
/**
* Returns the W3C trace context (`traceparent` / `tracestate`) of the
* currently active context, as produced by the configured propagator.
* Returns `undefined` when OpenTelemetry is not enabled, no span context is
* active, or the propagator doesn't produce a `traceparent`.
*/
getTraceContextHeaders(): TraceContextHeaders | undefined
/**
* Whether OpenTelemetry is enabled: a tracer provider is registered, or a
* recording span is active.
*/
isOpenTelemetryEnabled(): boolean
/**
* Whether the currently active context has a valid span context: an active
* span, or a remote parent extracted from incoming trace headers.
*/
hasActiveSpanContext(): boolean
/** /**
* Executes a function with the given span set as the active span in the context. * Executes a function with the given span set as the active span in the context.
* This allows child spans created within the function to automatically parent to this span. * This allows child spans created within the function to automatically parent to this span.
...@@ -211,6 +239,11 @@ export interface ClientTraceDataEntry { ...@@ -211,6 +239,11 @@ export interface ClientTraceDataEntry {
value: string value: string
} }
export interface TraceContextHeaders {
traceparent: string
tracestate?: string
}
const clientTraceDataSetter: TextMapSetter<ClientTraceDataEntry[]> = { const clientTraceDataSetter: TextMapSetter<ClientTraceDataEntry[]> = {
set(carrier, key, value) { set(carrier, key, value) {
carrier.push({ carrier.push({
...@@ -230,7 +263,7 @@ class NextTracerImpl implements NextTracer { ...@@ -230,7 +263,7 @@ class NextTracerImpl implements NextTracer {
return trace.getTracer('next.js', process.env.__NEXT_VERSION) return trace.getTracer('next.js', process.env.__NEXT_VERSION)
} }
private isOpenTelemetryEnabled(): boolean { public isOpenTelemetryEnabled(): boolean {
const activeSpan = trace.getSpan(context.active()) const activeSpan = trace.getSpan(context.active())
if (activeSpan?.isRecording()) { if (activeSpan?.isRecording()) {
return true return true
...@@ -260,6 +293,30 @@ class NextTracerImpl implements NextTracer { ...@@ -260,6 +293,30 @@ class NextTracerImpl implements NextTracer {
return entries return entries
} }
public getTraceContextHeaders(): TraceContextHeaders | undefined {
const activeContext = context.active()
if (
!trace.getSpanContext(activeContext) ||
!this.isOpenTelemetryEnabled()
) {
return undefined
}
const carrier: Record<string, string> = {}
propagation.inject(activeContext, carrier)
const { traceparent, tracestate } = carrier
if (!traceparent) {
return undefined
}
return { traceparent, tracestate }
}
public hasActiveSpanContext(): boolean {
const spanContext = trace.getSpanContext(context.active())
return Boolean(spanContext && trace.isSpanContextValid(spanContext))
}
public getActiveScopeSpan(): Span | undefined { public getActiveScopeSpan(): Span | undefined {
const localSpanRecorder = getLocalSpanRecorder() const localSpanRecorder = getLocalSpanRecorder()
const activeLocalSpan = localSpanRecorder?.getActiveLocalSpan() const activeLocalSpan = localSpanRecorder?.getActiveLocalSpan()
...@@ -277,19 +334,6 @@ class NextTracerImpl implements NextTracer { ...@@ -277,19 +334,6 @@ class NextTracerImpl implements NextTracer {
return activeLocalSpan return activeLocalSpan
} }
/**
* Run `fn` with the active span cleared, so spans created inside become new
* roots (or parent to incoming propagated context). Used for in-process
* Node.js middleware so its span is a sibling of the request span, matching
* edge middleware which runs in a detached sandbox.
*/
public runWithDetachedContext<T>(fn: () => T): T {
if (!NEXT_OTEL_PERFORMANCE_PREFIX && !this.isOpenTelemetryEnabled()) {
return fn()
}
return context.with(ROOT_CONTEXT, fn)
}
public withPropagatedContext<T, C>( public withPropagatedContext<T, C>(
carrier: C, carrier: C,
fn: () => T, fn: () => T,
......
...@@ -1866,24 +1866,21 @@ export default class NextNodeServer extends BaseServer< ...@@ -1866,24 +1866,21 @@ export default class NextNodeServer extends BaseServer<
try { try {
// Node.js middleware runs in-process, inside the active // Node.js middleware runs in-process, inside the active
// `handleRequest` span. Detach that span so the middleware span // `handleRequest` span, so the middleware span (and the trace context
// becomes a sibling root (or parents to an incoming traceparent), // it forwards to the render) stays in the request's trace.
// matching edge middleware which runs in a detached sandbox. result = await adapterFn({
result = await getTracer().runWithDetachedContext(() => handler:
adapterFn({ middlewareModule.proxy ||
handler: middlewareModule.middleware ||
middlewareModule.proxy || middlewareModule,
middlewareModule.middleware || request: {
middlewareModule, ...requestData,
request: { body: hasRequestBody
...requestData, ? requestData.body.cloneBodyStream()
body: hasRequestBody : undefined,
? requestData.body.cloneBodyStream() },
: undefined, page: 'middleware',
}, })
page: 'middleware',
})
)
} finally { } finally {
if (hasRequestBody) { if (hasRequestBody) {
await requestData.body.finalize() await requestData.body.finalize()
......
...@@ -220,6 +220,13 @@ export interface RequestMeta { ...@@ -220,6 +220,13 @@ export interface RequestMeta {
*/ */
middlewareInvoke?: boolean middlewareInvoke?: boolean
/**
* The W3C `traceparent` of the middleware invocation's `handleRequest` span,
* written onto the request headers so the middleware span and the render
* that follows are siblings under that span.
*/
middlewareInvokeTraceparent?: string
/** /**
* Whether the request should render the fallback shell or not. * Whether the request should render the fallback shell or not.
*/ */
......
...@@ -2,7 +2,7 @@ import type { IncomingMessage, ServerResponse } from 'node:http' ...@@ -2,7 +2,7 @@ import type { IncomingMessage, ServerResponse } from 'node:http'
import type { ParsedUrlQuery } from 'node:querystring' import type { ParsedUrlQuery } from 'node:querystring'
import { RouteKind } from '../../route-kind' import { RouteKind } from '../../route-kind'
import { BaseServerSpan } from '../../lib/trace/constants' import { BaseServerSpan } from '../../lib/trace/constants'
import { getTracer, SpanKind, type Span } from '../../lib/trace/tracer' import { getTracer, type Span } from '../../lib/trace/tracer'
import { formatUrl } from '../../../shared/lib/router/utils/format-url' import { formatUrl } from '../../../shared/lib/router/utils/format-url'
import { import {
addRequestMeta, addRequestMeta,
...@@ -100,6 +100,47 @@ export const getHandler = ({ ...@@ -100,6 +100,47 @@ export const getHandler = ({
const multiZoneDraftMode = process.env const multiZoneDraftMode = process.env
.__NEXT_MULTI_ZONE_DRAFT_MODE as any as boolean .__NEXT_MULTI_ZONE_DRAFT_MODE as any as boolean
// Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace.
// When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest(
req,
{
tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget: req.url,
instrumentation,
},
(requestSpan, requestRootSpan) =>
handleRequest(
req,
res,
ctx,
srcPage,
multiZoneDraftMode,
requestSpan,
requestRootSpan
)
)
}
async function handleRequest(
req: IncomingMessage,
res: ServerResponse,
ctx: {
waitUntil?: (prom: Promise<void>) => void
requestMeta?: RequestMeta
},
srcPage: string,
multiZoneDraftMode: boolean,
requestSpan: Span | undefined,
requestRootSpan: Span | undefined
) {
const prepareResult = await routeModule.prepare(req, res, { const prepareResult = await routeModule.prepare(req, res, {
srcPage, srcPage,
multiZoneDraftMode, multiZoneDraftMode,
...@@ -250,7 +291,6 @@ export const getHandler = ({ ...@@ -250,7 +291,6 @@ export const getHandler = ({
} }
const tracer = getTracer() const tracer = getTracer()
const activeSpan = tracer.getActiveScopeSpan()
const isWrappedByNextServer = Boolean( const isWrappedByNextServer = Boolean(
routerServerContext?.isWrappedByNextServer routerServerContext?.isWrappedByNextServer
) )
...@@ -265,7 +305,6 @@ export const getHandler = ({ ...@@ -265,7 +305,6 @@ export const getHandler = ({
query: hasStaticProps ? {} : originalQuery, query: hasStaticProps ? {} : originalQuery,
}) })
let parentSpan: Span | undefined
const handleResponse = async (span?: Span) => { const handleResponse = async (span?: Span) => {
const responseGenerator: ResponseGenerator = async ({ const responseGenerator: ResponseGenerator = async ({
previousCacheEntry, previousCacheEntry,
...@@ -822,41 +861,23 @@ export const getHandler = ({ ...@@ -822,41 +861,23 @@ export const getHandler = ({
}) })
span.updateName(name) span.updateName(name)
// Propagate http.route to the parent span if one exists // Propagate http.route to the request's root span if it isn't
// (e.g. a platform-created HTTP span in adapter // the request span (e.g. a platform-created HTTP span in adapter
// deployments). // deployments).
if (parentSpan && parentSpan !== span) { if (requestRootSpan && requestRootSpan !== span) {
parentSpan.setAttribute('http.route', route) requestRootSpan.setAttribute('http.route', route)
parentSpan.updateName(name) requestRootSpan.updateName(name)
} }
})() })()
} }
} }
// TODO: activeSpan code path is for when wrapped by // When invoked directly, the request span was started before `prepare`;
// next-server can be removed when this is no longer used // record the request target as normalized by it. next-server's request
if (isWrappedByNextServer && activeSpan) { // span already has it.
await handleResponse(activeSpan) if (!isWrappedByNextServer && req.url !== undefined) {
} else { requestSpan?.setAttribute('http.target', req.url)
parentSpan = tracer.getActiveScopeSpan()
await tracer.withPropagatedContext(
req.headers,
() =>
tracer.trace(
BaseServerSpan.handleRequest,
{
spanName: `${method} ${srcPage}`,
kind: SpanKind.SERVER,
attributes: {
'http.method': method,
'http.target': req.url,
},
},
handleResponse
),
undefined,
!isWrappedByNextServer
)
} }
await handleResponse(requestSpan)
} }
} }
...@@ -78,8 +78,8 @@ import { ...@@ -78,8 +78,8 @@ import {
import { decodePathParams } from '../lib/router-utils/decode-path-params' import { decodePathParams } from '../lib/router-utils/decode-path-params'
import { removeTrailingSlash } from '../../shared/lib/router/utils/remove-trailing-slash' import { removeTrailingSlash } from '../../shared/lib/router/utils/remove-trailing-slash'
import { isInterceptionRouteRewrite } from '../../lib/is-interception-route-rewrite' import { isInterceptionRouteRewrite } from '../../lib/is-interception-route-rewrite'
import { getTracer } from '../lib/trace/tracer' import { getTracer, SpanKind, type Span } from '../lib/trace/tracer'
import { RouteModuleSpan } from '../lib/trace/constants' import { BaseServerSpan, RouteModuleSpan } from '../lib/trace/constants'
/** /**
* RouteModuleOptions is the options that are passed to the route module, other * RouteModuleOptions is the options that are passed to the route module, other
...@@ -96,6 +96,26 @@ export interface RouteModuleOptions< ...@@ -96,6 +96,26 @@ export interface RouteModuleOptions<
readonly relativeProjectDir: string readonly relativeProjectDir: string
} }
/**
* Returned by `RouteModule.ensureInstrumentation` with `deferSpans`, and passed
* to `RouteModule.traceRequest`.
*
* @internal
*/
export interface RequestInstrumentation {
/**
* When the request started, before registering `instrumentation.ts` (same
* clock as the instrumentation lifecycle spans). The request span is
* backdated to it.
*/
startTime: number
/**
* Emits the lifecycle spans of `instrumentation.ts`. Only set for the request
* that triggered registration; called inside the request span.
*/
emitSpans?: () => void
}
/** /**
* RouteHandlerContext is the base context for a route handler. * RouteHandlerContext is the base context for a route handler.
*/ */
...@@ -632,6 +652,85 @@ export abstract class RouteModule< ...@@ -632,6 +652,85 @@ export abstract class RouteModule<
return { nextConfig, deploymentId } return { nextConfig, deploymentId }
} }
/**
* Runs a route handler that was invoked directly (e.g. by an adapter or a
* deployment, without next-server) inside its request span
* (`BaseServer.handleRequest`).
*
* The span is started before `fn` calls `prepare`, so everything done for
* the request, including route module preparation, belongs to the request's
* trace. When the handler is wrapped by next-server, which already started
* the request span, `fn` runs inside that active span instead.
*
* Callers must await `ensureInstrumentation` before calling this: on the
* first request, registering `instrumentation.ts` usually installs the
* tracer provider, and a span started earlier would not be recorded. With
* the `instrumentation` it returned, the request span is backdated to cover
* registration, and the lifecycle spans of `instrumentation.ts` are children
* of the request span when this request triggered registration.
*
* `fn` receives the request span and the request's root span: the span that
* was active when the handler was invoked (next-server's request span, or a
* platform-created HTTP span), or the request span itself when there is no
* such span.
*
* `tracer` must be the caller's tracer: route modules are bundled with their
* own copy of the tracer, whose root span bookkeeping (e.g.
* `getRootSpanAttributes`) the caller can't see.
*
* @internal - only called by the Next.js build templates. Kept out of the
* published type declarations, which can't reference the tracer's types.
*/
public async traceRequest<T>(
req: IncomingMessage,
{
tracer,
spanName,
httpTarget,
instrumentation,
}: {
tracer: ReturnType<typeof getTracer>
spanName: string
httpTarget: string | undefined
instrumentation: RequestInstrumentation | undefined
},
fn: (span: Span | undefined, rootSpan: Span | undefined) => Promise<T>
): Promise<T> {
const activeSpan = tracer.getActiveScopeSpan()
const isWrappedByNextServer = Boolean(
this.getRouterServerContext(req)?.isWrappedByNextServer
)
// TODO: activeSpan code path is for when wrapped by
// next-server can be removed when this is no longer used
if (isWrappedByNextServer && activeSpan) {
return fn(activeSpan, activeSpan)
}
return tracer.withPropagatedContext(
req.headers,
() =>
tracer.trace(
BaseServerSpan.handleRequest,
{
spanName,
kind: SpanKind.SERVER,
startTime: instrumentation?.startTime,
attributes: {
'http.method': req.method || 'GET',
'http.target': httpTarget,
},
},
(span) => {
instrumentation?.emitSpans?.()
return fn(span, activeSpan ?? span)
}
),
undefined,
!isWrappedByNextServer
)
}
public prepare( public prepare(
req: IncomingMessage | BaseNextRequest, req: IncomingMessage | BaseNextRequest,
res: ServerResponse | null, res: ServerResponse | null,
...@@ -647,6 +746,72 @@ export abstract class RouteModule< ...@@ -647,6 +746,72 @@ export abstract class RouteModule<
) )
} }
/**
* Makes sure `instrumentation.ts` is registered for the Node.js runtime.
* Safe to call more than once per request. Returns the absolute project
* directory (`undefined` in the edge runtime, which handles loading
* instrumentation at the edge adapter level).
*
* With `deferSpans` (callers of `traceRequest`), it also returns the
* `instrumentation` to pass to `traceRequest`: when this call triggered
* registration, the lifecycle spans of `instrumentation.ts` are not emitted
* yet but emitted inside the request span.
*
* @internal - called by `prepare` and by the Next.js build templates before
* `traceRequest`.
*/
public async ensureInstrumentation(
req: IncomingMessage | BaseNextRequest,
{ deferSpans = false }: { deferSpans?: boolean } = {}
): Promise<{
absoluteProjectDir: string | undefined
instrumentation: RequestInstrumentation | undefined
}> {
let absoluteProjectDir: string | undefined
const instrumentation: RequestInstrumentation | undefined = deferSpans
? { startTime: performance.timeOrigin + performance.now() }
: undefined
if (process.env.NEXT_RUNTIME !== 'edge') {
const { join, relative } =
require('node:path') as typeof import('node:path')
absoluteProjectDir = join(
/* turbopackIgnore: true */
process.cwd(),
getRequestMeta(req, 'relativeProjectDir') || this.relativeProjectDir
)
const absoluteDistDir = getRequestMeta(req, 'distDir')
if (absoluteDistDir) {
this.distDir = relative(absoluteProjectDir, absoluteDistDir)
}
const {
ensureInstrumentationRegistered,
ensureInstrumentationRegisteredWithDeferredSpans,
} = await import(
'../lib/router-utils/instrumentation-globals.external.js'
)
// ensure instrumentation is registered and pass
// onRequestError below. Awaited so any caller of `RouteModule.prepare`
// that bypasses `BaseServer.handleRequest` (where this is also awaited
// via `prepareImpl`) still observes the instrumentation hook completing
// before the userland route handler runs.
if (instrumentation) {
instrumentation.emitSpans =
await ensureInstrumentationRegisteredWithDeferredSpans(
absoluteProjectDir,
this.distDir
)
} else {
await ensureInstrumentationRegistered(absoluteProjectDir, this.distDir)
}
}
return { absoluteProjectDir, instrumentation }
}
private async prepareImpl( private async prepareImpl(
req: IncomingMessage | BaseNextRequest, req: IncomingMessage | BaseNextRequest,
res: ServerResponse | null, res: ServerResponse | null,
...@@ -701,38 +866,10 @@ export abstract class RouteModule< ...@@ -701,38 +866,10 @@ export abstract class RouteModule<
} }
| undefined | undefined
> { > {
let absoluteProjectDir: string | undefined if (process.env.NEXT_RUNTIME !== 'edge' && res) {
patchSetHeaderWithCookieSupport(req, res)
// edge runtime handles loading instrumentation at the edge adapter level
if (process.env.NEXT_RUNTIME !== 'edge') {
if (res) {
patchSetHeaderWithCookieSupport(req, res)
}
const { join, relative } =
require('node:path') as typeof import('node:path')
absoluteProjectDir = join(
/* turbopackIgnore: true */
process.cwd(),
getRequestMeta(req, 'relativeProjectDir') || this.relativeProjectDir
)
const absoluteDistDir = getRequestMeta(req, 'distDir')
if (absoluteDistDir) {
this.distDir = relative(absoluteProjectDir, absoluteDistDir)
}
const { ensureInstrumentationRegistered } = await import(
'../lib/router-utils/instrumentation-globals.external.js'
)
// ensure instrumentation is registered and pass
// onRequestError below. Awaited so any caller of `RouteModule.prepare`
// that bypasses `BaseServer.handleRequest` (where this is also awaited
// via `prepareImpl`) still observes the instrumentation hook completing
// before the userland route handler runs.
await ensureInstrumentationRegistered(absoluteProjectDir, this.distDir)
} }
const { absoluteProjectDir } = await this.ensureInstrumentation(req)
const manifests = getTracer().trace( const manifests = getTracer().trace(
RouteModuleSpan.loadManifests, RouteModuleSpan.loadManifests,
{ spanName: 'load route manifests' }, { spanName: 'load route manifests' },
......
This diff is collapsed.
import {
forwardTraceContextToRequest,
getSharedParentTraceContext,
} from './forward-trace-context'
import { NextResponse } from './spec-extension/response'
const TRACEPARENT = '00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01'
const TRACESTATE = 'vendor=value'
const RECEIVED_TRACEPARENT =
'00-11111111111111111111111111111111-2222222222222222-01'
function requestOverrides(headers: Headers) {
const list = headers.get('x-middleware-override-headers')
if (list === null) return undefined
return Object.fromEntries(
list
.split(',')
.filter(Boolean)
.map((key) => [key, headers.get(`x-middleware-request-${key}`)])
)
}
describe('getSharedParentTraceContext', () => {
it('uses the received trace headers verbatim when they are the active context', () => {
const received = new Headers({
traceparent: RECEIVED_TRACEPARENT,
// Not normalized by the propagator.
tracestate: 'a=1 ,b=2',
})
expect(
getSharedParentTraceContext(received, {
traceparent: RECEIVED_TRACEPARENT,
tracestate: 'a=1,b=2',
})
).toEqual({ traceparent: RECEIVED_TRACEPARENT, tracestate: 'a=1 ,b=2' })
})
it('keeps the tracestate key when the received trace headers have none', () => {
const received = new Headers({ traceparent: RECEIVED_TRACEPARENT })
expect(
getSharedParentTraceContext(received, {
traceparent: RECEIVED_TRACEPARENT,
tracestate: undefined,
})
).toStrictEqual({
traceparent: RECEIVED_TRACEPARENT,
tracestate: undefined,
})
})
it('uses the active context when the request carries none', () => {
expect(
getSharedParentTraceContext(new Headers(), {
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
).toEqual({ traceparent: TRACEPARENT, tracestate: TRACESTATE })
})
})
describe('forwardTraceContextToRequest', () => {
const requestHeaders = () =>
new Headers({ accept: 'text/html', cookie: 'a=b', 'x-custom': 'custom' })
it('adds nothing when the request already carries the trace context', () => {
const received = requestHeaders()
received.set('traceparent', RECEIVED_TRACEPARENT)
received.set('tracestate', 'received=1')
const response = NextResponse.next()
forwardTraceContextToRequest(response.headers, received, {
traceparent: RECEIVED_TRACEPARENT,
tracestate: 'received=1',
})
expect([...response.headers.keys()]).toEqual(['x-middleware-next'])
})
it('lists every request header plus the trace context when the request carries none', () => {
const response = NextResponse.next()
forwardTraceContextToRequest(response.headers, requestHeaders(), {
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
expect(requestOverrides(response.headers)).toEqual({
accept: 'text/html',
cookie: 'a=b',
'x-custom': 'custom',
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
})
it('appends the trace context to user overrides that omit it', () => {
const userHeaders = new Headers({ 'x-user': 'user' })
const response = NextResponse.next({ request: { headers: userHeaders } })
forwardTraceContextToRequest(response.headers, requestHeaders(), {
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
expect(requestOverrides(response.headers)).toEqual({
'x-user': 'user',
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
})
it('drops a received tracestate when the trace context has none', () => {
const received = requestHeaders()
received.set('tracestate', 'received=1')
const response = NextResponse.next()
forwardTraceContextToRequest(response.headers, received, {
traceparent: TRACEPARENT,
})
expect(requestOverrides(response.headers)).toEqual({
accept: 'text/html',
cookie: 'a=b',
'x-custom': 'custom',
traceparent: TRACEPARENT,
})
})
it('drops a tracestate copied into user overrides when the trace context has none', () => {
const received = requestHeaders()
received.set('traceparent', RECEIVED_TRACEPARENT)
received.set('tracestate', 'received=1')
const response = NextResponse.next({
request: { headers: new Headers(received) },
})
forwardTraceContextToRequest(response.headers, received, {
traceparent: TRACEPARENT,
})
expect(requestOverrides(response.headers)).toEqual({
accept: 'text/html',
cookie: 'a=b',
'x-custom': 'custom',
traceparent: TRACEPARENT,
})
})
it('replaces a tracestate set by the user when the traceparent is forwarded', () => {
const response = NextResponse.next({
request: { headers: new Headers({ tracestate: 'user=1' }) },
})
forwardTraceContextToRequest(response.headers, requestHeaders(), {
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
expect(requestOverrides(response.headers)).toEqual({
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
})
it('keeps a traceparent explicitly set by the user without a forwarded tracestate', () => {
const userTraceparent =
'00-33333333333333333333333333333333-4444444444444444-01'
const response = NextResponse.next({
request: { headers: new Headers({ traceparent: userTraceparent }) },
})
forwardTraceContextToRequest(response.headers, requestHeaders(), {
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
expect(requestOverrides(response.headers)).toEqual({
traceparent: userTraceparent,
})
})
it('keeps the trace headers explicitly set by the user', () => {
const userTraceparent =
'00-33333333333333333333333333333333-4444444444444444-01'
const received = requestHeaders()
received.set('tracestate', 'received=1')
const response = NextResponse.next({
request: {
headers: new Headers({
traceparent: userTraceparent,
tracestate: 'user=1',
}),
},
})
forwardTraceContextToRequest(response.headers, received, {
traceparent: TRACEPARENT,
tracestate: TRACESTATE,
})
expect(requestOverrides(response.headers)).toEqual({
traceparent: userTraceparent,
tracestate: 'user=1',
})
})
})
import type { TraceContextHeaders } from '../lib/trace/tracer'
const OVERRIDE_HEADERS = 'x-middleware-override-headers'
const REQUEST_HEADER_PREFIX = 'x-middleware-request-'
// The trace ID and parent ID of a W3C `traceparent`
// (`version-traceid-parentid-flags`), lower-cased.
function getTraceparentIds(traceparent: string | null | undefined) {
const [, traceId, parentId] = traceparent?.toLowerCase().split('-') ?? []
return traceId && parentId ? `${traceId}-${parentId}` : undefined
}
/**
* The trace context of the span that the middleware span and the downstream
* request (the render) share as their parent, given the middleware's active
* context (before the middleware span starts).
*
* When the active context is the one the request carries (an incoming
* `traceparent`, or the one next-server's middleware-invoke pass wrote onto
* the request), the received `traceparent` / `tracestate` are returned
* verbatim, so the middleware never changes them. Otherwise (an active host
* span, or a root span inserted by the middleware adapter) the active
* context is returned.
*/
export function getSharedParentTraceContext(
requestHeaders: Headers,
activeTraceContext: TraceContextHeaders | undefined
): TraceContextHeaders | undefined {
if (!activeTraceContext) {
return undefined
}
const receivedTraceparent = requestHeaders.get('traceparent')
const receivedIds = getTraceparentIds(receivedTraceparent)
if (
receivedTraceparent &&
receivedIds &&
receivedIds === getTraceparentIds(activeTraceContext.traceparent)
) {
const receivedTracestate = requestHeaders.get('tracestate')
return receivedTracestate
? { traceparent: receivedTraceparent, tracestate: receivedTracestate }
: { traceparent: receivedTraceparent, tracestate: undefined }
}
return activeTraceContext
}
/**
* Makes sure the downstream request of a middleware response that continues
* the request (`NextResponse.next()` or a rewrite) carries the trace context
* of the middleware span's parent, so the render is a sibling of the
* middleware span in the same trace. It is passed as `traceparent` /
* `tracestate` request-header overrides, which every host applies to the
* downstream request: hosts that invoke middleware separately from the render
* (deployments, adapters, direct entrypoints) as well as next-server's router
* server.
*
* `traceparent` and `tracestate` are forwarded as a pair: the downstream
* request never carries one trace context's `traceparent` with another's
* `tracestate`. Request-header overrides replace the whole downstream request
* header set, so:
* - Without user overrides (or with an empty list, which hosts ignore):
* nothing is added when the request already carries that trace context.
* Otherwise every request header the middleware received is listed
* (unchanged) alongside the trace context, except a received `tracestate`
* when the trace context has none.
* - With user overrides, they are kept and the trace context headers are
* added. A `traceparent` the user explicitly set (to a value other than the
* one the middleware received) wins, together with whatever `tracestate`
* the user set (if any). Otherwise (`traceparent` copied unchanged from the
* request, e.g. via `new Headers(request.headers)`, or omitted) both are set
* to the trace context, dropping a user `tracestate` when it has none.
*/
export function forwardTraceContextToRequest(
responseHeaders: Headers,
requestHeaders: Headers,
traceContext: TraceContextHeaders | undefined
): void {
if (!traceContext) {
return
}
// Redirects and direct responses have no downstream request.
if (
!responseHeaders.has('x-middleware-next') &&
!responseHeaders.has('x-middleware-rewrite')
) {
return
}
const userOverrides = responseHeaders.get(OVERRIDE_HEADERS)
// An empty override list is ignored by hosts (no headers are replaced), so
// treat it like no overrides rather than turning it into "only the trace
// context headers".
if (!userOverrides) {
if (
requestHeaders.get('traceparent') === traceContext.traceparent &&
requestHeaders.get('tracestate') === (traceContext.tracestate ?? null)
) {
// The downstream request already carries the trace context.
return
}
const keys: string[] = []
for (const [key, value] of requestHeaders) {
if (!isTraceHeader(key)) {
responseHeaders.set(REQUEST_HEADER_PREFIX + key, value)
keys.push(key)
}
}
setTraceContextOverrides(responseHeaders, keys, traceContext)
return
}
const keys = userOverrides
.split(',')
.map((key) => key.trim())
.filter(Boolean)
if (
keys.includes('traceparent') &&
responseHeaders.get(REQUEST_HEADER_PREFIX + 'traceparent') !==
requestHeaders.get('traceparent')
) {
// The user explicitly set the `traceparent`: keep both trace headers as
// the user set them.
return
}
setTraceContextOverrides(
responseHeaders,
keys.filter((key) => !isTraceHeader(key)),
traceContext
)
}
function isTraceHeader(key: string) {
return key === 'traceparent' || key === 'tracestate'
}
// Lists the trace context's `traceparent` and `tracestate` (or no
// `tracestate`, if it has none) as request-header overrides, after `keys`
// (the other overridden headers).
function setTraceContextOverrides(
responseHeaders: Headers,
keys: string[],
traceContext: TraceContextHeaders
) {
responseHeaders.set(
REQUEST_HEADER_PREFIX + 'traceparent',
traceContext.traceparent
)
keys.push('traceparent')
if (traceContext.tracestate) {
responseHeaders.set(
REQUEST_HEADER_PREFIX + 'tracestate',
traceContext.tracestate
)
keys.push('tracestate')
} else {
responseHeaders.delete(REQUEST_HEADER_PREFIX + 'tracestate')
}
responseHeaders.set(OVERRIDE_HEADERS, keys.join(','))
}
import { headers } from 'next/headers'
import { Suspense } from 'react'
import { getRenderSpans } from '../../trace-recorder'
import { markMetadataGenerated, metadataGenerated } from '../../metadata-ready'
async function RequestInfo() {
const requestHeaders = Object.fromEntries((await headers()).entries())
await metadataGenerated()
return (
<>
<pre id="spans">{JSON.stringify(getRenderSpans())}</pre>
<pre id="headers">{JSON.stringify(requestHeaders)}</pre>
</>
)
}
export default function Page() {
return (
<Suspense>
<RequestInfo />
</Suspense>
)
}
export async function generateMetadata() {
markMetadataGenerated('page')
return {}
}
import { connection } from 'next/server'
import { getInstrumentationLifecycleSpans } from '../../trace-recorder'
export async function GET() {
// Run at request time, in the instance serving this request.
await connection()
return Response.json({ spans: getInstrumentationLifecycleSpans() })
}
import type { ReactNode } from 'react'
import { markMetadataGenerated } from '../metadata-ready'
export default function Root({ children }: { children: ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
// Traced as `generateMetadata` spans in the render.
export async function generateMetadata() {
markMetadataGenerated('layout')
return {}
}
export default function Home() {
return <p>otel-middleware-trace</p>
}
export async function generateMetadata() {
return {}
}
import { headers } from 'next/headers'
import { Suspense } from 'react'
import { getRenderSpans } from '../../trace-recorder'
import { markMetadataGenerated, metadataGenerated } from '../../metadata-ready'
async function Spans() {
// Reading request headers also makes this run at request time, so the
// reported spans belong to the request.
await headers()
await metadataGenerated()
return <pre id="spans">{JSON.stringify(getRenderSpans())}</pre>
}
export default function Page() {
return (
<Suspense>
<Spans />
</Suspense>
)
}
export async function generateMetadata() {
markMetadataGenerated('page')
return {}
}
import { connection } from 'next/server'
import { getRenderSpans } from '../../trace-recorder'
export async function GET() {
// Run at request time so the reported spans belong to the request.
await connection()
return Response.json({ spans: getRenderSpans() })
}
import { connection } from 'next/server'
import { Suspense } from 'react'
import { getRenderSpans } from '../../trace-recorder'
import { markMetadataGenerated, metadataGenerated } from '../../metadata-ready'
async function Spans() {
// Run at request time so the reported spans belong to the request.
await connection()
await metadataGenerated()
return <pre id="spans">{JSON.stringify(getRenderSpans())}</pre>
}
// Not matched by the middleware: a baseline for requests without middleware.
export default function Page() {
return (
<Suspense>
<Spans />
</Suspense>
)
}
export async function generateMetadata() {
markMetadataGenerated('page')
return {}
}
const { createServer } = require('http')
const { parse } = require('url')
const next = require('next')
const getPort = require('get-port')
const { trace } = require('@opentelemetry/api')
// A custom server that wraps every request in an active host span, like an
// APM's HTTP instrumentation does. The tracer provider is the one
// `instrumentation.ts` registers (with the span recorder) during `prepare()`.
async function main() {
const hostname = 'localhost'
const port = await getPort()
const app = next({
dev: process.env.NODE_ENV === 'development',
hostname,
port,
dir: __dirname,
})
const handle = app.getRequestHandler()
await app.prepare()
const tracer = trace.getTracer('custom-server')
createServer((req, res) => {
tracer.startActiveSpan('custom-server-request', async (span) => {
try {
await handle(req, res, parse(req.url ?? '/', true))
} catch (err) {
span.recordException(err)
res.statusCode = 500
res.end('Internal Server Error')
} finally {
span.end()
}
})
}).listen(port, hostname, () => {
console.log(`- Local: http://${hostname}:${port}`)
})
}
main().catch((err) => {
console.error(err)
process.exit(1)
})
import { registerOTel } from '@vercel/otel'
import { spanRecorder } from './trace-recorder'
export function register() {
// No exporter is needed: the app itself reports the spans of a request, so
// the suite also works when deployed (where no local collector is
// reachable).
registerOTel({
serviceName: 'otel-middleware-trace',
spanProcessors: [spanRecorder],
})
}
import { cache } from 'react'
// `generateMetadata` runs concurrently with the page. The page reports the
// render's spans once `generateMetadata` of the layout and the page ran (per
// request, via `cache`), so the `generateMetadata` spans are always among them.
type Segment = 'layout' | 'page'
const getDeferred = cache((_segment: Segment) => {
let resolve!: () => void
const promise = new Promise<void>((r) => {
resolve = r
})
return { promise, resolve }
})
export function markMetadataGenerated(segment: Segment) {
getDeferred(segment).resolve()
}
export async function metadataGenerated() {
// Don't hang when `generateMetadata` doesn't run for this render (e.g. its
// result was prerendered).
let timeout: ReturnType<typeof setTimeout> | undefined
await Promise.race([
Promise.all([getDeferred('layout').promise, getDeferred('page').promise]),
new Promise<void>((resolve) => {
timeout = setTimeout(resolve, 5_000)
}),
])
clearTimeout(timeout)
}
import { NextResponse, type NextRequest } from 'next/server'
import { getMiddlewareSpans } from './trace-recorder'
export const USER_TRACE_ID = '0af7651916cd43dd8448eb211c80319c'
export const USER_SPAN_ID = 'b7ad6b7169203331'
// Shared by `proxy.ts` (Node.js runtime) and `middleware.ts` (Edge runtime).
export function handler(request: NextRequest) {
// The middleware's span and its ancestors started in this process, for the
// test to print the request's span tree.
const spans = getMiddlewareSpans()
const { pathname } = request.nextUrl
let response: NextResponse
if (pathname === '/rewrite') {
response = NextResponse.rewrite(new URL('/page', request.url))
} else if (pathname === '/redirect') {
response = NextResponse.redirect(new URL('/page', request.url))
} else if (pathname === '/direct') {
response = new NextResponse('direct response from middleware')
} else if (pathname === '/user-overrides') {
const headers = new Headers(request.headers)
headers.set('x-user-added', 'added-by-middleware')
headers.delete('x-delete-me')
response = NextResponse.rewrite(new URL('/headers-echo', request.url), {
request: { headers },
})
} else if (pathname === '/user-traceparent') {
const headers = new Headers(request.headers)
headers.set('traceparent', `00-${USER_TRACE_ID}-${USER_SPAN_ID}-01`)
response = NextResponse.rewrite(new URL('/headers-echo', request.url), {
request: { headers },
})
} else {
response = NextResponse.next()
}
response.headers.set('x-mw-spans', JSON.stringify(spans))
return response
}
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {}
export default nextConfig
import { trace, type Context } from '@opentelemetry/api'
import type {
ReadableSpan,
Span as SdkSpan,
SpanProcessor,
} from '@opentelemetry/sdk-trace-base'
// Records every span started in this process with its parent, using only
// public OpenTelemetry APIs (`onStart`'s parent context), so the app can report
// the spans of a request itself. This works in the Node.js and Edge runtimes
// and when deployed, where no collector is reachable.
export interface RecordedSpan {
id: string
parentId: string | undefined
traceId: string
name: string
/** The `next.span_type` attribute. */
type: string | undefined
/** The span's `PRINTED_ATTRIBUTES` that are set, when reported. */
attributes?: Record<string, string | number | boolean>
}
// Attributes shown in the span trees, set by Next.js at different points of a
// request (some of them only once the request span ends, so they're only
// shown if they were set when the spans were reported).
const PRINTED_ATTRIBUTES = [
'http.target',
'http.route',
'next.route',
'http.status_code',
]
interface Entry {
span: SdkSpan
parentId: string | undefined
}
const MAX_RECORDED_SPANS = 10_000
function getRecordedSpans(): Map<string, Entry> {
const store = globalThis as typeof globalThis & {
__otelTestRecordedSpans?: Map<string, Entry>
}
store.__otelTestRecordedSpans ??= new Map()
return store.__otelTestRecordedSpans
}
export const spanRecorder: SpanProcessor = {
onStart(span: SdkSpan, parentContext: Context) {
const spans = getRecordedSpans()
if (spans.size >= MAX_RECORDED_SPANS) {
// Drop the oldest entry (Map iterates in insertion order).
spans.delete(spans.keys().next().value!)
}
spans.set(span.spanContext().spanId, {
span,
parentId: trace.getSpanContext(parentContext)?.spanId,
})
},
onEnd(_span: ReadableSpan) {},
forceFlush: () => Promise.resolve(),
shutdown: () => Promise.resolve(),
}
// Read when reported, so spans renamed after they started (e.g. a request
// span named after its route) show their current name.
function toRecordedSpan({ span, parentId }: Entry): RecordedSpan {
const type = span.attributes['next.span_type']
const attributes: Record<string, string | number | boolean> = {}
for (const key of PRINTED_ATTRIBUTES) {
const value = span.attributes[key]
if (
typeof value === 'string' ||
typeof value === 'number' ||
typeof value === 'boolean'
) {
attributes[key] = value
}
}
return {
id: span.spanContext().spanId,
parentId,
traceId: span.spanContext().traceId,
name: span.name,
type: typeof type === 'string' ? type : undefined,
attributes: Object.keys(attributes).length > 0 ? attributes : undefined,
}
}
/** The given span and its ancestors, as far as they were started here. */
function getChain(spanId: string | undefined) {
const spans = getRecordedSpans()
const chain: RecordedSpan[] = []
let entry = spanId ? spans.get(spanId) : undefined
while (entry) {
chain.push(toRecordedSpan(entry))
entry = entry.parentId ? spans.get(entry.parentId) : undefined
}
return chain
}
/** The spans started here so far below the given span (not the span itself). */
function getDescendants(spanId: string) {
return [...getRecordedSpans().values()]
.filter(({ parentId }) =>
getChain(parentId).some((span) => span.id === spanId)
)
.map(toRecordedSpan)
}
/**
* The middleware's spans: its active span, that span's ancestors, and the
* descendants started so far of the topmost of them (e.g. spans started next
* to the middleware span in the root inserted by the middleware adapter).
*/
export function getMiddlewareSpans() {
const chain = getChain(trace.getActiveSpan()?.spanContext().spanId)
if (chain.length === 0) {
return chain
}
const ids = new Set(chain.map((span) => span.id))
return [
...chain,
...getDescendants(chain[chain.length - 1].id).filter(
(span) => !ids.has(span.id)
),
]
}
/**
* The `instrumentation.ts` lifecycle spans (`Instrumentation.*`) started in
* this process, with their ancestors as far as they were started here.
*/
export function getInstrumentationLifecycleSpans() {
const spans = new Map<string, RecordedSpan>()
for (const [id, { span }] of getRecordedSpans()) {
if (
String(span.attributes['next.span_type']).startsWith('Instrumentation.')
) {
for (const recorded of getChain(id)) {
spans.set(recorded.id, recorded)
}
}
}
return [...spans.values()]
}
/**
* The render's spans: its request span (the nearest `BaseServer.handleRequest`
* ancestor of the active span), that span's ancestors, and the request span's
* descendants started so far.
*/
export function getRenderSpans() {
const chain = getChain(trace.getActiveSpan()?.spanContext().spanId)
const index = chain.findIndex(
(span) => span.type === 'BaseServer.handleRequest'
)
if (index === -1) {
return chain
}
const requestSpanId = chain[index].id
return [...chain.slice(index), ...getDescendants(requestSpanId)]
}
...@@ -3,6 +3,12 @@ import { SavedSpan } from './constants' ...@@ -3,6 +3,12 @@ import { SavedSpan } from './constants'
export interface Collector { export interface Collector {
getSpans: () => SavedSpan[] getSpans: () => SavedSpan[]
/**
* Like `getSpans`, but also includes spans marked `next.bubble` (e.g.
* next-server's middleware-invoke `handleRequest` span), which are needed to
* check the full span hierarchy of a request.
*/
getAllSpans: () => SavedSpan[]
reset: () => void reset: () => void
shutdown: () => Promise<void> shutdown: () => Promise<void>
} }
...@@ -13,6 +19,7 @@ export async function connectCollector({ ...@@ -13,6 +19,7 @@ export async function connectCollector({
port: number port: number
}): Promise<Collector> { }): Promise<Collector> {
const spans: SavedSpan[] = [] const spans: SavedSpan[] = []
const allSpans: SavedSpan[] = []
const server = new HttpServer(async (req, res) => { const server = new HttpServer(async (req, res) => {
if (req.method !== 'POST') { if (req.method !== 'POST') {
...@@ -33,6 +40,7 @@ export async function connectCollector({ ...@@ -33,6 +40,7 @@ export async function connectCollector({
}) })
const newSpans = JSON.parse(body.toString('utf-8')) as SavedSpan[] const newSpans = JSON.parse(body.toString('utf-8')) as SavedSpan[]
allSpans.push(...newSpans)
const filteredSpans = newSpans.filter((span) => { const filteredSpans = newSpans.filter((span) => {
if (span.attributes?.['next.bubble'] === true) { if (span.attributes?.['next.bubble'] === true) {
return false return false
...@@ -61,8 +69,12 @@ export async function connectCollector({ ...@@ -61,8 +69,12 @@ export async function connectCollector({
getSpans() { getSpans() {
return spans return spans
}, },
getAllSpans() {
return allSpans
},
reset() { reset() {
spans.length = 0 spans.length = 0
allSpans.length = 0
}, },
shutdown() { shutdown() {
return new Promise<void>((resolve, reject) => return new Promise<void>((resolve, reject) =>
......
...@@ -189,6 +189,7 @@ async function main() { ...@@ -189,6 +189,7 @@ async function main() {
try { try {
await handle() await handle()
} catch (err) { } catch (err) {
console.error('Failed to handle request:', err)
res.statusCode = 500 res.statusCode = 500
res.end('Internal Server Error') res.end('Internal Server Error')
} }
......
...@@ -5,6 +5,15 @@ const { ...@@ -5,6 +5,15 @@ const {
const REJECT_REVALIDATION = Symbol.for('next.test.reject-revalidation') const REJECT_REVALIDATION = Symbol.for('next.test.reject-revalidation')
module.exports = class IncrementalCacheHandler extends FileSystemCache { module.exports = class IncrementalCacheHandler extends FileSystemCache {
constructor(ctx) {
// Lets a test fail a request while the middleware adapter sets it up,
// before it runs the middleware.
if (ctx._requestHeaders?.['x-test-fail-cache-handler'] === '1') {
throw new Error('cache handler failed for this request')
}
super(ctx)
}
async revalidateTag(tags, durations) { async revalidateTag(tags, durations) {
if (tags.includes('reject-after-headers')) { if (tags.includes('reject-after-headers')) {
return new Promise((_, reject) => { return new Promise((_, reject) => {
......
/**
* Prints OpenTelemetry spans as an indented tree, for snapshot tests of the
* trace shape. Span IDs are left out, but trace identity is kept: every tree
* root is labelled with its trace (the name of a known trace context, e.g.
* `incoming` for the test's incoming `traceparent`, otherwise `A`, `B`, … in
* order of appearance), and a span whose trace differs from its parent's is
* marked with its own label. Parents that aren't among the spans are shown as
* `[<name> traceparent]` (a known trace context) or `[remote]`. A span's
* `attributes`, if any, are appended in braces.
*
* ```
* [trace A] GET (Next.request) {http.target=/page}
* ├─ GET /page (BaseServer.handleRequest) {http.target=/page}
* │ └─ render route (app) /page (AppRender.getBodyResult)
* └─ middleware GET (Middleware.execute)
* ```
*/
export interface TreeSpan {
id: string
parentId?: string
traceId: string
name: string
/** The `next.span_type` attribute. */
type?: string
/** Attributes to print, in the given order. */
attributes?: Record<string, string | number | boolean>
}
interface Node {
label: string
traceId: string
children: Node[]
}
/** A trace context the test knows, e.g. the `traceparent` it sends. */
export interface KnownTraceContext {
name: string
traceId: string
spanId: string
}
/**
* One-time work outside of the request's own work: instrumentation
* registration, app route module loading (in the first request only), and
* route module prewarming at startup (root `LoadComponents.loadRouteModule`
* spans, which can still arrive during the first test). Omit it (`omit`) so
* span tree snapshots don't depend on the test order.
*/
export function isOneTimeSpan(span: TreeSpan) {
return (
span.type === 'Instrumentation.loadModule' ||
span.type === 'Instrumentation.register' ||
span.type === 'AppRouteRouteModule.loadUserland' ||
(!span.parentId && span.type === 'LoadComponents.loadRouteModule')
)
}
export function printSpanTree(
allSpans: TreeSpan[],
{
known = [],
omit = () => false,
}: {
known?: KnownTraceContext[]
/** Spans to leave out, with their descendants (e.g. `isOneTimeSpan`). */
omit?: (span: TreeSpan) => boolean
} = {}
): string {
const omitted = new Set<string>()
for (let changed = true; changed; ) {
changed = false
for (const span of allSpans) {
if (
!omitted.has(span.id) &&
(omit(span) || (span.parentId && omitted.has(span.parentId)))
) {
omitted.add(span.id)
changed = true
}
}
}
const spans = allSpans.filter((span) => !omitted.has(span.id))
const knownNames = new Set(known.map((context) => context.name))
const traceLabels = new Map<string, string>()
for (const context of known) {
traceLabels.set(context.traceId, context.name)
}
const traceLabel = (traceId: string) => {
let label = traceLabels.get(traceId)
if (!label) {
const count = [...traceLabels.values()].filter(
(value) => !knownNames.has(value)
).length
label =
count < 26
? String.fromCharCode('A'.charCodeAt(0) + count)
: String(count + 1)
traceLabels.set(traceId, label)
}
return label
}
const nodes = new Map<string, Node>()
for (const span of spans) {
const label =
span.type && span.type !== span.name
? `${span.name} (${span.type})`
: span.name
const attributes = Object.entries(span.attributes ?? {})
.map(([key, value]) => `${key}=${value}`)
.join(', ')
nodes.set(span.id, {
label: attributes ? `${label} {${attributes}}` : label,
traceId: span.traceId,
children: [],
})
}
// Parents that aren't among the spans, by parent span ID.
const missingParents = new Map<string, Node>()
const roots: Node[] = []
for (const span of spans) {
const node = nodes.get(span.id)!
if (!span.parentId) {
roots.push(node)
continue
}
let parent = nodes.get(span.parentId) ?? missingParents.get(span.parentId)
if (!parent) {
const context = known.find(({ spanId }) => spanId === span.parentId)
parent = {
label: context ? `[${context.name} traceparent]` : '[remote]',
traceId: span.traceId,
children: [],
}
missingParents.set(span.parentId, parent)
roots.push(parent)
}
parent.children.push(node)
}
const sortNodes = (list: Node[]) =>
list.sort((a, b) => a.label.localeCompare(b.label))
// Assign trace labels in a deterministic order (sorted roots, depth first),
// so the labels don't depend on the order the spans were collected in.
const assignLabels = (node: Node) => {
traceLabel(node.traceId)
sortNodes(node.children).forEach(assignLabels)
}
sortNodes(roots).forEach(assignLabels)
roots.sort(
(a, b) =>
traceLabel(a.traceId).localeCompare(traceLabel(b.traceId)) ||
a.label.localeCompare(b.label)
)
const lines: string[] = []
const print = (node: Node, prefix: string) => {
node.children.forEach((child, index) => {
const last = index === node.children.length - 1
const marker =
child.traceId !== node.traceId
? ` [trace ${traceLabel(child.traceId)}]`
: ''
lines.push(`${prefix}${last ? '└─ ' : '├─ '}${child.label}${marker}`)
print(child, prefix + (last ? ' ' : '│ '))
})
}
for (const root of roots) {
lines.push(`[trace ${traceLabel(root.traceId)}] ${root.label}`)
print(root, '')
}
return lines.join('\n')
}
...@@ -19,6 +19,7 @@ ...@@ -19,6 +19,7 @@
"test-data-service/*": ["./test/lib/test-data-service/*"], "test-data-service/*": ["./test/lib/test-data-service/*"],
"test-log": ["./test/lib/test-log"], "test-log": ["./test/lib/test-log"],
"router-act": ["./test/lib/router-act"], "router-act": ["./test/lib/router-act"],
"otel-span-tree": ["./test/lib/otel-span-tree"],
"@next/playwright": ["./packages/next-playwright/src/index.ts"] "@next/playwright": ["./packages/next-playwright/src/index.ts"]
} }
}, },
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment