Unverified Commit 65675c37 authored by Tobias Koppers's avatar Tobias Koppers Committed by GitHub

Emit instrumentation lifecycle spans in the request that registered it (#100350)

## Summary

Stacked on #100324. Includes the commits from #100392, which were folded
into this PR.

Next.js emits two lifecycle spans for `instrumentation.ts` in the
Node.js runtime: `load instrumentation module` and `register
instrumentation`. They're backdated, because they're emitted after
`register()` returns (`register()` usually installs the tracer
provider). Until now they were always separate root traces.

Direct entrypoints (Vercel, adapters, standalone handlers) register
instrumentation lazily, in the first request. That split the first
request's work across extra traces. Now the spans join the trace of the
request that triggered registration:

```
route / page entrypoint:
 [P →] GET /page (BaseServer.handleRequest, backdated to cover registration)
        ├─ load instrumentation module
        ├─ register instrumentation
        └─ prepare route module → render …

Node.js middleware entrypoint, no parent:
 GET (Next.request, backdated)
  ├─ middleware GET (backdated to cover registration)
  │   ├─ load instrumentation module
  │   └─ register instrumentation
  └─ GET /page

Node.js middleware entrypoint with host span / incoming traceparent P:
 P
  ├─ middleware GET (backdated to cover registration)
  │   ├─ load instrumentation module
  │   └─ register instrumentation
  └─ GET /page
```

### How

- `ensureInstrumentationRegisteredWithDeferredSpans`: when this call
starts registration and registration succeeds, the lifecycle spans are
recorded instead of emitted. The function returns an idempotent emitter
to that caller only. Concurrent callers, later callers and startup
callers get nothing.
- `RouteModule.ensureInstrumentation(req, { deferSpans: true })` is
called by the templates before `traceRequest`. It records when the
request started and returns that time, together with the emitter, as
`instrumentation`. `traceRequest` takes it as an option, backdates the
request span to that time, and emits the lifecycle spans as the span's
first children.
- The Node.js middleware template passes the emitter and the start time
to the adapter. The adapter calls the emitter first thing inside the
`Middleware.execute` span and backdates that span to the request start.
- The shared parent (incoming `traceparent`, host span, or the backdated
inserted `Next.request` root) is still captured before entering the
middleware span. The middleware and the render stay siblings, and
`traceparent` handling and forwarding are unchanged.
- Unchanged:
  - `next start` / `next dev` startup registration (separate roots).
- Registration errors: emitted immediately with the same statuses,
including non-`Error` throws.
  - Edge (no lifecycle spans).
  - No `instrumentation.ts`, or no `register` export.
- User-created spans in `instrumentation.ts` are out of scope (see
#100324).

## Verification

- Collector suite, cold direct entrypoints:
- Route/page requests, with and without an incoming `traceparent`, and
with a host span.
- Node.js middleware requests: no parent, incoming `traceparent`, host
span.
- Each test asserts exactly one load span and one register span in the
request's trace, parented as shown above and covered by the backdated
parent. For middleware, the parent is the middleware span, and it starts
no later than the lifecycle spans and no earlier than its own parent.
- Direct-entrypoint single-trace assertions no longer exclude lifecycle
spans as startup work. Under `next start`, startup spans stay roots.
Fail-first verified.
- Deploy-capable `test/e2e/app-dir/otel-middleware-trace`: span tree
snapshots of the lifecycle spans with their ancestors, plus the first
request's tree including them.
- Deployed: children of the request span that triggered registration, or
of `middleware GET` for Node.js middleware.
  - `next dev` / `next start`: startup roots.
  - No deploy gate.
- Unit tests for the deferral helper: once-only, concurrency, startup
ownership, no `register`, `register()` errors (including `throw
undefined`), and module-load failure.

<!-- NEXT_JS_LLM -->


<!-- fleet ce0ecb23-b14c-4679-92a9-e676130f618e -->

---------
Co-authored-by: 's avatarvercel-fleet-prod[bot] <318278635+vercel-fleet-prod[bot]@users.noreply.github.com>
Co-authored-by: 's avatarTobias Koppers <1365881+sokra@users.noreply.github.com>
Co-authored-by: 's avatarLuke Sandberg <210140+lukesandberg@users.noreply.github.com>
Co-authored-by: 's avatarNiklas Mischkulnig <4586894+mischnic@users.noreply.github.com>
parent 60e68070
...@@ -247,13 +247,18 @@ export function createAppPageEntrypoint({ ...@@ -247,13 +247,18 @@ export function createAppPageEntrypoint({
// Registering `instrumentation.ts` usually installs the tracer provider, so // Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before // do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace. // `prepare`, so route module preparation is part of the request's trace.
await routeModule.ensureInstrumentation(req) // When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest( return routeModule.traceRequest(
req, req,
{ {
tracer: getTracer(), tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`, spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget, httpTarget,
instrumentation,
}, },
(requestSpan, requestRootSpan) => (requestSpan, requestRootSpan) =>
handleRequest( handleRequest(
......
...@@ -133,13 +133,18 @@ export async function handler( ...@@ -133,13 +133,18 @@ export async function handler(
// Registering `instrumentation.ts` usually installs the tracer provider, so // Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before // do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace. // `prepare`, so route module preparation is part of the request's trace.
await routeModule.ensureInstrumentation(req) // When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest( return routeModule.traceRequest(
req, req,
{ {
tracer: getTracer(), tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`, spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget: req.url, httpTarget: req.url,
instrumentation,
}, },
(requestSpan, requestRootSpan) => (requestSpan, requestRootSpan) =>
handleRequest( handleRequest(
......
...@@ -75,12 +75,15 @@ function errorHandledHandler(fn: AdapterOptions['handler']) { ...@@ -75,12 +75,15 @@ function errorHandledHandler(fn: AdapterOptions['handler']) {
} }
const internalHandler: EdgeHandler = async (opts) => { const internalHandler: EdgeHandler = async (opts) => {
let emitInstrumentationSpans: (() => void) | undefined
let requestStartTime: number | undefined
if (process.env.NEXT_RUNTIME !== 'edge') { if (process.env.NEXT_RUNTIME !== 'edge') {
// This mirrors what `RouteModule#prepare` does for routes // This mirrors what `RouteModule#prepare` does for routes
// edge runtime handles loading instrumentation at the edge adapter level // edge runtime handles loading instrumentation at the edge adapter level
requestStartTime = performance.timeOrigin + performance.now()
const { join, relative } = const { join, relative } =
require('node:path') as typeof import('node:path') require('node:path') as typeof import('node:path')
const { ensureInstrumentationRegistered } = const { ensureInstrumentationRegisteredWithDeferredSpans } =
require('../../server/lib/router-utils/instrumentation-globals.external') as typeof import('../../server/lib/router-utils/instrumentation-globals.external') require('../../server/lib/router-utils/instrumentation-globals.external') as typeof import('../../server/lib/router-utils/instrumentation-globals.external')
const absoluteProjectDir = join( const absoluteProjectDir = join(
/* turbopackIgnore: true */ /* turbopackIgnore: true */
...@@ -92,7 +95,13 @@ const internalHandler: EdgeHandler = async (opts) => { ...@@ -92,7 +95,13 @@ const internalHandler: EdgeHandler = async (opts) => {
? relative(absoluteProjectDir, absoluteDistDir) ? relative(absoluteProjectDir, absoluteDistDir)
: '.next' : '.next'
await ensureInstrumentationRegistered(absoluteProjectDir, distDir) // When this request registers `instrumentation.ts`, the adapter emits its
// lifecycle spans inside the middleware span.
emitInstrumentationSpans =
await ensureInstrumentationRegisteredWithDeferredSpans(
absoluteProjectDir,
distDir
)
} }
return adapter({ return adapter({
...@@ -101,6 +110,8 @@ const internalHandler: EdgeHandler = async (opts) => { ...@@ -101,6 +110,8 @@ const internalHandler: EdgeHandler = async (opts) => {
incrementalCacheHandler, incrementalCacheHandler,
page, page,
handler: errorHandledHandler(handlerUserland), handler: errorHandledHandler(handlerUserland),
emitInstrumentationSpans,
requestStartTime,
}) })
} }
......
...@@ -70,13 +70,18 @@ export async function handler( ...@@ -70,13 +70,18 @@ export async function handler(
// Registering `instrumentation.ts` usually installs the tracer provider, so // Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before // do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace. // `prepare`, so route module preparation is part of the request's trace.
await routeModule.ensureInstrumentation(req) // When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest( return routeModule.traceRequest(
req, req,
{ {
tracer: getTracer(), tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`, spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget: req.url, httpTarget: req.url,
instrumentation,
}, },
(requestSpan, requestRootSpan) => (requestSpan, requestRootSpan) =>
handleRequest(req, res, ctx, srcPage, requestSpan, requestRootSpan) handleRequest(req, res, ctx, srcPage, requestSpan, requestRootSpan)
......
...@@ -19,16 +19,23 @@ type InstrumentationSpanTiming = { ...@@ -19,16 +19,23 @@ type InstrumentationSpanTiming = {
error?: unknown error?: unknown
} }
/** A lifecycle span of `instrumentation.ts`, recorded to be emitted later. */
type InstrumentationLifecycleSpan = {
type: InstrumentationSpan
spanName: string
timing: InstrumentationSpanTiming
}
function getInstrumentationTimestamp() { function getInstrumentationTimestamp() {
// Date.now() is treated as synchronous I/O while Cache Components render. // Date.now() is treated as synchronous I/O while Cache Components render.
return performance.timeOrigin + performance.now() return performance.timeOrigin + performance.now()
} }
function traceBackdatedInstrumentationSpan( function traceBackdatedInstrumentationSpan({
type: InstrumentationSpan, type,
spanName: string, spanName,
{ startTime, endTime, error }: InstrumentationSpanTiming timing: { startTime, endTime, error },
) { }: InstrumentationLifecycleSpan) {
getTracer().trace( getTracer().trace(
type, type,
{ spanName, startTime, endTime }, { spanName, startTime, endTime },
...@@ -47,16 +54,29 @@ function traceBackdatedInstrumentationSpan( ...@@ -47,16 +54,29 @@ function traceBackdatedInstrumentationSpan(
) )
} }
function traceInstrumentationModuleLoad() { /**
if (!instrumentationModuleLoadTiming) return * Takes the recorded timing of loading the instrumentation module, if not
* emitted yet, as a lifecycle span.
*/
function takeInstrumentationModuleLoadSpan():
| InstrumentationLifecycleSpan
| undefined {
if (!instrumentationModuleLoadTiming) return undefined
const timing = instrumentationModuleLoadTiming const timing = instrumentationModuleLoadTiming
instrumentationModuleLoadTiming = undefined instrumentationModuleLoadTiming = undefined
traceBackdatedInstrumentationSpan( return {
InstrumentationSpan.loadModule, type: InstrumentationSpan.loadModule,
'load instrumentation module', spanName: 'load instrumentation module',
timing timing,
) }
}
function traceInstrumentationModuleLoad() {
const span = takeInstrumentationModuleLoadSpan()
if (span) {
traceBackdatedInstrumentationSpan(span)
}
} }
export async function getInstrumentationModule( export async function getInstrumentationModule(
...@@ -102,7 +122,22 @@ export async function getInstrumentationModule( ...@@ -102,7 +122,22 @@ export async function getInstrumentationModule(
let instrumentationModulePromise: Promise<any> | null = null let instrumentationModulePromise: Promise<any> | null = null
async function registerInstrumentation(projectDir: string, distDir: string) { /**
* Loads `instrumentation.ts` and calls its `register()`.
*
* Its lifecycle spans (`load instrumentation module`, `register
* instrumentation`) are backdated and emitted afterwards, since `register()`
* commonly installs the tracer provider. They are emitted right away in the
* active context, unless `deferredSpans` is given: then successfully completed
* steps are added to it instead, for the caller to emit in another context
* (the request that triggered registration). Failed steps are always emitted
* right away.
*/
async function registerInstrumentation(
projectDir: string,
distDir: string,
deferredSpans?: InstrumentationLifecycleSpan[]
) {
// Ensure registerInstrumentation is not called in production build // Ensure registerInstrumentation is not called in production build
if (process.env.NEXT_PHASE === 'phase-production-build') { if (process.env.NEXT_PHASE === 'phase-production-build') {
return return
...@@ -121,9 +156,11 @@ async function registerInstrumentation(projectDir: string, distDir: string) { ...@@ -121,9 +156,11 @@ async function registerInstrumentation(projectDir: string, distDir: string) {
if (instrumentation?.register) { if (instrumentation?.register) {
const startTime = getInstrumentationTimestamp() const startTime = getInstrumentationTimestamp()
let error: unknown let error: unknown
let registered = false
try { try {
await instrumentation.register() await instrumentation.register()
extendInstrumentationAfterRegistration() extendInstrumentationAfterRegistration()
registered = true
} catch (err: any) { } catch (err: any) {
error = err error = err
err.message = `An error occurred while loading instrumentation hook: ${err.message}` err.message = `An error occurred while loading instrumentation hook: ${err.message}`
...@@ -133,15 +170,31 @@ async function registerInstrumentation(projectDir: string, distDir: string) { ...@@ -133,15 +170,31 @@ async function registerInstrumentation(projectDir: string, distDir: string) {
// register() commonly installs the provider, so emit both lifecycle // register() commonly installs the provider, so emit both lifecycle
// spans afterward with their original timestamps. // spans afterward with their original timestamps.
traceInstrumentationModuleLoad() const spans = [
traceBackdatedInstrumentationSpan( takeInstrumentationModuleLoadSpan(),
InstrumentationSpan.register, {
'register instrumentation', type: InstrumentationSpan.register,
{ startTime, endTime, error } spanName: 'register instrumentation',
) timing: { startTime, endTime, error },
},
].filter((span) => span !== undefined)
// Only defer after a successful registration: a failing request has no
// request span to emit them in.
if (deferredSpans && registered) {
deferredSpans.push(...spans)
} else {
spans.forEach(traceBackdatedInstrumentationSpan)
}
} }
} else { } else {
traceInstrumentationModuleLoad() const span = takeInstrumentationModuleLoadSpan()
if (span) {
if (deferredSpans) {
deferredSpans.push(span)
} else {
traceBackdatedInstrumentationSpan(span)
}
}
} }
} }
...@@ -172,3 +225,36 @@ export function ensureInstrumentationRegistered( ...@@ -172,3 +225,36 @@ export function ensureInstrumentationRegistered(
} }
return registerInstrumentationPromise return registerInstrumentationPromise
} }
/**
* Like `ensureInstrumentationRegistered`, for a request that may trigger
* registration (direct entrypoints, without next-server). When this call
* starts registration and it succeeds, the lifecycle spans aren't emitted yet:
* the returned function emits them in the context it's called in, so they can
* become children of the request's span. Returns `undefined` when
* registration was already started by another caller.
*/
export async function ensureInstrumentationRegisteredWithDeferredSpans(
projectDir: string,
distDir: string
): Promise<(() => void) | undefined> {
if (registerInstrumentationPromise) {
await registerInstrumentationPromise
return undefined
}
const deferredSpans: InstrumentationLifecycleSpan[] = []
registerInstrumentationPromise = registerInstrumentation(
projectDir,
distDir,
deferredSpans
)
await registerInstrumentationPromise
if (deferredSpans.length === 0) {
return undefined
}
return () => {
// Emit at most once.
deferredSpans.splice(0).forEach(traceBackdatedInstrumentationSpan)
}
}
import fs from 'node:fs'
import os from 'node:os'
import path from 'node:path'
const mockTraced: Array<{ type: string; spanName: string; error: boolean }> = []
jest.mock('../trace/tracer', () => ({
SpanStatusCode: { ERROR: 2 },
getTracer: () => ({
trace: (
type: string,
options: { spanName: string },
fn: (span: unknown) => void
) => {
let error = false
fn({
recordException: () => {},
setAttribute: () => {},
setStatus: () => {
error = true
},
})
mockTraced.push({ type, spanName: options.spanName, error })
},
}),
}))
jest.mock('./instrumentation-node-extensions', () => ({
afterRegistration: () => {},
}))
type InstrumentationGlobals =
typeof import('./instrumentation-globals.external')
const LOAD = 'Instrumentation.loadModule'
const REGISTER = 'Instrumentation.register'
let projectDir: string
function writeInstrumentation(source: string) {
const serverDir = path.join(projectDir, '.next', 'server')
fs.mkdirSync(serverDir, { recursive: true })
fs.writeFileSync(path.join(serverDir, 'instrumentation.js'), source)
}
function loadGlobals(): InstrumentationGlobals {
let globals: InstrumentationGlobals | undefined
jest.isolateModules(() => {
globals =
require('./instrumentation-globals.external') as typeof import('./instrumentation-globals.external')
})
return globals!
}
describe('instrumentation lifecycle spans', () => {
beforeEach(() => {
mockTraced.length = 0
projectDir = fs.mkdtempSync(path.join(os.tmpdir(), 'next-instrumentation-'))
})
afterEach(() => {
fs.rmSync(projectDir, { recursive: true, force: true })
})
it('emits them right away for startup registration', async () => {
writeInstrumentation('exports.register = () => {}')
const globals = loadGlobals()
await globals.ensureInstrumentationRegistered(projectDir, '.next')
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD, REGISTER])
})
it('defers them to the request that triggered registration, once', async () => {
writeInstrumentation('exports.register = () => {}')
const globals = loadGlobals()
const [emit, concurrentEmit] = await Promise.all([
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
),
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
),
])
// Only the initiating request gets them, and nothing was emitted yet.
expect(concurrentEmit).toBeUndefined()
expect(mockTraced).toEqual([])
emit!()
emit!()
expect(mockTraced).toEqual([
{
type: LOAD,
spanName: 'load instrumentation module',
error: false,
},
{ type: REGISTER, spanName: 'register instrumentation', error: false },
])
// Later requests don't emit them again.
expect(
await globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).toBeUndefined()
})
it('does not defer them when a request awaits startup registration', async () => {
writeInstrumentation('exports.register = () => {}')
const globals = loadGlobals()
const startup = globals.ensureInstrumentationRegistered(projectDir, '.next')
const emit = await globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
await startup
expect(emit).toBeUndefined()
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD, REGISTER])
})
it('defers the module load span without a register export', async () => {
writeInstrumentation('exports.onRequestError = () => {}')
const globals = loadGlobals()
const emit = await globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
expect(mockTraced).toEqual([])
emit!()
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD])
})
it('emits them right away when register() throws', async () => {
writeInstrumentation(
'exports.register = () => { throw new Error("register failed") }'
)
const globals = loadGlobals()
await expect(
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).rejects.toThrow(
'An error occurred while loading instrumentation hook: register failed'
)
// The load succeeded; only the register span has an error status.
expect(mockTraced).toEqual([
{
type: LOAD,
spanName: 'load instrumentation module',
error: false,
},
{ type: REGISTER, spanName: 'register instrumentation', error: true },
])
})
it('emits them right away when register() throws a non-error value', async () => {
writeInstrumentation('exports.register = () => { throw undefined }')
const globals = loadGlobals()
await expect(
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).rejects.toThrow()
// Nothing is left waiting for a request span that won't be created.
expect(mockTraced.map(({ type }) => type)).toEqual([LOAD, REGISTER])
})
it('emits the module load span right away when loading fails', async () => {
writeInstrumentation('throw new Error("load failed")')
const globals = loadGlobals()
await expect(
globals.ensureInstrumentationRegisteredWithDeferredSpans(
projectDir,
'.next'
)
).rejects.toThrow('load failed')
expect(mockTraced).toEqual([
{
type: LOAD,
spanName: 'load instrumentation module',
error: true,
},
])
})
})
...@@ -103,13 +103,18 @@ export const getHandler = ({ ...@@ -103,13 +103,18 @@ export const getHandler = ({
// Registering `instrumentation.ts` usually installs the tracer provider, so // Registering `instrumentation.ts` usually installs the tracer provider, so
// do it before starting the request span. The span is started before // do it before starting the request span. The span is started before
// `prepare`, so route module preparation is part of the request's trace. // `prepare`, so route module preparation is part of the request's trace.
await routeModule.ensureInstrumentation(req) // When this request triggers registration, the lifecycle spans of
// `instrumentation.ts` are emitted inside the request span.
const { instrumentation } = await routeModule.ensureInstrumentation(req, {
deferSpans: true,
})
return routeModule.traceRequest( return routeModule.traceRequest(
req, req,
{ {
tracer: getTracer(), tracer: getTracer(),
spanName: `${req.method || 'GET'} ${srcPage}`, spanName: `${req.method || 'GET'} ${srcPage}`,
httpTarget: req.url, httpTarget: req.url,
instrumentation,
}, },
(requestSpan, requestRootSpan) => (requestSpan, requestRootSpan) =>
handleRequest( handleRequest(
......
...@@ -96,6 +96,26 @@ export interface RouteModuleOptions< ...@@ -96,6 +96,26 @@ export interface RouteModuleOptions<
readonly relativeProjectDir: string readonly relativeProjectDir: string
} }
/**
* Returned by `RouteModule.ensureInstrumentation` with `deferSpans`, and passed
* to `RouteModule.traceRequest`.
*
* @internal
*/
export interface RequestInstrumentation {
/**
* When the request started, before registering `instrumentation.ts` (same
* clock as the instrumentation lifecycle spans). The request span is
* backdated to it.
*/
startTime: number
/**
* Emits the lifecycle spans of `instrumentation.ts`. Only set for the request
* that triggered registration; called inside the request span.
*/
emitSpans?: () => void
}
/** /**
* RouteHandlerContext is the base context for a route handler. * RouteHandlerContext is the base context for a route handler.
*/ */
...@@ -644,7 +664,10 @@ export abstract class RouteModule< ...@@ -644,7 +664,10 @@ export abstract class RouteModule<
* *
* Callers must await `ensureInstrumentation` before calling this: on the * Callers must await `ensureInstrumentation` before calling this: on the
* first request, registering `instrumentation.ts` usually installs the * first request, registering `instrumentation.ts` usually installs the
* tracer provider, and a span started earlier would not be recorded. * tracer provider, and a span started earlier would not be recorded. With
* the `instrumentation` it returned, the request span is backdated to cover
* registration, and the lifecycle spans of `instrumentation.ts` are children
* of the request span when this request triggered registration.
* *
* `fn` receives the request span and the request's root span: the span that * `fn` receives the request span and the request's root span: the span that
* was active when the handler was invoked (next-server's request span, or a * was active when the handler was invoked (next-server's request span, or a
...@@ -664,10 +687,12 @@ export abstract class RouteModule< ...@@ -664,10 +687,12 @@ export abstract class RouteModule<
tracer, tracer,
spanName, spanName,
httpTarget, httpTarget,
instrumentation,
}: { }: {
tracer: ReturnType<typeof getTracer> tracer: ReturnType<typeof getTracer>
spanName: string spanName: string
httpTarget: string | undefined httpTarget: string | undefined
instrumentation: RequestInstrumentation | undefined
}, },
fn: (span: Span | undefined, rootSpan: Span | undefined) => Promise<T> fn: (span: Span | undefined, rootSpan: Span | undefined) => Promise<T>
): Promise<T> { ): Promise<T> {
...@@ -690,12 +715,16 @@ export abstract class RouteModule< ...@@ -690,12 +715,16 @@ export abstract class RouteModule<
{ {
spanName, spanName,
kind: SpanKind.SERVER, kind: SpanKind.SERVER,
startTime: instrumentation?.startTime,
attributes: { attributes: {
'http.method': req.method || 'GET', 'http.method': req.method || 'GET',
'http.target': httpTarget, 'http.target': httpTarget,
}, },
}, },
(span) => fn(span, activeSpan ?? span) (span) => {
instrumentation?.emitSpans?.()
return fn(span, activeSpan ?? span)
}
), ),
undefined, undefined,
!isWrappedByNextServer !isWrappedByNextServer
...@@ -723,13 +752,25 @@ export abstract class RouteModule< ...@@ -723,13 +752,25 @@ export abstract class RouteModule<
* directory (`undefined` in the edge runtime, which handles loading * directory (`undefined` in the edge runtime, which handles loading
* instrumentation at the edge adapter level). * instrumentation at the edge adapter level).
* *
* With `deferSpans` (callers of `traceRequest`), it also returns the
* `instrumentation` to pass to `traceRequest`: when this call triggered
* registration, the lifecycle spans of `instrumentation.ts` are not emitted
* yet but emitted inside the request span.
*
* @internal - called by `prepare` and by the Next.js build templates before * @internal - called by `prepare` and by the Next.js build templates before
* `traceRequest`. * `traceRequest`.
*/ */
public async ensureInstrumentation( public async ensureInstrumentation(
req: IncomingMessage | BaseNextRequest req: IncomingMessage | BaseNextRequest,
): Promise<string | undefined> { { deferSpans = false }: { deferSpans?: boolean } = {}
): Promise<{
absoluteProjectDir: string | undefined
instrumentation: RequestInstrumentation | undefined
}> {
let absoluteProjectDir: string | undefined let absoluteProjectDir: string | undefined
const instrumentation: RequestInstrumentation | undefined = deferSpans
? { startTime: performance.timeOrigin + performance.now() }
: undefined
if (process.env.NEXT_RUNTIME !== 'edge') { if (process.env.NEXT_RUNTIME !== 'edge') {
const { join, relative } = const { join, relative } =
...@@ -746,7 +787,10 @@ export abstract class RouteModule< ...@@ -746,7 +787,10 @@ export abstract class RouteModule<
if (absoluteDistDir) { if (absoluteDistDir) {
this.distDir = relative(absoluteProjectDir, absoluteDistDir) this.distDir = relative(absoluteProjectDir, absoluteDistDir)
} }
const { ensureInstrumentationRegistered } = await import( const {
ensureInstrumentationRegistered,
ensureInstrumentationRegisteredWithDeferredSpans,
} = await import(
'../lib/router-utils/instrumentation-globals.external.js' '../lib/router-utils/instrumentation-globals.external.js'
) )
// ensure instrumentation is registered and pass // ensure instrumentation is registered and pass
...@@ -754,10 +798,18 @@ export abstract class RouteModule< ...@@ -754,10 +798,18 @@ export abstract class RouteModule<
// that bypasses `BaseServer.handleRequest` (where this is also awaited // that bypasses `BaseServer.handleRequest` (where this is also awaited
// via `prepareImpl`) still observes the instrumentation hook completing // via `prepareImpl`) still observes the instrumentation hook completing
// before the userland route handler runs. // before the userland route handler runs.
await ensureInstrumentationRegistered(absoluteProjectDir, this.distDir) if (instrumentation) {
instrumentation.emitSpans =
await ensureInstrumentationRegisteredWithDeferredSpans(
absoluteProjectDir,
this.distDir
)
} else {
await ensureInstrumentationRegistered(absoluteProjectDir, this.distDir)
}
} }
return absoluteProjectDir return { absoluteProjectDir, instrumentation }
} }
private async prepareImpl( private async prepareImpl(
...@@ -817,7 +869,7 @@ export abstract class RouteModule< ...@@ -817,7 +869,7 @@ export abstract class RouteModule<
if (process.env.NEXT_RUNTIME !== 'edge' && res) { if (process.env.NEXT_RUNTIME !== 'edge' && res) {
patchSetHeaderWithCookieSupport(req, res) patchSetHeaderWithCookieSupport(req, res)
} }
const absoluteProjectDir = await this.ensureInstrumentation(req) const { absoluteProjectDir } = await this.ensureInstrumentation(req)
const manifests = getTracer().trace( const manifests = getTracer().trace(
RouteModuleSpan.loadManifests, RouteModuleSpan.loadManifests,
{ spanName: 'load route manifests' }, { spanName: 'load route manifests' },
......
...@@ -101,6 +101,16 @@ export type AdapterOptions = { ...@@ -101,6 +101,16 @@ export type AdapterOptions = {
IncrementalCache?: typeof import('../lib/incremental-cache').IncrementalCache IncrementalCache?: typeof import('../lib/incremental-cache').IncrementalCache
incrementalCacheHandler?: typeof import('../lib/incremental-cache').CacheHandler incrementalCacheHandler?: typeof import('../lib/incremental-cache').CacheHandler
bypassNextUrl?: boolean bypassNextUrl?: boolean
/**
* When the Node.js middleware entrypoint registered `instrumentation.ts` for
* this request: emits its lifecycle spans, inside the middleware span.
*/
emitInstrumentationSpans?: () => void
/**
* When the request was received (epoch milliseconds), if before the
* adapter ran (e.g. while registering `instrumentation.ts`).
*/
requestStartTime?: number
} }
// This has to be compatible with what the Vercel builder does as well: // This has to be compatible with what the Vercel builder does as well:
...@@ -311,7 +321,9 @@ export async function adapter( ...@@ -311,7 +321,9 @@ export async function adapter(
// siblings: they share the currently active context as their parent. // siblings: they share the currently active context as their parent.
// That is an active host span, otherwise the request's `traceparent` // That is an active host span, otherwise the request's `traceparent`
// (incoming, or written by next-server's middleware-invoke pass), or // (incoming, or written by next-server's middleware-invoke pass), or
// the root span inserted below. // the root span inserted below. When this request registered
// `instrumentation.ts`, its lifecycle spans are children of the
// middleware span, which is backdated to cover registration.
const receivedHeaders = new Headers(request.headers) const receivedHeaders = new Headers(request.headers)
forwardedTraceContext = getSharedParentTraceContext( forwardedTraceContext = getSharedParentTraceContext(
receivedHeaders, receivedHeaders,
...@@ -326,12 +338,16 @@ export async function adapter( ...@@ -326,12 +338,16 @@ export async function adapter(
MiddlewareSpan.execute, MiddlewareSpan.execute,
{ {
spanName: `middleware ${request.method}`, spanName: `middleware ${request.method}`,
startTime: params.emitInstrumentationSpans
? params.requestStartTime
: undefined,
attributes: { attributes: {
'http.target': request.nextUrl.pathname, 'http.target': request.nextUrl.pathname,
'http.method': request.method, 'http.method': request.method,
}, },
}, },
async () => { async () => {
params.emitInstrumentationSpans?.()
try { try {
const onUpdateCookies = (cookies: Array<string>) => { const onUpdateCookies = (cookies: Array<string>) => {
cookiesFromResponse = cookies cookiesFromResponse = cookies
...@@ -421,12 +437,14 @@ export async function adapter( ...@@ -421,12 +437,14 @@ export async function adapter(
// middleware and the render are siblings in one trace instead of two // middleware and the render are siblings in one trace instead of two
// independent root spans. It's named after the method only (like // independent root spans. It's named after the method only (like
// next-server's request span before its route is known): the path // next-server's request span before its route is known): the path
// would make the span name high-cardinality. // would make the span name high-cardinality. It is backdated to when
// the request was received, to cover registering `instrumentation.ts`.
return tracer.trace( return tracer.trace(
NextSpan.request, NextSpan.request,
{ {
spanName: request.method, spanName: request.method,
kind: SpanKind.SERVER, kind: SpanKind.SERVER,
startTime: params.requestStartTime,
attributes: { attributes: {
'http.target': request.nextUrl.pathname, 'http.target': request.nextUrl.pathname,
'http.method': request.method, 'http.method': request.method,
......
import { connection } from 'next/server'
import { getInstrumentationLifecycleSpans } from '../../trace-recorder'
export async function GET() {
// Run at request time, in the instance serving this request.
await connection()
return Response.json({ spans: getInstrumentationLifecycleSpans() })
}
...@@ -147,25 +147,68 @@ describe.each(VARIANTS)( ...@@ -147,25 +147,68 @@ describe.each(VARIANTS)(
expectNoInternalHeaders(res) expectNoInternalHeaders(res)
const tree = await getSpanTree(res, 'html', { includeOneTimeSpans: true }) const tree = await getSpanTree(res, 'html', { includeOneTimeSpans: true })
if (isNextDeploy) { if (isNextDeploy) {
// Deployed, the render and Node.js middleware functions each register
// instrumentation in their first request: under the render's request
// span and under the middleware span.
if (renderMode === 'default') { if (renderMode === 'default') {
if (isNodeMiddleware) {
expect(tree).toMatchInlineSnapshot(`
"[trace A] GET (Next.request) {http.target=/page}
├─ GET /page/page (BaseServer.handleRequest) {http.target=/page}
│ ├─ load instrumentation module (Instrumentation.loadModule)
│ ├─ prepare route module (RouteModule.prepare)
│ ├─ register instrumentation (Instrumentation.register)
│ └─ render route (app) /page (AppRender.getBodyResult) {next.route=/page}
│ ├─ build component tree (NextNodeServer.createComponentTree)
│ │ ├─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
│ │ └─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
│ ├─ generateMetadata /layout (ResolveMetadata.generateMetadata)
│ └─ generateMetadata /page/page (ResolveMetadata.generateMetadata)
└─ middleware GET (Middleware.execute) {http.target=/page}
├─ load instrumentation module (Instrumentation.loadModule)
└─ register instrumentation (Instrumentation.register)"
`)
} else {
expect(tree).toMatchInlineSnapshot(`
"[trace A] GET (Next.request) {http.target=/page}
├─ GET /page/page (BaseServer.handleRequest) {http.target=/page}
│ ├─ load instrumentation module (Instrumentation.loadModule)
│ ├─ prepare route module (RouteModule.prepare)
│ ├─ register instrumentation (Instrumentation.register)
│ └─ render route (app) /page (AppRender.getBodyResult) {next.route=/page}
│ ├─ build component tree (NextNodeServer.createComponentTree)
│ │ ├─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
│ │ └─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
│ ├─ generateMetadata /layout (ResolveMetadata.generateMetadata)
│ └─ generateMetadata /page/page (ResolveMetadata.generateMetadata)
└─ middleware GET (Middleware.execute) {http.target=/page}"
`)
}
} else if (isNodeMiddleware) {
expect(tree).toMatchInlineSnapshot(` expect(tree).toMatchInlineSnapshot(`
"[trace A] GET (Next.request) {http.target=/page} "[trace A] GET (Next.request) {http.target=/page}
├─ GET /page/page (BaseServer.handleRequest) {http.target=/page} ├─ middleware GET (Middleware.execute) {http.target=/page}
│ ├─ prepare route module (RouteModule.prepare) │ ├─ load instrumentation module (Instrumentation.loadModule)
│ └─ render route (app) /page (AppRender.getBodyResult) {next.route=/page} │ └─ register instrumentation (Instrumentation.register)
│ ├─ build component tree (NextNodeServer.createComponentTree) └─ POST /page/page (BaseServer.handleRequest) {http.target=/page}
│ │ ├─ resolve segment modules (NextNodeServer.getLayoutOrPageModule) ├─ load instrumentation module (Instrumentation.loadModule)
│ │ └─ resolve segment modules (NextNodeServer.getLayoutOrPageModule) ├─ prepare route module (RouteModule.prepare)
│ ├─ generateMetadata /layout (ResolveMetadata.generateMetadata) ├─ register instrumentation (Instrumentation.register)
│ └─ generateMetadata /page/page (ResolveMetadata.generateMetadata) └─ render route (app) /page (AppRender.getBodyResult) {next.route=/page}
└─ middleware GET (Middleware.execute) {http.target=/page}" ├─ build component tree (NextNodeServer.createComponentTree)
│ ├─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
│ └─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
├─ generateMetadata /layout (ResolveMetadata.generateMetadata)
└─ generateMetadata /page/page (ResolveMetadata.generateMetadata)"
`) `)
} else { } else {
expect(tree).toMatchInlineSnapshot(` expect(tree).toMatchInlineSnapshot(`
"[trace A] GET (Next.request) {http.target=/page} "[trace A] GET (Next.request) {http.target=/page}
├─ middleware GET (Middleware.execute) {http.target=/page} ├─ middleware GET (Middleware.execute) {http.target=/page}
└─ POST /page/page (BaseServer.handleRequest) {http.target=/page} └─ POST /page/page (BaseServer.handleRequest) {http.target=/page}
├─ load instrumentation module (Instrumentation.loadModule)
├─ prepare route module (RouteModule.prepare) ├─ prepare route module (RouteModule.prepare)
├─ register instrumentation (Instrumentation.register)
└─ render route (app) /page (AppRender.getBodyResult) {next.route=/page} └─ render route (app) /page (AppRender.getBodyResult) {next.route=/page}
├─ build component tree (NextNodeServer.createComponentTree) ├─ build component tree (NextNodeServer.createComponentTree)
│ ├─ resolve segment modules (NextNodeServer.getLayoutOrPageModule) │ ├─ resolve segment modules (NextNodeServer.getLayoutOrPageModule)
...@@ -890,6 +933,38 @@ describe.each(VARIANTS)( ...@@ -890,6 +933,38 @@ describe.each(VARIANTS)(
`) `)
} }
}) })
// The `instrumentation.ts` lifecycle spans (`load instrumentation module`,
// `register instrumentation`) of the instance serving the request, with
// their ancestors.
it('traces instrumentation registration in the triggering request', async () => {
const res = await next.fetch('/instrumentation-spans')
expect(res.status).toBe(200)
const tree = printSpanTree((await res.json()).spans, {
known: [INCOMING, SET_BY_MIDDLEWARE],
})
if (isNextDeploy) {
// Deployed functions register instrumentation lazily, in the first
// request they handle: the lifecycle spans are children of that
// request's span (a route or page entrypoint) or of the middleware
// span (a Node.js middleware entrypoint). The function serving this
// route handler also serves `/route-handler`, which an earlier test
// requested first.
expect(tree).toMatchInlineSnapshot(`
"[trace A] [remote]
└─ GET /route-handler (BaseServer.handleRequest) {http.target=/route-handler, http.route=/route-handler, next.route=/route-handler, http.status_code=200}
├─ load instrumentation module (Instrumentation.loadModule)
└─ register instrumentation (Instrumentation.register)"
`)
} else {
// `next start` / `next dev` register instrumentation at startup,
// outside of any request, so the lifecycle spans are roots.
expect(tree).toMatchInlineSnapshot(`
"[trace A] load instrumentation module (Instrumentation.loadModule)
[trace B] register instrumentation (Instrumentation.register)"
`)
}
})
} }
) )
......
...@@ -128,6 +128,24 @@ export function getMiddlewareSpans() { ...@@ -128,6 +128,24 @@ export function getMiddlewareSpans() {
] ]
} }
/**
* The `instrumentation.ts` lifecycle spans (`Instrumentation.*`) started in
* this process, with their ancestors as far as they were started here.
*/
export function getInstrumentationLifecycleSpans() {
const spans = new Map<string, RecordedSpan>()
for (const [id, { span }] of getRecordedSpans()) {
if (
String(span.attributes['next.span_type']).startsWith('Instrumentation.')
) {
for (const recorded of getChain(id)) {
spans.set(recorded.id, recorded)
}
}
}
return [...spans.values()]
}
/** /**
* The render's spans: its request span (the nearest `BaseServer.handleRequest` * The render's spans: its request span (the nearest `BaseServer.handleRequest`
* ancestor of the active span), that span's ancestors, and the request span's * ancestor of the active span), that span's ancestors, and the request span's
......
...@@ -2042,11 +2042,20 @@ describe.each( ...@@ -2042,11 +2042,20 @@ describe.each(
if (isNextStart) { if (isNextStart) {
describe.each([ describe.each([
{ name: 'without parent span', withParentSpan: false }, {
{ name: 'with parent span', withParentSpan: true }, name: 'without parent span',
withParentSpan: false,
coldTraceparent: false,
},
{
name: 'without parent span, cold request with traceparent',
withParentSpan: false,
coldTraceparent: true,
},
{ name: 'with parent span', withParentSpan: true, coldTraceparent: false },
])( ])(
'opentelemetry route module preparation with direct entrypoint handler $name', 'opentelemetry route module preparation with direct entrypoint handler $name',
({ withParentSpan }) => { ({ withParentSpan, coldTraceparent }) => {
let collector: Collector | undefined let collector: Collector | undefined
const { next, skipped } = nextTestSetup({ const { next, skipped } = nextTestSetup({
files: __dirname, files: __dirname,
...@@ -2098,7 +2107,7 @@ if (isNextStart) { ...@@ -2098,7 +2107,7 @@ if (isNextStart) {
[] []
for (const [index, pathname] of pathnames.entries()) { for (const [index, pathname] of pathnames.entries()) {
for (const variant of [ for (const variant of [
{ traceparent: false }, { traceparent: index === 0 && coldTraceparent },
{ traceparent: false }, { traceparent: false },
...(withParentSpan ? [] : [{ traceparent: true }]), ...(withParentSpan ? [] : [{ traceparent: true }]),
]) { ]) {
...@@ -2189,6 +2198,46 @@ if (isNextStart) { ...@@ -2189,6 +2198,46 @@ if (isNextStart) {
expect(requestSpan.parentId).toBeUndefined() expect(requestSpan.parentId).toBeUndefined()
} }
} }
// The cold request registered instrumentation, so the lifecycle
// spans of `instrumentation.ts` are children of its request span
// instead of separate traces.
const coldRequest = requests.find((request) => request.cold)!
const coldRequestSpan = spans.find(
(span) =>
span.attributes?.['next.span_type'] ===
'BaseServer.handleRequest' &&
String(span.attributes?.['http.target'] ?? '').includes(
`id=${coldRequest.id}`
)
)!
expect(coldRequest.traceparent).toBe(coldTraceparent)
for (const [type, name] of [
['Instrumentation.loadModule', 'load instrumentation module'],
['Instrumentation.register', 'register instrumentation'],
]) {
const lifecycleSpans = spans.filter(
(span) => span.attributes?.['next.span_type'] === type
)
expect(lifecycleSpans).toEqual([
expect.objectContaining({
...(withParentSpan ? {} : { runtime: 'nodejs' }),
name,
parentId: coldRequestSpan.id,
traceId: coldRequestSpan.traceId,
attributes: {
'next.span_category': 'nextjs',
'next.span_name': name,
'next.span_type': type,
},
status: { code: 0 },
}),
])
// The request span is backdated to cover registration.
expect(coldRequestSpan.timestamp!).toBeLessThanOrEqual(
lifecycleSpans[0].timestamp!
)
}
}) })
}) })
} }
...@@ -2340,14 +2389,26 @@ describe.each( ...@@ -2340,14 +2389,26 @@ describe.each(
useDirectEntrypointHandler: true, useDirectEntrypointHandler: true,
withParentSpan: true, withParentSpan: true,
}, },
// Only the first request, with an incoming `traceparent`, in a fresh
// process.
isNextStart && {
name: 'direct entrypoints, cold request with traceparent',
useDirectEntrypointHandler: true,
coldTraceparent: true,
},
].filter(Boolean) as { ].filter(Boolean) as {
name: string name: string
useDirectEntrypointHandler: boolean useDirectEntrypointHandler: boolean
withParentSpan?: boolean withParentSpan?: boolean
coldTraceparent?: boolean
}[] }[]
)( )(
'opentelemetry - middleware $name', 'opentelemetry - middleware $name',
({ useDirectEntrypointHandler, withParentSpan = false }) => { ({
useDirectEntrypointHandler,
withParentSpan = false,
coldTraceparent = false,
}) => {
describe.each(['edge', 'nodejs'])('%s runtime', (runtime) => { describe.each(['edge', 'nodejs'])('%s runtime', (runtime) => {
const { const {
next: { next, skipped }, next: { next, skipped },
...@@ -2367,6 +2428,82 @@ describe.each( ...@@ -2367,6 +2428,82 @@ describe.each(
return return
} }
if (useDirectEntrypointHandler) {
// Runs first, in a fresh process: the first request reaches the
// Node.js middleware, which registers `instrumentation.ts`.
it('should trace instrumentation registration in the first middleware request', async () => {
const res = await next.fetch(
`/behind-middleware?id=${Math.random().toString(36).slice(2)}`,
{
headers: coldTraceparent
? {
traceparent: `00-${EXTERNAL.traceId}-${EXTERNAL.spanId}-01`,
}
: {},
}
)
expect(res.status).toBe(200)
await retry(async () => {
const spans = getCollector().getAllSpans()
const byId = new Map(spans.map((span) => [span.id, span]))
const [middlewareSpan] = spans.filter(
(span) =>
span.attributes?.['next.span_type'] === 'Middleware.execute'
)
expect(middlewareSpan).toBeDefined()
// The shared parent of the middleware span: the incoming
// `traceparent` (not exported here), the host span, or, without
// either, the root inserted by the middleware adapter, backdated
// to cover registration. The middleware triggered registration, so
// the lifecycle spans are its children.
const parent = byId.get(middlewareSpan.parentId!)
if (coldTraceparent) {
expect(parent).toBeUndefined()
expect(middlewareSpan).toMatchObject({
parentId: EXTERNAL.spanId,
traceId: EXTERNAL.traceId,
})
} else {
expect(parent).toBeDefined()
expect(parent!.attributes?.['next.span_type']).toBe(
withParentSpan ? undefined : 'Next.request'
)
}
for (const [type, name] of [
['Instrumentation.loadModule', 'load instrumentation module'],
['Instrumentation.register', 'register instrumentation'],
]) {
const lifecycleSpans = spans.filter(
(span) => span.attributes?.['next.span_type'] === type
)
expect(lifecycleSpans).toEqual([
expect.objectContaining({
name,
parentId: middlewareSpan.id,
traceId: middlewareSpan.traceId,
status: { code: 0 },
}),
])
// The middleware span is backdated to cover registration.
expect(middlewareSpan.timestamp!).toBeLessThanOrEqual(
lifecycleSpans[0].timestamp!
)
if (parent) {
expect(parent.timestamp!).toBeLessThanOrEqual(
middlewareSpan.timestamp!
)
}
}
})
})
}
if (coldTraceparent) {
return
}
for (const env of [ for (const env of [
{ {
name: 'root context', name: 'root context',
...@@ -2611,6 +2748,22 @@ describe.each( ...@@ -2611,6 +2748,22 @@ describe.each(
expect(loadModuleSpan?.timestamp).toBeLessThanOrEqual( expect(loadModuleSpan?.timestamp).toBeLessThanOrEqual(
registerSpan!.timestamp! registerSpan!.timestamp!
) )
if (useDirectEntrypointHandler) {
// Direct entrypoints register instrumentation in the first request,
// so the lifecycle spans are children of its request span.
for (const span of [loadModuleSpan, registerSpan]) {
const parent = spans.find(({ id }) => id === span?.parentId)
expect(parent?.attributes?.['next.span_type']).toBe(
'BaseServer.handleRequest'
)
expect(span?.traceId).toBe(parent!.traceId)
}
} else {
// `next start` registers instrumentation at startup, outside of any
// request, so the lifecycle spans are roots.
expect(loadModuleSpan?.parentId).toBeUndefined()
expect(registerSpan?.parentId).toBeUndefined()
}
}) })
}) })
} }
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment