Unverified Commit 52284fed authored by Jimmy Lai's avatar Jimmy Lai Committed by GitHub

misc: remove vendored node-fetch usages (#75741)

## What

This PR replaces the usage of node-fetch in the next/font/google loader
code, which runs as part of every next build. `node-fetch` relies on
`punycode`, which shows a deprecration warning from Node.js 22,
affecting every users on Next.js and Node 22.

This PR also removes any other references to the vendored node-fetch
used in other parts of the codebase.

~~Unfortunately, we can't remove it because other next.js dependencies
like the amp optimiser actually use node-fetch, so they would need to be
updated separately.~~

not a problem anymore because of @wowczarczyk suggestion of also
rewriting the external in the dependencies to point to the correct one

This PR does not remove the node-fetch usage in tests.

## How

This replaces the node-fetch call with a call to the native fetch
exposed in Node.js. However, this fetch does not support proxies without
tapping into Undici, so I've had to "polyfill" a usage because I did not
want to add a dependency to Undici for a minor use case.

<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:

## For Contributors

### Improving Documentation

- Run `pnpm prettier-fix` to fix formatting issues before opening the
PR.
- Read the Docs Contribution Guide to ensure your contribution follows
the docs guidelines:
https://nextjs.org/docs/community/contribution-guide

### Adding or Updating Examples

- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md

### Fixing a bug

- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md

### Adding a feature

- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md


## For Maintainers

- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Closes NEXT-


-->
Fixes #66289
Fixes #75313
Closes NDX-730

---------
Co-authored-by: 's avatarJJ Kasper <jj@jjsweb.site>
parent b76775a6
// @ts-ignore
import fetch from 'next/dist/compiled/node-fetch'
import { nextFontError } from '../next-font-error' import { nextFontError } from '../next-font-error'
import { getProxyAgent } from './get-proxy-agent' import { fetchResource } from './fetch-resource'
import { retry } from './retry' import { retry } from './retry'
/** /**
...@@ -16,56 +14,23 @@ export async function fetchCSSFromGoogleFonts( ...@@ -16,56 +14,23 @@ export async function fetchCSSFromGoogleFonts(
fontFamily: string, fontFamily: string,
isDev: boolean isDev: boolean
): Promise<string> { ): Promise<string> {
// Check if mocked responses are defined, if so use them instead of fetching from Google Fonts
let mockedResponse: string | undefined
if (process.env.NEXT_FONT_GOOGLE_MOCKED_RESPONSES) { if (process.env.NEXT_FONT_GOOGLE_MOCKED_RESPONSES) {
const mockFile = require(process.env.NEXT_FONT_GOOGLE_MOCKED_RESPONSES) const mockFile = require(process.env.NEXT_FONT_GOOGLE_MOCKED_RESPONSES)
mockedResponse = mockFile[url] const mockedResponse = mockFile[url]
if (!mockedResponse) { if (!mockedResponse) {
nextFontError('Missing mocked response for URL: ' + url) nextFontError('Missing mocked response for URL: ' + url)
} }
return mockedResponse
} }
let cssResponse: string const buffer = await retry(async () => {
if (mockedResponse) { return fetchResource(
// Just use the mocked CSS if it's set url,
cssResponse = mockedResponse isDev,
} else { `Failed to fetch font \`${fontFamily}\`: ${url}\n` +
// Retry the fetch a few times in case of network issues as some font files `Please check your network connection.`
// are quite large: )
// https://github.com/vercel/next.js/issues/45080 }, 3)
cssResponse = await retry(async () => {
const controller =
isDev && typeof AbortController !== 'undefined'
? new AbortController()
: undefined
const signal = controller?.signal
const timeoutId = controller
? setTimeout(() => controller.abort(), 3000)
: undefined
const res = await fetch(url, { return buffer.toString('utf8')
agent: getProxyAgent(),
// Add a timeout in dev
signal,
headers: {
// The file format is based off of the user agent, make sure woff2 files are fetched
'user-agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36',
},
}).finally(() => {
timeoutId && clearTimeout(timeoutId)
})
if (!res.ok) {
nextFontError(
`Failed to fetch font \`${fontFamily}\`.\nURL: ${url}\n\nPlease check if the network is available.`
)
}
return res.text()
}, 3)
}
return cssResponse
} }
// @ts-ignore import fs from 'node:fs'
import fetch from 'next/dist/compiled/node-fetch'
import { getProxyAgent } from './get-proxy-agent'
import { retry } from './retry' import { retry } from './retry'
import { fetchResource } from './fetch-resource'
/** /**
* Fetch the url and return a buffer with the font file. * Fetches a font file and returns its contents as a Buffer.
* If NEXT_FONT_GOOGLE_MOCKED_RESPONSES is set, we handle mock data logic.
*/ */
export async function fetchFontFile(url: string, isDev: boolean) { export async function fetchFontFile(url: string, isDev: boolean) {
// Check if we're using mocked data
if (process.env.NEXT_FONT_GOOGLE_MOCKED_RESPONSES) { if (process.env.NEXT_FONT_GOOGLE_MOCKED_RESPONSES) {
// If it's an absolute path, read the file from the filesystem
if (url.startsWith('/')) { if (url.startsWith('/')) {
return require('fs').readFileSync(url) return fs.readFileSync(url)
} }
// Otherwise just return a unique buffer
return Buffer.from(url) return Buffer.from(url)
} }
return await retry(async () => { return await retry(async () => {
const controller = new AbortController() return fetchResource(
const timeoutId = setTimeout(() => controller.abort(), 3000) url,
const arrayBuffer = await fetch(url, { isDev,
agent: getProxyAgent(), `Failed to fetch font file from \`${url}\`.`
// Add a timeout in dev )
signal: isDev ? controller.signal : undefined,
})
.then((r: any) => r.arrayBuffer())
.finally(() => {
clearTimeout(timeoutId)
})
return Buffer.from(arrayBuffer)
}, 3) }, 3)
} }
import http from 'node:http'
import https from 'node:https'
import { getProxyAgent } from './get-proxy-agent'
/**
* Makes a simple GET request and returns the entire response as a Buffer.
* - Throws if the response status is not 200.
* - Applies a 3000 ms timeout when `isDev` is `true`.
*/
export function fetchResource(
url: string,
isDev: boolean,
errorMessage?: string
): Promise<Buffer> {
return new Promise((resolve, reject) => {
const { protocol } = new URL(url)
const client = protocol === 'https:' ? https : http
const timeout = isDev ? 3000 : undefined
const req = client.request(
url,
{
agent: getProxyAgent(),
headers: {
// The file format is based off of the user agent, make sure woff2 files are fetched
'User-Agent':
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' +
'AppleWebKit/537.36 (KHTML, like Gecko) ' +
'Chrome/104.0.0.0 Safari/537.36',
},
},
(res) => {
if (res.statusCode !== 200) {
reject(
new Error(
errorMessage ||
`Request failed: ${url} (status: ${res.statusCode})`
)
)
return
}
const chunks: Buffer[] = []
res.on('data', (chunk) => chunks.push(Buffer.from(chunk)))
res.on('end', () => resolve(Buffer.concat(chunks)))
}
)
if (timeout) {
req.setTimeout(timeout, () => {
req.destroy(new Error(`Request timed out after ${timeout}ms`))
})
}
req.on('error', (err) => reject(err))
req.end()
})
}
import nextFontGoogleFontLoader from './loader' import nextFontGoogleFontLoader from './loader'
// @ts-ignore import { fetchResource } from './fetch-resource'
import fetch from 'next/dist/compiled/node-fetch'
jest.mock('next/dist/compiled/node-fetch') jest.mock('./fetch-resource')
const mockFetchResource = fetchResource as jest.Mock
describe('next/font/google loader', () => { describe('next/font/google loader', () => {
afterEach(() => { afterEach(() => {
...@@ -120,10 +121,7 @@ describe('next/font/google loader', () => { ...@@ -120,10 +121,7 @@ describe('next/font/google loader', () => {
fontFunctionArguments: any, fontFunctionArguments: any,
expectedUrl: any expectedUrl: any
) => { ) => {
fetch.mockResolvedValue({ mockFetchResource.mockResolvedValue(Buffer.from('OK'))
ok: true,
text: async () => 'OK',
})
const { css } = await nextFontGoogleFontLoader({ const { css } = await nextFontGoogleFontLoader({
functionName, functionName,
data: [ data: [
...@@ -141,8 +139,12 @@ describe('next/font/google loader', () => { ...@@ -141,8 +139,12 @@ describe('next/font/google loader', () => {
variableName: 'myFont', variableName: 'myFont',
}) })
expect(css).toBe('OK') expect(css).toBe('OK')
expect(fetch).toHaveBeenCalledTimes(1) expect(mockFetchResource).toHaveBeenCalledTimes(1)
expect(fetch).toHaveBeenCalledWith(expectedUrl, expect.any(Object)) expect(mockFetchResource).toHaveBeenCalledWith(
expectedUrl,
false,
expect.stringContaining('Failed to fetch font')
)
} }
) )
}) })
......
...@@ -2,7 +2,6 @@ import findUp from 'next/dist/compiled/find-up' ...@@ -2,7 +2,6 @@ import findUp from 'next/dist/compiled/find-up'
import fsPromise from 'fs/promises' import fsPromise from 'fs/promises'
import child_process from 'child_process' import child_process from 'child_process'
import assert from 'assert' import assert from 'assert'
import fetch from 'next/dist/compiled/node-fetch'
import os from 'os' import os from 'os'
import { createInterface } from 'readline' import { createInterface } from 'readline'
import { createReadStream } from 'fs' import { createReadStream } from 'fs'
......
...@@ -77,6 +77,7 @@ const externals = { ...@@ -77,6 +77,7 @@ const externals = {
'terser-webpack-plugin': 'terser-webpack-plugin':
'next/dist/build/webpack/plugins/terser-webpack-plugin/src', 'next/dist/build/webpack/plugins/terser-webpack-plugin/src',
punycode: 'punycode/',
// TODO: Add @swc/helpers to externals once @vercel/ncc switch to swc-loader // TODO: Add @swc/helpers to externals once @vercel/ncc switch to swc-loader
} }
// eslint-disable-next-line camelcase // eslint-disable-next-line camelcase
......
...@@ -351,12 +351,6 @@ declare module 'next/dist/compiled/@next/react-refresh-utils/dist/ReactRefreshWe ...@@ -351,12 +351,6 @@ declare module 'next/dist/compiled/@next/react-refresh-utils/dist/ReactRefreshWe
export = m export = m
} }
declare module 'next/dist/compiled/node-fetch' {
import fetch from 'node-fetch'
export * from 'node-fetch'
export default fetch
}
declare module 'next/dist/compiled/commander' { declare module 'next/dist/compiled/commander' {
import commander from 'commander' import commander from 'commander'
export * from 'commander' export * from 'commander'
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment