Unverified Commit e1ee42f7 authored by Gary Borton's avatar Gary Borton Committed by GitHub

Improve caching of next-app-loader to prevent more fs calls. (#67792)

I noticed in some local profiles that a significant % of the build was
being spent in the `metadataResolver` function of `next-app-loader`. It
seems to be spending a good chunk of its time inside
`fileExistsInDirectory`.

It looks like this was already updated to have some level of caching
(>99% of calls _do_ hit a cache already), but that a significant number
of calls are reaching out to the filesystem but don't have to.

For my build in particular, `36685` calls made it to the filesystem, of
which `35545 (~97%)` _could_ have been cached. ~~Because the cache here
isn't shared between instances, this change would only reduce the cache
misses by `~29191 (~79%)`. I think this can be optimized further, but
I'm wary of trying to make this change myself.~~

^ I've updated to get us to the 97% case by stuffing compilation in a
WeakMap. Some risk of something holding onto a compilation that
shouldn't be as a side effect. I've also added a comment to the code to
point this out.

Additionally, I spotted that `getFilesInDir` returns an array, but is
only ever used in a place that immediately converts it to a set. I've
updated to return the set directly, 🤞 that we can save some gc cycles as
well.

Below are two 60 second profiles. In each, I've disabled webpack
parallelism, and added a local cache that writes cache items directly to
the filesystem. I've also preloaded the entire cache into memory
_before_ the profiles so that I'm looking as closely at the raw CPU work
as I can.

**Before**
<img width="1642" alt="image"
src="https://github.com/user-attachments/assets/d27fea21-278f-4dbd-8b33-405fda3c9d08">

**After**

![image](https://github.com/user-attachments/assets/9edf3506-5417-4839-9a26-22f6b49fbb36)


<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:

## For Contributors

### Improving Documentation

- Run `pnpm prettier-fix` to fix formatting issues before opening the
PR.
- Read the Docs Contribution Guide to ensure your contribution follows
the docs guidelines:
https://nextjs.org/docs/community/contribution-guide

### Adding or Updating Examples

- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md

### Fixing a bug

- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md

### Adding a feature

- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md


## For Maintainers

- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Closes NEXT-
Fixes #

-->

---------
Co-authored-by: 's avatarZack Tanner <1939140+ztanner@users.noreply.github.com>
Co-authored-by: 's avatarJJ Kasper <jj@jjsweb.site>
parent 6f2fba4c
...@@ -896,7 +896,7 @@ export default async function build( ...@@ -896,7 +896,7 @@ export default async function build(
: []), : []),
] ]
const rootPaths = (await getFilesInDir(rootDir)) const rootPaths = Array.from(await getFilesInDir(rootDir))
.filter((file) => includes.some((include) => include.test(file))) .filter((file) => includes.some((include) => include.test(file)))
.sort(sortByPageExts(config.pageExtensions)) .sort(sortByPageExts(config.pageExtensions))
.map((file) => path.join(rootDir, file).replace(dir, '')) .map((file) => path.join(rootDir, file).replace(dir, ''))
......
...@@ -35,6 +35,7 @@ import { ...@@ -35,6 +35,7 @@ import {
import { getFilesInDir } from '../../../lib/get-files-in-dir' import { getFilesInDir } from '../../../lib/get-files-in-dir'
import type { PageExtensions } from '../../page-extensions-type' import type { PageExtensions } from '../../page-extensions-type'
import { PARALLEL_ROUTE_DEFAULT_PATH } from '../../../client/components/parallel-route-default' import { PARALLEL_ROUTE_DEFAULT_PATH } from '../../../client/components/parallel-route-default'
import type { Compilation } from 'webpack'
export type AppLoaderOptions = { export type AppLoaderOptions = {
name: string name: string
...@@ -521,6 +522,10 @@ function createAbsolutePath(appDir: string, pathToTurnAbsolute: string) { ...@@ -521,6 +522,10 @@ function createAbsolutePath(appDir: string, pathToTurnAbsolute: string) {
) )
} }
const filesInDirMapMap: WeakMap<
Compilation,
Map<string, Promise<Set<string>>>
> = new WeakMap()
const nextAppLoader: AppLoader = async function nextAppLoader() { const nextAppLoader: AppLoader = async function nextAppLoader() {
const loaderOptions = this.getOptions() const loaderOptions = this.getOptions()
const { const {
...@@ -634,20 +639,30 @@ const nextAppLoader: AppLoader = async function nextAppLoader() { ...@@ -634,20 +639,30 @@ const nextAppLoader: AppLoader = async function nextAppLoader() {
// This can be more efficient than checking them with `fs.stat` one by one // This can be more efficient than checking them with `fs.stat` one by one
// because all the thousands of files are likely in a few possible directories. // because all the thousands of files are likely in a few possible directories.
// Note that it should only be cached for this compilation, not globally. // Note that it should only be cached for this compilation, not globally.
const filesInDir = new Map<string, Set<string>>()
const fileExistsInDirectory = async (dirname: string, fileName: string) => { const fileExistsInDirectory = async (dirname: string, fileName: string) => {
const existingFiles = filesInDir.get(dirname) // I don't think we should ever hit this code path, but if we do we should handle it gracefully.
if (existingFiles) { if (this._compilation === undefined) {
return existingFiles.has(fileName) try {
return (await getFilesInDir(dirname).catch(() => new Set())).has(
fileName
)
} catch (e) {
return false
}
} }
try { const map =
const files = await getFilesInDir(dirname) filesInDirMapMap.get(this._compilation) ||
const fileNames = new Set<string>(files) new Map<string, Promise<Set<string>>>()
filesInDir.set(dirname, fileNames) if (!filesInDirMapMap.has(this._compilation)) {
return fileNames.has(fileName) filesInDirMapMap.set(this._compilation, map)
} catch (err) { }
return false if (!map.has(dirname)) {
map.set(
dirname,
getFilesInDir(dirname).catch(() => new Set())
)
} }
return ((await map.get(dirname)) || new Set()).has(fileName)
} }
const resolver: PathResolver = async (pathname) => { const resolver: PathResolver = async (pathname) => {
...@@ -761,7 +776,7 @@ const nextAppLoader: AppLoader = async function nextAppLoader() { ...@@ -761,7 +776,7 @@ const nextAppLoader: AppLoader = async function nextAppLoader() {
} }
// Clear fs cache, get the new result with the created root layout. // Clear fs cache, get the new result with the created root layout.
filesInDir.clear() if (this._compilation) filesInDirMapMap.get(this._compilation)?.clear()
treeCodeResult = await createTreeCodeFromPath(pagePath, { treeCodeResult = await createTreeCodeFromPath(pagePath, {
page, page,
resolveDir, resolveDir,
......
...@@ -2,9 +2,9 @@ import { join } from 'path' ...@@ -2,9 +2,9 @@ import { join } from 'path'
import fs from 'fs/promises' import fs from 'fs/promises'
import type { Dirent, StatsBase } from 'fs' import type { Dirent, StatsBase } from 'fs'
export async function getFilesInDir(path: string): Promise<string[]> { export async function getFilesInDir(path: string): Promise<Set<string>> {
const dir = await fs.opendir(path) const dir = await fs.opendir(path)
const results = [] const results = new Set<string>()
for await (const file of dir) { for await (const file of dir) {
let resolvedFile: Dirent | StatsBase<number> = file let resolvedFile: Dirent | StatsBase<number> = file
...@@ -14,7 +14,7 @@ export async function getFilesInDir(path: string): Promise<string[]> { ...@@ -14,7 +14,7 @@ export async function getFilesInDir(path: string): Promise<string[]> {
} }
if (resolvedFile.isFile()) { if (resolvedFile.isFile()) {
results.push(file.name) results.add(file.name)
} }
} }
......
...@@ -47,12 +47,8 @@ describe('getFilesInDir', () => { ...@@ -47,12 +47,8 @@ describe('getFilesInDir', () => {
await fs.remove(testDir) await fs.remove(testDir)
await setupTestDir() await setupTestDir()
expect(await getFilesInDir(srcDir)).toIncludeAllMembers([ expect(await getFilesInDir(srcDir)).toEqual(
'.hidden', new Set(['.hidden', 'file', 'link', 'link-level-2', 'link-level-3'])
'file', )
'link',
'link-level-2',
'link-level-3',
])
}) })
}) })
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment