Unverified Commit ff756b2c authored by Benjamin Woodruff's avatar Benjamin Woodruff Committed by GitHub

Turbopack: Define built-in webpack conditions using an enum and typescript union (#82765)

## Context

We have some publicly documented syntax for configuring webpack loaders here: https://nextjs.org/docs/app/api-reference/config/next-config-js/turbopack#configuring-webpack-loaders

However, we also have some undocumented features. Specifically, we have two different "condition" concepts, both of which are shown here: https://github.com/vercel/next.js/blob/1766a1aba6a7c92a03c08073114652d61da1a388/packages/next/src/build/swc/index.ts#L836-L860

1. The `conditions` field in the `turbopack` object of the config can be used to specify predicates (defined by file name glob, regex, or content of file) that can be referenced by rules.
2. There are special built-in internal "conditions" that can be matched by passing an object to a `rule` that does not contain a `loader` field. These special conditions allow us to match on some things we couldn't otherwise declare conditions for.

This is about improving the current state of #2, which I'm calling here to "webpack loader built-in conditions".

Confusingly, there are also very similar conditions used by the resolver context. I'm not changing those, but I've done a tiny bit of work in this PR to better clarify the difference with naming and types.

## This PR

- Instead of using an `RcStr`, for these conditions, use an enum. Theoretically that gives us less flexibility, but it makes it way easier to discover, document, and keep track of what built-in conditions we support. I feel that it's very important that we do this because this enum is effectively a public API, and I'd like to document some version of this API in a subsequent PR.
- Use the exact union of supported fields in the zod schema and typescript definitions.
- Represent the collection of conditions as a set instead of a vec. In reality, this doesn't have much impact because the set is small and always constructed in the same order, but the intent is clearer. Implemented `TaskInput` for `BTreeSet`.
- Some minor consistency cleanup between `next_server` and `next_client`.
- Get rid of some extra type conversions using the `rcstr!()` macro.
parent 523ae09b
use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::TaskInput;
use turbopack_ecmascript_runtime::RuntimeType;
use crate::next_shared::webpack_rules::WebpackLoaderBuiltinCondition;
/// The mode in which Next.js is running.
#[turbo_tasks::value(shared)]
#[derive(Debug, Copy, Clone, TaskInput, Ord, PartialOrd, Hash)]
......@@ -20,12 +23,23 @@ impl NextMode {
}
}
/// Returns the exports condition for the current mode.
pub fn condition(&self) -> &'static str {
/// Returns conditions that can be used in the Next.js config's turbopack "rules" section for
/// defining webpack loader configuration.
pub fn webpack_loader_conditions(&self) -> impl Iterator<Item = WebpackLoaderBuiltinCondition> {
match self {
NextMode::Development => "development",
NextMode::Build => "production",
NextMode::Development => [WebpackLoaderBuiltinCondition::Development],
NextMode::Build => [WebpackLoaderBuiltinCondition::Production],
}
.into_iter()
}
/// Returns conditions used by `ResolveOptionsContext`.
pub fn custom_resolve_conditions(&self) -> impl Iterator<Item = RcStr> {
match self {
NextMode::Development => [rcstr!("development")],
NextMode::Build => [rcstr!("production")],
}
.into_iter()
}
/// Returns true if the development React runtime should be used.
......
use std::iter::once;
use std::collections::BTreeSet;
use anyhow::Result;
use serde::{Deserialize, Serialize};
......@@ -58,7 +58,7 @@ use crate::{
styled_jsx::get_styled_jsx_transform_rule,
swc_ecma_transform_plugins::get_swc_ecma_transform_plugin_rule,
},
webpack_rules::webpack_loader_options,
webpack_rules::{WebpackLoaderBuiltinCondition, webpack_loader_options},
},
transform_options::{
get_decorators_transform_options, get_jsx_transform_options,
......@@ -150,7 +150,7 @@ pub async fn get_client_resolve_options_context(
get_next_client_resolved_map(project_path.clone(), project_path.clone(), *mode.await?)
.to_resolved()
.await?;
let custom_conditions = vec![mode.await?.condition().into()];
let custom_conditions = mode.await?.custom_resolve_conditions().collect();
let resolve_options_context = ResolveOptionsContext {
enable_node_modules: Some(project_path.root().owned().await?),
custom_conditions,
......@@ -238,26 +238,21 @@ pub async fn get_client_module_options_context(
.to_resolved()
.await?;
// A separate webpack rules will be applied to codes matching
// foreign_code_context_condition. This allows to import codes from
// node_modules that requires webpack loaders, which next-dev implicitly
// does by default.
let conditions = vec![rcstr!("browser"), mode.await?.condition().into()];
let foreign_enable_webpack_loaders = webpack_loader_options(
project_path.clone(),
next_config,
true,
conditions
.iter()
.cloned()
.chain(once(rcstr!("foreign")))
.collect(),
)
.await?;
let mut loader_conditions = BTreeSet::new();
loader_conditions.insert(WebpackLoaderBuiltinCondition::Browser);
loader_conditions.extend(mode.await?.webpack_loader_conditions());
// A separate webpack rules will be applied to codes matching foreign_code_context_condition.
// This allows to import codes from node_modules that requires webpack loaders, which next-dev
// implicitly does by default.
let mut foreign_conditions = loader_conditions.clone();
foreign_conditions.insert(WebpackLoaderBuiltinCondition::Foreign);
let foreign_enable_webpack_loaders =
webpack_loader_options(project_path.clone(), next_config, true, foreign_conditions).await?;
// Now creates a webpack rules that applies to all codes.
// Now creates a webpack rules that applies to all code.
let enable_webpack_loaders =
webpack_loader_options(project_path.clone(), next_config, false, conditions).await?;
webpack_loader_options(project_path.clone(), next_config, false, loader_conditions).await?;
let tree_shaking_mode_for_user_code = *next_config
.tree_shaking_mode_for_user_code(next_mode.is_development())
......
use std::{collections::BTreeSet, str::FromStr};
use anyhow::{Context, Result, bail};
use rustc_hash::FxHashSet;
use serde::{Deserialize, Deserializer, Serialize};
......@@ -27,8 +29,11 @@ use turbopack_ecmascript_plugins::transform::{
use turbopack_node::transforms::webpack::{WebpackLoaderItem, WebpackLoaderItems};
use crate::{
mode::NextMode, next_import_map::mdx_import_source_file,
next_shared::transforms::ModularizeImportPackageConfig,
mode::NextMode,
next_import_map::mdx_import_source_file,
next_shared::{
transforms::ModularizeImportPackageConfig, webpack_rules::WebpackLoaderBuiltinCondition,
},
};
#[turbo_tasks::value]
......@@ -1277,7 +1282,7 @@ impl NextConfig {
#[turbo_tasks::function]
pub async fn webpack_rules(
&self,
active_conditions: Vec<RcStr>,
active_conditions: BTreeSet<WebpackLoaderBuiltinCondition>,
project_path: FileSystemPath,
) -> Result<Vc<OptionWebpackRules>> {
let Some(turbo_rules) = self.turbopack.as_ref().and_then(|t| t.rules.as_ref()) else {
......@@ -1286,7 +1291,6 @@ impl NextConfig {
if turbo_rules.is_empty() {
return Ok(Vc::cell(None));
}
let active_conditions = active_conditions.into_iter().collect::<FxHashSet<_>>();
let mut rules = FxIndexMap::default();
for (ext, rule) in turbo_rules.iter() {
fn transform_loaders(loaders: &[LoaderItem]) -> ResolvedVc<WebpackLoaderItems> {
......@@ -1310,13 +1314,17 @@ impl NextConfig {
}
fn find_rule<'a>(
rule: &'a RuleConfigItem,
active_conditions: &FxHashSet<RcStr>,
active_conditions: &BTreeSet<WebpackLoaderBuiltinCondition>,
) -> FindRuleResult<'a> {
match rule {
RuleConfigItem::Options(rule) => FindRuleResult::Found(rule),
RuleConfigItem::Conditional(map) => {
for (condition, rule) in map.iter() {
if condition == "default" || active_conditions.contains(condition) {
let condition = WebpackLoaderBuiltinCondition::from_str(condition);
if let Ok(condition) = condition
&& (condition == WebpackLoaderBuiltinCondition::Default
|| active_conditions.contains(&condition))
{
match find_rule(rule, active_conditions) {
FindRuleResult::Found(rule) => {
return FindRuleResult::Found(rule);
......
......@@ -147,14 +147,8 @@ pub async fn get_edge_resolve_options_context(
)];
// https://github.com/vercel/next.js/blob/bf52c254973d99fed9d71507a2e818af80b8ade7/packages/next/src/build/webpack-config.ts#L96-L102
let mut custom_conditions = vec![mode.await?.condition().into()];
custom_conditions.extend(
NextRuntime::Edge
.conditions()
.iter()
.map(ToString::to_string)
.map(RcStr::from),
);
let mut custom_conditions: Vec<_> = mode.await?.custom_resolve_conditions().collect();
custom_conditions.extend(NextRuntime::Edge.custom_resolve_conditions());
if ty.should_use_react_server_condition() {
custom_conditions.push(rcstr!("react-server"));
......
use std::iter::once;
use std::collections::BTreeSet;
use anyhow::{Result, bail};
use serde::{Deserialize, Serialize};
......@@ -66,7 +66,7 @@ use crate::{
styled_jsx::get_styled_jsx_transform_rule,
swc_ecma_transform_plugins::get_swc_ecma_transform_plugin_rule,
},
webpack_rules::webpack_loader_options,
webpack_rules::{WebpackLoaderBuiltinCondition, webpack_loader_options},
},
transform_options::{
get_decorators_transform_options, get_jsx_transform_options,
......@@ -211,14 +211,8 @@ pub async fn get_server_resolve_options_context(
.to_resolved()
.await?;
let mut custom_conditions = vec![mode.await?.condition().to_string().into()];
custom_conditions.extend(
NextRuntime::NodeJs
.conditions()
.iter()
.map(ToString::to_string)
.map(RcStr::from),
);
let mut custom_conditions: Vec<_> = mode.await?.custom_resolve_conditions().collect();
custom_conditions.extend(NextRuntime::NodeJs.custom_resolve_conditions());
if ty.should_use_react_server_condition() {
custom_conditions.push(rcstr!("react-server"));
......@@ -489,34 +483,21 @@ pub async fn get_server_module_options_context(
let enable_postcss_transform = Some(postcss_transform_options.resolved_cell());
let enable_foreign_postcss_transform = Some(postcss_foreign_transform_options.resolved_cell());
let mut conditions = vec![mode.await?.condition().into()];
conditions.extend(
next_runtime
.conditions()
.iter()
.map(ToString::to_string)
.map(RcStr::from),
);
let mut loader_conditions = BTreeSet::new();
loader_conditions.extend(mode.await?.webpack_loader_conditions());
loader_conditions.extend(next_runtime.webpack_loader_conditions());
// A separate webpack rules will be applied to codes matching
// foreign_code_context_condition. This allows to import codes from
// node_modules that requires webpack loaders, which next-dev implicitly
// does by default.
let foreign_enable_webpack_loaders = webpack_loader_options(
project_path.clone(),
next_config,
true,
conditions
.iter()
.cloned()
.chain(once(rcstr!("foreign")))
.collect(),
)
.await?;
// A separate webpack rules will be applied to codes matching foreign_code_context_condition.
// This allows to import codes from node_modules that requires webpack loaders, which next-dev
// implicitly does by default.
let mut foreign_conditions = loader_conditions.clone();
foreign_conditions.insert(WebpackLoaderBuiltinCondition::Foreign);
let foreign_enable_webpack_loaders =
webpack_loader_options(project_path.clone(), next_config, true, foreign_conditions).await?;
// Now creates a webpack rules that applies to all codes.
// Now creates a webpack rules that applies to all code.
let enable_webpack_loaders =
webpack_loader_options(project_path.clone(), next_config, false, conditions).await?;
webpack_loader_options(project_path.clone(), next_config, false, loader_conditions).await?;
let tree_shaking_mode_for_user_code = *next_config
.tree_shaking_mode_for_user_code(next_mode.is_development())
......
use std::{collections::BTreeSet, str::FromStr};
use anyhow::Result;
use turbo_rcstr::{RcStr, rcstr};
use turbo_tasks::{ResolvedVc, Vc};
use serde::{Deserialize, Serialize};
use turbo_rcstr::rcstr;
use turbo_tasks::{NonLocalValue, ResolvedVc, TaskInput, Vc, trace::TraceRawVcs};
use turbo_tasks_fs::FileSystemPath;
use turbopack::module_options::WebpackLoadersOptions;
use turbopack_core::resolve::{ExternalTraced, ExternalType, options::ImportMapping};
......@@ -11,14 +14,96 @@ use crate::next_config::NextConfig;
pub(crate) mod babel;
pub(crate) mod sass;
/// Built-in conditions provided by the Next.js Turbopack integration for configuring webpack
/// loaders. These can be used in the `next.config.js` `turbopack.rules` section.
///
/// These are different from than the user-configurable "conditions" field.
//
// Note: If you add a field here, make sure to also add it in:
// - The typescript definition in `packages/next/src/server/config-shared.ts`
// - The zod schema in `packages/next/src/server/config-schema.ts`
//
// Note: Sets of conditions could be stored more efficiently as a bitset, but it's probably not used
// in enough places for it to matter.
#[derive(
Copy,
Clone,
Debug,
PartialEq,
Eq,
PartialOrd,
Ord,
Hash,
Serialize,
Deserialize,
TaskInput,
TraceRawVcs,
NonLocalValue,
)]
pub enum WebpackLoaderBuiltinCondition {
/// Treated as always-present.
Default,
/// Client-side code.
Browser,
/// Code in `node_modules` that should typically not be modified by webpack loaders.
Foreign,
// These are provided by NextMode:
Development,
Production,
// These are provided by NextRuntime:
/// Server code on the Node.js runtime.
Node,
/// Server code on the edge runtime.
EdgeLight,
}
impl WebpackLoaderBuiltinCondition {
fn as_str(self) -> &'static str {
match self {
Self::Default => "default",
Self::Browser => "browser",
Self::Foreign => "foreign",
Self::Development => "development",
Self::Production => "production",
Self::Node => "node",
Self::EdgeLight => "edge-light",
}
}
}
impl FromStr for WebpackLoaderBuiltinCondition {
type Err = ();
fn from_str(s: &str) -> Result<Self, Self::Err> {
match s {
"default" => Ok(Self::Default),
"browser" => Ok(Self::Browser),
"foreign" => Ok(Self::Foreign),
"development" => Ok(Self::Development),
"production" => Ok(Self::Production),
"node" => Ok(Self::Node),
"edge-light" => Ok(Self::EdgeLight),
_ => Err(()),
}
}
}
impl PartialEq<WebpackLoaderBuiltinCondition> for &str {
fn eq(&self, other: &WebpackLoaderBuiltinCondition) -> bool {
*self == other.as_str()
}
}
pub async fn webpack_loader_options(
project_path: FileSystemPath,
next_config: Vc<NextConfig>,
foreign: bool,
condition_strs: Vec<RcStr>,
loader_conditions: BTreeSet<WebpackLoaderBuiltinCondition>,
) -> Result<Option<ResolvedVc<WebpackLoadersOptions>>> {
let rules = *next_config
.webpack_rules(condition_strs, project_path.clone())
.webpack_rules(loader_conditions, project_path.clone())
.await?;
let rules = *maybe_add_sass_loader(next_config.sass_config(), rules.map(|v| *v)).await?;
let rules = if foreign {
......
......@@ -40,6 +40,7 @@ use crate::{
next_config::{NextConfig, RouteHas},
next_import_map::get_next_package,
next_manifests::MiddlewareMatcher,
next_shared::webpack_rules::WebpackLoaderBuiltinCondition,
};
const NEXT_TEMPLATE_PATH: &str = "dist/esm/build/templates";
......@@ -228,11 +229,23 @@ pub enum NextRuntime {
}
impl NextRuntime {
pub fn conditions(&self) -> &'static [&'static str] {
/// Returns conditions that can be used in the Next.js config's turbopack "rules" section for
/// defining webpack loader configuration.
pub fn webpack_loader_conditions(&self) -> impl Iterator<Item = WebpackLoaderBuiltinCondition> {
match self {
NextRuntime::NodeJs => &["node"],
NextRuntime::Edge => &["edge-light"],
NextRuntime::NodeJs => [WebpackLoaderBuiltinCondition::Node],
NextRuntime::Edge => [WebpackLoaderBuiltinCondition::EdgeLight],
}
.into_iter()
}
/// Returns conditions used by `ResolveOptionsContext`.
pub fn custom_resolve_conditions(&self) -> impl Iterator<Item = RcStr> {
match self {
NextRuntime::NodeJs => [rcstr!("node")],
NextRuntime::Edge => [rcstr!("edge-light")],
}
.into_iter()
}
}
......
......@@ -879,7 +879,7 @@ function bindingToApi(
nextConfigSerializable.exportPathMap = {}
nextConfigSerializable.webpack = nextConfig.webpack && {}
if (nextConfigSerializable.experimental?.turbo?.rules) {
if (nextConfigSerializable.turbopack?.rules) {
ensureLoadersHaveSerializableOptions(
nextConfigSerializable.turbopack?.rules
)
......@@ -974,10 +974,9 @@ function bindingToApi(
if ('loaders' in rule) {
checkLoaderItems((rule as TurbopackRuleConfigItemOptions).loaders, glob)
} else {
for (const key in rule) {
const inner = rule[key]
if (typeof inner === 'object' && inner) {
checkConfigItem(inner, glob)
for (const value of Object.values(rule)) {
if (typeof value === 'object' && value) {
checkConfigItem(value, glob)
}
}
}
......
......@@ -14,6 +14,7 @@ import type {
TurbopackRuleConfigItemOptions,
TurbopackRuleConfigItemOrShortcut,
TurbopackRuleCondition,
TurbopackLoaderBuiltinCondition,
} from './config-shared'
import type {
Header,
......@@ -102,7 +103,7 @@ const zHeader: zod.ZodType<Header> = z.object({
internal: z.boolean().optional(),
})
const zTurboLoaderItem: zod.ZodType<TurbopackLoaderItem> = z.union([
const zTurbopackLoaderItem: zod.ZodType<TurbopackLoaderItem> = z.union([
z.string(),
z.object({
loader: z.string(),
......@@ -111,31 +112,42 @@ const zTurboLoaderItem: zod.ZodType<TurbopackLoaderItem> = z.union([
}),
])
const zTurboRuleConfigItemOptions: zod.ZodType<TurbopackRuleConfigItemOptions> =
const zTurbopackRuleConfigItemOptions: zod.ZodType<TurbopackRuleConfigItemOptions> =
z.object({
loaders: z.array(zTurboLoaderItem),
loaders: z.array(zTurbopackLoaderItem),
as: z.string().optional(),
})
const zTurboRuleConfigItem: zod.ZodType<TurbopackRuleConfigItem> = z.union([
const zTurbopackLoaderBuiltinCondition: zod.ZodType<TurbopackLoaderBuiltinCondition> =
z.union([
z.literal('default'),
z.literal('browser'),
z.literal('foreign'),
z.literal('development'),
z.literal('production'),
z.literal('node'),
z.literal('edge-light'),
])
const zTurbopackRuleConfigItem: zod.ZodType<TurbopackRuleConfigItem> = z.union([
z.literal(false),
z.record(
z.string(),
z.lazy(() => zTurboRuleConfigItem)
zTurbopackLoaderBuiltinCondition,
z.lazy(() => zTurbopackRuleConfigItem)
),
zTurboRuleConfigItemOptions,
zTurbopackRuleConfigItemOptions,
])
const zTurboRuleConfigItemOrShortcut: zod.ZodType<TurbopackRuleConfigItemOrShortcut> =
z.union([z.array(zTurboLoaderItem), zTurboRuleConfigItem])
const zTurbopackRuleConfigItemOrShortcut: zod.ZodType<TurbopackRuleConfigItemOrShortcut> =
z.union([z.array(zTurbopackLoaderItem), zTurbopackRuleConfigItem])
const zTurboCondition: zod.ZodType<TurbopackRuleCondition> = z.object({
const zTurbopackCondition: zod.ZodType<TurbopackRuleCondition> = z.object({
path: z.union([z.string(), z.instanceof(RegExp)]).optional(),
content: z.instanceof(RegExp).optional(),
})
const zTurbopackConfig: zod.ZodType<TurbopackOptions> = z.strictObject({
rules: z.record(z.string(), zTurboRuleConfigItemOrShortcut).optional(),
conditions: z.record(z.string(), zTurboCondition).optional(),
rules: z.record(z.string(), zTurbopackRuleConfigItemOrShortcut).optional(),
conditions: z.record(z.string(), zTurbopackCondition).optional(),
resolveAlias: z
.record(
z.string(),
......@@ -155,8 +167,8 @@ const zTurbopackConfig: zod.ZodType<TurbopackOptions> = z.strictObject({
// properties are duplicated here as `ZodType`s do not export `extend()`.
const zDeprecatedExperimentalTurboConfig: zod.ZodType<DeprecatedExperimentalTurboOptions> =
z.strictObject({
loaders: z.record(z.string(), z.array(zTurboLoaderItem)).optional(),
rules: z.record(z.string(), zTurboRuleConfigItemOrShortcut).optional(),
loaders: z.record(z.string(), z.array(zTurbopackLoaderItem)).optional(),
rules: z.record(z.string(), zTurbopackRuleConfigItemOrShortcut).optional(),
resolveAlias: z
.record(
z.string(),
......
......@@ -261,6 +261,15 @@ export type TurbopackLoaderItem =
options: Record<string, JSONValue>
}
export type TurbopackLoaderBuiltinCondition =
| 'default'
| 'browser'
| 'foreign'
| 'development'
| 'production'
| 'node'
| 'edge-light'
export type TurbopackRuleCondition = {
path?: string | RegExp
content?: RegExp
......@@ -277,7 +286,7 @@ export type TurbopackRuleConfigItemOptions = {
export type TurbopackRuleConfigItem =
| TurbopackRuleConfigItemOptions
| { [condition: string]: TurbopackRuleConfigItem }
| { [condition in TurbopackLoaderBuiltinCondition]?: TurbopackRuleConfigItem }
| false
export interface TurbopackOptions {
......
use std::{
collections::BTreeMap, fmt::Debug, future::Future, hash::Hash, sync::Arc, time::Duration,
collections::{BTreeMap, BTreeSet},
fmt::Debug,
future::Future,
hash::Hash,
sync::Arc,
time::Duration,
};
use anyhow::Result;
......@@ -278,7 +283,7 @@ where
fn is_resolved(&self) -> bool {
self.iter()
.all(|(k, v)| TaskInput::is_resolved(k) || TaskInput::is_resolved(v))
.all(|(k, v)| TaskInput::is_resolved(k) && TaskInput::is_resolved(v))
}
fn is_transient(&self) -> bool {
......@@ -286,6 +291,28 @@ where
.any(|(k, v)| TaskInput::is_transient(k) || TaskInput::is_transient(v))
}
}
impl<T> TaskInput for BTreeSet<T>
where
T: TaskInput + Ord,
{
async fn resolve_input(&self) -> Result<Self> {
let mut new_map = BTreeSet::new();
for value in self {
new_map.insert(TaskInput::resolve_input(value).await?);
}
Ok(new_map)
}
fn is_resolved(&self) -> bool {
self.iter().all(TaskInput::is_resolved)
}
fn is_transient(&self) -> bool {
self.iter().any(TaskInput::is_transient)
}
}
macro_rules! tuple_impls {
( $( $name:ident )+ ) => {
impl<$($name: TaskInput),+> TaskInput for ($($name,)+)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment