Unverified Commit ff3a2cfa authored by Luke Sandberg's avatar Luke Sandberg Committed by GitHub

[turbopack] Strip leading BOM before parsing CSS (#96678)

Fork PR #96379 by @lazerg, re-opened as a branch PR so the "when
deployed" CI jobs can run — those require Vercel deployment secrets that
GitHub does not expose to pull requests from forks, so they can never
pass on the original.

**The fix commits are unchanged and still authored by @lazerg.** This PR
only adds tests on top. Please credit them; #96379 should be closed in
favor of this one.

### What?

A CSS file beginning with a UTF-8 BOM (`EF BB BF`) is mishandled by
Turbopack. Lightning CSS does not skip the BOM, so it is tokenized as
content and the first token is misparsed:

```
./app/bom.css:1:2
Error: Parsing CSS source code failed
Unexpected token AtKeyword("layer")
```

The user-visible symptom is broader than a failed build. Turbopack
parses with `error_recovery: true`, and under that setting a leading BOM
makes Lightning CSS return `Ok` with **zero rules** — so a BOM-prefixed
stylesheet could silently drop all of its styles instead of erroring.
dart-sass (compressed style) and PostCSS >= 8.5.24 both emit or
round-trip such a BOM, so real projects hit this.

Fixes #96374

### How?

Strip a leading `U+FEFF` in `parse_css_stylesheet` before handing the
source to Lightning CSS, covering both `StyleSheet::parse` call sites
while leaving `ParseCssResult.code` as the original bytes that code
frames are rendered from.

That split makes parser positions relative to the stripped copy while
code frames still render the original line, so first-line positions need
compensating. `source_pos_for_loc` adds the stripped character back for
line 0 of BOM files. Only line 0 is affected, because the BOM contains
no newline.

### Tests

`test/e2e/app-dir/css-bom` — a BOM-prefixed stylesheet compiles and its
rules reach the page. Verified failing without the fix with the exact
error above, and passing with it, in dev-turbo, start-turbo and
start-webpack.

`test/development/app-dir/css-bom-code-frame` — covers the position
correction. Two fixtures hold the same invalid `@media (min-width: {})`
on line 1 and differ only by the leading BOM; the test asserts the BOM
file's reported column is exactly one greater:

| | `no-bom` | `bom` | |
|---|---|---|---|
| without `source_pos_for_loc` | 18 | 18 | fails |
| with it | 18 | 19 | passes |

Asserting the relationship rather than a literal column keeps this
robust if Lightning CSS changes its absolute column convention. It is
kept separate from the e2e suite because the fixtures are intentionally
invalid CSS, and scoped to Turbopack in dev, where the warning reaches
the CLI as the page is requested.

---------
Co-authored-by: 's avatarlazerg <lazerg2@gmail.com>
Co-authored-by: 's avatarvercel-gh-bot-3[bot] <282332853+vercel-gh-bot-3[bot]@users.noreply.github.com>
parent 64571418
@media (min-width: {}) {
.first-line-error {
color: rgb(0, 128, 0);
}
}
import './error.css'
export default function Page() {
return <p className="first-line-error">bom</p>
}
export default function Root({ children }: { children: React.ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
@media (min-width: {}) {
.no-bom {
color: rgb(0, 128, 0);
}
}
import './error.css'
export default function Page() {
return <p className="no-bom">no bom</p>
}
export default function Page() {
return <p>home</p>
}
import { readFileSync } from 'fs'
import { join } from 'path'
import { nextTestSetup } from 'e2e-utils'
import { retry } from 'next-test-utils'
import stripAnsi from 'strip-ansi'
// A leading BOM is stripped before the CSS is handed to the parser, so parser
// positions are relative to the stripped copy while the code frame is rendered
// from the original (un-stripped) file. Only errors on the first line are
// affected, because the BOM contains no newline.
//
// Turbopack-only: this asserts on positions reported by Turbopack's CSS parser.
;(process.env.IS_TURBOPACK_TEST ? describe : describe.skip)(
'css BOM code frame',
() => {
const { next } = nextTestSetup({
files: __dirname,
})
it('keeps the fixtures byte-exact', () => {
// The whole point of the fixtures is the leading bytes, and an editor or
// formatter dropping them would silently turn the test below into a pass.
const bom = readFileSync(join(__dirname, 'app', 'bom', 'error.css'))
expect([bom[0], bom[1], bom[2]]).toEqual([0xef, 0xbb, 0xbf])
const noBom = readFileSync(join(__dirname, 'app', 'no-bom', 'error.css'))
expect(noBom[0]).not.toBe(0xef)
})
it('reports the right column for an error on the first line', async () => {
// A warning for one file can be re-emitted while the other is compiled,
// so each column is matched against its own path rather than whatever
// happens to appear first in the output.
const columnFor = async (dir: string) => {
// Anchored on `app/` so that the `bom` pattern does not also match
// inside `no-bom/error.css`.
const pattern = new RegExp(`app/${dir}/error\\.css:1:(\\d+)\\b`)
await next.fetch(`/${dir}`)
let column: number | undefined
await retry(async () => {
const match = stripAnsi(next.cliOutput).match(pattern)
expect(match).not.toBeNull()
column = Number(match[1])
})
return column
}
// Both files hold the same invalid `@media (min-width: {})` on line 1 and
// differ only by the leading BOM. The BOM occupies one character position
// in the line the code frame renders, so the reported column for the BOM
// file must be exactly one greater. Without that correction both report
// the same column and the BOM file's caret is rendered one column early.
const withoutBom = await columnFor('no-bom')
const withBom = await columnFor('bom')
expect(withBom).toBe(withoutBom + 1)
})
}
)
@layer bom-layer {
.bom {
color: rgb(0, 0, 255);
}
}
import './bom.css'
export default function Root({ children }: { children: React.ReactNode }) {
return (
<html>
<body>{children}</body>
</html>
)
}
export default function Page() {
return <p className="bom">hello world</p>
}
import { readFileSync } from 'fs'
import { join } from 'path'
import { nextTestSetup } from 'e2e-utils'
describe('app dir - css with a UTF-8 BOM', () => {
const { next } = nextTestSetup({
files: __dirname,
})
it('keeps the BOM in the fixture', () => {
// Sanity check: the regression only happens when the CSS file literally
// starts with the UTF-8 BOM bytes (EF BB BF).
const bytes = readFileSync(join(__dirname, 'app', 'bom.css'))
expect([bytes[0], bytes[1], bytes[2]]).toEqual([0xef, 0xbb, 0xbf])
})
it('compiles and serves a CSS file that starts with a BOM', async () => {
const $ = await next.render$('/')
expect($('p.bom').text()).toBe('hello world')
const hrefs = $('link[rel="stylesheet"]')
.map((_, el) => $(el).attr('href'))
.get()
const inlined = $('style')
.map((_, el) => $(el).text())
.get()
const stylesheets = await Promise.all(
hrefs.map(async (href) => (await next.fetch(href)).text())
)
expect([...stylesheets, ...inlined].join('\n')).toContain('.bom')
})
})
...@@ -402,6 +402,30 @@ pub async fn parse_css( ...@@ -402,6 +402,30 @@ pub async fn parse_css(
.await .await
} }
/// Strips a leading UTF-8 byte-order mark, if present.
fn strip_bom(code: &str) -> &str {
code.strip_prefix('\u{feff}').unwrap_or(code)
}
/// Builds the [`SourcePos`] for a lightningcss error/warning location.
///
/// Lightning CSS counts columns from the start of the string it's given.
/// Since we hand it a BOM-stripped copy, its column numbers on line 0 are one
/// character behind the same position in the original (un-stripped) source
/// that code frames are rendered from. Add that character back when the
/// original code had a BOM and the location is on its first line.
fn source_pos_for_loc(loc: &lightningcss::error::ErrorLocation, code_had_bom: bool) -> SourcePos {
SourcePos {
line: loc.line,
// lightningcss::ErrorLocation is 1-based for column only
column: if code_had_bom && loc.line == 0 {
loc.column
} else {
loc.column - 1
},
}
}
/// Parse a CSS stylesheet and run CSS module validation. /// Parse a CSS stylesheet and run CSS module validation.
/// ///
/// Does not handle parser warnings — the caller is responsible for configuring /// Does not handle parser warnings — the caller is responsible for configuring
...@@ -412,6 +436,9 @@ fn parse_css_stylesheet<'a, 'o>( ...@@ -412,6 +436,9 @@ fn parse_css_stylesheet<'a, 'o>(
ty: CssModuleType, ty: CssModuleType,
source: ResolvedVc<Box<dyn Source>>, source: ResolvedVc<Box<dyn Source>>,
) -> Result<StyleSheet<'a, 'o>, lightningcss::error::Error<lightningcss::error::ParserError<'a>>> { ) -> Result<StyleSheet<'a, 'o>, lightningcss::error::Error<lightningcss::error::ParserError<'a>>> {
// Lightning CSS tokenizes a leading byte-order mark as content instead of
// skipping it, which misaligns the parser and rejects the first token.
let code = strip_bom(code);
let mut ss = StyleSheet::parse(code, config)?; let mut ss = StyleSheet::parse(code, config)?;
if matches!(ty, CssModuleType::Module) { if matches!(ty, CssModuleType::Module) {
...@@ -449,6 +476,8 @@ async fn process_content( ...@@ -449,6 +476,8 @@ async fn process_content(
} }
} }
let code_had_bom = code.starts_with('\u{feff}');
// `@custom-media` is draft syntax and behind a parser flag. // `@custom-media` is draft syntax and behind a parser flag.
// //
// See: https://lightningcss.dev/transpilation.html#custom-media-queries // See: https://lightningcss.dev/transpilation.html#custom-media-queries
...@@ -521,11 +550,7 @@ async fn process_content( ...@@ -521,11 +550,7 @@ async fn process_content(
let issue_source = match &err.loc { let issue_source = match &err.loc {
Some(loc) => IssueSource::from_single_line_col( Some(loc) => IssueSource::from_single_line_col(
source, source,
SourcePos { source_pos_for_loc(loc, code_had_bom),
// lightningcss::ErrorLocation is 1-based for column only
line: loc.line,
column: loc.column - 1,
},
), ),
None => IssueSource::from_source_only(source), None => IssueSource::from_source_only(source),
}; };
...@@ -558,11 +583,7 @@ async fn process_content( ...@@ -558,11 +583,7 @@ async fn process_content(
let issue_source = match &e.loc { let issue_source = match &e.loc {
Some(loc) => IssueSource::from_single_line_col( Some(loc) => IssueSource::from_single_line_col(
source, source,
SourcePos { source_pos_for_loc(loc, code_had_bom),
// lightningcss::ErrorLocation is 1-based for column only
line: loc.line,
column: loc.column - 1,
},
), ),
None => IssueSource::from_source_only(source), None => IssueSource::from_source_only(source),
}; };
...@@ -598,11 +619,7 @@ async fn process_content( ...@@ -598,11 +619,7 @@ async fn process_content(
let issue_source = match &e.loc { let issue_source = match &e.loc {
Some(loc) => IssueSource::from_single_line_col( Some(loc) => IssueSource::from_single_line_col(
source, source,
SourcePos { source_pos_for_loc(loc, code_had_bom),
// lightningcss::ErrorLocation is 1-based for column only
line: loc.line,
column: loc.column - 1,
},
), ),
None => IssueSource::from_source_only(source), None => IssueSource::from_source_only(source),
}; };
...@@ -815,7 +832,7 @@ mod tests { ...@@ -815,7 +832,7 @@ mod tests {
visitor::Visit, visitor::Visit,
}; };
use super::{CssError, CssValidator}; use super::{CssError, CssValidator, source_pos_for_loc, strip_bom};
fn lint_lightningcss(code: &str) -> Vec<CssError> { fn lint_lightningcss(code: &str) -> Vec<CssError> {
let mut ss = StyleSheet::parse( let mut ss = StyleSheet::parse(
...@@ -976,4 +993,43 @@ mod tests { ...@@ -976,4 +993,43 @@ mod tests {
}", }",
); );
} }
#[test]
fn strip_bom_lets_lightningcss_parse() {
let with_bom = "\u{feff}@layer a {}";
assert!(StyleSheet::parse(with_bom, ParserOptions::default()).is_err());
assert!(StyleSheet::parse(strip_bom(with_bom), ParserOptions::default()).is_ok());
assert_eq!(strip_bom("@layer a {}"), "@layer a {}");
}
#[test]
fn source_pos_for_loc_corrects_line_one_column_for_bom_files() {
// Column reported against the BOM-stripped copy we hand to lightningcss.
let loc = lightningcss::error::ErrorLocation {
filename: String::new(),
line: 0,
column: 12,
};
// No BOM: the existing 1-based-to-0-based conversion is unaffected.
assert_eq!(source_pos_for_loc(&loc, false).column, 11);
// With a BOM: add the stripped character back so the position still
// lines up with the original (un-stripped) source that code frames
// are read from.
assert_eq!(source_pos_for_loc(&loc, true).column, 12);
// Only line 0 (the file's first line) is affected — the BOM never
// contains a newline, so later lines are already aligned.
let later_line = lightningcss::error::ErrorLocation {
filename: String::new(),
line: 1,
column: 12,
};
assert_eq!(
source_pos_for_loc(&later_line, true).column,
source_pos_for_loc(&later_line, false).column,
);
}
} }
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment