Unverified Commit c8f9fcb5 authored by Jiwon Choi's avatar Jiwon Choi Committed by GitHub

[15.5.x][ci] Port sccache fallback from #91873/#92911/#93164/#94319 (#94848)

### Why?

Stacked replacement for #94844. This is the top PR in the `next-15-5`
backport stack and is based on #94847.

This was discovered while investigating the fork PR CI failure on
#94829.

Fork PRs to `next-15-5` do not receive `secrets.TURBO_TOKEN`, but the
pinned `ijjk/rust-cache` action throws when `TURBO_TOKEN` is missing.
Canary no longer uses that action for these Rust cache paths; it starts
the local `sccache` action and only configures Vercel Artifacts when a
token is available.

This ports the fork-safe cache behavior from canary without taking the
broader native-builder and workflow rewrites from the source PRs.

Source PRs / ported parts:

- #91873: add the local `.github/actions/sccache` action and move Rust
workflows off `ijjk/rust-cache`.
- #92911: install `cargo-binstall` and use the `vercel/sccache`
cargo-binstall path.
- #93164: keep the Vercel Artifacts token/team wiring used by canary.
- #94319: remove the remaining `build_and_deploy.yml` `ijjk/rust-cache`
step.

### How?

- Add the canary `.github/actions/sccache` action and install
`cargo-binstall` from `setup-rust`.
- Replace the remaining `ijjk/rust-cache` usage in reusable Rust
workflows with the local `sccache` action.
- Remove the remaining `build_and_deploy.yml` rust-cache step so fork
PRs do not load the token-requiring external action.

<!-- NEXT_JS_LLM_PR -->
parent 019fcca5
name: 'sccache with Vercel Artifacts'
description: 'Start sccache with multi-level disk+Vercel Artifacts caching for per-crate Rust compilation caching. Automatically stops and prints stats after the job completes.'
inputs:
base-dir:
description: 'SCCACHE_BASE_DIR for path normalization (default: $GITHUB_WORKSPACE)'
required: false
default: ''
turbo-token:
description: 'Fallback TURBO_TOKEN if not set in environment (e.g. from secrets.TURBO_TOKEN)'
required: false
default: ''
runs:
using: 'node20'
main: 'main.js'
post: 'main.js'
post-if: 'always()'
const { execFileSync } = require('child_process')
const path = require('path')
const fs = require('fs')
const isPost = !!process.env.STATE_isPost
// Signal to the post phase that main ran
fs.appendFileSync(process.env.GITHUB_STATE, 'isPost=true\n')
const script = path.join(__dirname, isPost ? 'stop.sh' : 'start.sh')
try {
execFileSync('bash', [script], { stdio: 'inherit', env: process.env })
} catch (e) {
if (isPost) return // don't fail the job on cleanup
throw e
}
#!/usr/bin/env bash
set -euo pipefail
# Normalize TURBO_API and TURBO_TOKEN: prefer environment values and
# fall back to Vercel's public API and the secret passed as an action input.
if [ -z "${TURBO_API:-}" ]; then
export TURBO_API="https://api.vercel.com"
echo "TURBO_API=${TURBO_API}" >> "$GITHUB_ENV"
fi
if [ -z "${TURBO_TOKEN:-}" ]; then
if [ -n "${INPUT_TURBO_TOKEN:-}" ]; then
export TURBO_TOKEN="${INPUT_TURBO_TOKEN}"
echo "TURBO_TOKEN=${TURBO_TOKEN}" >> "$GITHUB_ENV"
else
echo "WARNING: no TURBO_TOKEN available"
fi
fi
if [ -z "${TURBO_TEAM:-}" ]; then
export TURBO_TEAM="vtest314-next-adapter-e2e-tests"
echo "TURBO_TEAM=${TURBO_TEAM}" >> "$GITHUB_ENV"
fi
echo "::add-mask::${TURBO_TOKEN:-}"
echo "Cache endpoint: ${TURBO_API:0:9}..."
echo "TURBO_TOKEN: ${TURBO_TOKEN:0:3}..."
echo "TURBO_TEAM: ${TURBO_TEAM}"
# Kill stale processes from cancelled builds on self-hosted runners.
sccache --stop-server 2>/dev/null || true
pkill -9 -x cargo 2>/dev/null || true
pkill -9 -x rustc 2>/dev/null || true
# Install vercel/sccache fork via cargo-binstall.
# cargo-binstall is installed by .github/actions/setup-rust.
cargo binstall --no-confirm --git https://github.com/vercel/sccache sccache
sccache --version
# Set env vars for the sccache server (export) and subsequent steps (GITHUB_ENV).
set_env() {
export "$1=$2"
echo "$1=$2" >> "$GITHUB_ENV"
}
# Temporary disable while we work out binstall issue
#set_env RUSTC_WRAPPER sccache
set_env SCCACHE_BASEDIRS "${INPUT_BASE_DIR:-${GITHUB_WORKSPACE}}"
set_env CARGO_INCREMENTAL 0
set_env SCCACHE_IDLE_TIMEOUT 0
set_env SCCACHE_DIR "${HOME}/.sccache"
set_env SCCACHE_RUST_CRATE_TYPE_ALLOW_HASH v1
set_env SCCACHE_ERROR_LOG "${RUNNER_TEMP:-/tmp}/sccache-error.log"
# Gracefully fall back to local compilation if the remote cache is unreachable.
set_env SCCACHE_IGNORE_SERVER_IO_ERROR 1
# Configure remote cache if token is available, otherwise disk-only.
if [ -n "${TURBO_TOKEN:-}" ]; then
set_env SCCACHE_MULTILEVEL_CHAIN "disk,vercel_artifacts"
set_env SCCACHE_VERCEL_ARTIFACTS_ENDPOINT "${TURBO_API}"
set_env SCCACHE_VERCEL_ARTIFACTS_TOKEN "${TURBO_TOKEN}"
set_env SCCACHE_VERCEL_ARTIFACTS_TEAM_SLUG "${TURBO_TEAM}"
fi
# Start the sccache daemon.
echo "SCCACHE_BASEDIRS=${SCCACHE_BASEDIRS}"
sccache --start-server 2>&1 || echo "WARNING: sccache failed to start"
sccache --show-stats 2>&1 | grep -E "Cache location|Base directories" || true
#!/usr/bin/env bash
echo "=== sccache stats ==="
sccache --show-stats || true
sccache --stop-server 2>/dev/null || true
# Show server error log if present
LOG="${RUNNER_TEMP:-/tmp}/sccache-error.log"
if [ -f "$LOG" ]; then
echo "=== sccache error log (last 30 lines) ==="
tail -30 "$LOG"
fi
...@@ -23,6 +23,9 @@ runs: ...@@ -23,6 +23,9 @@ runs:
sudo apt-get -y -o DPkg::Lock::Timeout=60 update sudo apt-get -y -o DPkg::Lock::Timeout=60 update
sudo apt-get -o DPkg::Lock::Timeout=60 -y install lld sudo apt-get -o DPkg::Lock::Timeout=60 -y install lld
- name: 'Install cargo-binstall'
uses: cargo-bins/cargo-binstall@dc19f1e48450eefe5a29b8da6c6b00a87d730b37 # v1.18.1
- name: 'Add cargo problem matchers' - name: 'Add cargo problem matchers'
shell: bash shell: bash
run: echo "::add-matcher::${{ github.action_path }}/matchers.json" run: echo "::add-matcher::${{ github.action_path }}/matchers.json"
...@@ -368,14 +368,6 @@ jobs: ...@@ -368,14 +368,6 @@ jobs:
run: ${{ matrix.settings.setup }} run: ${{ matrix.settings.setup }}
if: ${{ matrix.settings.setup }} if: ${{ matrix.settings.setup }}
- name: Cache on ${{ github.ref_name }}
uses: ijjk/rust-cache@a34594c450817c9860143c79797a8770c4e587f5 # turbo-cache-v1.0.9
if: ${{ !env.NEXT_SKIP_BUILD_CACHE }}
with:
save-if: 'true'
cache-provider: 'turbo'
shared-key: build-${{ matrix.settings.target }}-${{ hashFiles('.cargo/config.toml') }}
- name: Clear native build - name: Clear native build
run: rm -rf packages/next-swc/native run: rm -rf packages/next-swc/native
......
...@@ -210,14 +210,6 @@ jobs: ...@@ -210,14 +210,6 @@ jobs:
- run: corepack prepare --activate yarn@1.22.19 && npm i -g "@napi-rs/cli@${NAPI_CLI_VERSION}" - run: corepack prepare --activate yarn@1.22.19 && npm i -g "@napi-rs/cli@${NAPI_CLI_VERSION}"
- name: Cache on ${{ github.ref_name }}
uses: ijjk/rust-cache@a34594c450817c9860143c79797a8770c4e587f5 # turbo-cache-v1.0.9
if: ${{ inputs.rustCacheKey }}
with:
cache-provider: 'turbo'
save-if: ${{ github.ref_name == 'canary' }}
shared-key: ${{ inputs.rustCacheKey }}-${{ inputs.buildNativeTarget }}-build-${{ inputs.rustBuildProfile }}-${{ hashFiles('.cargo/config.toml') }}
# clean up any previous artifacts to avoid hitting disk space limits # clean up any previous artifacts to avoid hitting disk space limits
- run: git clean -xdf && rm -rf /tmp/next-repo-*; rm -rf /tmp/next-install-* /tmp/yarn-* /tmp/ncc-cache target - run: git clean -xdf && rm -rf /tmp/next-repo-*; rm -rf /tmp/next-install-* /tmp/yarn-* /tmp/ncc-cache target
...@@ -234,6 +226,12 @@ jobs: ...@@ -234,6 +226,12 @@ jobs:
- run: node scripts/normalize-version-bump.js - run: node scripts/normalize-version-bump.js
name: normalize versions name: normalize versions
- name: Start sccache
uses: ./.github/actions/sccache
if: ${{ runner.os != 'Windows' && (inputs.skipNativeBuild != 'yes' || inputs.needsNextest == 'yes' || inputs.needsRust == 'yes') }}
with:
turbo-token: ${{ secrets.TURBO_TOKEN }}
- run: pnpm dlx turbo@${TURBO_VERSION} run build-native-${{ inputs.rustBuildProfile }} -v --env-mode loose --remote-cache-timeout 90 --summarize -- --target ${{ inputs.buildNativeTarget }} - run: pnpm dlx turbo@${TURBO_VERSION} run build-native-${{ inputs.rustBuildProfile }} -v --env-mode loose --remote-cache-timeout 90 --summarize -- --target ${{ inputs.buildNativeTarget }}
if: ${{ inputs.skipNativeBuild != 'yes' }} if: ${{ inputs.skipNativeBuild != 'yes' }}
......
...@@ -49,6 +49,11 @@ jobs: ...@@ -49,6 +49,11 @@ jobs:
- run: pnpm install - run: pnpm install
working-directory: turbopack/benchmark-apps working-directory: turbopack/benchmark-apps
- name: Start sccache
uses: ./.github/actions/sccache
with:
turbo-token: ${{ secrets.TURBO_TOKEN }}
- name: Build benchmarks for tests - name: Build benchmarks for tests
timeout-minutes: 120 timeout-minutes: 120
run: | run: |
......
...@@ -43,6 +43,11 @@ jobs: ...@@ -43,6 +43,11 @@ jobs:
with: with:
tool: cargo-codspeed@2.10.1 tool: cargo-codspeed@2.10.1
- name: Start sccache
uses: ./.github/actions/sccache
with:
turbo-token: ${{ secrets.TURBO_TOKEN }}
- name: Build app build benchmarks - name: Build app build benchmarks
run: cargo codspeed build -p next-api run: cargo codspeed build -p next-api
...@@ -66,12 +71,10 @@ jobs: ...@@ -66,12 +71,10 @@ jobs:
with: with:
tool: cargo-codspeed@2.10.1 tool: cargo-codspeed@2.10.1
- name: Cache on ${{ github.ref_name }} - name: Start sccache
uses: ijjk/rust-cache@a34594c450817c9860143c79797a8770c4e587f5 # turbo-cache-v1.0.9 uses: ./.github/actions/sccache
with: with:
save-if: 'true' turbo-token: ${{ secrets.TURBO_TOKEN }}
cache-provider: 'turbo'
shared-key: build-turbopack-benchmark-small-apps-${{ hashFiles('.cargo/config.toml') }}
- name: Install pnpm dependencies - name: Install pnpm dependencies
working-directory: turbopack/benchmark-apps working-directory: turbopack/benchmark-apps
...@@ -104,6 +107,11 @@ jobs: ...@@ -104,6 +107,11 @@ jobs:
with: with:
tool: cargo-codspeed@2.10.1 tool: cargo-codspeed@2.10.1
- name: Start sccache
uses: ./.github/actions/sccache
with:
turbo-token: ${{ secrets.TURBO_TOKEN }}
- name: Build the benchmark target(s) - name: Build the benchmark target(s)
run: cargo codspeed build -p turbopack -p turbopack-bench run: cargo codspeed build -p turbopack -p turbopack-bench
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment