diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 7529d8eb3b1..d754e21eb4e 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -34,7 +34,11 @@ Each package includes a DESIGN.md file, read that to gain a general understandin ### Example projects -- `examples/todo-app/` — A simple To-Do app demonstrating FAST usage patterns. +- `examples/design-system/` — Shared semantic design tokens (`@microsoft/fast-examples-design-system`) consumed by all example apps. +- `examples/csr/todo-app/` — A simple To-Do app demonstrating FAST usage patterns and using the shared `@microsoft/fast-examples-design-system` tokens. +- `examples/csr/todo-mobx-app/` — A To-Do app demonstrating how to integrate MobX state with `@microsoft/fast-element` using a single `autorun` per component (no custom bridge code). +- `examples/ssr/chat-app/` — A declarative FAST chat demo that pre-renders with `@microsoft/fast-build` and streams canned assistant replies into the transcript. Dark theme. +- `examples/ssr/webui-todo-app/` — A To-Do app demonstrating FAST declarative HTML with `@microsoft/webui` prerendering and `@microsoft/fast-element`'s `declarativeTemplate()` + `enableHydration()` for client-side hydration. Light theme. ## Skills diff --git a/.github/workflows/README.md b/.github/workflows/README.md index a41fb9b0330..20651bb431e 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -9,4 +9,17 @@ For more information see the [GitHub CLI documentation](https://cli.github.com/m All CI workflows that run against pull requests are configured to skip draft PRs: - **GitHub Actions** (`ci-validate-pr.yml`, `ci-validate-platforms.yml`, `ci-validate-rust.yml`): The `pull_request` trigger includes `ready_for_review` in its event types, and each job has a condition that skips execution when the PR is a draft. When a draft PR is marked as ready for review, the workflows will automatically trigger. -- **Azure Pipelines** (`azure-pipelines-bench.yml`, `azure-pipelines-ci.yml`): The `pr` trigger uses `drafts: false` to prevent pipeline runs on draft PRs. \ No newline at end of file +- **Azure Pipelines** (`azure-pipelines-bench.yml`, `azure-pipelines-ci.yml`): The `pr` trigger uses `drafts: false` to prevent pipeline runs on draft PRs. + +## Continuous Deployment + +Nightly publishing is split into two coordinated jobs so that npm credentials never leave the Azure environment. GitHub Releases are the source of truth, and `deployed/` git marker tags track which releases have already been published. + +- **`cd-github-releases.yml`** (GitHub Actions) runs nightly via cron (`0 8 * * *` UTC, ~12am PST) and on `workflow_dispatch`. It does **not** bump versions or push source changes to `main` — version bumps land on `main` through ordinary human-authored pull requests (for example, by running `npm run bump` locally and opening a PR). The cron is scheduled ~1 hour before the Azure CD pipeline (09:00 UTC) so any GitHub releases this job creates are picked up by that same night's publish run. The workflow has two jobs: + 1. **`detect`** — checks out `main` with `fetch-depth: 0` and runs [`build/scripts/create-github-releases.mjs --check-only`](../../build/scripts/create-github-releases.mjs). The script walks the workspaces tree (no `npm ci` required), computes `${name}_v${version}` for every non-private workspace, and emits `hasMissingReleases=true` if any of those git tags do not yet exist. + 2. **`release`** runs only when missing releases exist. Installs Node, the Rust toolchain (for `cargo package`), and the npm workspace dependencies, builds the repo, then runs the script in default mode. For every missing release the script: packs the npm tarball into `publish_artifacts_npm/`, packs the paired Rust crate (if `crates//Cargo.toml` exists) into `publish_artifacts_crates/`, and creates the GitHub release with both assets attached via `gh release create --target `. The `gh` CLI creates the git tag atomically with the release, so "tag exists" and "release exists" are always the same fact — a failed release is safely retried on the next workflow run, with no orphan tag stranded behind. The script errors if a paired crate's version does not match the npm package's version — but this is purely a safety net: the [`postbump` hook in `beachball.config.js`](../../beachball.config.js) rewrites the crate's `Cargo.toml` (and the matching entry in `Cargo.lock`) automatically whenever `npm run bump` bumps the paired npm package, so the two stay in sync from the same commit. +- **`azure-pipelines-cd.yml`** (Azure Pipelines) runs every night at **1am PST (`0 9 * * *` UTC)** with `always: true` so it still runs on no-op nights (it is checking external GitHub state, not repo commits). It is split into two stages so the heavy publish work is skipped on no-op nights: + 1. **`Check`** — runs [`build/scripts/download-github-releases.mjs --check-only`](../../build/scripts/download-github-releases.mjs). The script lists every git tag matching the beachball-style `${name}_v${version}` pattern that does **not** also have a `deployed/` counterpart and emits the comma-separated list via Azure Pipelines `setvariable` output variables (`needsDeployment`, `undeployedTags`). No network calls to npm or crates.io. + 2. **`Package`** — depends on `Check` and runs only when `needsDeployment == 'true'`. Downloads every undeployed release's assets via `gh release download`, sorts them into `publish_artifacts_npm/` (`.tgz`) and `publish_artifacts_crates/` (`.crate`), hands off to `FAST.Release.PipelineTemplate.yml@fastPipelines` for the actual `npm publish` / `cargo publish`, and on success pushes a `deployed/` git marker tag for each release that was just published. The next nightly run will see those markers and skip the corresponding releases. + +Both scripts are thin Node.js wrappers around `gh`, `npm`, `cargo`, and `git` — no extra npm dependencies and no custom GitHub API client. Idempotency is enforced entirely through git tags (`${name}_v${version}` on the GitHub side, `deployed/${name}_v${version}` on the Azure side), so neither side needs to talk to npm.org or crates.io to decide whether work is required. \ No newline at end of file diff --git a/.github/workflows/cd-github-releases.yml b/.github/workflows/cd-github-releases.yml new file mode 100644 index 00000000000..b9a4f5344a0 --- /dev/null +++ b/.github/workflows/cd-github-releases.yml @@ -0,0 +1,78 @@ +name: Release packages to GitHub releases + +on: + workflow_dispatch: + schedule: + # 12am PST = 08:00 UTC (drifts to ~11pm PDT during US daylight time). + # Runs ~1 hour before the Azure CD pipeline (09:00 UTC) so any new + # GitHub releases this job creates are ready for that night's publish. + - cron: '0 8 * * *' + +permissions: { contents: write } + +jobs: + # Lightweight detection: walks the workspaces directory tree (no `npm ci`, + # no build, no `gh` API call) and checks whether each `${name}_v${version}` + # tag already exists in git. The downstream `release` job is skipped if + # every current version already has a tag. + detect: + runs-on: ubuntu-latest + outputs: + hasMissingReleases: ${{ steps.check.outputs.hasMissingReleases }} + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # need all tags for `git rev-parse refs/tags/`. + + - uses: actions/setup-node@v6 + with: + node-version: 22 + + - id: check + name: Detect publishable workspaces without a release tag + run: node build/scripts/create-github-releases.mjs --check-only + + release: + needs: detect + if: needs.detect.outputs.hasMissingReleases == 'true' + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # need all tags so `git rev-parse` can detect existing releases. + + - uses: actions/setup-node@v6 + with: + node-version: 22 + + - name: Cache multiple paths + uses: actions/cache@v4 + env: + cache-name: cache-node-modules + with: + path: ~/.npm + key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }} + restore-keys: | + ${{ runner.os }}-build-${{ env.cache-name }}- + ${{ runner.os }}-build- + ${{ runner.os }}- + + - name: Install package dependencies + run: npm ci + + - name: Install Rust and wasm-pack + run: | + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable + export PATH="$HOME/.cargo/bin:$PATH" + rustup target add wasm32-unknown-unknown + cargo install wasm-pack + echo "$HOME/.cargo/bin" >> "$GITHUB_PATH" + + - name: Build workspaces + run: npm run build + + - name: Pack and create GitHub releases + run: node build/scripts/create-github-releases.mjs + env: + GH_TOKEN: ${{ secrets.GH_TOKEN }} diff --git a/.github/workflows/ci-validate-pr.yml b/.github/workflows/ci-validate-pr.yml index 80612cbecd5..f400ab514cc 100644 --- a/.github/workflows/ci-validate-pr.yml +++ b/.github/workflows/ci-validate-pr.yml @@ -67,5 +67,8 @@ jobs: - name: Testing unit tests run: npx lage test:node test:chromium ${{ github.event_name == 'pull_request' && format('--since origin/{0}', github.event.pull_request.base.ref) || '' }} --allow-no-target-runs --verbose + - name: End-to-end tests for example apps + run: npm run test:e2e + - name: Testing final validation run: npm run test:validation diff --git a/.gitignore b/.gitignore index 85b932b26ca..413d124619d 100644 --- a/.gitignore +++ b/.gitignore @@ -71,7 +71,10 @@ obj/ test-results/ # Pack directory -publish_artifacts/ +publish_artifacts_npm/ +publish_artifacts_crates/ +publish_artifacts_meta/ +publish_artifacts_stage/ # Rust build artifacts crates/**/target/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b95744d0e5f..d971b6e000b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -146,6 +146,83 @@ Example of how to format `MIGRATION.md`: - `Bat` has been updated to use the new API [`BatConfig`](link/to/api). ``` +### Publishing + +Releases are produced from a dedicated **bump pull request** authored by a maintainer (not by CI). Once the bump PR lands on `main`, the [`cd-github-releases.yml`](.github/workflows/cd-github-releases.yml) workflow attaches the freshly-packed tarballs to a GitHub release per package, and the nightly Azure pipeline ([`azure-pipelines-cd.yml`](azure-pipelines-cd.yml)) downloads those assets and publishes them to npm and crates.io. The detailed CD design is documented in [`.github/workflows/README.md`](.github/workflows/README.md). + +This section covers the maintainer workflow for opening the bump PR. + +#### 1. Prerequisites + +Every PR that touches a publishable workspace should include a beachball [change file](#change-files). Bump PRs consume the accumulated change files; if a workspace needs to be bumped but has no change file (e.g. on first release, or because a contributor forgot to add one), generate one yourself with `npm run change` before bumping. + +#### 2. Create the bump branch + +```bash +git checkout main +git pull origin main +git checkout -b chore/bump-$(date +%Y-%m-%d) +``` + +#### 3. Run `npm run bump` + +```bash +npm run bump +``` + +This single command: + +- Reads every file under `change/`. +- Bumps `package.json` and updates `CHANGELOG.md` / `CHANGELOG.json` for every affected workspace. +- Runs the `postbump` hook in [`beachball.config.js`](beachball.config.js) — for any bumped npm package that has a paired Rust crate at `crates//Cargo.toml` (where `` is the npm name with the leading `@` dropped and `/` replaced with `-`, e.g. `@microsoft/fast-build` → `microsoft-fast-build`), the hook rewrites the crate's `Cargo.toml` and `Cargo.lock` to the new version so they stay in lock-step with the npm package. +- Deletes the consumed change files. + +No commit, push, npm publish, or git tag is made by `npm run bump`. + +#### 4. Review the result + +```bash +git status +git diff +node build/scripts/create-github-releases.mjs --check-only +``` + +The third command previews exactly which workspaces the post-merge CD will publish, by listing every workspace whose freshly-bumped `${name}_v${version}` tag is not present in the local git tag list. Run `git fetch --tags --prune origin` beforehand if you want the preview to reflect the current state on `origin` rather than your stale local refs — though for a fresh bump that hasn't been pushed yet, your local tag list is the source of truth anyway. + +A typical bump PR touches: + +- `packages//package.json` (`version` field) for each bumped workspace +- `packages//CHANGELOG.md` and `CHANGELOG.json` for each bumped workspace +- `crates//Cargo.toml` and `crates//Cargo.lock` for any paired Rust crate (handled automatically by the postbump hook) +- Deletions under `change/` + +#### 5. Open the PR + +```bash +git add -A +git commit -m "chore: release packages" +git push origin chore/bump-$(date +%Y-%m-%d) +gh pr create --fill --base main +``` + +The bump PR goes through normal review. `npm run checkchange` will pass because the change files have all been consumed; the PR itself does **not** publish anything. + +:::note +Do not edit `package.json` or `Cargo.toml` versions by hand. Let `npm run bump` and the postbump hook do it. [`create-github-releases.mjs`](build/scripts/create-github-releases.mjs) refuses to release a workspace whose npm version and paired crate version disagree. +::: + +#### 6. After merge + +After merge, [`cd-github-releases.yml`](.github/workflows/cd-github-releases.yml) runs on its nightly cron (`0 8 * * *` UTC, ~12am PST) — or you can trigger it immediately via `gh workflow run cd-github-releases.yml` if you don't want to wait. Its `detect` job notices the new `${name}_v${version}` tags don't yet exist; the `release` job packs each `.tgz` (and any paired `.crate`) and atomically creates one GitHub release per bumped package. The next nightly run of [`azure-pipelines-cd.yml`](azure-pipelines-cd.yml) (scheduled ~1 hour later at 09:00 UTC) downloads those assets, hands off to `FAST.Release.PipelineTemplate` for the actual `npm publish` / `cargo publish`, and on success pushes `deployed/` marker tags so the publish is never repeated. + +#### Hotfix or single-package bump + +The same flow works for a single-package release — just keep only the relevant change file(s) before running `npm run bump`, or pass `--package` to beachball directly: + +```bash +npx beachball bump --package "@microsoft/fast-build" +``` + ### Recommended Settings for Visual Studio Code You can use any code editor you like when working with the FAST monorepo. One of our favorites is [Visual Studio Code](https://code.visualstudio.com/). VS Code has great autocomplete support for TypeScript and JavaScript APIs, as well as a rich ecosystem of plugins. diff --git a/README.md b/README.md index 62a143a7ea7..ae9ed061c2b 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,7 @@ For an in-depth explanation of FAST [see our docs introduction](https://fast.des [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) [![npm version](https://badge.fury.io/js/%40microsoft%2Ffast-element.svg)](https://badge.fury.io/js/%40microsoft%2Ffast-element) -The `@microsoft/fast-element` library is a lightweight means to easily build performant, memory-efficient, standards-compliant Web Components. FAST Elements work in every major browser and can be used in combination with any front-end framework or even without a framework. To get up and running with `@microsoft/fast-element` see [the Getting Started guide](https://fast.design/docs/fast-element/getting-started). +The `@microsoft/fast-element` library is a lightweight means to easily build performant, memory-efficient, standards-compliant Web Components. FAST Elements work in every major browser and can be used in combination with any front-end framework or even without a framework. To get up and running with `@microsoft/fast-element` see [the Getting Started guide](https://fast.design/docs/2.x/getting-started/quick-start/). ### `@fluentui/web-components` @@ -46,13 +46,13 @@ The source for `@fluentui/web-components` is hosted in [the Fluent UI monorepo]( We hope you're excited by the possibilities that FAST presents. But, you may be wondering where to start. Here are a few statements that describe various members of our community. We recommend that you pick the statement you most identify with and follow the links where they lead. You can always come back and explore another topic at any time. * "I just want ready-made components!" - * [Check out the FluentUI Web Components.](https://docs.microsoft.com/en-us/fluent-ui/web-components/) + * [Check out the FluentUI Web Components.](https://learn.microsoft.com/en-us/fluent-ui/web-components/) * "I want to build my own components." - * [Jump to the fast-element docs.](https://fast.design/docs/getting-started/quick-start) + * [Jump to the fast-element docs.](https://fast.design/docs/2.x/getting-started/quick-start/) * "I need to integrate FAST with another framework or build system." - * [Jump to the integration docs.](https://fast.design/docs/integrations) + * [Jump to the integration docs.](https://fast.design/docs/2.x/integrations/) * "I want to look at a quick reference." - * [Jump to the Cheat Sheet](https://fast.design/docs/1.x/resources/cheat-sheet) + * [Jump to the Cheat Sheet](https://fast.design/docs/1.x/resources/cheat-sheet) *(only available in 1.x)* ## Roadmap diff --git a/azure-pipelines-cd.yml b/azure-pipelines-cd.yml index a9cf28778b4..6845c1fe920 100644 --- a/azure-pipelines-cd.yml +++ b/azure-pipelines-cd.yml @@ -2,11 +2,12 @@ trigger: none pr: none schedules: -- cron: '0 0 * * *' - displayName: Daily Release +- cron: '0 9 * * *' # 09:00 UTC daily (~1am Pacific in standard time, ~2am during DST). + displayName: Daily npm/crates publish from GitHub Releases branches: include: - main + always: true # run even when there are no new commits — we are checking external GitHub releases, not repo changes. # The `resources` specify the location and version of the 1ES PT. resources: @@ -39,21 +40,57 @@ extends: settings: networkIsolationPolicy: Permissive stages: - - stage: Stage + + # ── Stage 1: Lightweight detection (runs every night, ~1-2 min) ── + # This stage is separate so that when no deployment is needed, the + # heavyweight Package stage (with all its 1ES SDL tasks) is skipped + # entirely — avoiding ~30 min of unnecessary agent provisioning and + # compliance scans on no-op nights. + - stage: Check + displayName: Detect undeployed GitHub releases jobs: - - job: HostJob + - job: CheckVersion + steps: + - checkout: self + persistCredentials: "true" + fetchTags: true + + - task: UseNode@1 + inputs: + version: "22.x" + displayName: "Install Node.js" - variables: - npm_config_cache: $(Pipeline.Workspace)/.npm + # The check uses local `git tag` rather than `npm view` / `cargo search` + # because external network calls from 1ES agents to npm/crates.io are + # unreliable. The Package stage pushes a `deployed/` marker tag + # after a successful publish, and the next pipeline run sees that + # marker via `git tag -l`. + - script: | + node build/scripts/download-github-releases.mjs --check-only + displayName: "Detect releases whose tarballs have not been published" + name: deploymentCheck + env: + GITHUB_REPOSITORY: microsoft/fast + # ── Stage 2: Download, publish, and mark deployed (skipped on no-op nights) ── + - stage: Package + displayName: Publish tarballs to npm and crates.io + dependsOn: Check + condition: eq(dependencies.Check.outputs['CheckVersion.deploymentCheck.needsDeployment'], 'true') + variables: + npm_config_cache: $(Pipeline.Workspace)/.npm + jobs: + - job: Deploy steps: - checkout: self persistCredentials: "true" + fetchTags: true - script: | git config --global user.email fastsvc@microsoft.com git config --global user.name "Microsoft FAST Builds" git remote set-url origin https://$(GH_TOKEN)@github.com/microsoft/fast.git + displayName: "Configure git for tag push" - task: UseNode@1 inputs: @@ -73,21 +110,38 @@ extends: displayName: "Install package dependencies" - script: | - curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable - export PATH="$HOME/.cargo/bin:$PATH" - echo "##vso[task.prependpath]$HOME/.cargo/bin" - rustup target add wasm32-unknown-unknown - cargo install wasm-pack - displayName: "Install Rust and wasm-pack" - - - script: | - npm run build - displayName: "Build workspaces" - - - script: | - npm run publish-ci - displayName: "Use Beachball publish to bump, pack, and commit without publishing" + node build/scripts/download-github-releases.mjs + displayName: "Download undeployed release assets and sort into npm/crates folders" env: GH_TOKEN: $(GH_TOKEN) + GITHUB_REPOSITORY: microsoft/fast - template: FAST.Release.PipelineTemplate.yml@fastPipelines # Template reference + + # Push a `deployed/` marker tag for each release that was just + # published so that the next nightly Check stage sees it and skips + # the release. Reads the list of tags written by the download script. + # Idempotent: a marker tag that already exists locally (i.e. fetched + # from origin via `fetchTags: true`) is left alone instead of failing. + - script: | + set -euo pipefail + META_FILE=publish_artifacts_meta/undeployed-tags.txt + if [ ! -s "$META_FILE" ]; then + echo "No tags to mark as deployed." + exit 0 + fi + while IFS= read -r tag; do + [ -z "$tag" ] && continue + DEPLOY_TAG="deployed/${tag}" + if git rev-parse --verify --quiet "refs/tags/${DEPLOY_TAG}" >/dev/null; then + echo "Already marked deployed: ${DEPLOY_TAG} (skipping)" + continue + fi + echo "Marking deployed: ${DEPLOY_TAG}" + # Point the marker at the release tag's commit (not the + # agent's current HEAD) so the marker stays accurate even + # if the pipeline is re-run from a later commit. + git tag "${DEPLOY_TAG}" "refs/tags/${tag}" + git push origin "${DEPLOY_TAG}" + done < "$META_FILE" + displayName: "Mark releases as deployed" diff --git a/beachball.config.js b/beachball.config.js index d89d4006d30..993182ac7f0 100644 --- a/beachball.config.js +++ b/beachball.config.js @@ -1,3 +1,110 @@ +const { existsSync, readFileSync, writeFileSync } = require("node:fs"); +const { join } = require("node:path"); + +/** + * Convert an npm package name into the paired Rust crate name by dropping + * the leading `@` and replacing `/` with `-`. + * + * Example: `@microsoft/fast-build` -> `microsoft-fast-build`. + */ +function npmNameToCrateName(npmName) { + return npmName.replace(/^@/, "").replace(/\//g, "-"); +} + +/** + * Rewrite the `version = "..."` line for a specific `[package]` or + * `[[package]]` block (matched by the crate name) within Cargo TOML + * content. Returns the new content, or `null` if no change was made. + * + * Hand-rolled (instead of using a TOML library) so beachball.config.js + * does not pull in any new runtime dependencies. + */ +function rewriteCargoVersion(content, crateName, newVersion, { manifest }) { + const lines = content.split("\n"); + let inTargetBlock = false; + let nameMatched = manifest; + let changed = false; + + for (let i = 0; i < lines.length; i++) { + const trimmed = lines[i].trim(); + + if (trimmed.startsWith("[")) { + if (manifest) { + inTargetBlock = trimmed === "[package]"; + } else { + inTargetBlock = trimmed === "[[package]]"; + nameMatched = false; + } + continue; + } + + if (!inTargetBlock) continue; + + if (!manifest && !nameMatched) { + const nameMatch = /^name\s*=\s*"([^"]+)"/.exec(trimmed); + if (nameMatch && nameMatch[1] === crateName) { + nameMatched = true; + continue; + } + } + + if (nameMatched) { + const versionMatch = /^(\s*version\s*=\s*")([^"]+)(")/.exec(lines[i]); + if (versionMatch) { + if (versionMatch[2] !== newVersion) { + lines[i] = `${versionMatch[1]}${newVersion}${versionMatch[3]}`; + changed = true; + } + if (manifest) break; + inTargetBlock = false; + nameMatched = false; + } + } + } + + return changed ? lines.join("\n") : null; +} + +/** + * Beachball `postbump` hook: when an npm package with a paired Rust + * crate is bumped, rewrite the crate's `Cargo.toml` (and matching entry + * in `Cargo.lock`, if present) so the crate version stays in lock-step + * with the npm version. + * + * Beachball commits any modified files in the same bump commit, so the + * Cargo updates land in the same PR as the package.json bump. + */ +function syncPairedCrateVersion(packagePath, name, version) { + const crateName = npmNameToCrateName(name); + const cargoTomlPath = join(__dirname, "crates", crateName, "Cargo.toml"); + if (!existsSync(cargoTomlPath)) return; + + const updatedToml = rewriteCargoVersion( + readFileSync(cargoTomlPath, "utf8"), + crateName, + version, + { manifest: true }, + ); + if (updatedToml !== null) { + writeFileSync(cargoTomlPath, updatedToml); + console.log(`[beachball] Synced ${cargoTomlPath} to ${version}`); + } + + const cargoLockPath = join(__dirname, "crates", crateName, "Cargo.lock"); + if (existsSync(cargoLockPath)) { + const updatedLock = rewriteCargoVersion( + readFileSync(cargoLockPath, "utf8"), + crateName, + version, + { manifest: false }, + ); + if (updatedLock !== null) { + writeFileSync(cargoLockPath, updatedLock); + console.log(`[beachball] Synced ${cargoLockPath} to ${version}`); + } + } +} + module.exports = { ignorePatterns: [ ".ignore", @@ -11,5 +118,7 @@ module.exports = { // This one is especially important (otherwise dependabot would be blocked by change file requirements) "package-lock.json", ], - packToPath: "publish_artifacts" + hooks: { + postbump: syncPairedCrateVersion, + }, }; diff --git a/build/scripts/create-github-releases.mjs b/build/scripts/create-github-releases.mjs new file mode 100644 index 00000000000..aad4c2a078b --- /dev/null +++ b/build/scripts/create-github-releases.mjs @@ -0,0 +1,289 @@ +#!/usr/bin/env node +/** + * Create one GitHub release per non-private workspace whose + * `${name}_v${version}` tag does not yet exist in git. + * + * Invoked by `.github/workflows/cd-github-releases.yml`. The workflow does + * NOT bump versions or commit source changes — version bumps land on `main` + * through ordinary human-authored pull requests. This script: + * + * 1. Walks the root `package.json` `workspaces` globs to find every + * workspace's `package.json` (no `node_modules` required, so the + * `--check-only` mode can run before `npm ci`). + * 2. Skips workspaces whose package.json sets `private: true`. + * 3. For each remaining workspace, looks for a paired Rust crate at + * `crates//Cargo.toml`, where the crate name is derived + * from the npm name by dropping the leading `@` and replacing `/` + * with `-` (so `@microsoft/fast-build` -> `microsoft-fast-build`). + * When a pair exists, the script errors if the two versions are not + * identical, forcing the version-bump PR to keep them in sync. + * 4. Computes `tag = ${name}_v${version}` (matching beachball's tag + * format) and skips the workspace if the git tag already exists + * (idempotent across re-runs). + * 5. Otherwise (when not `--check-only`) packs the npm tarball into + * `publish_artifacts_npm/` with `npm pack`, optionally packs the + * paired crate into `publish_artifacts_crates/` with + * `cargo package`, and creates the GitHub release with both + * assets attached (`gh release create --target `). The `gh` + * CLI creates the git tag atomically with the release, so the + * tag and the release exist if and only if each other does. + * + * Modes: + * + * - default: pack + tag + create any missing releases. + * - `--check-only`: only enumerate missing releases. Sets the + * `hasMissingReleases` GitHub Actions output (via `$GITHUB_OUTPUT`) + * to `"true"` or `"false"`. Performs no packing, no tagging, no + * release creation. Safe to run without `node_modules` populated. + * + * Authentication: the `gh` CLI reads `GH_TOKEN` from the environment. + */ + +import { execFileSync } from "node:child_process"; +import { + appendFileSync, + copyFileSync, + existsSync, + mkdirSync, + readdirSync, + readFileSync, +} from "node:fs"; +import { basename, join, resolve } from "node:path"; + +const NPM_DIR = "publish_artifacts_npm"; +const CRATES_DIR = "publish_artifacts_crates"; +const CHECK_ONLY = process.argv.includes("--check-only"); + +function run(file, args, opts = {}) { + return execFileSync(file, args, { encoding: "utf8", ...opts }); +} + +function gitTagExists(tag) { + try { + execFileSync("git", ["rev-parse", "--verify", `refs/tags/${tag}`], { + stdio: "ignore", + }); + return true; + } catch { + return false; + } +} + +function npmNameToCrateName(npmName) { + return npmName.replace(/^@/, "").replace(/\//g, "-"); +} + +function readCargoTomlVersion(cargoTomlPath) { + const content = readFileSync(cargoTomlPath, "utf8"); + let inPackage = false; + for (const rawLine of content.split("\n")) { + const line = rawLine.trim(); + if (line.startsWith("[")) { + inPackage = line === "[package]"; + continue; + } + if (!inPackage) continue; + const m = /^version\s*=\s*"([^"]+)"/.exec(line); + if (m) return m[1]; + } + return null; +} + +function listPublishableWorkspaces() { + const rootPkg = JSON.parse(readFileSync("package.json", "utf8")); + const patterns = rootPkg.workspaces || []; + const locations = new Set(); + + for (const pattern of patterns) { + if (pattern.endsWith("/*")) { + const parent = pattern.slice(0, -2); + if (!existsSync(parent)) continue; + for (const entry of readdirSync(parent, { withFileTypes: true })) { + if (entry.isDirectory()) { + locations.add(join(parent, entry.name)); + } + } + } else { + locations.add(pattern); + } + } + + const workspaces = []; + for (const location of locations) { + const pkgPath = join(location, "package.json"); + if (!existsSync(pkgPath)) continue; + const pkg = JSON.parse(readFileSync(pkgPath, "utf8")); + if (pkg.private === true) continue; + if (!pkg.name || !pkg.version) continue; + + const crateName = npmNameToCrateName(pkg.name); + const cargoTomlPath = join("crates", crateName, "Cargo.toml"); + const hasCrate = existsSync(cargoTomlPath); + + if (hasCrate) { + const crateVersion = readCargoTomlVersion(cargoTomlPath); + if (crateVersion !== pkg.version) { + throw new Error( + `Version mismatch for ${pkg.name}: package.json is ${pkg.version} but ${cargoTomlPath} is ${crateVersion}. Update one to match the other.`, + ); + } + } + + workspaces.push({ + location, + name: pkg.name, + version: pkg.version, + crateName: hasCrate ? crateName : null, + cargoTomlPath: hasCrate ? cargoTomlPath : null, + }); + } + + return workspaces; +} + +function setGitHubOutput(name, value) { + if (!process.env.GITHUB_OUTPUT) return; + appendFileSync(process.env.GITHUB_OUTPUT, `${name}=${value}\n`); +} + +const publishable = listPublishableWorkspaces(); + +if (publishable.length === 0) { + console.log("No publishable workspaces found."); + setGitHubOutput("hasMissingReleases", "false"); + process.exit(0); +} + +const missing = publishable.filter( + ({ name, version }) => !gitTagExists(`${name}_v${version}`), +); + +console.log(`Publishable workspaces: ${publishable.length}`); +console.log(`With existing git tag: ${publishable.length - missing.length}`); +console.log(`Missing git tag / release: ${missing.length}`); + +if (missing.length > 0) { + console.log("\nPackages that need a release:"); + for (const { name, version, crateName } of missing) { + const suffix = crateName ? ` (+ crate ${crateName})` : ""; + console.log(` - ${name}@${version}${suffix}`); + } +} + +setGitHubOutput("hasMissingReleases", missing.length > 0 ? "true" : "false"); + +if (CHECK_ONLY || missing.length === 0) { + process.exit(0); +} + +// `--check-only` only enumerates missing releases via local git state, +// but creating releases requires `gh release create`, which needs a +// token. Fail fast here so the workflow surfaces a clear error rather +// than a generic `gh` auth failure mid-loop. +if (!process.env.GH_TOKEN) { + console.error("GH_TOKEN must be set so the `gh` CLI can create GitHub releases."); + process.exit(1); +} + +mkdirSync(NPM_DIR, { recursive: true }); +mkdirSync(CRATES_DIR, { recursive: true }); + +let created = 0; +let hasErrors = false; + +for (const { name, version, location, crateName, cargoTomlPath } of missing) { + const tag = `${name}_v${version}`; + const assets = []; + + try { + console.log(`\nPacking ${name}@${version} from ${location}...`); + const packJson = run("npm", [ + "pack", + "--silent", + "--json", + `--workspace=${location}`, + `--pack-destination=${resolve(NPM_DIR)}`, + ]); + assets.push(join(NPM_DIR, JSON.parse(packJson)[0].filename)); + + if (cargoTomlPath) { + console.log(`Packaging crate ${crateName}@${version}...`); + run( + "cargo", + [ + "package", + "--no-verify", + "--allow-dirty", + "--manifest-path", + cargoTomlPath, + ], + { stdio: "inherit" }, + ); + const srcCrate = join( + "crates", + crateName, + "target", + "package", + `${crateName}-${version}.crate`, + ); + if (!existsSync(srcCrate)) { + throw new Error( + `Expected ${srcCrate} after cargo package, but it does not exist.`, + ); + } + const destCrate = join(CRATES_DIR, basename(srcCrate)); + copyFileSync(srcCrate, destCrate); + assets.push(destCrate); + } + + const notes = [ + `Nightly release for \`${name}@${version}\`.`, + "", + "Version bumps were landed via a regular pull request. The attached", + "assets will be downloaded and published to npm" + + (cargoTomlPath ? " and crates.io" : "") + + " by the nightly Azure release pipeline.", + ].join("\n"); + + // Let `gh release create` create the tag atomically with the + // release so that "tag exists" and "release exists" are always + // the same fact. If we created the tag separately and pushed + // it, and then `gh release create` failed, the next workflow + // run would think the release was already done (because the + // tag exists) and skip it forever. + const targetSha = ( + process.env.GITHUB_SHA || run("git", ["rev-parse", "HEAD"]) + ).trim(); + + console.log(`Creating release ${tag} at ${targetSha.slice(0, 7)}...`); + run( + "gh", + [ + "release", + "create", + tag, + ...assets, + "--target", + targetSha, + "--title", + `${name}@${version}`, + "--notes", + notes, + ], + { stdio: "inherit" }, + ); + + console.log(`Created release ${tag} with ${assets.length} asset(s)`); + created += 1; + } catch (error) { + hasErrors = true; + const message = error instanceof Error ? error.message : String(error); + console.error(`Failed to release ${name}@${version}: ${message}`); + } +} + +console.log(`\nReleases created: ${created}/${missing.length}`); + +if (hasErrors) { + process.exitCode = 1; +} diff --git a/build/scripts/download-github-releases.mjs b/build/scripts/download-github-releases.mjs new file mode 100644 index 00000000000..6d0a1234c8a --- /dev/null +++ b/build/scripts/download-github-releases.mjs @@ -0,0 +1,192 @@ +#!/usr/bin/env node +/** + * Download GitHub release assets whose `${name}_v${version}` git tag has + * no `deployed/` counterpart, and sort them into separate folders + * for the publish template: + * + * - `.tgz` -> `publish_artifacts_npm/` + * - `.crate` -> `publish_artifacts_crates/` + * + * Invoked by `azure-pipelines-cd.yml` before the existing + * `FAST.Release.PipelineTemplate` runs. We use a `deployed/` git + * marker tag instead of `npm view` / `cargo search` because external + * calls from 1ES agents to npm/crates.io are unreliable. + * + * After a successful publish the Azure pipeline pushes the + * `deployed/` marker tag for each release that was published; the + * next run sees it via `git tag -l` and skips that release. + * + * Modes: + * + * - default: download assets for every undeployed release. Writes the + * processed tag list to `publish_artifacts_meta/undeployed-tags.txt` + * so the Azure pipeline can push `deployed/` markers after the + * publish template completes. + * - `--check-only`: enumerate undeployed releases only. Sets the + * `needsDeployment` and `undeployedTags` Azure Pipelines output + * variables (when running under `TF_BUILD`). No downloads. + * + * Inputs: + * + * - `GITHUB_REPOSITORY` env var (`owner/repo`) — required. + * - `GH_TOKEN` env var — required for non-`--check-only` mode so the + * `gh` CLI can authenticate. + * + * The script never reads any package.json or Cargo.toml: the + * source of truth for "what should be published" is the set of git + * tags. This keeps the script working on a freshly-cloned 1ES agent + * with no `node_modules` or cargo registry. + */ + +import { execFileSync } from "node:child_process"; +import { + mkdirSync, + readdirSync, + renameSync, + rmSync, + unlinkSync, + writeFileSync, +} from "node:fs"; +import { join } from "node:path"; + +const NPM_DIR = "publish_artifacts_npm"; +const CRATES_DIR = "publish_artifacts_crates"; +const META_DIR = "publish_artifacts_meta"; +const STAGE_DIR = "publish_artifacts_stage"; + +const CHECK_ONLY = process.argv.includes("--check-only"); + +const repo = process.env.GITHUB_REPOSITORY; +if (!repo) { + console.error("GITHUB_REPOSITORY must be set to owner/repo"); + process.exit(1); +} + +// `--check-only` works against the local git tag list and never invokes +// `gh`, so it does not need a token. In non-check mode we call +// `gh release download`, which fails with a generic auth error if +// `GH_TOKEN` is missing; fail fast here with a clearer message. +if (!CHECK_ONLY && !process.env.GH_TOKEN) { + console.error("GH_TOKEN must be set so the `gh` CLI can download release assets."); + process.exit(1); +} + +function run(file, args, opts = {}) { + return execFileSync(file, args, { encoding: "utf8", ...opts }); +} + +function listGitTags() { + return run("git", ["tag", "--list"]) + .split("\n") + .map(t => t.trim()) + .filter(Boolean); +} + +function setAzureOutput(name, value) { + if (!process.env.TF_BUILD) return; + console.log(`##vso[task.setvariable variable=${name};isOutput=true]${value}`); +} + +const allTags = listGitTags(); +const deployed = new Set( + allTags.filter(t => t.startsWith("deployed/")).map(t => t.slice("deployed/".length)), +); +// Match beachball's release tag format: `${name}_v${major}.${minor}.${patch}`, +// where the version is the publishable portion (`1.0.0` or `1.0.0-alpha.3`). +// Anchored at the end of the tag name so trailing junk (e.g. a stray +// suffix accidentally appended after a real version) does not get +// misclassified as a release. `deployed/` marker tags are excluded +// explicitly so they aren't re-processed as if they were releases. +const RELEASE_TAG_RE = /_v\d+\.\d+\.\d+(?:-[\w.-]+)?$/; +const releaseTags = allTags.filter( + t => !t.startsWith("deployed/") && RELEASE_TAG_RE.test(t), +); +const undeployed = releaseTags.filter(t => !deployed.has(t)).sort(); + +console.log(`Release tags total: ${releaseTags.length}`); +console.log(`Already deployed: ${releaseTags.length - undeployed.length}`); +console.log(`Undeployed: ${undeployed.length}`); + +if (undeployed.length > 0) { + console.log("\nUndeployed tags:"); + for (const tag of undeployed) { + console.log(` - ${tag}`); + } +} + +setAzureOutput("needsDeployment", undeployed.length > 0 ? "true" : "false"); +setAzureOutput("undeployedTags", undeployed.join(",")); + +if (CHECK_ONLY || undeployed.length === 0) { + process.exit(0); +} + +// Clear the publish artifact directories before recreating them so a +// re-run (locally or after a partially-failed Azure attempt) cannot +// pick up stale `.tgz` / `.crate` files from a previous invocation. +for (const dir of [NPM_DIR, CRATES_DIR, META_DIR]) { + rmSync(dir, { recursive: true, force: true }); + mkdirSync(dir, { recursive: true }); +} + +let hasErrors = false; +const processed = []; + +for (const tag of undeployed) { + try { + console.log(`\nDownloading assets for ${tag}...`); + // Clear the stage dir before each iteration so leftover unknown-type + // files from a previous release are not reprocessed (or warned about + // repeatedly). + rmSync(STAGE_DIR, { recursive: true, force: true }); + mkdirSync(STAGE_DIR, { recursive: true }); + run( + "gh", + ["release", "download", tag, "--repo", repo, "--dir", STAGE_DIR, "--clobber"], + { stdio: "inherit" }, + ); + + let foundAssets = 0; + for (const file of readdirSync(STAGE_DIR)) { + const src = join(STAGE_DIR, file); + if (file.endsWith(".tgz")) { + renameSync(src, join(NPM_DIR, file)); + foundAssets += 1; + } else if (file.endsWith(".crate")) { + renameSync(src, join(CRATES_DIR, file)); + foundAssets += 1; + } else { + console.warn(` Ignoring unknown asset type: ${file}`); + unlinkSync(src); + } + } + + // Refuse to mark a tag as processed if its release had no + // recognised publish assets. Otherwise the Azure pipeline would + // push the `deployed/` marker, and the (presumably broken) + // release would never be retried. + if (foundAssets === 0) { + throw new Error( + `Release ${tag} contained no .tgz or .crate assets; refusing to mark as deployed.`, + ); + } + + processed.push(tag); + } catch (error) { + hasErrors = true; + const message = error instanceof Error ? error.message : String(error); + console.error(`Failed to download release ${tag}: ${message}`); + } +} + +writeFileSync( + join(META_DIR, "undeployed-tags.txt"), + processed.join("\n") + (processed.length ? "\n" : ""), +); + +console.log(`\nProcessed ${processed.length}/${undeployed.length} release(s).`); +console.log(`Tag list written to ${join(META_DIR, "undeployed-tags.txt")}`); + +if (hasErrors) { + process.exitCode = 1; +} diff --git a/change/@microsoft-fast-build-b3bddb40-ecab-4fd0-b47f-780c158e9dd7.json b/change/@microsoft-fast-build-b3bddb40-ecab-4fd0-b47f-780c158e9dd7.json new file mode 100644 index 00000000000..079d9a7433f --- /dev/null +++ b/change/@microsoft-fast-build-b3bddb40-ecab-4fd0-b47f-780c158e9dd7.json @@ -0,0 +1,7 @@ +{ + "type": "minor", + "comment": "feat: propagate host attributes from inner