diff --git a/docs/SKILL.md b/docs/SKILL.md index 91ba059d..fcff6a74 100644 --- a/docs/SKILL.md +++ b/docs/SKILL.md @@ -41,6 +41,7 @@ repository's local catalog with `docs/skills/hive.md` as the preflight guard. | Edit `system_files/shared/`, `bluefin/`, or `nvidia/` | [`submodule-boundary.md`](skills/submodule-boundary.md) | | Touch any image reference or registry path | [`image-registry.md`](skills/image-registry.md) | | Modify the `Containerfile` or add a binary | [`containerfile.md`](skills/containerfile/SKILL.md) | +| Author or debug BuildStream elements (dakota, fsdk-containers) | [`buildstream/SKILL.md`](skills/buildstream/SKILL.md) | | Use Context7 to look up external tools | [`context7.md`](skills/context7.md) | | Change `.github/workflows/` | [`ci-tooling.md`](skills/ci-tooling/SKILL.md) + [`workflow-map.md`](skills/workflow-map.md) | | Debug a CI failure | [`ci-pitfalls.md`](skills/ci-pitfalls/SKILL.md) | diff --git a/docs/skills/buildstream/SKILL.md b/docs/skills/buildstream/SKILL.md new file mode 100644 index 00000000..27e65381 --- /dev/null +++ b/docs/skills/buildstream/SKILL.md @@ -0,0 +1,175 @@ +--- +name: buildstream +version: "1.0" +last_updated: "2026-08-09" +id: buildstream +one_line_purpose: Apply the BuildStream 2 patterns shared by every factory BST repo. +entry_point: docs/skills/buildstream/SKILL.md +category: ci-ops +mcp_compliance_level: partial +optimization_status: draft +status: active +dependencies: [context7] +tags: [buildstream, bst, elements, junctions, caching] +description: >- + Cross-repo BuildStream 2 conventions for the factory's BST repos (dakota, + fsdk-containers): element kinds, dependency types, cache keys, junction + hygiene, source tracking. Use when authoring .bst elements, debugging BST + builds, or bumping junction refs. +metadata: + type: reference + context7-sources: + - /apache/buildstream +--- + +# BuildStream (BST) — factory-wide patterns + +## When to Use + +Use when authoring or editing `.bst` elements, debugging BST graph/build/cache +failures, tracking sources or bumping junction refs, or deciding whether a BST +pattern belongs in `common` or stays repo-local. + +## When Not to Use + +- Repo-specific recipes: dakota's OS packaging (`dakota/docs/skills/add-package.md`, + `packaging-*.md`, `oci-layers.md`) and fsdk-containers' distroless recipe + (`fsdk-containers/docs/skills/slim-an-image.md`, `verify-distroless/`) live in + those repos and are deliberately NOT promoted here. +- Containerfile-based repos (common, bluefin, bluefin-lts) — see + [`containerfile`](../containerfile/SKILL.md). +- Asserting BST behavior from memory — Context7 lookup first + (`/apache/buildstream`), per [`context7.md`](../context7.md). + +## Scope rule: what is factory-general + +This skill records only what is **true in every factory BST repo** (today: +`dakota` and `fsdk-containers`). Both are BuildStream 2 projects that junction +freedesktop-sdk and pull from shared upstream artifact caches. + +The following are **per-repo product decisions, not factory rules** — never +copy them across repos: + +| Decision | dakota | fsdk-containers | +|---|---|---| +| What is composed | Bootable bootc OS; desktop platform content in scope | `components/*` only; `platform.bst` banned (AGENTS.md hard rule) | +| Micro-arch | Opt-in `x86_64_v3` option (`project.conf`) | `x86_64_v3` banned — broad-compatibility baseline | +| Output contract | Bootable OS image (systemd, shell, dconf, ldconfig-for-bootc) | Distroless OCI: no shell, SLIM recipe, `just verify` gates | + +When a pattern is general but its parameters differ, this skill names the +pattern and points at the per-repo parameter source (`project.conf` options, +repo `AGENTS.md`). + +## Core patterns + +### 1. Graph-first workflow + +`bst show` before `bst build` — graph/YAML errors surface in seconds; a build +is the slowest feedback loop. Classify a failure (graph / fetch / compile / +install / composition) before opening a `bst shell --build` sandbox. Inspect +`bst artifact log` and `bst artifact list-contents` before guessing. Failed +builds are cached as failed artifacts — `bst artifact delete ` before +retrying. + +### 2. Element kinds are load-bearing + +`stack` aggregates dependencies and produces **no filesystem output**; +`compose` is the filesystem-producing filter step (build-deps only); +`script` runs assembly commands; `junction` is a subproject boundary. A layer +element written as `stack` builds successfully and ships an empty layer — +the most common composition bug. (Source: BuildStream docs → +`handling-files/composition.md`.) Available plugin kinds are per-repo (dakota's +graph registers `meson`/`cmake`/etc.; fsdk-containers hand-writes +`kind: manual` only) — check the repo's `project.conf` before choosing a kind. + +### 3. Dependency types are about *when*, not just *what* + +`build-depends` is staged only for this element's own build; `runtime-depends` +is NOT staged for this element's build, only for its consumers; plain +`depends` is both. Listing a build-time tool under `runtime-depends` fails +silently until a confusing downstream error (e.g. `ModuleNotFoundError` deep +in an install script). + +### 4. Cache keys and junction blast radius + +An element's strong cache key covers its own config plus all dependency keys, +recursively. A junction ref bump therefore invalidates every element the +junction provides, and a junction's `patch_queue` is part of its source hash +— so the queue decides which upstream cache your keys align with (pattern 5). +Non-strict (weak-key) builds can serve stale artifacts — symptom: a package +builds but is missing from the composed image. (Source: BuildStream docs → +`arch_cachekeys.md`.) Details and policy: +[`references/cache-keys-and-junctions.md`](references/cache-keys-and-junctions.md). + +### 5. Junction hygiene: drift control, not "never patch" + +A junction's `patch_queue` is part of its source hash, so the queue decides +which upstream artifact cache your keys align with. Patches **diverging** +from the parent project's own queue are cache-destroying and prohibited; +patches **replicating** the parent's queue byte-for-byte at the pinned ref +are cache-aligning and mandatory when the parent carries one. The rule is +drift control against the parent's queue (dakota: `just patch-drift-check` +in CI), not "never patch a junction". Prefer upstream fixes or ref bumps +over local overrides; every override carries a written exit condition and is +re-audited at every junction bump. + +### 6. Source and ref discipline + +`bst source track` writes `ref:` fields — never hand-write a `git_repo` ref. +A version selector and its cryptographic `ref` update atomically (no +Renovate-style version-only bumps). Derived vendor blocks (`cargo2`, +`go_module`) are NOT regenerated by `source track` — regenerate them after +every ref bump. Variables do not expand in `sources[].url:` — use URL +aliases. Keep install commands hermetic: no `$(date)`, `$(curl ...)`, or +network access at build time. + +### 7. Authoring gotchas verified in both repos + +Bare/minimal sandboxes may have **no shell** (`Staged artifacts do not +provide command 'sh'`) and no `find`; remote-execution sandboxes may lack +`/dev/stdin` — write inline files with `install -Dm644 /dev/null` + +`cat > target <<'EOF'`. Option names are alphanumeric + underscore only +(BuildStream docs → `format_project.md`); option names/values are per-repo — +read `project.conf` before writing `(?):` conditionals. `overlap-whitelist` +only *permits* an overlap; staging order (and `integration-commands`, which +run after all staging) decides which file wins. + +Full authoring reference: +[`references/element-authoring.md`](references/element-authoring.md). + +## Remote execution + +Both BST repos run builds on the shared ghost-cluster BuildBarn grid via +per-repo wrappers; success evidence in the log is `Waiting for the remote +build to complete` per built element. Endpoints, auth, and opt-outs are +per-repo: `dakota/docs/skills/debugging.md` (RE-first policy), +`fsdk-containers/docs/skills/remote-execution.md`. Do not copy endpoint +configuration between repos. + +## Red Flags + +- Copying a composition baseline (`platform.bst` vs `components/*`), arch + option, or output recipe from one BST repo into the other. +- `kind: stack` where filesystem output is expected. +- A junction `patch_queue` that diverges from the upstream parent project's + queue at the pinned ref — divergence invalidates every downstream cache + key. Byte-alignment with the parent is the requirement. +- Hand-written `git_repo` refs, or a version bump without its `ref` bump. +- Building before `bst show` validates; sandboxing before reading the log. +- Asserting BST flags/config keys without a Context7 `/apache/buildstream` + lookup in the session. + +## Verification + +- [ ] `bst show` on the top-level target resolves before any build. +- [ ] Element kind matches the expected output (filesystem ⇒ `compose`). +- [ ] Junction changes state their cache blast radius and exit condition. +- [ ] Derived vendor blocks regenerated after any `source track`. +- [ ] Repo `AGENTS.md` hard rules checked for per-repo parameters. + +## References + +| File | Description | +|---|---| +| [`references/cache-keys-and-junctions.md`](references/cache-keys-and-junctions.md) | Strong/weak cache keys, invalidation cones, junction hygiene and upstream-first override policy. | +| [`references/element-authoring.md`](references/element-authoring.md) | Element/source kinds, dependency types, variables, conditionals, sandbox constraints, overlap rules. | diff --git a/docs/skills/buildstream/references/cache-keys-and-junctions.md b/docs/skills/buildstream/references/cache-keys-and-junctions.md new file mode 100644 index 00000000..f574cf23 --- /dev/null +++ b/docs/skills/buildstream/references/cache-keys-and-junctions.md @@ -0,0 +1,102 @@ +# Cache keys and junction hygiene + +Reference for [`../SKILL.md`](../SKILL.md). Sources: BuildStream docs +(`/apache/buildstream`) sections `arch_cachekeys.md`, `format_public.md`, +`junctions/junction-elements.md`; dakota `docs/skills/bst-overrides.md`, +`patch-junctions.md`, `oci-layers.md`; fsdk-containers +`docs/skills/bump-fsdk-version.md`. + +## The cache-key model + +BuildStream has two cache-key types (`arch_cachekeys.md`): + +- **Strong key** — captures everything that influences build output: the + element's own config, variables, environment, source refs, and the strong + keys of all build *and* runtime dependencies, recursively. +- **Weak key** — includes only the *names* of build dependencies. It changes + when the element itself changes, not when a dependency is updated. + +Strict builds (the default build plan) use strong keys. Non-strict builds +reuse any artifact matching the weak key — faster, but reverse dependencies +are not automatically rebuilt. + +### Consequence: invalidation cones + +| Change | What is invalidated | +|---|---| +| Junction ref bump | Every element the junction provides, recursively downstream (widest cone) | +| `patch_queue` diverging from the parent project's queue | Same width as a ref change: keys no longer match the parent's published cache. A queue matching the parent's byte-for-byte is how cache reuse is *preserved* — see drift control below | +| `project.conf` options/variables | Project-wide | +| Leaf element ref bump | That element plus its reverse dependencies | +| Workflow/Justfile/docs edits | Nothing — no cache impact | + +Merge-order rule that follows from this: leaf bumps first, junction bumps +last, one at a time, each verified green before the next. + +### Consequence: weak-key staleness + +Symptom verified in dakota: a new package is added to a `stack` element, the +build succeeds, but the package is missing from the composed image — the +stack's weak key did not change, so the downstream `compose` was served from +cache. If the graph is right but the output is stale, suspect cache +invalidation before debugging the element. + +### Consequence: failed builds are cached + +A failed build is cached as a failed artifact; retries exit immediately. +Delete it before rebuilding: `bst artifact delete `. + +## Junction hygiene: drift control, not "never patch" + +Both factory BST repos junction `freedesktop-sdk` (via a `gnome-build-meta` +junction) and rely on upstream public artifact caches. A junction's sources — +**including any `patch_queue`** — feed its source hash, which feeds every +imported element's cache key. The patch queue therefore determines *which* +upstream artifact cache you can reuse: + +- A patch that **diverges** from the parent project's own queue is + cache-destroying. dakota measured this: a downstream-only patch on the + `gnome-build-meta` junction silently forced from-scratch compiles of the + imported graph (removing it restored 1053 of 1090 cached elements). This is + the thing to prohibit — not patches in general. +- A patch queue that **replicates** the parent project's queue byte-for-byte + at your pinned ref is cache-*aligning* — mandatory when the parent carries + one, because matching it is what makes your keys line up with the parent's + published cache. Both repos do this today: dakota carries 7 patches in + `patches/freedesktop-sdk/` (byte-identity with GBM's queue at the pinned + ref is documented in `dakota/docs/skills/bst-overrides.md`), and + fsdk-containers carries 0001+0002, with `elements/gnome-build-meta.bst` + noting its fdsdk junction "has to match what gnome-build-meta is using". + +The enforceable rule is **drift control against the parent project's queue at +the pinned ref**: dakota implements this as `just patch-drift-check` (diffs +GBM's queue at the pinned commit against the local one) run in CI; +fsdk-containers re-checks that its two CAS-config patches still apply at +every FSDK bump. The per-repo parameter is *which* upstream project's queue +you must match — currently `gnome-build-meta` at the repo's pinned ref in +both repos. + +Supporting rules: + +1. **Check upstream first.** If the fix is in a newer upstream ref, bump the + junction ref instead of patching. +2. **Never edit junction `.bst` content directly** — changes go through the + junction element's sources/overrides so the cache impact is explicit. +3. **Local overrides are last-resort debt.** Every override or temporary + patch carries an exit condition (`# Exit condition: Drop after fdsdk + ships X`) and, where upstreamable, an `Upstream-Status: Submitted ` + header. Re-audit overrides at every junction bump. +4. **Byte-alignment is the bar where cache reuse matters** — even + semantically equal diffs to the parent's junction file or patch queue + change keys. + +Patch ordering note: patches within a queue apply in alphabetical filename +order; numbering gaps are deliberate insertion room. + +## Remote execution evidence + +Both repos dispatch build actions to the shared BuildBarn grid. The log line +`Waiting for the remote build to complete` per element is the evidence that +remote execution is active; local sandbox staging messages for build actions +mean RE is not engaged. RE and artifact caching are separate mechanisms — +per-repo endpoints and auth live in the repo's own skills. diff --git a/docs/skills/buildstream/references/element-authoring.md b/docs/skills/buildstream/references/element-authoring.md new file mode 100644 index 00000000..cafa4067 --- /dev/null +++ b/docs/skills/buildstream/references/element-authoring.md @@ -0,0 +1,105 @@ +# Element authoring reference + +Reference for [`../SKILL.md`](../SKILL.md). Sources: BuildStream docs +(`/apache/buildstream`) sections `format_project.md`, `format_public.md`, +`handling-files/composition.md`; dakota `docs/skills/buildstream.md`, +`add-package.md`, `update-refs.md`; fsdk-containers +`docs/skills/add-fsdk-component/SKILL.md`, `add-new-image.md`. + +## Element kinds + +| Kind | Produces filesystem output? | Use | +|---|---|---| +| `stack` | **No** — dependency aggregation only | Dep lists consumed by a compose | +| `compose` | Yes — filtered staging area | Layer/filter step (`include:`/`exclude:` domains) | +| `script` | Yes — via explicit commands | OCI/image assembly | +| `import` | Yes — files staged directly | Config-only / direct file placement | +| `manual` | Yes — hand-written shell | Custom builds; the only kind available in every repo | +| `junction` | No — subproject boundary | Importing freedesktop-sdk / gnome-build-meta | + +Build-system kinds (`meson`, `cmake`, `autotools`, `make`) come from plugins +registered per-repo — dakota's junction graph provides them, fsdk-containers +deliberately registers none and hand-writes `kind: manual`. Check the target +repo's `project.conf` before choosing; do not assume a kind exists. + +Compose filtering uses `split-rules` domains declared in elements' public +data (`format_public.md`); `compose` elements take build dependencies only — +no transient/runtime deps (`handling-files/composition.md`). + +## Dependency types + +| Keyword | Staged for this element's build? | Required by consumers? | +|---|---|---| +| `build-depends` | Yes | No | +| `runtime-depends` | **No** | Yes | +| `depends` (or `type: all`) | Yes | Yes | + +A tool your own `build-commands`/`install-commands` invoke must be +`build-depends` or `depends`. Under `runtime-depends` it is invisible to your +sandbox and the failure surfaces as a confusing mid-build error, not a clear +"dependency missing". + +## Sources and refs + +- `bst source track ` is the only way refs are written. For + `tar`/`remote` sources the ref is the downloaded file's sha256; for + `git_repo` it is a git-describe string (`--g`), not a plain + tag or SHA — never hand-write one. (Source: `downloadablefilesource.py` + `track()`.) +- A version selector and its `ref` change **atomically**. Version-only + automation (e.g. a Renovate bump of `v%{version}` without the ref) leaves + the element unfetchable. +- `source track` does NOT regenerate derived source blocks (`cargo2`, + `go_module`, vendored lock data). Regenerate them after every ref bump or + the next cold build fails at the fetch step. +- Variables do not expand in `sources[].url:` — define URL aliases + (conventionally `include/aliases.yml`) and reference those. +- Keep commands hermetic: no `$(date)`, `$(hostname)`, `$(curl ...)` — + they break reproducibility and caching. + +## Variables and paths + +Both repos consume freedesktop-sdk, which is merged-usr and provides the +standard variables: `%{prefix}` = `/usr`, `%{bindir}` = `/usr/bin`, +`%{indep-libdir}` = `/usr/lib`, `%{sysconfdir}` = `/etc` (use sparingly), +`%{install-root}` = staging dir (prefix every install path with it). +FSDK also defines `strip-binaries`: set `strip-binaries: ""` in an element's +`variables:` whenever the payload is not ELF (fonts, configs, scripts, +pre-built tarballs) or the strip step fails the build. + +## Options and conditionals + +- Option names are alphanumeric + underscore only and cannot begin with a + digit (`format_project.md`) — `my_option`, never `my-option`. +- Option *names and values are per-repo*, declared in `project.conf`. Both + current BST repos define `arch` (`aarch64`/`x86_64`); dakota additionally + defines `x86_64_v3` (opt-in) which fsdk-containers bans. Read + `project.conf` before writing a conditional — a `(?):` block referencing + an undefined option name fails at load time. +- Conditional syntax: `(?):` blocks; command-hook composition uses `(>):` + (append), `(<):` (prepend), `(@):` (YAML include). + +## Sandbox constraints + +- A bare `manual` element has **no shell**: `mkdir`, `cat`, etc. fail with + `Staged artifacts do not provide command 'sh'`. Add a shell/coreutils + provider to `build-depends` when commands need one. `kind: script` + assembly elements likewise do not inherit a shell from the runtime they + stage. +- Minimal sandboxes may lack `find` — use shell globs + `case`. +- Remote-execution sandboxes may lack `/dev/stdin` (backend-dependent: + bubblewrap-based runners mount `/proc`, the BuildBarn `bb_runner` + chroot does not). Write inline files as: + `install -Dm644 /dev/null target` then `cat > target <<'EOF'`. +- In container-wrapped BST invocations, artifact checkouts must use + project-relative paths — the container only sees the repo mount. + +## Overlaps and post-staging steps + +`overlap-whitelist` (public data, `format_public.md`) only *permits* two +elements to ship the same path — it does not choose the winner. Staging +order decides; a `depends:` on the element shipping the original forces your +override to stage later. For deterministic replacement of a junction-owned +file, overwrite it in `integration-commands`, which run after all staging — +both repos rely on this ordering (e.g. FSDK's `integration/ldconfig.bst` +rebuilds the linker cache post-staging). diff --git a/docs/skills/context7.md b/docs/skills/context7.md index 117504ca..716d1eb5 100644 --- a/docs/skills/context7.md +++ b/docs/skills/context7.md @@ -108,6 +108,7 @@ Library IDs can change; always resolve fresh. | Tool | Library ID (last confirmed) | Used for | |---|---|---| | bootc | `/bootc-dev/bootc` | image build, kargs, filesystem layout | +| BuildStream | `/apache/buildstream` | element kinds, cache keys, junctions, source tracking | | WirePlumber | `/websites/pipewire_pages_freedesktop_wireplumber` | OEM hook config fragments | Add rows here as you confirm new IDs during sessions, with the date. diff --git a/docs/skills/index.json b/docs/skills/index.json index 9aeb6819..7342a5cd 100644 --- a/docs/skills/index.json +++ b/docs/skills/index.json @@ -1,5 +1,5 @@ { - "generated_at": "2026-08-08", + "generated_at": "2026-08-09", "schema_version": "1.0", "skills": [ { @@ -71,6 +71,25 @@ "last_updated": "2026-08-08", "doc_type": "procedure" }, + { + "id": "buildstream", + "name": "buildstream", + "one_line_purpose": "Apply the BuildStream 2 patterns shared by every factory BST repo.", + "entry_point": "docs/skills/buildstream/SKILL.md", + "category": "ci-ops", + "status": "active", + "tags": [ + "buildstream", + "bst", + "elements", + "junctions", + "caching" + ], + "description": "Cross-repo BuildStream 2 conventions for the factory's BST repos (dakota, fsdk-containers): element kinds, dependency types, cache keys, junction hygiene, source tracking. Use when authoring .bst elements, debugging BST builds, or bumping junction refs.", + "version": "1.0", + "last_updated": "2026-08-09", + "doc_type": "reference" + }, { "id": "ci-pitfalls", "name": "ci-pitfalls", diff --git a/docs/skills/index.md b/docs/skills/index.md index 47f81abb..c6f9e5ab 100644 --- a/docs/skills/index.md +++ b/docs/skills/index.md @@ -3,7 +3,7 @@ This file is a human-readable mirror of `index.json`. Both are generated by `scripts/generate_skill_index.py` — do not hand-edit either file. -Generated: 2026-08-08 · schema 1.0 · 40 skills +Generated: 2026-08-09 · schema 1.0 · 41 skills | id | category | status | one-line purpose | |---|---|---|---| @@ -11,6 +11,7 @@ Generated: 2026-08-08 · schema 1.0 · 40 skills | [bonedigger](bonedigger/SKILL.md) | ci-ops | active | Operate bonedigger and kubestellar-bot issue/report automation. | | [bootc](bootc.md) | ci-ops | active | Work with bootc image build, update, and Containerfile mechanics. | | [brew-lifecycle](brew-lifecycle/SKILL.md) | ci-ops | active | Manage OS-managed Homebrew packages and RPM/brew placement. | +| [buildstream](buildstream/SKILL.md) | ci-ops | active | Apply the BuildStream 2 patterns shared by every factory BST repo. | | [ci-pitfalls](ci-pitfalls/SKILL.md) | ci-ops | active | Diagnose CI gotchas and silent workflow failures across repos. | | [ci-tooling](ci-tooling/SKILL.md) | ci-ops | active | Apply CI policy for SHA pinning, pre-commit, and Renovate tooling. | | [containerfile](containerfile/SKILL.md) | ci-ops | active | Modify and locally test the common Containerfile build. |