Proposal: buildx replay

[tonistiigi]

buildx replay

buildx replay lets users rebuild an image on their own instead of trusting a
published artifact. Replaying from the same pinned materials makes it possible
to verify that the artifact was actually built from the sources recorded in
provenance, and a separate verification attestation makes it possible to
express stronger source policies based on independent rebuilds.

This can replace the current manual snapshotting of source files in DHI pipeline.

Scope

Replay targets release builds from remote sources. Builds that used local
directory context are not supported because local sources are not recorded in
provenance materials. Replay requires that provenance was captured with
sufficient completeness for the requested replay mode.

Commands

• docker buildx replay build [flags] <subject> -o ...
• docker buildx replay snapshot [flags] <subject> -o ...
• docker buildx replay verify [flags] <subject> -o ...

Subject input

All commands take one positional subject argument.

Supported forms:

• docker-image://...
• oci-layout://...
• local attestation file, for example .sigstore.json

Resolution rules:

• If the input is an image or OCI layout, replay loads provenance attached to
  that subject.
• If the input is a local file, replay expects an attestation document and
  loads provenance from it.

replay build

Replay a build from provenance and produce a normal build result.

Usage:

docker buildx replay build [flags] <subject> -o ...

Flags:

• -replay-mode=<mode>

  • Values:
    • materials
    • frontend
    • llb
  • Default: materials
  • Meaning:
    • materials: replay from pinned source materials recorded in provenance
    • frontend: replay from the same frontend with the same attributes
      recorded in provenance
    • llb: replay from serialized LLB recorded in provenance; requires
      mode=max provenance

• -materials=<source>

  • Optional override for where source materials are resolved from
  • Supported values:
    • provenance
    • oci-layout://...
    • registry://...
    • local path to replay material store
  • Default: provenance
  • Possibly, we would also want to support key=value format in here, and the user could provide direct mapping from a specific material to the snapshot location. This would be used to patch errors if a specific material is causing the build to fail.

• -network=<mode>

  • Values:
    • default
    • none
  • Default: default
  • Meaning:
    • controls whether replay can fetch materials or other runtime dependencies
      from the network
    • none is useful when materials are local and have been independently
      verified

• -secret

  • Same syntax as docker buildx build --secret
  • Required when the original build used secrets; replay fails if a required
    secret is missing

• -ssh

  • Same syntax as docker buildx build --ssh
  • Required when the original build used SSH mounts; replay fails if a
    required SSH identity is missing

• -platform=<platform>

  • Replay only the specified platform from a multi-platform build
  • Default: replay all platforms recorded in provenance

• -output, -o <output-spec>

  • Same output model as normal buildx build

Behavior:

• Reconstruct the build according to -replay-mode.
• Resolve materials according to -materials.
• Export the rebuilt result via -output.
• Default mode is faithful replay from pinned materials.
• Fail if provenance is insufficient for the requested replay mode.
• Fail if the original build used secrets or SSH and the corresponding
  -secret or -ssh flags are not provided.

Examples:

docker buildx replay build --replay-mode=materials docker-image://example/app:latest -o type=oci,dest=result.oci
docker buildx replay build --materials=./materials --network=none docker-image://example/app:latest -o type=docker

replay snapshot

Preserve replay inputs for later rebuilds, even if original sources change or
disappear.

Usage:

docker buildx replay snapshot [flags] <subject> -o ...

Flags:

• -include-materials
  • Boolean
  • Default: true
  • Meaning:
    • when true, snapshot immutable source materials together with replay
      metadata
    • when false, snapshot replay metadata only
• -output, -o <output-spec>
  • Required
  • Supported targets should include:
    • local directory
    • OCI layout
    • registry

Behavior:

• Export replay metadata describing how to replay the build.
• By default also store reusable copies of immutable source materials.
• Output can later be used as a -materials source for replay build or
  replay verify.

Examples:

docker buildx replay snapshot docker-image://example/app:latest -o type=oci,dest=replay
docker buildx replay snapshot --include-materials=false docker-image://example/app:latest -o type=registry,ref=example/app-replay:meta

replay verify

Replay a build, compare the result, and emit a verification result.

Usage:

docker buildx replay verify [flags] <subject> -o ...

Flags:

• -materials=<source>
  • Same meaning as in replay build
• -network=<mode>
  • Same meaning as in replay build
• -secret
  • Same meaning as in replay build
• -ssh
  • Same meaning as in replay build
• -platform=<platform>
  • Verify only the specified platform from a multi-platform build
  • Default: verify all platforms recorded in provenance
• -compare=<mode>
  • Values:
    • digest
    • artifact
    • semantic
  • Default: digest
  • Meaning:
    • digest: exact output digest match
    • artifact: compare image manifest digests
    • semantic: reserved for future higher-level comparison
• -output, -o <output-spec>
  • Writes verification result or attestation

Behavior:

• Replay the build using the same replay-relevant attributes and same materials
  as the original provenance.
• Compare replay result against expected artifact.
• Emit a verification result suitable for saving or pushing.
• On failure, report where the output diverged (layer, config, or manifest
  level).
• Verification attestation may be signed by external tooling.
• Resulting verification evidence can later be used by source policy.

Examples:

docker buildx replay verify --materials=./materials docker-image://example/app:latest -o type=local,dest=./verify
docker buildx replay verify --network=none docker-image://example/app:latest -o type=registry,ref=example/app:verify

Replay metadata

Replay snapshot output should contain:

• provenance-derived replay instructions
• source material descriptors
• enough metadata to drive replay build and replay verify

Default policy

• strict by default
• faithful replay uses pinned materials
• missing required material data is an error
• fallback to weaker replay modes must be explicit

Verification rules

• Verification is only valid when replay uses the same replay-relevant
  attributes and the same materials as recorded in the original provenance.
• If replay uses different attributes or different materials, it is not a
  verification of the original build.
• Digest comparison requires the same compatibility-version as the original
  build. The effective compatibility-version is recorded in provenance.
• If the replay requires materials that are not part of the original provenance, verification should probably fail by default, but possibly override could be passed.

Security model

• Local materials let the user independently verify source checksums before
  running replay.
• If the user then disables network, they know source inputs used by the replay
  must come from those verified local materials.

Verification output

• Verification output should be representable as an attestation.
• Target predicate type is SLSA Verification Summary Attestation (VSA); exact
  predicate fit needs investigation to confirm replay-specific fields
  (replay mode, materials source, comparison method, compatibility-version)
  are representable.
• It should be possible to save it locally or push it remotely via -output.
• Signing remains external to replay itself.

Non-goals

• hermetic build guarantees
• semantic comparison in the first version

@AkihiroSuda @crazy-max

[tonistiigi]

I think we will also need some diff on images builtin to ease working with cases where there is a mismatch and debugging what needs to be changed. Prior art in https://github.com/reproducible-containers/diffoci . This isn't a direct dependency but UX would be poor if there is no way to debug issues. Recommendations welcome how this could look like.

[AkihiroSuda]

Prior art in https://github.com/reproducible-containers/diffoci .

https://pkg.go.dev/github.com/reproducible-containers/diffoci@v0.1.8/pkg/diff#Diff is designed to be embeddable in other projects, so I guess buildx can just use this function

func Diff(ctx context.Context, cs content.Provider, descs [2]ocispec.Descriptor,
	platMC platforms.MatchComparer, opts *Options) (*EventTreeNode, error)

[AkihiroSuda]

replay build

How will it reproduce apt/dnf/apk packages?
We can't expect upstream image builders to use snapshot.debian.org due to the network issues, so a replayer has to somehow pull old packages by itself.
Will it patch /etc/apt/sources.list before running ExecOp and revert it after the execution?
What change will be needed on buildkit?

Relevant discussions:

• dockerfile: implement hooks for RUN instructions moby/buildkit#4669
• sourcepolicy: add INJECT action for package registry redirection moby/buildkit#6490

[tonistiigi]

How will it reproduce apt/dnf/apk packages?

This depends on the build definition. In Dockerfiles the build author must make sure the RUN commands will work in the future (eg. use a stable package repository). For things like buildkit-alpine or DHI frontend, they should be able to track packages individually so they can be backed up with replay snapshot.

The objective for this is not that any build possible with BuildKit will automatically become binary reproducible forever. I don't think that is just possible. But if you create your build definition in reproducible way then these commands should help you.