Architecture

Module layout and dependency rules for the chunk Go CLI.

Directory Structure

chunk-cli/
├── main.go                    # Entry point: cobra bootstrap + usererr handling
├── skills/                    # Skill definitions (go:embed) and skill subdirectories
├── acceptance/                # Acceptance tests
└── internal/
    ├── cmd/                   # Cobra command definitions (thin wrappers)
    │   ├── root.go            # Root command, registers all subcommands
    │   ├── auth.go            # auth set, auth status, auth remove
    │   ├── completion.go      # completion install/uninstall/zsh
    │   ├── config.go          # config show/set
    │   ├── init.go            # init (project setup, git pre-commit hook)
    │   ├── sidecar.go         # sidecar list/create/exec/add-ssh-key/ssh/sync/env/build/setup/snapshot
    │   ├── skills.go          # skill install/list
    │   ├── task.go            # task run/config
    │   ├── upgrade.go         # upgrade
    │   └── validate.go        # validate
    ├── anthropic/             # Anthropic Messages API client
    ├── circleci/              # CircleCI REST API client
    ├── claudecode/            # Runs claude -p on a sidecar and explains how the run ended
    ├── config/                # User config (XDG_CONFIG_HOME/chunk/config.json)
    ├── factoryplan/           # The flow screen's plan, relatives and session in factory
    ├── filecache/             # Generic JSON-on-disk cache (FileCache[T])
    ├── github/                # GitHub GraphQL client (pull requests, repos)
    ├── gitexec/               # Low-level local Git process execution
    ├── gitremote/             # Git remote URL parsing for org/repo detection
    ├── gitutil/               # Git utility helpers, working-tree fingerprints
    ├── hashutil/              # Collision-free digests of string parts (cache keys)
    ├── httpcl/                # HTTP client library (JSON + retries)
    ├── iostream/              # I/O stream abstraction
    ├── sidecar/               # CircleCI sidecar operations
    ├── skills/                # Skill definitions (go:embed) and installation
    ├── task/                  # Task run config and CircleCI trigger
    ├── secrets/               # Secret resolution (env var value expansion)
    ├── session/               # Which agent session this invocation belongs to
    ├── telemetry/             # Usage telemetry (Segment)
    │   └── receiver/          # Forwards buffered events to Segment (used by receive-telemetry)
    ├── testing/recorder/      # HTTP recorder for tests
    ├── tui/                   # Terminal UI components (confirm, input, select)
    ├── ui/                    # Colors, formatting, spinner
    ├── upgrade/               # CLI self-upgrade
    ├── validate/              # Validation command logic
    └── chunkd/                # The daemon's API: wire types, client, paths, lifecycle
        └── server/            # The daemon itself: poll loop, stores, handlers, plan builds

Layering Rules

Dependencies flow strictly downward:

main.go → internal/cmd/ → internal/{business packages} → internal/httpcl/

main.go creates the root command and handles top-level errors
internal/cmd/ contains thin cobra wrappers that parse flags and delegate
Business packages (factory/, task/, etc.) contain the logic
internal/httpcl/ is an independent library — no imports are allowed to other internal/ packages
config/ is a leaf — no imports from other internal/ packages
hashutil/ is a leaf too: stdlib only, so any package can share its digest primitive rather than re-deriving one

No upward or lateral imports between business packages, except where a package naturally composes another (e.g. task/ uses circleci/).

Entry Point

main() → cmd.NewRootCmd(version) → rootCmd.Execute()

Errors are caught in main(). If the error is a usererr.Error, only the user-facing message is printed (no stack trace). Otherwise the raw error is printed. Both exit with code 1.

Data Flow: `factory bootstrap`

1. Config            writeBootstrapConfig() → validate.DetectCommands()
                     → .chunk/config.json
                     (skipped when the config exists)

2. Sidecar image     sidecar.Create() from cimg-base:2026.09-claude (or --image)
                     → sidecar.Provision() → CreateSnapshot()
                     → validation.sidecarImage
                     (skipped when it is set, or with --skip-sidecar)

Configuration and Environment Variables

Single source of truth for env var names

Define every environment variable name as a const in the config package. Use these constants in user-facing messages and t.Setenv calls. Never use bare os.Getenv("CIRCLE_TOKEN") strings.

const (
    EnvCircleToken     = "CIRCLE_TOKEN"
    EnvAnthropicAPIKey = "ANTHROPIC_API_KEY"
    EnvGitHubToken     = "GITHUB_TOKEN"
    EnvModel           = "CODE_REVIEW_CLI_MODEL"
    // ...
)

Struct-based env loading

Declare all environment variables once in an EnvVars struct with env struct tags (via go-envconfig). Express defaults as tag values, not if-empty checks:

type EnvVars struct {
    CircleToken      string `env:"CIRCLE_TOKEN"`
    CircleCIBaseURL  string `env:"CIRCLECI_BASE_URL,default=https://circleci.com"`
    AnthropicAPIKey  string `env:"ANTHROPIC_API_KEY"`
    AnthropicBaseURL string `env:"ANTHROPIC_BASE_URL,default=https://api.anthropic.com"`
    // ...
}

LoadEnv(ctx) populates the struct via envconfig.Process.

When adding a new environment variable:

Add a const Env... for user-facing messages and test code.
Add a field to EnvVars with an env tag (and default= if needed).
Wire it into Resolve() or consume it from the struct directly.

Layered resolution with explicit precedence

Config resolves through a strict priority chain:

flag > env var > config file > default

Resolve() returns a ResolvedConfig struct. Each value is paired with a source string (e.g. "Environment variable (CIRCLE_TOKEN)") so status/diagnostic output can show where the value came from.

Not all values support all layers — for example, CircleCI and GitHub tokens have no flag, so their chain is env var > config file. The Anthropic API key and model support the full chain.

User config lives at $XDG_CONFIG_HOME/chunk/config.json (default: ~/.config/chunk/config.json):

{
  "anthropicAPIKey": "sk-...",
  "model": "claude-sonnet-4-6"
}

Project config lives in .chunk/config.json (per repository):

{
  "orgID": "f22b6566-597d-46d5-ba74-99ef5bb3d85c",
  "commands": []
}

Client constructors accept config, not env

Client New() functions receive values from the resolved config. They must not call os.Getenv themselves. This keeps env reading centralised in config.Resolve and makes clients testable.

Environment variable reference

Variable	Used by	Purpose
`ANTHROPIC_API_KEY`	anthropic, config, validate	Anthropic authentication
`ANTHROPIC_BASE_URL`	anthropic, claudecode, validate	API endpoint override
`CLAUDE_CODE_OAUTH_TOKEN`	config, claudecode	Claude subscription authentication for Claude Code runs
`GITHUB_TOKEN`	github	GitHub authentication
`GITHUB_API_URL`	github, upgrade	GitHub API endpoint override (also used by the update check)
`CIRCLE_TOKEN` / `CIRCLECI_TOKEN`	circleci	CircleCI authentication
`CIRCLECI_ORG_ID`	sidecar	CircleCI organization ID (overrides `orgID` in `.chunk/config.json`)
`CIRCLECI_BASE_URL`	circleci	CircleCI endpoint override
`CLAUDE_CODE_SESSION_ID`	session, sidecar	Which agent session is running the command, so parallel sessions in one working tree get their own sidecar pool
`CHUNK_SESSION_ID`	session, sidecar	Pins that session identity by hand; wins over `CLAUDE_CODE_SESSION_ID`
`XDG_CONFIG_HOME`	config	User config directory (default: `~/.config`)
`XDG_DATA_HOME`	sidecar, validate	Per-project state directory (default: `~/.local/share`)
`XDG_STATE_HOME`	upgrade, chunkd	User state directory for the 24 h update-check cache and the daemon's socket, pid file and log (default: `~/.local/state`)
`CHUNK_SIGNOFF_LOCAL`	signoff	Set to `1` to run `chunk signoff`'s commands on this machine, in the snapshot's checkout, instead of on a sidecar; any other value uses the sidecar
`CHUNK_NO_TELEMETRY`	telemetry	Disable usage telemetry (any non-empty value)
`NO_ANALYTICS`	telemetry	Disable usage telemetry (any non-empty value)
`DO_NOT_TRACK`	telemetry	Disable usage telemetry (any non-empty value)
`CI`	telemetry, upgrade	Also disables usage telemetry and the update check (set by most CI systems)
`CHUNK_NO_UPDATE_CHECK`	upgrade	Disable the new-version check and its notice (any non-empty value)
`CHUNK_TELEMETRY_LOG`	telemetry	Log telemetry events to stderr instead of (or alongside) sending them
`HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`	sidecar	Standard Go proxy env vars, honored when dialing the sidecar's WebSocket SSH tunnel (needed in devcontainers/networks where egress is proxy-only)

Telemetry (`internal/telemetry/`)

Modeled on circleci-cli's internal/telemetry package. Every command reports a single command_invocation event containing the command path and the names (never values) of flags the user set — no flag values, argument values, file paths, or other PII. chunk-cli and circleci-cli share the same Segment write key/workspace; events are distinguished by context_app_name (chunk-cli vs circleci-cli). The install UUID is sent as AnonymousId (not UserId) to avoid mixing machine identifiers with real user IDs in shared event counts. When the user has authenticated, their CircleCI user UUID is also sent as UserId.

The event's outcome is one of success, blocked (the user could not proceed: bad args, not authenticated, not configured, not found), check_failed (chunk ran and the project's own checks exited non-zero), or failure (an actual error: API failure, internal error, anything unclassified). Errors choose their outcome by implementing telemetry.Classified; an error that does not is a failure.

When an auth flow validates a token, it persists the user UUID (config.SaveUserID) and calls telemetry.IdentifyUser. That attaches the user ID to the rest of the current run, and sends a Segment identify joining the anonymous history to the user: one for the current session tracking ID (if any) and one for the instance ID, which is the AnonymousId of every run made outside an agent session. chunk auth remove circleci clears the stored user ID once no CircleCI token resolves anywhere.

Installs that authenticated before the user ID was recorded (or via an environment variable token) have no saved ID, so ensureCircleCIClient backfills it: when a token resolves, telemetry is on and no ID is saved, it looks the user up once via /api/v2/me (3 s timeout), persists the ID and calls telemetry.IdentifyUser. A failed lookup leaves the install anonymous and is retried on the next command.

Telemetry is flushed by cmd.ExecuteRoot rather than PersistentPostRunE, which cobra skips when RunE errors — otherwise a login followed by a failed command would lose its one-time identify. Buffered events reach the receive-telemetry subprocess as tagged receiver.Message envelopes; the receiver also accepts the older bare-track array.

Every event's Context also carries the operating system (runtime.GOOS) and, if detected, the AI coding agent chunk-cli was invoked from (e.g. claude-code, cursor) — see internal/telemetry/agent.go's DetectCodingAgent, which checks for well-known environment variables those agents set in the shells they spawn. The agent list is intentionally conservative: only signals confirmed to be set by the agent itself are included, to avoid misattributing invocations. Detection returns "" when no known agent is found, the common case for a human running chunk directly.

internal/cmd/root.go's setupTelemetry resolves the user's preference (opt-out env var → telemetry config field, first match wins) and attaches a telemetry.Sender to the command's context; RecordForSubcommands wraps every subcommand's RunE so it reports its event automatically, with no per-command changes needed.
Events are never sent inline. Sender.Close() marshals buffered events to JSON and hands them to a detached chunk receive-telemetry subprocess (see internal/telemetry/delegate.go and the hidden command in internal/cmd/receivetelemetry.go), so a slow or unreachable Segment endpoint never delays CLI exit. The subprocess forwards them via internal/telemetry/receiver.
CHUNK_TELEMETRY_LOG=1 logs event payloads to stderr instead of (or alongside) sending them — useful for verifying what a command reports without touching the network.
The Segment write key (cmd.writeKey) is a hardcoded constant, like circleci-cli's — write keys are not secret, they can only send events, not read data, so checking one into git is safe. Events sent with it are tagged as chunk-cli invocations via Context.App.Name ("chunk-cli") in Meta.toContext (internal/telemetry/telemetry.go), keeping them distinguishable from circleci-cli's own telemetry. CHUNK_TELEMETRY_LOG still works without touching the network, independent of the write key.
setupTelemetry also checks testing.Testing() before sending, so a real write key committed to source still can't trigger sends (or the recursive subprocess spawning that implies) from a go test binary.

Pre-Commit Hooks

chunk init installs .git/hooks/pre-commit, which runs chunk validate before every commit; a non-zero exit blocks the commit. It writes no agent hook files.

chunk validate always runs its commands, whatever is on stdin and whether or not the tree has changed since the last run.

Chunk Daemon (`internal/chunkd/`, `internal/chunkd/server/`)

The daemon is split in two. internal/chunkd is its API: the types that go over the wire, the HTTP client, the socket and pid paths, and starting the process. Its internal dependencies are config, eventlog, iostream and version. internal/chunkd/server is the daemon itself and imports chunkd for the types it serves. Clients — chunk ui, chunk factory plan, hooks, chunk validate — import chunkd only, so they depend on the daemon's API and never on the plan builds or the sidecar pool behind it. Only cmd/daemon.go, which runs the daemon, imports server (and tests that start a real one).

Every client in a process reaches a daemon through one shared connection pool per socket or TCP address (sharedTransport in internal/chunkd/transport.go), so a long-running client such as chunk watch reuses its connections instead of opening a new one per call. New client code must get its transport there rather than build an http.Transport of its own. Both ends close connections left idle.

Starting the daemon belongs to chunkd too. It is this binary re-executed as the hidden chunk _daemon (chunkd.DaemonSubcommand, which cmd registers under that name); callers only say when they need one, with EnsureRunning at startup, which also replaces a daemon from another build. A dashboard that keeps polling opts into FetchSnapshotRelaunching, which starts a daemon again if it has gone away but never replaces one that answers.

The split is in the imports, not yet everywhere at runtime:

Validation still runs through the CLI. /validate takes the caller's command line (ValidateRequest.Args) and answers with the output a terminal would have shown. The daemon runs it by executing the cobra command tree in-process, through the ValidateRunner that cmd/daemon.go passes in, so the code doing the work lives in cmd/validate.go.
Projects are registered by writing a file, not by asking the daemon. chunk ui, chunk validate and chunk review write the breadcrumbs AllProjectRoots reads, and the daemon finds projects by scanning them.

The daemon backs chunk ui and builds the plans chunk factory plan approves. It polls every registered project every 5 s, tailing each project's events.jsonl by byte offset, and serves snapshots as JSON over a Unix socket at $XDG_STATE_HOME/chunk/daemon/daemon.sock, default ~/.local/state/chunk/daemon/daemon.sock (CHUNK_DAEMON_DIR overrides the directory). When nothing answers there, clients also try ~/.chunk/daemon/daemon.sock, so they reach a daemon started by an older release until it exits.

Variable	Default	Purpose
`CHUNK_DAEMON_DIR`	`~/.local/state/chunk/daemon/`	Override the directory for the socket, pid file and log. The socket path (`<dir>/daemon.sock`) must fit the OS limit (104 bytes on macOS, 108 on Linux); a longer one is refused up front with this variable named as the fix
`CHUNK_DAEMON_TCP_ADDR`	(disabled)	Bind a TCP listener on this address in addition to the Unix socket (e.g. `127.0.0.1:7777`); requires `CHUNK_DAEMON_TCP_TOKEN`
`CHUNK_DAEMON_REMOTE_ADDR`	(local socket)	Connect to a remote daemon at this TCP address instead of the local socket (e.g. `127.0.0.1:7777` via an SSH tunnel)
`CHUNK_DAEMON_TCP_TOKEN`	(required when TCP is enabled)	Bearer token required on every TCP request; set the same value on the daemon and all clients. TCP traffic is not encrypted — use an SSH tunnel (`ssh -L`) when the daemon is not on loopback.

Remote viewing

Clients of /snapshot and /output open no local state, so they work unchanged against a daemon reached over TCP. chunk ui is not one of them: its flows and plans are planned and built from this machine, so it refuses to start with CHUNK_DAEMON_REMOTE_ADDR set. What differs for the others:

EnsureRunning / EnsureLaunched are no-ops when CHUNK_DAEMON_REMOTE_ADDR is set. A remote daemon is managed where it runs.
Project filters are matched on the daemon's host. snapshot accepts a root by its own spelling or its symlink-resolved one, because a remote viewer can only type a path as it knows it and that need not be the daemon's key.
A 401 surfaces as chunkd.ErrUnauthorized and an unreachable address is reported with the address in the message (requestError), so a client can say which daemon failed and why.

Beyond that it owns three things that outlive the processes they came from.

Command output buffering

chunk validate and chunk sidecar exec submit a remote command and register it with the daemon before consuming any output. Both go through one helper (submitAndStream), so a new command path cannot quietly skip registration and leave its output unreachable:

POST /command   {command_id, sidecar_id, project_root, op, name, submitted_at}
GET  /output?command_id=<id>&offset=<n>
    → {data, next_offset, running, exit_code, truncated, found}
GET  /snapshot  → ... plus per-project `commands` and a top-level `auth_error`

The daemon then streams that command's output itself and buffers it. This is what makes hook-driven runs observable: the hook process exits the moment its command finishes, so anything holding the only copy of the output loses it.

Design constraints worth preserving:

The daemon has no working directory worth trusting. The socket is user-global, so one daemon serves every repo on the machine. launchDaemon runs it in the daemon's state directory (EnsureDir) rather than the checkout that launched it, so a removed worktree cannot break later commands at getcwd — but that cwd is no project's root. Anything the daemon runs on a caller's behalf must therefore be told which project it is for: ValidateRequest.ProjectRoot carries it, and makeValidateRunner turns it into --project so the run cannot fall through to os.Getwd(). A run that resolves the project itself validates the daemon's repo and reports the answer under whichever project asked — and on the async path the staleness check, comparing the requested project against itself, confirms it as current.
Registration is best-effort and never starts the daemon. RegisterCommand reports no error and uses a 2 s timeout. It sits on the hook path in front of a command the developer is waiting on, so a missing daemon must cost nothing. Launching one as a side effect of a hook firing would be intrusive, and a hook that hangs on a daemon launch is far worse than a missing logs pane.
Offsets are the daemon's own byte positions, not the API's opaque SSE cursor. The server cursor stays private to the daemon, so no reader has to reason about reconnects. Positions are logical — they count evicted bytes — so an offset that lands before the retained window is exactly the signal that the reader missed some, which is what sets truncated.
Buffers are capped in bytes, not lines (MaxCommandBytes), keeping the tail. Output is neither small nor uniform, so a line count says nothing about memory.
Streamers never run on the poll path. Each is its own goroutine with its own context, so a hung stream cannot stall the dashboard for every other project.
Credentials resolve lazily and never prompt. authprompt.ResolveCircleCIClient returns ErrNeedsAuth rather than asking; the daemon reports that in Snapshot.AuthError and carries on serving. It has no terminal, so a process blocked on a hidden prompt would be indistinguishable from one that has hung. Resolution is deferred to first use because it can read the OS keychain, and the daemon starts on every chunk ui whether or not anything needs a token.

Plan builds

The daemon's factory sessions are plan builds. A session is the daemon's record of building one approved factory plan for a project: a Claude Code session in the plan's worktree, with factory's f plugin, that works through factory's tasks for the plan through to a pull request. The daemon is the local one (Unix socket); a build works from files on this machine. Approving a plan in chunk factory plan (or in chunk ui) starts one. A build's progress is the plan's flow in factory; the daemon's session only records that the build is running.

POST /factory/plans        {project_root, plan_id, model?} → 202 {id}: build a factory plan
                             no session holds, in a fresh build session that claims it (409
                             claimed by another session, no flow, plan over or already
                             building; 404 unknown project; 502 factory unreachable; 503 not
                             authenticated to CircleCI)
GET  /factory/plans/{plan}/build[?after=N] → PlanBuildActivity: the plan's build
                             session (the running one, else the latest) and its feed
                             lines after the first N, numbered from first; total is the
                             next after (404 no build of the plan)
POST /factory/plans/{plan}/approve {project_root, model?} → 202 {id}: approve the plan in
                                   factory, stop its planner and start its build, in one call
                                   that a client going away does not cut short (409 being
                                   approved, not ready, no units, questions open or already being
                                   built; 502 factory failed, or approved but the build would not
                                   start; 503 not authenticated to CircleCI; 404 unknown project)
POST /factory/plans/{plan}/planner/request {project_root, text, model?} → 200 SessionLog: start
                                   the plan's planner on a new plan's request, once (409 already
                                   requested, claimed by another session, or the plan being
                                   approved or approved; 503 no Claude Code, factory or factory
                                   daemon, or not authenticated to CircleCI; 404 unknown project)
POST /factory/plans/{plan}/planner/resume  {project_root, model?, release_holder?} → 200 SessionLog: have the
                                   planner claim the plan again, starting or resuming it, when it
                                   holds no claim; nothing while it does (409/503/404 as request).
                                   release_holder is the session the developer confirmed taking
                                   over: the daemon re-reads the claim, refuses unless that same
                                   session still holds it and it is a foreign session the
                                   daemon may release, releases it, then resumes
GET  /factory/plans/{plan}/planner[?after=N] → {session_id, log_id, oldest, busy, activity?,
                                   next, events[]}: the planner's events from cursor N (the
                                   `next` of an earlier read), and whether it is mid-turn;
                                   without after, no events
POST /factory/{id}/cancel  → 202 (idempotent): stop a build without ending its plan, and
                                   free factory's claim if the build still holds it, so a new
                                   build can start at once (a daemon shutdown frees nothing)
DELETE /factory/{id}       → 204: forget a finished session; 409 while it is still running
POST /chat                 {root, text, context?} → 200 SessionLog, once the session has the
                           message: send a message to root's chat, starting or
                           resuming its session (400 no text, or a root that is not an
                           absolute path to a git repository's top level; 503 while the
                           daemon shuts down)
GET  /chat?root=&after=N → SessionLog: the conversation from cursor N; without
                           after, no events
GET  /snapshot             → each project carries `sessions` (state only), and the
                             top level a `review_auth_error` when plan builds cannot run

A plan's planner is a long-lived Claude Code session with factory's f plugin that plans the plan through factory's tasks, one per plan. Planners are kept apart from sessions, in a registry of their own: they are not listed in GET /factory or the snapshot, so an idle planner never keeps an outdated daemon from being replaced. A planner starts on a request or resume, and stops on approval, when the plan's flow ends, or when the daemon stops. It outlives the screens that follow it: chunk factory plan and chunk ui poll GET …/planner while a plan is open, and closing them only ends that polling.

The chat. The daemon runs one chat per repository: a Claude Code session in factory.ModeChat, in the repository's root, that answers questions about its factory plans and, when asked, acts on their flows with factory's own commands: what chatTools allows, and never what chatDeniedTools denies. It starts on the repository's first message, resuming the session recorded in factory.OpenChatSession(root). It runs on the same session runner as the planner, so its conversation is the runner's log, including the developer's messages (message events, without the context sent with them), and a client reads it by cursor and log_id as it reads a planner's. The daemon stops the session after 30 idle minutes, keeping the log; the next message resumes it. A restarted daemon has no log to show, and a new log_id says so, but its next message resumes the same Claude session.

A build records a chunkd.Session whose factory field (FactoryRun) names the plan, and runs a factory build session in the plan's worktree. While it runs, the daemon follows the plan's flow in factory: it stops the build once the flow has ended, so ending a plan stops its build, and tells the build session to claim the plan again when another session has held its claim and released it.

Test seams. server.WithPlans replaces factory, the worktree and the build session in tests; left out, the daemon uses the real ones.
Detach is not cancel. A viewer disconnecting changes nothing; the build belongs to the daemon and outlives the screen that approved it. A build ends with its plan, or with POST /factory/{id}/cancel.
In memory only. Like async validate tasks, a daemon restart loses the session record (the last 10 per project are kept while it runs). The build's work is not lost with it: it is on the plan's branch, and the plan's flow is in factory.

Resource sampling

There is no metrics endpoint, so usage is sampled inside the sidecar. The naive implementation is a trap: sidecar.ExecOverSSH opens a fresh WebSocket and SSH handshake per call, so a 2 s interval would mean a handshake every 2 s per sidecar. Instead the daemon submits one long-lived shell loop through the exec API (SubmitExec with sh -c, then StreamOutput) and parses frames off its output stream.

Going through the exec API rather than SSH is deliberate. The SSH exec channel fork/execs the command string instead of interpreting it, which is why every ExecOverSSH call in this repo is a bare binary arg invocation — a shell loop cannot run over it at all. The exec API takes argv as structured data, so it runs scripts correctly, and its output stream is already long-lived and resumable. That removes the handshake-per-sample problem outright with no persistent connection for the daemon to supervise.

Sampling is gated on an attached dashboard. A snapshot request is the signal; after idleShutdown with none, samplers stop. Cost stays proportional to benefit.
Memory must be read cgroup-aware — v2, then v1, then /proc/meminfo. The sidecar is containerised, so /proc/meminfo reports the host's memory and would make every sidecar look like it had hundreds of gigabytes free.
CPU is a delta. /proc/stat is cumulative since boot, so a percentage exists only between two frames. One frame yields no CPU reading rather than a number that looks plausible and means nothing.
reconcile runs once per poll across all projects, not per project. It stops any sampler absent from what it is given, so feeding it one project's sidecars at a time would make each project tear down every other project's samplers.
The remote loop is bounded, not infinite (samplerIterations). Cancelling the stream stops the daemon reading, but nothing guarantees the remote process dies with it; a loop that ends on its own caps how long an orphan can linger. run simply submits another while a dashboard is still attached.
Repeated failure pauses a sidecar, it does not disqualify it. After maxSamplerFailures consecutive failures the daemon stops sampling that sidecar for sampleCooloff and then retries. Without the ceiling an unsampleable sidecar is retried forever, each attempt costing an API call; without the retry one bad minute costs that sidecar its numbers for the rest of the session. The sampler slot survives either way, so the last reading stays on screen to be dimmed as stale rather than vanishing.

Parsing is separated from transport (parseSampleFrame, cpuPercent, splitFrames, consumeSamples) so it is testable against captured fixtures with no SSH involved.

Background validation

A run delegated to the daemon (POST /validate) can set allow_async, offering to be released before the answer exists. The daemon weighs the change (risk.go) and either runs it while the caller waits or starts a tracked task (asyncvalidate.go) and answers with its ID. chunk validate results reads finished results back later. The CLI does not currently set allow_async; its own background runs go through POST /validate/async (chunk validate --async).

POST /validate         {args, env, project_root, allow_async}
    → {exit_code, stdout, stderr}          # ran here
    → {task_id, reason}                    # released; nothing has run yet
POST /validate/async   {args, env, project_root}   → {task_id}
GET  /validate/collect?root=<path>                 → {tasks}

The decision is made before validateMu is taken. A caller offering to be released must not first queue behind whatever run is already in flight, since that wait is the whole thing being avoided. The run itself still takes the lock, so background runs serialise with synchronous ones exactly as before — what changes is who waits, not how many run at once.
Async breaks the assumption every other validate path rests on: that somebody is waiting. taskStore holds which project a run was for, whether it is still going, and what it concluded, until a later turn collects it.
A live-tree verdict is only reported if it still describes the tree. Each task records a fingerprintTree reading at the start, compared again when the run ends and when the result is read. Any difference discards the verdict: a pass describing code that is no longer on disk reads as a green light for work already changed. The task is still handed over, stripped of its exit code and output, so the discard itself is visible. A snapshot-backed run is the exception and keeps its verdict — see the snapshot bullet below.
A failure leaves a debt, so a discarded result is not a lost one. riskMemory remembers a project whose last run failed — including one whose failure went stale — and the next run is held rather than released, where its caller can act on it. Any passing run clears it.
Both stores are in memory. A daemon restart loses in-flight tasks and forgets the debt. A lost task reports nothing rather than reporting wrongly, and a forgotten debt costs one turn's delay before the failure is found again.
Judgement lives in one place, next to the git read it depends on. Splitting it between the client and the daemon would mean two git status walks that can disagree about the same tree.
Size is measured from the last passing state, not from HEAD. Every run takes a gitutil.SnapshotTree before validating, and a pass records it as the baseline (riskMemory.recordGreen); the next change is a gitutil.ChangesBetween from there. Against HEAD the number only grows until something commits — five 100-line turns read as 500 by the fifth — and resets to nothing when anything does, validated or not. A snapshot is content, so neither happens.
A new run supersedes the live-tree run in flight. A run is released because the tree moved, so whatever is already running is validating code that is no longer there and its result is headed for the bin; taskStore.supersede cancels it rather than leaving two runs of the same commands queued behind each other. A cancelled run records nothing — not a failure, not a known-good state — because it concluded nothing. Snapshot-backed runs are exempt: their verdict survives the tree moving, so they are the one thing here worth letting finish.
A snapshot run validates a copy. gitutil.MaterializeTree checks the snapshot out into a temp worktree (opt-in: asyncValidateWorktree), so the checks cannot be raced by an edit — and a result about a copy is reported even when the live tree has moved, since it is still exactly true about the state it ran against. --attribute-to keeps the event log and the project breadcrumb under the real repository rather than the copy.
History tightens, never loosens. riskHistory keeps each project's finished runs (risk-history.jsonl: sizes and verdicts, no paths, no content) and answers what an absolute threshold cannot — is this change large for this repo, and do changes this size fail here. It can hold a run the rules would have released and raise a score; it can never release one or lower a score. A codebase where every change is enormous must not teach the daemon that enormous is fine.
Where git cannot answer, the daemon measures the tree itself. A repository with no commits has no HEAD to diff against, and a directory that was never one has nothing at all; both used to be unmeasurable, and so blocked every run. filestate walks and hashes instead (Index, Changes, Digest), which also gives the staleness check an identity to compare against — see fingerprintTree. It is the fallback, not the default: git knows what is ignored, and knows how much of a modified file an edit touched, where filestate can only tell that it changed and so counts the whole file. That over-measures, which errs towards blocking.
The snapshot never touches the developer's repository state. It stages into a throwaway index (GIT_INDEX_FILE in a temp file, seeded from the real index for its stat cache, without which every file is re-hashed per call). The objects it writes are unreferenced, so git's gc may collect a baseline; ChangesBetween then errors and the assessment falls back to HEAD, which reads larger and so errs towards blocking.

Sidecar Sync Strategy (`internal/sidecar/`)

Everything syncs with rsync. RsyncSync / RsyncSyncEphemeral back chunk sidecar sync and variants; rsyncPoolSidecar backs the sidecar pool (seed, fan-out, stale replacement, dead-sidecar replacement), and so pooled chunk validate, chunk review and chunk mutate. All of them funnel into rsyncTo.

The git-bundle strategy and the checkout/patch strategy are gone. #539 removed them, #564 (pool primitives) brought bundle sync back by accident because the pool branch predated #539, and the pool was moved to rsync afterwards. Do not add a second sync strategy without changing this section.

Consequences worth knowing:

rsync is stateless, so the pool no longer records a last_synced_ref; there is no incremental-bundle bookkeeping. Old pool state files that still carry the key load fine.
The synced .git directory carries branch and remote-tracking refs, so the sidecar mirrors them without the ref-fixup scripts bundle sync needed.
Each pool member opens its own SSH proxy and walks the tree, where bundle sync built once and sent N times. Concurrency is capped by syncFanOutConcurrency, currently 8, so a pool that size or smaller syncs every member at once. Measured on this repo against warm sidecars: fan-out to 4 cost 1.01x a single sync (10.2s against 10.1s) and to 8 cost 1.13x (10.3s against 9.1s). Wall time is dominated by the per-sidecar round trip, not by repeating the walk, so the N-times cost bundle sync avoided does not show up until a pool exceeds syncFanOutConcurrency.
Pool tests replace poolSync, since rsync cannot run against the fake SSH server.

Data Flow: merge conflict advisories

The chunkd daemon answers "does this branch still merge cleanly?" out of band, so the hook that reports it never computes anything:

chunkd daemon (one process, all projects)
  pollLoop            every 5s   → sidecar state, event log, git HEAD
  checkConflictsLoop  every 60s  → per project:
                                   gitutil.FetchRemoteBranch  (every 3 min)
                                   gitutil.RevParseCtx        (target + HEAD)
                                   gitutil.PreviewMerge       (merge-tree)
                                   → projectState.conflict

chunk conflicts --hook
  → GET /conflicts?root=…  (Unix socket)
  → chunkd.ConflictNotice  → hookSpecificOutput.additionalContext on stdout

Three properties the layering exists to hold:

The conflict check never runs on the poll path. A fetch reaches the network and a merge preview reads the object database; neither may delay the dashboard, or the snapshot read a hook is waiting on. It is a separate goroutine with its own ticker, and every git call happens outside daemon.mu — only the final state assignment takes the lock.
Nothing mutates a developer's checkout. gitutil.PreviewMerge uses git merge-tree --write-tree, which resolves the merge in the object database and touches no working tree, index, or HEAD. A background process must not be able to disturb a tree somebody is editing.
"No answer" is distinct from "no conflict". ConflictState.Unavailable and ConflictReport.Known keep a detached HEAD, an unfetched target, and an absent daemon from being reported as a clean merge. Collapsing them would have the hook tell an agent a branch is clean when nothing ever looked.

Change Measurement (`internal/changeset/`, `internal/gitutil/`, `internal/filestate/`)

changeset.Changes is the shape of one answer — how far a working tree has moved from an earlier state — and lives in its own package because there is more than one way to find it and they have to agree on what the answer looks like.

gitutil measures with git: WorkingChanges against HEAD, SnapshotTree and ChangesBetween against an earlier uncommitted state.
filestate measures by walking and hashing, for trees git cannot answer for.

The two are not equivalent, and the difference is documented where it bites: git knows what is ignored and how much of a modified file changed; filestate only knows a file changed, so it counts the file. Both err in the same direction — over-measuring, which makes somebody wait — because the opposite error is a change that reads smaller than it is and gets waved through.

HTTP Client (`internal/httpcl/`)

Shared HTTP infrastructure used by anthropic/, circleci/, and github/:

JSON request/response encoding by default
Automatic retry via hashicorp/go-retryablehttp (up to 3 retries)
Configurable auth (Bearer token or custom header like x-api-key)
Fluent request builder: httpcl.NewRequest(method, path, opts...)

Error Handling

Two-tier error model

Every error returned from a command carries two perspectives:

Developer error — the wrapped error chain for logs and debugging.
User-facing message — a plain-English sentence shown on stderr.

The userError struct in internal/cmd/usererr.go satisfies both error and a set of display interfaces:

type userError struct {
    msg        string // brief user-facing headline
    detail     string // optional clarification
    suggestion string // optional actionable hint
    err        error  // underlying Go error (for errors.Is / As)
}

func (e *userError) UserMessage() string  { return e.msg }
func (e *userError) Detail() string       { return e.detail }
func (e *userError) Suggestion() string   { return e.suggestion }
func (e *userError) Unwrap() error        { return e.err }

The display interfaces (UserMessage, Detail, Suggestion) are checked via type assertion at the top-level error handler — they are not imported as a named interface. This keeps the error type private to cmd.

Single error-rendering boundary

All formatting happens in main(), never inside command handlers:

func main() {
    if err := rootCmd.Execute(); err != nil {
        msg, detail, suggestion := errorDetails(err)
        fmt.Fprint(os.Stderr, ui.FormatError(msg, detail, suggestion))
        os.Exit(1)
    }
}

errorDetails probes the error for the three display interfaces via duck typing, then falls back to sensible defaults (the raw .Error() string as detail, pattern-matched hints as suggestion).

Rules:

Command handlers must never call ui.FormatError or print styled error text themselves. Return the error; let the boundary format it.
Never use a sentinel "silent" error to suppress output. Every non-nil error produces output through the single boundary.
Helpers like notAuthorized(action, err) and sshSessionError(err) can inspect an error and return a *userError (or nil to signal "not my error"). The caller chains them:
```
if err := notAuthorized("sync files", err); err != nil { return err }
```

Typed package-level errors

API client packages export sentinel errors and typed error structs so callers use errors.Is / errors.As instead of string matching:

// internal/anthropic
var ErrKeyNotFound = errors.New("api key not found")
var ErrTokenLimit  = errors.New("prompt exceeds context window")

type StatusError struct {
    Op         string
    StatusCode int
}

HTTP client packages must not leak the shared httpcl.HTTPError type to callers. Instead, wrap it into a package-local StatusError via a mapErr helper:

func mapErr(op string, err error) error {
    var he *hc.HTTPError
    if !errors.As(err, &he) { return err }
    return &StatusError{Op: op, StatusCode: he.StatusCode}
}

Display and UI Decoupling

Business logic never imports `ui`

The ui package owns all ANSI styling (Bold, Dim, Red, Green, Warning, Success, FormatError). Business logic in internal/ must not import it.

Use callback injection for progress reporting via iostream.StatusFunc:

// iostream/status.go
type Level int
const (
    LevelStep Level = iota
    LevelInfo
    LevelWarn
    LevelDone
)
type StatusFunc func(level Level, msg string)

The cmd layer wires the callback to styled output:

func newStatusFunc(streams iostream.Streams) iostream.StatusFunc {
    return func(level iostream.Level, msg string) {
        switch level {
        case iostream.LevelStep:
            streams.ErrPrintln(ui.Bold(msg))
        case iostream.LevelInfo:
            streams.ErrPrintf("  %s\n", ui.Dim(msg))
        case iostream.LevelWarn:
            streams.ErrPrintf("  %s\n", ui.Warning(msg))
        case iostream.LevelDone:
            streams.ErrPrintf("  %s\n", ui.Success(msg))
        }
    }
}

Business logic accepts StatusFunc as a parameter and calls it for progress output. Tests can pass a no-op or capturing stub.

Test Assertions

Use gotest.tools/v3/assert for test assertions:

Prefer assert.Check to keep the test running and collect as many failures as possible in a single run
Use assert.Assert / assert.NilError as gates — only when failure means the remaining assertions are pointless or unsafe (e.g. a nil pointer would panic, or a missing resource means nothing else can be verified)
Do not call functions or methods directly inside the assertion; always use a temporary variable
Use cmp comparisons from gotest.tools/v3/assert/cmp for semantic matchers over raw boolean expressions

assert.Assert and assert.Check both accept three kinds of argument: a bool expression, a cmp.Comparison, or an error.

Assert vs Check

assert.Check calls t.Fail and returns false, allowing the test to continue collecting failures — prefer it by default. assert.Assert calls t.FailNow and stops immediately — use it only as a gate.

The canonical pattern: use assert.NilError (or assert.Assert) to gate on preconditions, then use assert.Check for everything else:

result, err := doSomething()
assert.NilError(t, err)                              // gate: no point checking result if err != nil
assert.Check(t, result.OK)
assert.Check(t, cmp.Equal(result.Status, "ready"))
assert.Check(t, cmp.Len(result.Items, 3))
assert.Check(t, cmp.Contains(result.Name, "prefix"))

Named functions are all fatal. assert.Equal, assert.DeepEqual, assert.Error, assert.ErrorContains, and assert.ErrorIs all call t.FailNow. To get the non-fatal equivalent, use assert.Check with the corresponding cmp comparison:

Fatal (gate only)	Non-fatal equivalent
`assert.Equal(t, a, b)`	`assert.Check(t, cmp.Equal(a, b))`
`assert.DeepEqual(t, a, b)`	`assert.Check(t, cmp.DeepEqual(a, b))`
`assert.Error(t, err, "msg")`	`assert.Check(t, cmp.Error(err, "msg"))`
`assert.ErrorContains(t, err, "sub")`	`assert.Check(t, cmp.ErrorContains(err, "sub"))`
`assert.ErrorIs(t, err, target)`	`assert.Check(t, cmp.ErrorIs(err, target))`
`assert.Assert(t, <bool or cmp>)`	`assert.Check(t, <bool or cmp>)`

Note: assert.Assert must be called from the goroutine running the test function. assert.Check is safe to call from any goroutine.

Temporary variable rule

Never pass a function or method call directly as an assertion argument. Always capture the result in a variable first. This applies to all calls, including error-returning functions, getters, and string conversions.

// ❌ BAD
assert.NilError(t, os.WriteFile(path, data, perm))
assert.Check(t, cmp.Equal(st.Code(), codes.NotFound))
assert.Check(t, cmp.Len(registry.List(), 0))

// ✅ GOOD
err := os.WriteFile(path, data, perm)
assert.NilError(t, err)

stCode := st.Code()
assert.Check(t, cmp.Equal(stCode, codes.NotFound))

sidecars := registry.List()
assert.Check(t, cmp.Len(sidecars, 0))

Type conversions (int32(x), string(b)) and built-in functions (len) are exempt from this rule.

Message argument

assert.Check (and assert.Assert) accept a trailing msgAndArgs ...interface{} that is appended to the failure output. Pass a message when the comparison alone does not make the intent obvious — for example, when checking a boolean derived from non-obvious logic, when the variable name is ambiguous, or when the test loops over cases and you need to identify which iteration failed.

// ❌ Opaque — failure says "false" with no context
assert.Check(t, got.ExpiresAt.Before(deadline))

// ✅ Clear — failure says what the check was verifying
assert.Check(t, got.ExpiresAt.Before(deadline), "token must expire before session deadline")

// ❌ In a loop — impossible to tell which item failed
for _, item := range items {
    assert.Check(t, cmp.Equal(item.State, "ready"))
}

// ✅ In a loop — failure identifies the offending item
for _, item := range items {
    assert.Check(t, cmp.Equal(item.State, "ready"), "item %q", item.ID)
}

Skip the message when the comparison is already self-documenting — cmp.Equal, cmp.DeepEqual, cmp.Len, and cmp.ErrorIs all produce structured failure messages that include the values involved, so they rarely need extra annotation.

Examples

import (
    "gotest.tools/v3/assert"
    "gotest.tools/v3/assert/cmp"
)

func TestSomething(t *testing.T) {
    // Gate: fail immediately if setup fails — nothing else can run
    err := startContainer(ctx)
    assert.NilError(t, err)

    result, err := doSomething()
    assert.NilError(t, err)  // gate: result is meaningless if err != nil

    // Check everything else — collects all failures in one run
    assert.Check(t, cmp.Equal(result.Status, "ok"))
    assert.Check(t, cmp.Len(result.Items, 3))
    assert.Check(t, result.Ready)
}

Semantic Matchers

Use the most specific assertion for the situation. Prefer named functions over raw boolean expressions when a named function exists. Fall back to assert.Check(t, <bool>) for comparisons that have no dedicated function — the expression source code appears verbatim in the failure message, which is good enough.

Core functions

Situation	Preferred form (non-fatal)	Gate form (fatal)
`err` must be nil	—	`assert.NilError(t, err)`
Two scalar values must be equal (`==`)	`assert.Check(t, cmp.Equal(actual, expected))`	`assert.Equal(t, actual, expected)`
Complex values must be equal (go-cmp diff)	`assert.Check(t, cmp.DeepEqual(actual, expected))`	`assert.DeepEqual(t, actual, expected)`
Error must match exact message	`assert.Check(t, cmp.Error(err, "msg"))`	`assert.Error(t, err, "msg")`
Error must contain substring	`assert.Check(t, cmp.ErrorContains(err, "sub"))`	`assert.ErrorContains(t, err, "sub")`
Error must match sentinel / wrapped error	`assert.Check(t, cmp.ErrorIs(err, target))`	`assert.ErrorIs(t, err, target)`
Anything else	`assert.Check(t, <bool or cmp>)`	`assert.Assert(t, <bool or cmp>)`

Nil and emptiness

There are no dedicated NotNil or NotEmpty helpers. Use boolean expressions — the source is included in the failure message:

// Use Assert as a gate when nil would cause a panic below
assert.Assert(t, result != nil)

// Use Check for non-fatal emptiness assertions
assert.Check(t, cmp.Nil(result))
assert.Check(t, len(items) != 0)

Numeric comparisons

Express comparisons directly as boolean expressions:

assert.Check(t, x > 0)
assert.Check(t, a >= b)
assert.Check(t, count < limit)

String contains

To check if a string contains a substring:

result := "this is the haystack"
assert.Check(t, cmp.Contains(result, "needle"))

Length and containment

Use cmp comparisons for richer failure messages:

// Length — prints expected vs actual length on failure
assert.Check(t, cmp.Len(items, 3))

// Containment — works for slices, maps, and strings
assert.Check(t, cmp.Contains(slice, item))
assert.Check(t, cmp.Contains(mapping, "key"))
assert.Check(t, cmp.Contains(str, "substr"))

Structured data

Use assert.Check(t, cmp.DeepEqual(...)) for structs, slices, and maps. It uses go-cmp and produces a clear diff on failure:

assert.Check(t, cmp.DeepEqual(result, myStruct{Name: "title"}))
// assertion failed: ... (diff of the two values)

For unordered slice comparison, pass cmpopts.SortSlices from github.com/google/go-cmp/cmp/cmpopts:

assert.Check(t, cmp.DeepEqual(actual, expected,
    cmpopts.SortSlices(func(a, b string) bool { return a < b }),
))

For JSON, unmarshal first and use DeepEqual:

var actual, expected MyType
err := json.Unmarshal(data, &actual)
assert.NilError(t, err)
assert.Check(t, cmp.DeepEqual(actual, expected))

Pattern matchers

assert.Check(t, cmp.Regexp(`^\d{4}-\d{2}-\d{2}$`, dateStr))

Custom comparisons

assert.Check (and assert.Assert) accept any cmp.Comparison — a function that returns a cmp.Result. Use this for domain-specific checks:

withinTolerance := func(got, want, delta float64) cmp.Comparison {
    return func() cmp.Result {
        if math.Abs(got-want) <= delta {
            return cmp.ResultSuccess
        }
        return cmp.ResultFailure(fmt.Sprintf("%v not within %v of %v", got, delta, want))
    }
}

assert.Check(t, withinTolerance(actual, expected, 0.01))

FilesExpand file tree

ARCHITECTURE.md

Latest commit

History