Run prisma dev converges in the prisma process

.drive/projects/alchemy-programmatic-api/deferred.md

@@ -0,0 +1,41 @@
+# Deferred
+
+## Deploy still sets the reporter variables in `process.env`
+
+Slice 4 sends every Management API request through the engine's `ctx.api` and writes no credential into `process.env`. The Prisma Cloud provider, its client and the state store take the client as a value. What remains in the converge's environment overlay is each reporter's variables (`PRISMA_BUILD_ID`, read by the state store's resource reporting), plus the change of working directory for Alchemy's `.env` and `.alchemy`. Both are Composer's own and are why one process runs one converge at a time (`DEPLOY.CONVERGE_RUNNING`).
+
+Wanted: the reporter hands its build id to the lowering as a value, as `LowerOptions.credentials` now carries the client, so the overlay is empty. The working directory remains until Alchemy takes a directory for `.env` and `.alchemy` as a value.
+
+Origin: Will, 2026-11-06, while discussing the `prisma login` token gap in #405. Slice 4 did the credential part.
+
+## The engine does not tell commands where `ctx.api` sends requests
+
+With a supplied client, Composer recognises Management API requests by the origin it computes from `PRISMA_MANAGEMENT_API_URL`, the variable the `prisma` CLI reads to build `ctx.api`. That only works while both read the variable the same way. A caller that passes its own client must also set `PRISMA_MANAGEMENT_API_URL` to match it.
+
+Wanted: `@prisma/cli-engine` exposes the client's base URL to commands (it already holds it as `apiBaseUrl` in its Management API client config), the deploy command passes it with the client in `CallerCredentials`, and Composer routes by that value instead of reading any environment variable. `@prisma/cli-engine` is our own package in prisma-cli, so this is our follow-up, not an upstream ask.
+
+Origin: slice 4 review (D02).
+
+## Destroy still authenticates with a token in a child process
+
+Deploy runs Alchemy in process and authenticates every request through `ctx.api`. Destroy still runs Alchemy in a child process with `needs: { credentials: 'child' }`, so it gets a token snapshot that cannot refresh, and a `prisma login` session works differently for the two commands.
+
+Wanted: destroy runs in process like deploy and passes `ctx.api` the same way.
+
+Origin: slice 4 review (D03).
+
+## Alchemy's HTTP state store requires a token
+
+`makeHttpStateStore` takes a required `authToken` and always sets `Authorization: Bearer <authToken>`. Composer passes an empty string, and the deploy's `HttpClient` removes the header before the request reaches the client. That works, but only because the adapter removes every `Authorization` header.
+
+Wanted: upstream makes `authToken` optional and sets no header without one, the same change alchemy-run/alchemy#3101 made for `PrismaEnvironment`. Then Composer passes no token.
+
+Origin: slice 4 review (S03).
+
+## Alchemy's `PrismaEnvironment` has no type for "the HTTP client authenticates"
+
+Upstream's `PrismaEnvironment` only allows `type: 'serviceToken'`, and `source.type` is `'stored'` or `'env'`. With a supplied client, Composer has no token and the client authenticates each request, so Composer sets `type: 'serviceToken'` and `source: { type: 'stored', details: … }`, with `details` saying the caller's client holds the credential. That is the closest honest value the type allows, but any upstream code that branches on `type` would still read it as a service token.
+
+Wanted: upstream adds a variant for a caller-authenticated HTTP client (for example `type: 'transport'` with no token), on alchemy-run/alchemy#3101 or a follow-up.
+
+Origin: slice 4 review (S05).

.drive/projects/alchemy-programmatic-api/design-notes.md

@@ -0,0 +1,30 @@
+# Design notes
+
+## Decisions
+
+- **No generated code.** The operation calls `lower()` and passes the stack value to `Alchemist.plan`. Containers, preflight results and bundles go in as arguments.
+- **No child process.** Alchemy runs in the `prisma` process. The Alchemy copy is the one `@prisma/composer-cli` depends on, like every other module the host loads.
+- **Output through the engine.** `Alchemist.onEvent` maps Alchemy's typed events (`apply.resource.status` and friends) to engine `step-started` / `step-finished` events under the existing `deploy.apply` step. `SigilRenderer` is not used: it would give `prisma deploy` a second output format and nothing in JSON mode. `Alchemist.layer()`'s console sink is replaced with a logger that sends Alchemy's log lines to stderr.
+- **Result without a file.** `apply` returns the stack output. The `report` hook `lower()` already takes is called in the same process, so `DeploymentSummary` comes back as a value. `captureEngineFailure` and the result file go away for deploy and dev.
+- **Ctrl-C.** The handler runs the effect with an interrupt on the engine's abort signal. Exit code comes from the operation's result; the ADR-0044 child-status exception no longer applies to deploy and dev.
+- **Dev.** Each converge is its own `plan` + `apply` in its own scope, so the lease is held only during a converge. Local providers stay Composer's own local target (`localTargetProviders`), not Alchemy's `dev: true`.
+
+## Credentials
+
+Alchemy's Prisma provider reads `PRISMA_SERVICE_TOKEN` (and Composer's target reads `PRISMA_WORKSPACE_ID`) from config, and `Alchemist` builds config only from `process.env` plus `.env`.
+
+- When the token is already in the environment (CI, the e2e workflow, `deploy.env`), it works unchanged.
+- In an interactive `prisma login` session there is no token in the environment, and the engine never gives a handler the credential. Today `needs: { credentials: 'child' }` writes it into the child's environment.
+
+Workaround for this project: set `PRISMA_SERVICE_TOKEN` / `PRISMA_WORKSPACE_ID` in `process.env` for the duration of the apply. That needs the engine to hand the handler the active access token.
+
+Upstream asks:
+
+1. Alchemy: let `plan` take config overrides (a `ConfigProvider` or a record) instead of reading only `process.env`.
+2. `@prisma/cli-engine`: a credentials need for in-process use, for example `needs: { credentials: 'process' }`, that gives the handler the active access token (refreshed) and workspace id.
+
+## Alternatives considered
+
+- **Keep a child process running a Composer-owned runner.** Rejected: extra process and IPC for nothing the in-process path lacks.
+- **Keep generating `alchemy.run.ts` and load it with `Alchemist.importStack`.** Rejected: the file existed so a failed run could be rerun by hand with the Alchemy CLI; with the API, writing code to import it back is circuitous.
+- **`SigilRenderer` for output.** Rejected above.

.drive/projects/alchemy-programmatic-api/handover.md

@@ -0,0 +1,72 @@
+# Handover: Alchemy in process (`alchemy-programmatic-api`)
+
+Written 2026-11-11 by maat-73, at a rate-limit break. Previous session's full transcript (read it for any detail this brief lacks): `/Users/wmadden/.claude/projects/-Users-wmadden-Projects-prisma-composer--claude-worktrees-composer-issue-191-033e6e/f959da3b-bf5f-4f06-9416-edab3001f618.jsonl`. It is long; search it rather than reading it end to end.
+
+## What the project is
+
+Composer used to deploy and run dev by generating `.prisma-composer/alchemy.run.ts` and spawning the `alchemy` CLI as a child process. This project replaces that with Alchemy's in-process programmatic API, `alchemy/Alchemist` (`plan` / `apply`), called from the `prisma` process. Deploy first, then dev. Destroy is out of scope and still uses the child process.
+
+Read first, in this folder: `spec.md`, `plan.md`, `design-notes.md`, `slice-1-spec.md` to `slice-4-spec.md`, `deferred.md`, `upstream-asks.md`, and `reviews/`. In the repo: `docs/design/01-principles/`, ADR-0064 (deploy runs Alchemy in process, amended for dev), ADR-0056 (dev rebuild behaviour), ADR-0005 / ADR-0047, `docs/design/10-domains/deploy-cli.md` and `local-dev.md`. The repo's `CLAUDE.md` requires reading the design docs before any design discussion.
+
+## The PRs
+
+| PR | Branch | What it does | State on 2026-11-11 |
+|---|---|---|---|
+| prisma/composer#404 | `claude/alchemy-api-evaluation-5f1d17` | Alchemy `2.0.0-beta.85`, effect 4.1, all pins in the pnpm catalog, `check:publish-deps` | **Merged** (squash `3a8006d0`). |
+| prisma/composer#405 | `claude/alchemy-deploy-in-process` | `prisma deploy` runs Alchemy in process. Slice 4: deploy works after `prisma login` with no `PRISMA_SERVICE_TOKEN` | Draft, base `main`. **Conflicts with `main` again** (new main commits #424 and #426, see below). |
+| prisma/composer#411 | `claude/alchemy-dev-in-process` | `prisma dev` runs Alchemy in process | Draft, base #405's branch. One `Test` failure on the last run, not investigated. This branch also carries this Drive folder. |
+
+Both branches are pushed to the `bot` remote (`git@github-wmadden-electric:prisma/composer.git`). Push only through `bot`. Never force-push. Never rebase.
+
+## The blocker: an Alchemy release
+
+Slice 4 needed two upstream changes, both **merged** on 2026-11-07:
+
+- alchemy-run/distilled#1076: `@distilled.cloud/prisma` `Credentials.apiToken` optional; header sent only when set.
+- alchemy-run/alchemy#3101: `PrismaEnvironment.serviceToken` optional; a caller-built environment may omit it and authenticate through the `HttpClient` it supplies. Env and profile resolution still require a token.
+
+**No Alchemy release contains them yet.** npm `latest` is still `alchemy` `2.0.0-beta.85`, `@alchemy.run/prisma` `2.0.0-beta.85`, `@distilled.cloud/prisma` `1.0.0-rc.18`. Until then #405 and #411 pin preview builds in `pnpm-workspace.yaml` (`pkg.alchemy.run` URLs; `alchemy` pinned to the exact tarball URL because `check:npm-effect-resolution` needs it). `check:publish-deps` refuses URL pins, so these PRs cannot merge until the release. When it ships: swap the three catalog entries to the released versions (follow `skills-contrib/upgrade-alchemy-effect/SKILL.md`), `pnpm install`, rerun checks, then mark #405 ready.
+
+## How slice 4 authenticates (decided with Will; do not reopen)
+
+`ctx.api`, the CLI engine's Management API client, is passed to Composer and becomes the `HttpClient` the Alchemy Prisma provider and Composer's state store use. Reason, in Will's words: the CLI manages authentication centrally; Composer does not manage its own token; when authentication fails, the CLI refreshes and retries (later perhaps re-login or switching credentials) inside the pending request, so Alchemy and Composer never see it. A token getter, even one called per request, cannot do that. Sam Goodwin (Alchemy) asked about this on #3101; Will answered and Sam merged.
+
+- Adapter: `packages/1-prisma-cloud/0-lowering/lowering/src/deploy-http-client.ts`. Management API requests go through the client's per-method calls; other hosts (presigned uploads) go to the node transport unauthenticated. `FetchHttpClient.Fetch` is provided per request (the live 401 bug, fixed in `d7125d31`, regression test in `state/__tests__/state-api.test.ts`).
+- `credentials.ts`: origin from `PRISMA_MANAGEMENT_API_URL` (default `https://api.prisma.io`); conflicts with `PRISMA_API_URL` refuse with `DEPLOY.MANAGEMENT_API_URL_CONFLICT`.
+- Core type `CallerCredentials` (renamed from `ContainerCredentials`, no alias) on `LowerOptions.credentials`. CLI `family/commands/deploy.ts` passes `{ client: ctx.api, workspaceId }`.
+- `CONVERGE_CONTRACT` is 2 (no released app speaks an earlier contract; keep it 2).
+- Known gap, not blocking: log tailing (`tailDeploymentLogs`) opens a WebSocket outside `HttpClient`, so a caller with no token cannot tail. Composer never calls `tail`. Closing it needs a CLI mechanism for WebSockets equivalent to the HTTP one. Record it in `deferred.md` if not there.
+
+Live QA of slice 4 passed (`reviews/slice-4/manual-qa.md`). The QA environment's bucket and service were deleted by Will. The empty Branch `maat73-login-qa` in project `bucket-example` (`proj_cfkzdu54l1qzgmjgrsfq9jkp`) still exists with old Alchemy state rows; deleting it is Will's call.
+
+## Slice 3 (dev in process): the open design decision, now settled
+
+In process, Node and Bun cache the app's definition modules, so a rebuild would re-apply the graph loaded at start. #411 currently detects edits to any app file the entry loaded (`recordAppFiles` in `entry-resolution.ts`, `files` on `LoadedEntry`) and emits `definition-changed`, which pauses rebuilds and tells the user to restart dev.
+
+**Will agreed on 2026-11-11 to option C:** on a definition change, load the app definition in a short-lived child process or worker that hands the graph back; the converge itself stays in process. Same behaviour on Node and Bun, and wiring edits keep applying live as ADR-0056 promised. Do not do "restart required" (Will rejected it: the CLI keeps emulators running, possibly detached) and not Node-only live reload.
+
+To do for slice 3:
+
+1. Design the loader (child vs worker; how the graph crosses the boundary; what happens to `files` tracking; error reporting for a broken edit). Check the graph is serialisable or what has to be reconstructed; if it is not, bring the finding to Will before building.
+2. Implement on `claude/alchemy-dev-in-process`, replacing the `definition-changed` pause.
+3. Record the decision in a new ADR that amends ADR-0056; point to it from ADR-0064's dev amendment (review S08).
+4. Address the rest of `reviews/slice-3/`: F02 (Ctrl-C during a rebuild waits for the pipeline), F03 (resource steps left open after a failed converge), S03 (docs over-claim detection; four names for one concept), S05–S07, S09 (an integration test that edits a definition file while dev runs, under Node and Bun if CI runs Bun). F01/S01 are mostly superseded by option C. S04 is done (contract stays 2).
+5. `/drive-code-review` round 2, fix, then manual QA of `prisma dev` (foreground and `--detach`).
+
+## Other remaining work
+
+- **Merge `main` into #405, then #405 into #411.** New on main since `3a8006d0`: #424 "stop the app's services before exiting, so Ctrl-C under npm frees their ports" (dev) and #426 "prisma deploy stops copying secrets from your local environment to Prisma Cloud" (deploy). #426 touches deploy, so port it into the in-process path deliberately, as the previous merges did for main's app configuration file and secret redaction (see merge commit `1602a7f6` on #405 and `19a1aeb1` on #411). Last time this took an Opus implementer agent per PR with a precise brief.
+- **Check #411's failing `Test` job** (run 42763154117). Several local-dev tests are flaky in CI on main too (timeouts in `local-target-current.test.ts`, dev-emulators); confirm whether this is one before re-running.
+- **Update the #405 and #411 PR descriptions** for slice 4 and the merges: deploy needs no service token after `prisma login`; preview pins until the Alchemy release; the in-process path writes no app configuration file; the secret-in-plain-value refusal now applies only to destroy.
+- **Run `/drive-code-review` on #405** again: the state-store fix and three merges of `main` happened after its last review.
+- **Note for the PR description / release notes:** removing `ContainerCredentials` without an alias breaks extensions importing it from `@prisma/composer/config`.
+- **Close-out** (`plan.md`): verify the spec's done-when; move `deferred.md` items to `.drive/deferred.md` or Linear; record destroy's migration; delete this folder.
+
+## Rules that bit this project
+
+- Will's global `CLAUDE.md`: design with Will, then execute without asking; `/drive-code-review` every slice; Opus (`model: "opus"`) for every subagent; never run `pnpm test:integration`, `test:e2e` or `test:all` in full (run single files with `bun run test/<file>` from `test/integration`); no `/tmp`, use `wip/`; plain English, short; no question UI; no floating task cards.
+- Commits: `git commit -s --trailer "Signed-off-by: Will Madden <madden@prisma.io>"` plus `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`. PR bodies end with `Agent: <name>` above the Claude Code line.
+- Stage explicit paths. A `git add -A --` with an empty path list once staged this whole untracked folder into a merge commit.
+- When upstream (Alchemy, Distilled) is involved, read the provider source first, and follow each repo's written contributor rules exactly. Prisma owns every `@prisma/*` package, including the CLI engine in prisma-cli; Alchemy, Distilled and Effect are upstream.
+- Do not accept an external maintainer's design advice without checking it against the CLI's design (`prisma-cli` `packages/cli-engine/src/execution/api-client.ts` and `.drive/projects/prisma-cli-v8/specs/s3-composer.md`). Discuss with Will before proposing.
+- Deploy credentials for live QA: `set -a; . ~/.config/prisma-compose/deploy.env; set +a`, `PRISMA_REGION=us-east-1`. Never print them. Slice 4 QA instead used only a `prisma login` session.

.drive/projects/alchemy-programmatic-api/plan.md

@@ -0,0 +1,35 @@
+# Plan: Alchemy in-process
+
+**Spec:** `.drive/projects/alchemy-programmatic-api/spec.md`
+
+Stack: 1 → 2 → 3. Each slice is one PR.
+
+## Slice 1: Alchemy preview and effect 4.1
+
+Move every `alchemy` pin to the preview tarball `https://pkg.alchemy.run/alchemy/972f294ae0c7982a8ebcb551fadd07d29c9c0fca`, and the `effect` set to the 4.1 line, following the `upgrade-alchemy-effect` skill. Fix what breaks between beta.83 and that commit. The subprocess path is unchanged.
+
+- Hands to: a tree where `alchemy/Alchemist` is importable and e2e deploy is green.
+- Done when: `pnpm test:packages`, the local-dev integration tests and the e2e deploy workflow pass.
+
+## Slice 2: Deploy in-process
+
+`deploy` lowers the app and calls `Alchemist.plan` / `apply` in the `prisma` process. Alchemy events become engine step events. Ctrl-C interrupts. Credentials per design-notes.md § Credentials. `lower()` takes containers and preflight results as values. New ADR superseding ADR-0007 for deploy; `deploy-cli.md` updated. Destroy and dev still spawn. Rebuild the lowering scratch-stack test harness on `alchemy/Alchemist`, or delete it (slice 1 review S07).
+
+- Builds on: slice 1.
+- Hands to: an in-process converge function dev can reuse.
+- Done when: e2e deploy workflow passes; `prisma deploy` writes no `alchemy.run.ts`; Ctrl-C manual QA passes.
+
+## Slice 3: Dev in-process
+
+Each `prisma dev` converge runs `plan` / `apply` in-process through slice 2's function. Retires the dev stack file. ADR and `deploy-cli.md` extended to dev.
+
+- Builds on: slice 2.
+- Done when: local-dev integration tests pass; `prisma dev` writes no `alchemy.run.ts` and spawns no Alchemy child.
+
+## Close-out
+
+- [ ] Verify the spec's Project DoD.
+- [x] Swap the preview URL for the first Alchemy release containing the API: 2.0.0-beta.85, commit 398ed309 on slice 1, merged up. Core-concepts skill sentence updated.
+- [ ] Record destroy's migration or deletion in `.drive/deferred.md`.
+- [ ] Move the items in this project's `deferred.md` to `.drive/deferred.md` (or Linear) before deleting the project folder.
+- [ ] Delete `.drive/projects/alchemy-programmatic-api/`.

.drive/projects/alchemy-programmatic-api/slice-1-spec.md

@@ -0,0 +1,18 @@
+# Slice 1: Alchemy preview and effect 4.1
+
+## Goal
+
+Every package in the repo depends on the Alchemy preview build `https://pkg.alchemy.run/alchemy/972f294ae0c7982a8ebcb551fadd07d29c9c0fca` instead of `2.0.0-beta.83`, and the `effect` set (`effect` and every `@effect/*` companion) moves to one 4.1 version that satisfies that build's peers. Alchemy still runs as a child process. No behaviour changes.
+
+## Done when
+
+- `import * as Alchemist from 'alchemy/Alchemist'` resolves from `@prisma/composer-cli`.
+- One copy each of `alchemy` and `effect` in the workspace; `check:npm-effect-resolution` and the CLI's effect-resolution check agree with the new pins.
+- `pnpm build`, `pnpm typecheck`, `pnpm lint`, `pnpm test:packages` pass.
+- The local-dev integration test files pass locally (run only those files).
+- The e2e deploy workflow passes in CI on the PR.
+- The `upgrade-alchemy-effect` skill and any docs that name the old versions are updated, including how to pin a preview URL.
+
+## Out of scope
+
+Any use of `alchemy/Alchemist`. That is slice 2.