diff --git a/.gitignore b/.gitignore index eb24547..baa30cb 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ .env .env.bak* +!*.env.enc data htpasswd *.log diff --git a/.mcp.json b/.mcp.json deleted file mode 100644 index c942808..0000000 --- a/.mcp.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "mcpServers": { - "code-review-graph": { - "command": "uvx", - "args": [ - "code-review-graph", - "serve" - ], - "type": "stdio" - } - } -} diff --git a/.opencode.json b/.opencode.json deleted file mode 100644 index 81d79fa..0000000 --- a/.opencode.json +++ /dev/null @@ -1,13 +0,0 @@ -{ - "mcpServers": { - "code-review-graph": { - "command": "uvx", - "args": [ - "code-review-graph", - "serve" - ], - "type": "stdio", - "env": [] - } - } -} diff --git a/.sops.yaml b/.sops.yaml index 938ab94..ddfd93e 100644 --- a/.sops.yaml +++ b/.sops.yaml @@ -1,7 +1,9 @@ -# SOPS configuration for encrypting secrets committed to the repo +# SOPS configuration for encrypting service-local .env files +# Each service directory (e.g., postgres/, traefik/) stores an encrypted .env.enc +# alongside its .env.example. Decrypt with: stackctl.sh secrets decrypt [service] creation_rules: - - path_regex: secrets/.*\.(env|yaml|yml)$ - # Replace with your age public key(s) - age: - - "age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # Replace with your real age public key + - path_regex: \.env(\.enc)?$ + key_groups: + - age: + - age12ph7sgtptrrcxzxdue28j3lesnu9gj73ae9ewuvg66awp6jpae8smyvspx encrypted_regex: '^(?!#)' diff --git a/README.md b/README.md index afe5656..7e7eb31 100644 --- a/README.md +++ b/README.md @@ -234,7 +234,7 @@ cd ../portainer docker-compose up -d ``` -### 5. Access the services +### 6. Access the services - Grafana: - Prometheus: @@ -261,12 +261,22 @@ local-stack/ ## Configuration -Each component has its own environment file for configuration. Copy the example files and modify as needed: +Each component has its own environment file for configuration. For local development without secrets management: ```sh find . -name ".env.example" -exec sh -c 'cp "$1" "${1%.example}"' _ {} \; ``` +For production or shared environments, use the SOPS + age secrets workflow (see [Managing Secrets](docs/Managing%20Secrets.md)): + +```sh +# Decrypt and deploy in one step +./stackctl.sh secrets deploy + +# Or decrypt manually for inspection +./stackctl.sh secrets decrypt +``` + ## License ```txt diff --git a/docs/Managing Secrets.md b/docs/Managing Secrets.md index aba6f67..adfe01f 100644 --- a/docs/Managing Secrets.md +++ b/docs/Managing Secrets.md @@ -1,68 +1,142 @@ # Managing Secrets Securely -Never store real secrets in `.env` files in the repo. Keep `.env.example` as documentation with placeholders only. +Secrets are encrypted at rest in git using SOPS + age and decrypted just-in-time at deploy time. Each service directory stores an encrypted `.env.enc` alongside its `.env.example`. -## Recommended approach (encrypted files committed to git) +## Workflow Overview -Use `sops` + age (or GPG) to encrypt per-environment secrets that can be safely committed. +``` +.env.example → .env (fill values) → .env.enc (encrypt) → git commit + ↓ +.git (encrypted) → .env.enc → .env (decrypt) → deploy → shred .env +``` + +## Setup -### 1) Install tools -- age: https://age-encryption.org -- sops: https://github.com/getsops/sops +### 1. Install tools -### 2) Create an age key pair (once) ```bash -# writes to ~/.config/sops/age/keys.txt +# macOS +brew install sops age + +# Linux +# sops: https://github.com/getsops/sops/releases +# age: https://github.com/FiloSottile/age/releases +``` + +### 2. Generate an age key pair (once per machine) + +```bash +mkdir -p ~/.config/sops/age age-keygen -o ~/.config/sops/age/keys.txt +# The public key is printed — add it to .sops.yaml key_groups ``` -Add the public recipient from that file (starts with `age1...`) to your repository SOPS config. -### 3) Add a SOPS config -Create `.sops.yaml` at repo root: +### 3. Add your public key to `.sops.yaml` + +Edit `.sops.yaml` and add your age public key to the `key_groups` list. Multiple recipients can be listed for team access: + ```yaml -# Encrypt files matching these globs with the recipient below creation_rules: - - path_regex: secrets/.*\.(env|yaml|yml)$ - age: ["AGE1_PUBLIC_KEY_HERE"] - encrypted_regex: '^(?!#)' + - path_regex: \.env(\.enc)?$ + key_groups: + - age: + - age1existingkey... # existing recipient + - age1yournewkey... # your new key ``` -Replace `AGE1_PUBLIC_KEY_HERE` with your public age key. -### 4) Create encrypted secret files -Place per-environment secrets under `secrets/` and encrypt with sops: +## Encrypting secrets + +For a single service: + ```bash -mkdir -p secrets -printf "TRAEFIK_ENABLE=true\nSSO_CREDENTIALS=admin:$apr1$...\n" > secrets/traefik.dev.env -sops -e -i secrets/traefik.dev.env +# Create .env from .env.example and fill in real values +cp postgres/.env.example postgres/.env +# Edit postgres/.env with real values... + +# Encrypt +./stackctl.sh secrets encrypt postgres ``` -The file is now encrypted at rest and safe to commit. -### 5) Decrypt for local use +For all services at once: + ```bash -# Produces a plaintext file for docker usage (do not commit this) -sops -d secrets/traefik.dev.env > traefik/.env +./stackctl.sh secrets encrypt ``` -You can add a simple make/script target to automate decrypt -> deploy -> clean. -### 6) CI/CD or remote deploy -On a deployment host, provision the age private key (read-only, secured). Decrypt secrets just-in-time before `docker stack deploy`. +This runs `sops --encrypt --input-type dotenv --output-type dotenv .env > .env.enc` for each service directory that has a `.env` file. -## Using Docker Swarm secrets (optional/advanced) -Docker Swarm supports native secrets. You can combine sops+age with `docker secret create`: +## Deploying + +Deploy decrypts, renders, deploys, and shreds in one step: -1) Decrypt locally in memory and pipe to secret create: ```bash -sops -d secrets/traefik.dev.env | docker secret create traefik_env - +# Deploy a specific service's stack +./stackctl.sh secrets deploy postgres + +# Deploy all services +./stackctl.sh secrets deploy ``` -2) Reference the secret in your stack file using `secrets:` and `env_file` alternatives where appropriate. -This is more granular and keeps values out of env vars in the container filesystem, but requires adjusting service configs to read from files or environment sourced from secrets. +The deploy operation: +1. Decrypts `.env.enc` → `.env` for each target service +2. Regenerates rendered stack files (variable substitution) +3. Deploys the relevant Docker Swarm stacks +4. Shreds all plaintext `.env` files + +## Decrypting (manual) + +If you need to inspect or edit secrets without deploying: + +```bash +# Decrypt a single service +./stackctl.sh secrets decrypt postgres + +# Decrypt all services +./stackctl.sh secrets decrypt +``` + +**Remember to clean up plaintext files after editing:** + +```bash +./stackctl.sh secrets clean +``` + +## Cleaning up + +Remove all plaintext `.env` files that have a corresponding `.env.enc`: + +```bash +./stackctl.sh secrets clean +``` + +This uses `shred -u` when available, falling back to `rm -f` on systems without `shred`. + +## Key rotation + +After adding a new recipient to `.sops.yaml`: + +```bash +find . -name '.env.enc' -exec sops updatekeys --yes {} \; +``` + +## Re-encrypting after editing + +```bash +# Decrypt, edit, re-encrypt +./stackctl.sh secrets decrypt postgres +# Edit postgres/.env... +./stackctl.sh secrets encrypt postgres +./stackctl.sh secrets clean +``` ## Git hygiene -- Commit only `.env.example` files and encrypted files under `secrets/`. -- Never commit plaintext `.env`. -- Add a `.gitignore` rule for `**/.env` and `**/*.env.decrypted` as needed. + +- **Commit**: `.env.example` (placeholders) and `.env.enc` (encrypted secrets) +- **Never commit**: `.env` (plaintext, gitignored) +- The `.gitignore` rule `!*.env.enc` ensures encrypted files are tracked ## Troubleshooting -- If `sops` can’t decrypt: ensure the age private key is in `~/.config/sops/age/keys.txt`. -- For team usage: include multiple recipients in `.sops.yaml` so each developer can decrypt. + +- **`sops` can't decrypt**: Ensure the age private key is at `~/.config/sops/age/keys.txt` and the corresponding public key is in `.sops.yaml`. +- **`shred` not found**: The script falls back to `rm -f`. Install `shred` (part of `coreutils` on Linux) for secure deletion. +- **`sops` or `age` not found**: Run `./stackctl.sh doctor` to check prerequisites, or install them manually. diff --git a/docs/proposals/Secrets Management Proposal.md b/docs/proposals/Secrets Management Proposal.md index 6fcd2e7..5198856 100644 --- a/docs/proposals/Secrets Management Proposal.md +++ b/docs/proposals/Secrets Management Proposal.md @@ -25,9 +25,9 @@ Cons: Concrete steps: 1) Keep only placeholders in `.env.example`. Do not commit real `.env`. -2) Store real values in encrypted files under `secrets/` (SOPS + age). See `docs/secrets.md`. +2) Store real values in encrypted `.env.enc` files per service directory (SOPS + age). See `docs/Managing Secrets.md`. 3) At deploy time, decrypt just-in-time and create Docker secrets, e.g.: - - `sops -d secrets/postgres.dev.env | grep POSTGRES_PASSWORD= | cut -d= -f2 | docker secret create postgres_password -` + - `sops -d postgres/.env.enc | grep POSTGRES_PASSWORD= | cut -d= -f2 | docker secret create postgres_password -` 4) Reference secrets in stacks: ```yaml services: @@ -63,7 +63,7 @@ Cons: Concrete steps: 1) Generate Portainer API key (Settings → API Keys). 2) Write a small script to: - - `sops -d secrets/.env |` extract values → POST to `/api/endpoints/{id}/docker/secrets/create`. + - `sops -d /.env.enc |` extract values → POST to `/api/endpoints/{id}/docker/secrets/create`. - Trigger stack redeploy via Portainer Stack API (optional) or `docker stack deploy` locally. 3) Reference secrets in stacks as in Option A. @@ -86,10 +86,10 @@ Cons: ## Recommended plan (Option A) -Phase 1: Foundation -- Keep `.env.example` placeholders. Done. -- Add `docs/secrets.md` and `.sops.yaml`. Done. -- Create age key(s) and commit encrypted files under `secrets/` (team recipients in `.sops.yaml`). +Phase 1: Foundation ✅ +- Keep `.env.example` placeholders. ✅ Done. +- Add `.sops.yaml` and `docs/Managing Secrets.md`. ✅ Done — now uses service-local `.env.enc` pattern with `stackctl.sh secrets`. +- Create age key(s) and commit encrypted files per service directory (team recipients in `.sops.yaml`). ✅ Done — see `stackctl.sh secrets encrypt`. Phase 2: Convert priority services - Databases: switch to `*_FILE` and define `secrets:` in `stacks/infrastructure.yml`. @@ -110,13 +110,13 @@ Phase 4: (Optional) Portainer integration ## Open questions - Which services require secrets and support `*_FILE`? (Postgres yes; Redis no; Mongo users might need env vars or runtime files.) - Who holds the age private key for CI/CD? (One or more maintainers; store on deployment hosts only.) -- Rotation cadence? (Document per-service rotation procedure in `docs/secrets.md`.) +- Rotation cadence? (Document per-service rotation procedure in `docs/Managing Secrets.md`.) ## Appendix: Example secret extraction Extract a single key from an encrypted env file without writing plaintext to disk: ```bash # Create postgres password secret from an encrypted env file -sops -d secrets/postgres.dev.env \ +sops -d postgres/.env.enc \ | awk -F= '/^POSTGRES_PASSWORD=/{print $2}' \ | docker secret create postgres_password - ``` diff --git a/docs/superpowers/specs/2026-06-16-sops-age-secrets-design.md b/docs/superpowers/specs/2026-06-16-sops-age-secrets-design.md new file mode 100644 index 0000000..c9e2e5d --- /dev/null +++ b/docs/superpowers/specs/2026-06-16-sops-age-secrets-design.md @@ -0,0 +1,116 @@ +# SOPS + age Secrets Management Setup + +Date: 2026-06-16 +Status: Approved +Branch: chore/sops-age-secrets-setup + +## Context + +Local-Stack currently uses plaintext `.env` files per service, with `.env.example` as documentation. The existing `.sops.yaml` targets a `secrets/` directory that was never adopted, and `docs/Managing Secrets.md` describes a workflow that doesn't match the repo's actual tooling. The homelab repo at `../../docker/` has a working SOPS+age pattern using service-local `.env.enc` files with a `deploy.sh` that decrypts, deploys, and shreds. + +Goal: adopt the same service-local `.env.enc` pattern in local-stack, integrated with `stackctl.sh`, so secrets are encrypted at rest in git and decrypted just-in-time at deploy. + +## Architecture + +Each service folder (e.g., `postgres/`, `traefik/`, `redis/`) keeps a committed `.env.enc` — the SOPS-encrypted version of its `.env`. Plaintext `.env` is decrypted just-in-time by `stackctl.sh secrets deploy`, used for rendering and stack deployment, then shredded. Generated Swarm stacks continue referencing service-local `.env` via `env_file` — no stack file changes needed. + +``` +postgres/ + .env.example # placeholders (committed) + .env.enc # encrypted secrets (committed) + .env # plaintext (gitignored, created by decrypt) + docker-compose.yml + swarm.fragment.yml +``` + +## File Changes + +### `.sops.yaml` + +Replace the current `secrets/`-targeted config with a service-local `.env` pattern matching `../../docker`: + +```yaml +creation_rules: + - path_regex: \.env$ + key_groups: + - age: + - + encrypted_regex: '^(?!#)' +``` + +- `path_regex: \.env$` matches any `.env` file in any directory. +- `encrypted_regex: '^(?!#)'` skips comment lines during encryption (safe for dotenv). +- The age public key placeholder will be replaced at setup time. + +### `.gitignore` + +Add an allowlist rule so `.env.enc` files are committed while `.env` stays ignored: + +```gitignore +.env +.env.bak* +!*.env.enc +``` + +The `!*.env.enc` negation pattern ensures encrypted files are tracked even though `.env` is ignored. + +### `stackctl.sh` + +Add a `secrets` subcommand with four operations: + +``` +stackctl.sh secrets encrypt [service] # sops -e -i service/.env → service/.env.enc +stackctl.sh secrets decrypt [service] # sops -d service/.env.enc → service/.env +stackctl.sh secrets deploy [service] # decrypt → render → stack deploy → shred +stackctl.sh secrets clean # shred all plaintext .env files +``` + +- `[service]` is optional; omit to operate on all services that have `.env.enc`. +- `encrypt`: runs `sops --encrypt --input-type dotenv --output-type dotenv service/.env > service/.env.enc`, reading plaintext `.env` and writing `.env.enc` without modifying the original. Requires `sops` and `age` on PATH. +- `decrypt`: runs `sops --decrypt --input-type dotenv --output-type dotenv` from `.env.enc` to `.env`. +- `deploy`: chains decrypt → `render_stack_file` → `docker stack deploy` → `shred -u` on `.env`. Reuses existing `discover_env_example_dirs` to find service folders. +- `clean`: finds all `.env` files that have a corresponding `.env.enc` and shreds them. +- Prerequisite checks: `sops` and `age` must be on PATH; warn if missing. + +### `docs/Managing Secrets.md` + +Rewrite to document the service-local `.env.enc` workflow: + +1. Install `sops` and `age`. +2. Generate an age key pair: `age-keygen -o ~/.config/sops/age/keys.txt`. +3. Add the public key to `.sops.yaml`. +4. Create `.env` from `.env.example`, fill in real values, then `stackctl.sh secrets encrypt `. +5. Deploy with `stackctl.sh secrets deploy [service]`. +6. Clean up plaintext with `stackctl.sh secrets clean`. +7. Key rotation: `sops updatekeys --yes **/.env.enc` after adding a new recipient to `.sops.yaml`. + +### `README.md` + +Update the secrets/setup section to reference `stackctl.sh secrets deploy` instead of the legacy `cp .env.example .env` + `docker-compose up` flow. Keep the legacy instructions as a fallback for non-Swarm local development. + +### `docs/proposals/Secrets Management Proposal.md` + +Update Phase 1 status to reflect completion. No structural changes to the proposal — Phase 2 (Docker Swarm native secrets) and Phase 3 (Portainer integration) remain as future work. + +## What Stays the Same + +- Generated `stacks/*.yml` — untouched, already reference service-local `.env` via `env_file`. +- `.env.example` files — remain as documentation with placeholder values. +- `stackctl.sh env` — still works for listing/recreating `.env` from examples. +- `stackctl.sh doctor` — still warns about missing `.env`. +- `stackctl.sh up` — continues to work with plaintext `.env` for local development. + +## Key Management + +| Item | Location | Committed? | +|---|---|---| +| Age public key | `.sops.yaml` | Yes | +| Age private key | `~/.config/sops/age/keys.txt` | Never | +| Encrypted secrets | `**/.env.enc` | Yes | +| Plaintext secrets | `**/.env` | No (gitignored) | + +Recovery: if the server dies, you need both the `.env.enc` files (in git) and the age private key (backed up to a password manager and offline drive). + +## Phase 2 (Future, Not This PR) + +Docker Swarm native secrets (`*_FILE` env vars, `docker secret create`) as a later hardening step per the existing proposal. This would require per-service compose changes and is out of scope for this setup. \ No newline at end of file diff --git a/stackctl.sh b/stackctl.sh index cbce92d..a2e0bd8 100755 --- a/stackctl.sh +++ b/stackctl.sh @@ -19,6 +19,7 @@ Commands: logs Follow logs for key services or specified services doctor Run preflight checks and optional fixes env List or recreate .env files from .env.example (safe-guarded) + secrets Encrypt, decrypt, deploy, or clean .env.enc files generate (Re-)generate stacks/ from compose file sources sync Check if stacks/ matches compose sources; exits 1 on drift help Show this help message and exit @@ -873,10 +874,318 @@ cmd_doctor() { fi } +cmd_secrets() { + local OPERATION="${1:-}" + shift || true + + case "$OPERATION" in + encrypt|decrypt|deploy|clean) + ;; + -h|--help) + log "Manage encrypted .env.enc files with SOPS + age." + log "" + log "Usage: $SCRIPT_NAME secrets [service]" + log "" + log "Operations:" + log " encrypt [service] Encrypt .env → .env.enc for one or all services" + log " decrypt [service] Decrypt .env.enc → .env for one or all services" + log " deploy [service] Decrypt, render, deploy, then shred .env" + log " clean Shred all plaintext .env files that have .env.enc" + log "" + log "Services are discovered from .env.example files in the repo." + log "[service] accepts a directory basename (e.g., postgres) or" + log "a repo-relative path (e.g., apisix/api-gateway)." + log "If omitted, operates on all discovered services." + log "" + log "Requires: sops and age on PATH (for encrypt, decrypt, deploy)." + exit 0 + ;; + *) + err "Unknown secrets operation: ${OPERATION:-}" + log "Usage: $SCRIPT_NAME secrets [service]" + log "Run: $SCRIPT_NAME secrets --help" + exit 2 + ;; + esac + + # Discover service directories that have .env.example (reuse existing discovery) + local -a all_dirs=() + while IFS= read -r dir; do + [[ -z "$dir" ]] || all_dirs+=("$dir") + done < <(discover_env_example_dirs) + + if [[ ${#all_dirs[@]} -eq 0 ]]; then + log "No service directories found (no .env.example files)." + exit 0 + fi + + # If a service name is given, filter to that directory + # Accepts basename (e.g., postgres) or repo-relative path (e.g., apisix/api-gateway) + local -a target_dirs=() + if [[ $# -gt 0 ]]; then + local svc="$1" + local found=false + for dir in "${all_dirs[@]}"; do + local dirname + dirname="$(basename "$dir")" + local rel_path="${dir#$SCRIPT_DIR/}" + if [[ "$dirname" == "$svc" || "$rel_path" == "$svc" ]]; then + target_dirs=("$dir") + found=true + break + fi + done + if [[ "$found" = false ]]; then + # Build a concise list: show basename, and rel-path when it differs + local available + available="$(for d in "${all_dirs[@]}"; do + local bn="$(basename "$d")" + local rp="${d#$SCRIPT_DIR/}" + if [[ "$bn" == "$rp" ]]; then + printf '%s ' "$bn" + else + printf '%s(%s) ' "$bn" "$rp" + fi + done)" + err "Service '$svc' not found. Available: $available" + exit 2 + fi + else + target_dirs=("${all_dirs[@]}") + fi + + case "$OPERATION" in + encrypt) + check_command sops + check_command age + _secrets_encrypt "${target_dirs[@]}" + ;; + decrypt) + check_command sops + check_command age + _secrets_decrypt "${target_dirs[@]}" + ;; + deploy) + check_command sops + check_command age + _secrets_deploy "${target_dirs[@]}" + ;; + clean) + _secrets_clean "${all_dirs[@]}" + ;; + esac +} + +_secrets_encrypt() { + local -a dirs=("$@") + local encrypted=0 + local skipped=0 + + for dir in "${dirs[@]}"; do + local env_file="$dir/.env" + local enc_file="$dir/.env.enc" + + if [[ ! -f "$env_file" ]]; then + log "SKIP: $dir — no .env file to encrypt" + skipped=$((skipped+1)) + continue + fi + + log "Encrypting: $env_file → $enc_file" + local tmp_enc + tmp_enc="$(mktemp "${enc_file}.tmp.XXXXXXXXXX")" + if sops --encrypt --input-type dotenv --output-type dotenv "$env_file" > "$tmp_enc"; then + mv "$tmp_enc" "$enc_file" + encrypted=$((encrypted+1)) + else + err "Failed to encrypt $env_file" + rm -f "$tmp_enc" + continue + fi + done + + log "Encrypt complete: $encrypted encrypted, $skipped skipped" +} + +_secrets_decrypt() { + local -a dirs=("$@") + local decrypted=0 + local skipped=0 + + for dir in "${dirs[@]}"; do + local enc_file="$dir/.env.enc" + local env_file="$dir/.env" + + if [[ ! -f "$enc_file" ]]; then + log "SKIP: $dir — no .env.enc file to decrypt" + skipped=$((skipped+1)) + continue + fi + + log "Decrypting: $enc_file → $env_file" + local tmp_env + tmp_env="$(mktemp "${env_file}.tmp.XXXXXXXXXX")" + local decrypt_ok=false + # Write to temp file with umask 077, then atomic move into place + if ( umask 077; sops --decrypt --input-type dotenv --output-type dotenv "$enc_file" > "$tmp_env" ); then + mv "$tmp_env" "$env_file" + decrypt_ok=true + else + err "Failed to decrypt $enc_file" + rm -f "$tmp_env" + fi + if [[ "$decrypt_ok" = true ]]; then + decrypted=$((decrypted+1)) + else + skipped=$((skipped+1)) + fi + done + + log "Decrypt complete: $decrypted decrypted, $skipped skipped" +} + +_secrets_deploy() { + local -a dirs=("$@") + local deployed=0 + local skipped=0 + + # Determine which stacks to deploy based on service dirs + # Map repo-relative path → stack name + local -A dir_to_stack=() + for dir in "${dirs[@]}"; do + local rel_path="${dir#$SCRIPT_DIR/}" + # Match precise env_file reference in stack content: ./REL_PATH/.env + # Avoids false positives from loose basename-only grep matches + local found_stack=false + for stack in "${STACK_FILES[@]}"; do + local stack_file + if stack_file="$(find_stack_file "$stack")"; then + if grep -F "./${rel_path}/.env" "$stack_file" >/dev/null 2>&1; then + dir_to_stack["$rel_path"]="$stack" + found_stack=true + break + fi + fi + done + if [[ "$found_stack" = false ]]; then + log "NOTE: $rel_path not found in any stack file — will decrypt but not deploy" + fi + done + + # Decrypt all target services first (temp file + umask 077 + atomic move) + local -a decrypted_dirs=() + for dir in "${dirs[@]}"; do + local enc_file="$dir/.env.enc" + local env_file="$dir/.env" + + if [[ ! -f "$enc_file" ]]; then + log "SKIP: $dir — no .env.enc file" + skipped=$((skipped+1)) + continue + fi + + log "Decrypting: $enc_file → $env_file" + local tmp_env + tmp_env="$(mktemp "${env_file}.tmp.XXXXXXXXXX")" + local decrypt_ok=false + if ( umask 077; sops --decrypt --input-type dotenv --output-type dotenv "$enc_file" > "$tmp_env" ); then + mv "$tmp_env" "$env_file" + decrypt_ok=true + decrypted_dirs+=("$dir") + else + err "Failed to decrypt $enc_file — skipping deploy for this service" + rm -f "$tmp_env" + skipped=$((skipped+1)) + fi + done + + # Deploy the relevant stacks + local -a stacks_to_deploy=() + + for dir in "${dirs[@]}"; do + local rel_path="${dir#$SCRIPT_DIR/}" + if [[ -n "${dir_to_stack[$rel_path]:-}" ]]; then + local stack="${dir_to_stack[$rel_path]}" + # Deduplicate stacks + local already=false + for s in "${stacks_to_deploy[@]}"; do + [[ "$s" == "$stack" ]] && already=true + done + if [[ "$already" = false ]]; then + stacks_to_deploy+=("$stack") + fi + fi + done + + if [[ ${#stacks_to_deploy[@]} -gt 0 ]]; then + # Regenerate stacks if needed (reuse up logic) + if command -v python3 >/dev/null 2>&1; then + log "Regenerating stacks before deploy..." + python3 "$SCRIPT_DIR/tools/generate_stacks.py" 2>/dev/null || log "Warning: stack generation failed" + fi + + for stack in "${stacks_to_deploy[@]}"; do + local file + if file="$(find_stack_file "$stack")"; then + local render_file + render_file="$(render_stack_file "$file")" + log "Deploying stack: $stack" + docker stack deploy -c "$render_file" "$stack" + deployed=$((deployed+1)) + else + err "Stack file not found for '$stack'" + fi + done + else + log "No stacks to deploy (services not found in any stack file)" + fi + + # Shred only .env files that were successfully decrypted in this run + if [[ ${#decrypted_dirs[@]} -gt 0 ]]; then + log "Cleaning up plaintext .env files..." + for dir in "${decrypted_dirs[@]}"; do + local env_file="$dir/.env" + if [[ -f "$env_file" ]]; then + if command -v shred >/dev/null 2>&1; then + shred -u "$env_file" + else + rm -f "$env_file" + log "Warning: shred not available, used rm -f for $env_file" + fi + fi + done + fi + + log "Deploy complete: $deployed stack(s) deployed, $skipped skipped" +} + +_secrets_clean() { + local -a dirs=("$@") + local cleaned=0 + + for dir in "${dirs[@]}"; do + local env_file="$dir/.env" + local enc_file="$dir/.env.enc" + + if [[ -f "$env_file" && -f "$enc_file" ]]; then + log "Shredding: $env_file" + if command -v shred >/dev/null 2>&1; then + shred -u "$env_file" + else + rm -f "$env_file" + log "Warning: shred not available, used rm -f for $env_file" + fi + cleaned=$((cleaned+1)) + fi + done + + log "Clean complete: $cleaned plaintext .env file(s) removed" +} + # Determine subcommand (default: up) SUBCOMMAND="${1:-}" case "$SUBCOMMAND" in - up|down|status|logs|doctor|env|generate|sync|help|-h|--help) + up|down|status|logs|doctor|env|secrets|generate|sync|help|-h|--help) [[ $# -gt 0 ]] && shift || true ;; *) SUBCOMMAND="up" ;; @@ -899,6 +1208,8 @@ case "$SUBCOMMAND" in cmd_sync "$@" ;; doctor) cmd_doctor "$@" ;; + secrets) + cmd_secrets "$@" ;; help|-h|--help) print_usage ;; *) diff --git a/stacks/README.md b/stacks/README.md index abbf979..3ad0564 100644 --- a/stacks/README.md +++ b/stacks/README.md @@ -46,7 +46,7 @@ docker stack rm infrastructure ### Using stackctl.sh (recommended) -The repo includes a helper script at the root, `./stackctl.sh`, which wraps the common lifecycle with preflight checks and nicer ergonomics. +The repo includes a helper script at the root, `./stackctl.sh`, which wraps the common lifecycle with preflight checks and nicer ergonomics. For encrypted secrets, use `./stackctl.sh secrets deploy` to decrypt, render, deploy, and clean up in one step (see [Managing Secrets](../docs/Managing%20Secrets.md)). Prerequisites: - Docker Engine with Swarm enabled (single-node is fine) @@ -80,7 +80,7 @@ Quick start: Notes: - `stackctl.sh` finds stack files from either `stacks/*.yml` or the repo root (`infrastructure.yml`, etc.). -- The `doctor` command validates Compose syntax for each stack and reminds you to create `.env` files where a `.env.example` exists. +- The `doctor` command validates Compose syntax for each stack and reminds you to create `.env` files where a `.env.example` exists. For encrypted secrets, use `./stackctl.sh secrets deploy` instead. - If you use local HTTPS, make sure `traefik/certs/local-cert.pem` and `traefik/certs/local-key.pem` exist; see below for generation. ### Rendered output naming @@ -95,7 +95,7 @@ These files are ignored by Git and safe to regenerate at any time. ## Notes -- Ensure each service folder has a `.env` copied from its `.env.example` where applicable. +- Ensure each service folder has a `.env` available. For local development, copy from `.env.example`; for production, use `./stackctl.sh secrets deploy` (see [Managing Secrets](../docs/Managing%20Secrets.md)). - APISIX dashboard uses `apisix/api-dashboard/config/conf.yaml` (generated from `conf.example.yml`). - Consider adding healthchecks for critical dependencies to improve startup reliability.