feat(secrets): add SOPS + age secrets management with stackctl.sh integration

.gitignore

@@ -1,5 +1,6 @@
 .env
 .env.bak*
+!*.env.enc
 data
 htpasswd
 *.log

.sops.yaml

@@ -1,7 +1,9 @@
-# SOPS configuration for encrypting secrets committed to the repo
+# SOPS configuration for encrypting service-local .env files
+# Each service directory (e.g., postgres/, traefik/) stores an encrypted .env.enc
+# alongside its .env.example. Decrypt with: stackctl.sh secrets decrypt [service]
 creation_rules:
-  - path_regex: secrets/.*\.(env|yaml|yml)$
-    # Replace with your age public key(s)
-    age:
-      - "age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" # Replace with your real age public key
+  - path_regex: \.env$
+    key_groups:
+      - age:
+          - age12ph7sgtptrrcxzxdue28j3lesnu9gj73ae9ewuvg66awp6jpae8smyvspx
     encrypted_regex: '^(?!#)'

README.md

@@ -234,7 +234,7 @@ cd ../portainer
 docker-compose up -d
 ```
 
-### 5. Access the services
+### 6. Access the services
 
 - Grafana: <https://grafana.your-domain.com>
 - Prometheus: <https://prometheus.your-domain.com>
@@ -261,12 +261,22 @@ local-stack/
 
 ## Configuration
 
-Each component has its own environment file for configuration. Copy the example files and modify as needed:
+Each component has its own environment file for configuration. For local development without secrets management:
 
 ```sh
 find . -name ".env.example" -exec sh -c 'cp "$1" "${1%.example}"' _ {} \;
 ```
 
+For production or shared environments, use the SOPS + age secrets workflow (see [Managing Secrets](docs/Managing%20Secrets.md)):
+
+```sh
+# Decrypt and deploy in one step
+./stackctl.sh secrets deploy
+
+# Or decrypt manually for inspection
+./stackctl.sh secrets decrypt
+```
+
 ## License
 
 ```txt

docs/Managing Secrets.md

@@ -1,68 +1,142 @@
 # Managing Secrets Securely
 
-Never store real secrets in `.env` files in the repo. Keep `.env.example` as documentation with placeholders only.
+Secrets are encrypted at rest in git using SOPS + age and decrypted just-in-time at deploy time. Each service directory stores an encrypted `.env.enc` alongside its `.env.example`.
 
-## Recommended approach (encrypted files committed to git)
+## Workflow Overview
 
-Use `sops` + age (or GPG) to encrypt per-environment secrets that can be safely committed.
+```
+.env.example  →  .env (fill values)  →  .env.enc (encrypt)  →  git commit
+                                                          ↓
+.git (encrypted)  →  .env.enc  →  .env (decrypt)  →  deploy  →  shred .env
+```
+
+## Setup
 
-### 1) Install tools
-- age: https://age-encryption.org
-- sops: https://github.com/getsops/sops
+### 1. Install tools
 
-### 2) Create an age key pair (once)
 ```bash
-# writes to ~/.config/sops/age/keys.txt
+# macOS
+brew install sops age
+
+# Linux
+# sops: https://github.com/getsops/sops/releases
+# age: https://github.com/FiloSottile/age/releases
+```
+
+### 2. Generate an age key pair (once per machine)
+
+```bash
+mkdir -p ~/.config/sops/age
 age-keygen -o ~/.config/sops/age/keys.txt
+# The public key is printed — add it to .sops.yaml key_groups
 ```
-Add the public recipient from that file (starts with `age1...`) to your repository SOPS config.
 
-### 3) Add a SOPS config
-Create `.sops.yaml` at repo root:
+### 3. Add your public key to `.sops.yaml`
+
+Edit `.sops.yaml` and add your age public key to the `key_groups` list. Multiple recipients can be listed for team access:
+
 ```yaml
-# Encrypt files matching these globs with the recipient below
 creation_rules:
-  - path_regex: secrets/.*\.(env|yaml|yml)$
-    age: ["AGE1_PUBLIC_KEY_HERE"]
-    encrypted_regex: '^(?!#)'
+  - path_regex: \.env$
+    key_groups:
+      - age:
+          - age1existingkey...   # existing recipient
+          - age1yournewkey...    # your new key
 ```
-Replace `AGE1_PUBLIC_KEY_HERE` with your public age key.
 
-### 4) Create encrypted secret files
-Place per-environment secrets under `secrets/` and encrypt with sops:
+## Encrypting secrets
+
+For a single service:
+
 ```bash
-mkdir -p secrets
-printf "TRAEFIK_ENABLE=true\nSSO_CREDENTIALS=admin:$apr1$...\n" > secrets/traefik.dev.env
-sops -e -i secrets/traefik.dev.env
+# Create .env from .env.example and fill in real values
+cp postgres/.env.example postgres/.env
+# Edit postgres/.env with real values...
+
+# Encrypt
+./stackctl.sh secrets encrypt postgres
 ```
-The file is now encrypted at rest and safe to commit.
 
-### 5) Decrypt for local use
+For all services at once:
+
 ```bash
-# Produces a plaintext file for docker usage (do not commit this)
-sops -d secrets/traefik.dev.env > traefik/.env
+./stackctl.sh secrets encrypt
 ```
-You can add a simple make/script target to automate decrypt -> deploy -> clean.
 
-### 6) CI/CD or remote deploy
-On a deployment host, provision the age private key (read-only, secured). Decrypt secrets just-in-time before `docker stack deploy`.
+This runs `sops --encrypt --input-type dotenv --output-type dotenv .env > .env.enc` for each service directory that has a `.env` file.
 
-## Using Docker Swarm secrets (optional/advanced)
-Docker Swarm supports native secrets. You can combine sops+age with `docker secret create`:
+## Deploying
+
+Deploy decrypts, renders, deploys, and shreds in one step:
 
-1) Decrypt locally in memory and pipe to secret create:
 ```bash
-sops -d secrets/traefik.dev.env | docker secret create traefik_env -
+# Deploy a specific service's stack
+./stackctl.sh secrets deploy postgres
+
+# Deploy all services
+./stackctl.sh secrets deploy
 ```
-2) Reference the secret in your stack file using `secrets:` and `env_file` alternatives where appropriate.
 
-This is more granular and keeps values out of env vars in the container filesystem, but requires adjusting service configs to read from files or environment sourced from secrets.
+The deploy operation:
+1. Decrypts `.env.enc` → `.env` for each target service
+2. Regenerates rendered stack files (variable substitution)
+3. Deploys the relevant Docker Swarm stacks
+4. Shreds all plaintext `.env` files
+
+## Decrypting (manual)
+
+If you need to inspect or edit secrets without deploying:
+
+```bash
+# Decrypt a single service
+./stackctl.sh secrets decrypt postgres
+
+# Decrypt all services
+./stackctl.sh secrets decrypt
+```
+
+**Remember to clean up plaintext files after editing:**
+
+```bash
+./stackctl.sh secrets clean
+```
+
+## Cleaning up
+
+Remove all plaintext `.env` files that have a corresponding `.env.enc`:
+
+```bash
+./stackctl.sh secrets clean
+```
+
+This uses `shred -u` when available, falling back to `rm -f` on systems without `shred`.
+
+## Key rotation
+
+After adding a new recipient to `.sops.yaml`:
+
+```bash
+sops updatekeys --yes **/.env.enc
+```
+
+## Re-encrypting after editing
+
+```bash
+# Decrypt, edit, re-encrypt
+./stackctl.sh secrets decrypt postgres
+# Edit postgres/.env...
+./stackctl.sh secrets encrypt postgres
+./stackctl.sh secrets clean
+```
 
 ## Git hygiene
-- Commit only `.env.example` files and encrypted files under `secrets/`.
-- Never commit plaintext `.env`.
-- Add a `.gitignore` rule for `**/.env` and `**/*.env.decrypted` as needed.
+
+- **Commit**: `.env.example` (placeholders) and `.env.enc` (encrypted secrets)
+- **Never commit**: `.env` (plaintext, gitignored)
+- The `.gitignore` rule `!*.env.enc` ensures encrypted files are tracked
 
 ## Troubleshooting
-- If `sops` can’t decrypt: ensure the age private key is in `~/.config/sops/age/keys.txt`.
-- For team usage: include multiple recipients in `.sops.yaml` so each developer can decrypt.
+
+- **`sops` can't decrypt**: Ensure the age private key is at `~/.config/sops/age/keys.txt` and the corresponding public key is in `.sops.yaml`.
+- **`shred` not found**: The script falls back to `rm -f`. Install `shred` (part of `coreutils` on Linux) for secure deletion.
+- **`sops` or `age` not found**: Run `./stackctl.sh doctor` to check prerequisites, or install them manually.

docs/proposals/Secrets Management Proposal.md

@@ -25,9 +25,9 @@ Cons:
 
 Concrete steps:
 1) Keep only placeholders in `.env.example`. Do not commit real `.env`.
-2) Store real values in encrypted files under `secrets/` (SOPS + age). See `docs/secrets.md`.
+2) Store real values in encrypted `.env.enc` files per service directory (SOPS + age). See `docs/Managing Secrets.md`.
 3) At deploy time, decrypt just-in-time and create Docker secrets, e.g.:
-   - `sops -d secrets/postgres.dev.env | grep POSTGRES_PASSWORD= | cut -d= -f2 | docker secret create postgres_password -`
+   - `sops -d postgres/.env.enc | grep POSTGRES_PASSWORD= | cut -d= -f2 | docker secret create postgres_password -`
 4) Reference secrets in stacks:
 ```yaml
 services:
@@ -63,7 +63,7 @@ Cons:
 Concrete steps:
 1) Generate Portainer API key (Settings → API Keys).
 2) Write a small script to:
-   - `sops -d secrets/<svc>.env |` extract values → POST to `/api/endpoints/{id}/docker/secrets/create`.
+   - `sops -d <service>/.env.enc |` extract values → POST to `/api/endpoints/{id}/docker/secrets/create`.
    - Trigger stack redeploy via Portainer Stack API (optional) or `docker stack deploy` locally.
 3) Reference secrets in stacks as in Option A.
 
@@ -86,10 +86,10 @@ Cons:
 
 ## Recommended plan (Option A)
 
-Phase 1: Foundation
-- Keep `.env.example` placeholders. Done.
-- Add `docs/secrets.md` and `.sops.yaml`. Done.
-- Create age key(s) and commit encrypted files under `secrets/` (team recipients in `.sops.yaml`).
+Phase 1: Foundation ✅
+- Keep `.env.example` placeholders. ✅ Done.
+- Add `.sops.yaml` and `docs/Managing Secrets.md`. ✅ Done — now uses service-local `.env.enc` pattern with `stackctl.sh secrets`.
+- Create age key(s) and commit encrypted files per service directory (team recipients in `.sops.yaml`). ✅ Done — see `stackctl.sh secrets encrypt`.
 
 Phase 2: Convert priority services
 - Databases: switch to `*_FILE` and define `secrets:` in `stacks/infrastructure.yml`.
@@ -110,13 +110,13 @@ Phase 4: (Optional) Portainer integration
 ## Open questions
 - Which services require secrets and support `*_FILE`? (Postgres yes; Redis no; Mongo users might need env vars or runtime files.)
 - Who holds the age private key for CI/CD? (One or more maintainers; store on deployment hosts only.)
-- Rotation cadence? (Document per-service rotation procedure in `docs/secrets.md`.)
+- Rotation cadence? (Document per-service rotation procedure in `docs/Managing Secrets.md`.)
 
 ## Appendix: Example secret extraction
 Extract a single key from an encrypted env file without writing plaintext to disk:
 ```bash
 # Create postgres password secret from an encrypted env file
-sops -d secrets/postgres.dev.env \
+sops -d postgres/.env.enc \
   | awk -F= '/^POSTGRES_PASSWORD=/{print $2}' \
   | docker secret create postgres_password -
 ```