A comprehensive Python CLI tool that creates isolated development environments using Git worktrees, Docker Compose, and Caddy reverse proxy. Each feature branch gets its own complete environment with isolated databases, Redis, and media storage.
# Install via pip
pip install ilha
# Or install from source
git clone https://github.com/catalpainternational/ilha.git
cd ilha
pip install -e .# Initialize ilha in your project
ilha setup
# Start global Caddy proxy
ilha start-proxy
# Create and start a worktree
ilha create feature-auth
ilha feature-auth up -d
# Access your isolated environment (note: domain includes project name)
# If your project is named "myapp", the URL will be:
open http://myapp-feature-auth.localhost
# Export environment as shareable package (includes code by default)
ilha packages export feature-auth
# Import environment from package
ilha packages import myapp-feature-auth-2024-01-15.ilha-package.tar.gz
# Stop and clean up
ilha feature-auth down
ilha remove feature-authilha CLI provides complete environment isolation through:
- Git Worktrees: Isolated working directories for each branch
- Docker Compose: Isolated containers with branch-specific volumes
- Caddy Proxy: Dynamic routing based on branch names
- Volume Isolation: Branch-specific databases, Redis, and media storage
- Package Management: Export/import complete environments as shareable packages
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Global Caddy │ │ Worktree A │ │ Worktree B │
│ (Port 80) │ │ feature-auth │ │ feature-pay │
│ │ │ │ │ │
│ Routes to: │───▶│ • PostgreSQL │ │ • PostgreSQL │
│ • *.localhost │ │ • Redis │ │ • Redis │
│ │ │ • Web App │ │ • Web App │
└─────────────────┘ └──────────────────┘ └─────────────────┘
Important: ilha uses a worktree-name-first pattern. For up and down commands, you can use:
ilha <worktree_name> upandilha <worktree_name> down(explicit worktree name)ilha upandilha down(auto-detected from current directory or current git branch)
The shorthand ilha up and ilha down automatically resolve the worktree from your current directory (if you're in a worktree) or from your current git branch (if you're at the project root).
| Command | Description | Example |
|---|---|---|
setup |
Initialize ilha for this project | ilha setup [--monkey-patch] |
| Command | Description | Example |
|---|---|---|
start-proxy |
Start global Caddy proxy container | ilha start-proxy |
stop-proxy |
Stop global Caddy proxy container | ilha stop-proxy |
start |
Alias for start-proxy | ilha start |
stop |
Alias for stop-proxy | ilha stop |
| Alias | Full Command | Description |
|---|---|---|
-D |
delete |
Delete worktree and branch completely |
-r |
remove |
Remove worktree but keep git branch |
| Command | Description | Example |
|---|---|---|
create <branch> |
Create worktree | ilha create feature-auth |
<branch> up -d |
Start worktree environment | ilha feature-auth up -d |
up |
Start worktree environment (auto-detected) | ilha up (from worktree directory or project root) |
<branch> down |
Stop worktree environment | ilha feature-auth down |
down |
Stop worktree environment (auto-detected) | ilha down (from worktree directory or project root) |
remove <branch> |
Remove worktree (keep branch) | ilha remove feature-auth |
delete <branch> |
Delete worktree and branch | ilha delete feature-auth |
Note: The ilha up and ilha down commands automatically detect the worktree from:
- Your current directory (if you're inside a worktree directory)
- Your current git branch (if you're at the project root and a worktree exists for that branch)
Important: Branch names must match exactly. The delete and remove commands check worktrees, branches, and docker volumes for exact matches only. If no exact match is found, an error message will show what was checked.
ilha supports direct passthrough to Docker Compose commands using the pattern: ilha <worktree_name> <compose-command>
| Command Pattern | Description | Example |
|---|---|---|
<branch> exec <service> <cmd> |
Execute command in container | ilha feature-auth exec web python manage.py migrate |
<branch> logs <service> |
View container logs | ilha feature-auth logs web |
<branch> ps |
List containers | ilha feature-auth ps |
<branch> run <service> <cmd> |
Run one-off command | ilha feature-auth run --rm web python manage.py test |
<branch> build |
Build services | ilha feature-auth build |
<branch> restart <service> |
Restart service | ilha feature-auth restart web |
<branch> <compose-cmd> |
Any docker compose command | ilha feature-auth pull, ilha feature-auth config |
Supported Docker Compose Commands:
exec,logs,ps,run,build,pull,push,restartstart,stop,up,down,config,images,porttop,events,kill,pause,unpause,scale
| Command | Description | Example |
|---|---|---|
remove <pattern> |
Remove worktrees matching pattern | ilha remove test-* |
delete <pattern> |
Delete worktrees and branches matching pattern | ilha delete feature-* |
Wildcard Patterns:
*- Matches any characters (e.g.,test-*matchestest-feature,test-bugfix)?- Matches single character (e.g.,test-?matchestest-1,test-a)[abc]- Matches any character in brackets (e.g.,test-[abc]matchestest-a,test-b,test-c)- Case-insensitive matching (e.g.,
test-*matchesTest-Feature,TEST-FEATURE)
| Command | Description | Example |
|---|---|---|
remove-all |
Remove all worktrees (keep branches) | ilha remove-all |
delete-all |
Delete all worktrees and branches | ilha delete-all --force |
| Command | Description | Example |
|---|---|---|
list |
List active worktrees | ilha list |
prune |
Remove prunable worktrees | ilha prune |
help |
Show help information | ilha help |
clean-legacy |
Clean legacy ilha elements | ilha clean-legacy |
| Command | Description | Example |
|---|---|---|
volumes list |
List all worktree volumes | ilha volumes list |
volumes size |
Show volume sizes | ilha volumes size |
volumes backup <branch> |
Backup worktree volumes | ilha volumes backup feature-auth |
volumes restore <branch> <file> |
Restore from backup | ilha volumes restore feature-auth backup.tar |
volumes clean <branch> |
Clean up volumes | ilha volumes clean feature-auth |
| Command | Description | Example |
|---|---|---|
image build [branch] |
Build and tag local Docker images for buildable compose services | ilha image build staging --registry catalpa --save |
image push [branch] |
Push already-built image tags to the configured registry | ilha image push staging |
image publish [branch] |
Build, push, and write .ilha/images/<branch>.json with exact image refs/digests |
ilha image publish staging |
image list [branch] |
List published image manifests; add --remote to include DOCR tags |
ilha image list staging --remote |
image destroy <branch> |
Delete published registry tags and local manifest (comma-separated branches supported) | ilha image destroy staging --force |
Image List Options:
branch_name(optional argument) - Filter to one branch manifest--remote- Query DOCR via DigitalOcean API (requiresDIGITALOCEAN_API_TOKEN)--registry <name>- DigitalOcean Container Registry name (for--remote)--repository <path>- Repository path prefix (e.g.bower/web); defaults from manifest when listing one branch--json- Output as JSON
Image Destroy Options:
<branch>(required argument) - One or more comma-separated branch names--service <name>- Apply registry deletion to matching compose service(s) only (repeatable)--manifest-only- Remove.ilha/images/<branch>.jsononly; skip DOCR tag deletion--registry-only- Delete DOCR tags only; keep local manifest--force- Skip confirmation prompts--registry <name>- Override DigitalOcean registry name for tag deletion
Image preferences are project-local and live in .ilha/config.yml:
image:
provider: digitalocean
registry: catalpa
repository: bower
platform: linux/amd64
tag_template: "{branch}-{sha7}"CLI flags override config for one run; add --save to persist reusable registry/repository/platform settings. DigitalOcean Container Registry images use registry.digitalocean.com/<registry>/<repository>/<service>:<tag> and authenticate via the DigitalOcean API (docker login credentials). Published manifests are written to .ilha/images/<branch>.json; use ilha image list --remote to query DOCR tags via the DO API. Worker deploy reads digest refs from those manifests via ImageManager.image_ref_for_service. Future deploy orchestration should consume the manifest digest when available rather than relying on mutable tags.
Do not bake secrets into images. ilha image build rejects sensitive-looking Docker build.args (for example DJANGO_SECRET_KEY, *_TOKEN, *_PASSWORD). It also refuses to build when the context contains .ilha/, secrets/, or backups/ unless those paths are listed in .dockerignore.
Store production secrets in encrypted SOPS files committed to git. Decrypt only at command boundaries (local compose, deploy). Worker env is copied over SSH after the droplet boots — never via cloud-init user_data.
Configure profiles in .ilha/config.yml:
secrets:
provider: sops
default_profile: staging
profiles:
staging:
file: .ilha/env.staging.sops.env
production:
file: .ilha/env.production.sops.env| Command | Description | Example |
|---|---|---|
secrets list |
List configured profiles | ilha secrets list |
secrets edit <profile> |
Edit encrypted file with sops | ilha secrets edit staging |
secrets print <profile> |
Show decrypted env (--redact to hide values) |
ilha secrets print staging --redact |
secrets exec <profile> -- <cmd> |
Run a command with profile env | ilha secrets exec staging -- ilha worker deploy staging |
secrets materialize <profile> --output <path> |
Write plaintext env (requires --force to overwrite) |
ilha secrets materialize staging --output .ilha/env.ilha --force |
Use --secrets <profile> on deploy/runtime commands:
ilha staging up -d --secrets staging
ilha droplet deploy staging --domain app.example.com --secrets production
ilha worker deploy staging --secrets staging --count 2 --central-droplet-name centralRequires sops and age keys on the machine running ilha. SOPS protects secrets at rest in git; runtime droplets still need plaintext env for Docker — restrict worker exposure with worker_pool.env_allowlist.
Spin up short-lived RQ worker droplets that pull jobs from central Redis and report to central DB. Cloud-init only installs Docker and writes compose YAML (image refs, not secrets). App env and DOCR_READ_TOKEN are copied over SSH after the droplet is up — DigitalOcean stores user_data in plaintext metadata.
| Command | Description | Example |
|---|---|---|
worker bundle [branch] |
Generate minimal rq-worker compose + env for a droplet | ilha worker bundle staging --central-droplet-name central |
worker deploy [branch] |
Create worker droplet(s), then SSH-copy env and start the worker | ilha worker deploy staging --count 2 --central-droplet-name central |
worker list |
List worker-tagged droplets | ilha worker list --branch staging |
worker reap |
Destroy workers older than worker_pool.idle_destroy_after_seconds |
ilha worker reap --branch staging --dry-run |
worker destroy |
Destroy worker droplets | ilha worker destroy --all --branch staging --force |
Recommended flow:
ilha image publish staging— build, push, and write the image manifest- Deploy central via
ilha droplet deploy(db, redis, web on one droplet in a VPC) ilha worker deploy staging --count N --central-droplet-name <central>when jobs need capacity- Autoscale via Django
autoscale_workers+ cron (seescripts/worker_autoscale.cronin consuming projects) ilha worker reap --branch stagingorilha worker destroy --all --branch stagingwhen the queue is empty
Configure defaults in .ilha/config.yml under worker_pool (central droplet name, size, region, max_workers, ssh_keys). Requires DIGITALOCEAN_API_TOKEN on the operator machine only (never allowlisted onto workers), DOCR_READ_TOKEN for SSH docker login, SSH keys, and a published image manifest for the branch. Deploy fails before creating a droplet if the manifest, registry token, SSH keys, or max_workers cap is missing.
| Command | Description | Example |
|---|---|---|
droplet deploy [branch] |
Create a droplet and deploy via rsync + remote docker compose (slim mode). Default restores local volumes onto a new droplet; pass --no-restore-volumes for empty volumes. Prefer ilha image publish so the server pulls digest refs; --build compiles on the droplet. |
ilha droplet deploy staging --domain app.example.com --build |
droplet deploy --no-create |
Deploy to an existing server (skip droplet creation) | ilha droplet deploy --no-create --scp-target root@1.2.3.4:/root |
droplet put [branch] |
Update an existing deployment. Default is code + config only (does not copy the laptop DB). --restore-volumes replaces remote data. |
ilha droplet put staging --domain app.example.com |
droplet dump-db [branch] |
Custom-format pg_dump from the remote db container into backups/ |
ilha droplet dump-db staging --domain app.example.com |
droplet status [branch] |
Compare remote /app/.ilha/deploy.json to local HEAD |
ilha droplet status staging |
droplet ssh [branch] |
Interactive SSH (or run a remote command) | ilha droplet ssh staging |
droplet exec [branch] SERVICE CMD… |
docker compose exec on the droplet |
ilha droplet exec staging web python manage.py migrate |
droplet rename OLD NEW |
Rename a DigitalOcean droplet; --set-hostname updates the OS hostname |
ilha droplet rename staging-new staging --set-hostname |
droplet logs [branch] |
Tail Docker Compose logs from a remote droplet | ilha droplet logs staging or ilha droplet logs -s web -f |
ilha doctor [branch] |
Diagnose a local worktree (not a remote droplet) | ilha doctor staging |
droplet list |
List all droplets (formatted table) | ilha droplet list |
droplet list --as-json |
List droplets as JSON | ilha droplet list --as-json |
droplet list --as-csv |
List droplets as CSV | ilha droplet list --as-csv |
droplet regions |
List available Digital Ocean regions (table) | ilha droplet regions |
droplet regions --show-all |
Include unavailable regions in region list | ilha droplet regions --show-all |
droplet regions --as-json |
List regions as JSON | ilha droplet regions --as-json |
droplet regions --as-csv |
List regions as CSV | ilha droplet regions --as-csv |
droplet info <id> |
Get droplet information | ilha droplet info 12345678 |
droplet destroy <id> |
Destroy a droplet (supports ID or name, comma-separated) | ilha droplet destroy 12345678 or ilha droplet destroy my-droplet or ilha droplet destroy 123,456,789 |
Droplet Deploy Options:
branch_name(optional argument) - Branch/worktree name (auto-detected from current directory if not provided)--region <region>- Droplet region (e.g. nyc1, sfo3). Useilha droplet regionsto list available regions (defaults from env or nyc1)--size <size>- Droplet size (defaults from env or s-1vcpu-1gb)--image <image>- Droplet image (defaults from env or ubuntu-22-04-x64)--ssh-keys <key>- SSH key names (comma-separated)--tags <tag>- Tags for the droplet (can be specified multiple times)--api-token <token>- Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)--create-only- Only create droplet, skip deploy--no-create- Skip droplet creation, deploy to existing server (requires --scp-target or --domain/--ip)--scp-target <target>- SCP target (optional, defaults to root@:/root)--vpc-uuid <uuid>- VPC UUID for the droplet (if not provided, uses default VPC for the region)--central-droplet-name <name>- Name of central droplet to reuse VPC UUID from (for worker deployments)--domain- Domain for HTTPS deployment--ip- IP for HTTP-only deployment--dns-token- DNS API token for domain management--build- Rebuild Docker images on server (docker compose buildbefore app start)--no-build- Never run a remote compose build; fail if the web image is missing (mutually exclusive with--build)--restore-volumes- Backup local worktree volumes and restore them on the server (opt-in; avoids accidental DB password mismatches on fresh droplets)--force-password-reset- After--restore-volumes, skip the Postgres password vs.envvalidation (dangerous)--debug- Enable DEBUG mode on server--use-staging-certificates- Use Let's Encrypt staging certificates
Droplet Logs Options:
--service, -s <name>- Service name(s) to show logs for (can be repeated, default: all)--tail, -n <lines>- Number of lines to show (default: 100)--follow, -f- Follow log output (stream, Ctrl+C to stop)--scp-target/--domain/--ip- Target override (falls back to stored config)
Droplet Name Auto-Detection:
- If
--domainis provided: uses subdomain as droplet name (e.g.,app.example.com→app) - Otherwise: uses branch/worktree name as droplet name (auto-detected if not provided)
Droplet List Options:
--as-jsonor--json- Output as JSON format--as-csv- Output as CSV format--api-token <token>- Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)
Droplet Regions Options:
--show-all- Include unavailable regions--as-jsonor--json- Output as JSON format--as-csv- Output as CSV format--api-token <token>- Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)
Droplet Destroy Options:
<id>- Single droplet ID or name, or comma-separated list of IDs/names (e.g.,123,456,789ormy-droplet,another-droplet,123)--force- Skip confirmation (destroys without typing droplet name)--only-droplet- Only destroy droplet, skip DNS deletion--only-domain- Only destroy DNS records, skip droplet deletion--domain <domain>- Domain name for DNS deletion (optional, auto-detects if not provided)--api-token <token>- Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)--dns-token <token>- DNS API token (if different from droplet token)--json- Output as JSON format
Droplet Destroy Behavior:
- Default: Destroys droplet only (backward compatible)
- Multiple Droplets: Accepts comma-separated IDs or names (e.g.,
123,456,789ormy-droplet,another-droplet,123) and processes all droplets sequentially - Name Resolution: Automatically resolves droplet names to IDs. If multiple droplets share the same name, use the droplet ID instead.
- Error Handling: Continues destroying remaining droplets even if one fails
- Summary Output: Shows summary of destroyed/failed droplets when processing multiple IDs
- Confirmation: Requires typing the exact droplet name to confirm (unless
--force) - each droplet requires confirmation separately - DNS Auto-detection: When destroying droplet, automatically finds and deletes DNS records pointing to droplet IP
- DNS-only mode: Use
--only-domainto delete DNS records without destroying the droplet - Domain confirmation: When deleting DNS records, requires typing the full domain name (e.g., "app.example.com") unless
--force - Domain override: Use
--domain <domain>to limit DNS search to specific domain
| Command | Description | Example |
|---|---|---|
packages export <branch> |
Export worktree as shareable package (includes code by default) | ilha packages export feature-auth |
packages import <file> |
Import environment from package (auto-detects standalone mode) | ilha packages import my-package.tar.gz |
packages import <file> --domain <sub.domain.tld> |
Import with domain override (HTTPS via Caddy) | ilha packages import pkg.tar.gz --domain myapp.example.com |
packages import <file> --ip <x.x.x.x> |
Import with IP override (HTTP-only) | ilha packages import pkg.tar.gz --ip 203.0.113.10 |
packages import <file> --standalone |
Force standalone import (create new project) | ilha packages import my-package.tar.gz --standalone --target-dir ./myproject |
packages list |
List available packages | ilha packages list |
packages validate <file> |
Validate package integrity | ilha packages validate my-package.tar.gz |
ilha intelligently detects whether you're in an existing project and automatically switches between normal and standalone import modes.
Automatic Detection (Default):
# In an empty directory - automatically creates new project
cd /path/to/new-location
ilha packages import my-package.tar.gz
# Auto-detects: No git repo → standalone mode
# Creates: ./myproject-standalone/
# In existing ilha project - automatically imports as worktree
cd /path/to/existing-project
ilha packages import my-package.tar.gz
# Auto-detects: Existing .ilha → normal mode
# Creates: new worktree in projectExplicit Standalone Mode:
# Force standalone import with custom directory
ilha packages import my-package.tar.gz --standalone --target-dir ./my-new-project
# Force standalone even if in existing project
ilha packages import my-package.tar.gz --standaloneRequirements:
- Standalone imports require packages exported with code (
--include-code, which is the default) - Normal imports work with or without code
What Gets Created in Standalone Mode:
- Complete project structure extracted from package
- Project root
.ilha/configuration - Worktree with its own
.ilha/(fractal structure) - All code files
- Docker volumes (if
--restore-data) - Ready-to-use isolated environment
Note: Standalone import is simple - it extracts the project tar and applies domain/IP overrides. No git initialization or additional setup needed since the package contains the complete fractal structure.
| Command | Description | Example |
|---|---|---|
completion install [shell] |
Install shell completion | ilha completion install |
completion uninstall |
Remove shell completion | ilha completion uninstall |
completion status |
Show completion status | ilha completion status |
During ilha setup, if a Django project is detected (manage.py present and a settings.py found), ilha validates your settings are environment-driven for reverse proxy use:
- ALLOWED_HOSTS (comma-separated)
- CSRF_TRUSTED_ORIGINS (space-separated)
- USE_X_FORWARDED_HOST (boolean)
- SECURE_PROXY_SSL_HEADER (tuple as
HTTP_X_FORWARDED_PROTO,https)
If any are missing, ilha prints a nicely formatted guidance block and, when --monkey-patch is supplied, appends a safe snippet to settings.py to read these values from environment variables.
Suggested snippet (auto-added with --monkey-patch):
import os
ALLOWED_HOSTS = os.getenv("ALLOWED_HOSTS", "localhost,127.0.0.1").split(",")
CSRF_TRUSTED_ORIGINS = os.getenv("CSRF_TRUSTED_ORIGINS", "").split()
USE_X_FORWARDED_HOST = os.getenv("USE_X_FORWARDED_HOST", "False") == "True"
_hdr = os.getenv("SECURE_PROXY_SSL_HEADER")
if _hdr:
SECURE_PROXY_SSL_HEADER = tuple(_hdr.split(",", 1))ilha automatically configures SQLite persistence when you run ilha setup:
Automatic Configuration:
- ✅ Creates
sqlite_dataDocker volume - ✅ Mounts volume at
/datain web containers - ✅ Copies SQLite databases between worktrees
- ✅ Includes SQLite in export/import packages
Django Configuration: Set your database path to use the volume:
# settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': os.getenv('SQLITE_PATH', '/data/db.sqlite3'), # Use volume path
}
}Migration:
If your database is currently in the source tree (e.g., db/db.sqlite3), migrate it:
# Copy to Docker volume
docker compose cp db/db.sqlite3 web:/data/db.sqlite3
# Or use ilha setup --monkey-patch to auto-configure
ilha setup --monkey-patchWhy This Matters:
- 📁 Each worktree gets its own isolated database
- 📦 Databases are included in ilha packages
- 🔄 Database state persists across container rebuilds
- ❌ Without volume: all worktrees share the same database file
When you run ilha setup, it creates a .ilha/ directory with:
config.yml- Project configurationdocker-compose.worktree.yml- Transformed compose file (generated from existing docker-compose.yml)Caddyfile.ilha- Caddy configurationREADME.md- User guide optimized for coding agentsworktrees/- Worktree directories
Important: ilha always uses worktrees/{branch}/.ilha/docker-compose.worktree.yml for all runtime operations. ilha never looks for or uses other compose files (such as docker-compose.yml in the project root or worktree directory) during runtime operations.
- Setup Phase: During
ilha setup, the basedocker-compose.ymlin the project root is read and transformed into.ilha/docker-compose.worktree.yml - Runtime Phase: All commands (
up,down,exec,logs, etc.) exclusively useworktrees/{branch}/.ilha/docker-compose.worktree.yml - No Fallback: There is no backward compatibility or fallback logic - ilha will only use the worktree compose file
Droplet defaults can be configured in .env or .ilha/env.ilha:
# .env or .ilha/env.ilha
DROPLET_DEFAULT_REGION=nyc1
DROPLET_DEFAULT_SIZE=s-1vcpu-1gb
DROPLET_DEFAULT_IMAGE=ubuntu-22-04-x64
# SSH key names (comma-separated, e.g., anders,peter)
# Only key names are supported, not numeric IDs or fingerprints
DROPLET_DEFAULT_SSH_KEYS=anders,peter
DIGITALOCEAN_API_TOKEN=your_token_hereThe .ilha/config.yml file contains:
project_name: myproject
caddy_network: ilha_caddy_proxy
worktree_dir: worktrees
services:
web:
container_name_template: ${COMPOSE_PROJECT_NAME}-web
db:
container_name_template: ${COMPOSE_PROJECT_NAME}-db
redis:
container_name_template: ${COMPOSE_PROJECT_NAME}-redis
volumes:
- postgres_data
- redis_data
- media_files
environment:
DEBUG: "True"
ALLOWED_HOSTS: "localhost,127.0.0.1,*.localhost,web"Worktree-specific volumes follow the pattern: {branch_name}_{volume_type}
feature-auth_postgres_data- Database datafeature-auth_redis_data- Cache datafeature-auth_media_files- User uploads
- Global Network:
ilha_caddy_proxy(external) - Worktree Networks:
{branch_name}_internal,{branch_name}_web
# 1. Initialize ilha in your project
ilha setup
# 2. Start global infrastructure
ilha start-proxy
# 3. Create feature branch environment
ilha create feature-new-auth
ilha feature-new-auth up -d
# 4. Develop and test
open http://feature-new-auth.localhost
# Make changes, test database migrations, etc.
# 5. Clean up when done
ilha feature-new-auth down
ilha remove feature-new-auth# Work on multiple features simultaneously
ilha create feature-auth
ilha create feature-payments
ilha create feature-notifications
# Start all environments
ilha feature-auth up -d
ilha feature-payments up -d
ilha feature-notifications up -d
# Access each independently
open http://feature-auth.localhost
open http://feature-payments.localhost
open http://feature-notifications.localhost# Test database migrations in isolation
ilha create test-migration
ilha test-migration up -d
# Run migrations, test data changes
# Each worktree has its own database
# Clean up test environment
ilha delete test-migration# Execute Django management commands
ilha feature-auth exec web python manage.py migrate
ilha feature-auth exec web python manage.py collectstatic
# View application logs
ilha feature-auth logs web
ilha feature-auth logs -f web # Follow logs
# Run one-off commands
ilha feature-auth run --rm web python manage.py test
ilha feature-auth run --rm web bash
# Check container status
ilha feature-auth ps
# Restart services
ilha feature-auth restart webDocker not running
Error: Docker is not running. Please start Docker and try again.Solution: Start Docker Desktop or Docker daemon
PostgreSQL Database Corruption
Error: invalid primary checkpoint record
Error: could not locate a valid checkpoint recordSolution: This has been fixed! ilha now stops the original database container before copying volumes, then restarts it after copying. This ensures data consistency and prevents corruption.
Worktree already exists
Error: Worktree for branch 'feature-auth' already existsSolution: Use ilha list to see existing worktrees, or remove the existing one
Port conflicts
Error: Port 80 is already in useSolution: Stop other services using port 80, or check if global Caddy is already running
Volume creation fails
Error: Failed to create volumeSolution: Check Docker disk space, ensure Docker has sufficient permissions
Branch not found during deletion
Error: No exact match found for 'feature-auth'. Checked: worktrees, branches, and docker volumes.Solution: Use ilha list to see exact branch names. Branch names are case-sensitive and must match exactly.
HTTPS hangs or certificate acquisition fails (Let's Encrypt rate limit)
Error: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of identifiers in the last 168h0m0sSymptoms: HTTPS connections hang or timeout, HTTP works fine. Browser shows connection timeout when accessing https://your-domain.com.
Root Cause: Let's Encrypt limits certificate issuance to 5 certificates per domain per 168 hours (7 days). If you've deployed the same domain multiple times, you may hit this limit.
Solution: Use Let's Encrypt staging certificates temporarily until the rate limit expires:
-
Manual Fix (Immediate): SSH to your server and update Caddy configuration:
# Get current config curl -s http://localhost:2019/config/ > /tmp/caddy_config.json # Edit config to use staging endpoint (add "ca" field to issuer) # Update the issuer in apps.tls.automation.policies[0].issuers[0]: # Add: "ca": "https://acme-staging-v02.api.letsencrypt.org/directory" # Reload Caddy curl -X POST http://localhost:2019/load -H "Content-Type: application/json" -d @/tmp/caddy_config.json
-
Automatic Fix (Long-term): ilha's dynamic configuration script (
caddy-dynamic-config.py) now automatically detects rate limit errors and falls back to staging certificates. The script checks Caddy logs for rate limit patterns and automatically switches to staging when detected. -
Force Staging Mode for Testing: You can force staging certificates for testing without hitting rate limits by setting the
USE_STAGING_CERTIFICATESenvironment variable:# Option 1: Set in your shell before running ilha commands export USE_STAGING_CERTIFICATES=1 # Option 2: Add to your .ilha/env.ilha file (recommended for droplet deploy/put) echo "USE_STAGING_CERTIFICATES=1" >> .ilha/env.ilha
For
droplet deployanddroplet put: AddUSE_STAGING_CERTIFICATES=1to your.ilha/env.ilhafile before deploying. This ensures the setting is included and used on the remote server. Perfect for testing environments where you don't want to consume your production certificate quota. Staging certificates don't count against Let's Encrypt's rate limits. -
Staging Certificates:
- Staging certificates work for HTTPS but show browser security warnings
- Users can proceed after accepting the warning
- Connection is still encrypted, just not trusted by default
- Switch back to production certificates after rate limit expires (check Caddy logs for "retry after" timestamp)
- Staging certificates don't count against Let's Encrypt rate limits - perfect for testing!
-
Monitor Certificate Health: The
caddy-docker-monitor.pyscript monitors certificate health and logs warnings when rate limits are detected.
Note: Rate limits expire after 168 hours (7 days) from the first certificate issuance. Check Caddy logs for the exact expiration time.
# Check Docker status
docker ps
# Check worktree status
ilha list
# Check volume status
ilha volumes list
# Check network status
docker network ls | grep ilha_caddy_proxy# View container logs
docker logs ilha_caddy_proxy
docker logs {branch-name}-web
# Check worktree directory
ls -la worktrees/{branch-name}/
# Verify environment files
cat worktrees/{branch-name}/.ilha/env.ilhailha CLI provides intelligent tab completion for Bash and Zsh shells, making it faster and easier to use.
- Command Completion: Tab complete all ilha commands and subcommands
- Worktree Names: Auto-complete worktree branch names for commands that need them
- Volume Operations: Complete worktree names for volume backup/restore/clean operations
- Flag Completion: Auto-complete command flags like
--force,-d,--detach
Shell completion is automatically offered during project setup:
ilha setup
# After successful setup, you'll be prompted:
# "Would you like to install bash completion? [Y/n]"# Install for current shell (auto-detected)
ilha completion install
# Install for specific shell
ilha completion install bash
ilha completion install zsh
# Check installation status
ilha completion status
# Uninstall completion
ilha completion uninstall# Tab complete commands
ilha <TAB>
# Shows: create delete down help list prune remove remove-all setup start-proxy stop-proxy start stop up volumes
# Tab complete worktree names
ilha feature-auth <TAB>
# Shows: up down exec logs ps run build restart
# Tab complete volume operations
ilha volumes backup <TAB>
# Shows: feature-auth feature-payments feature-notifications
# Tab complete flags
ilha delete <TAB>
# Shows: feature-auth --forceCompletion not working after installation:
# Restart your shell or source the configuration
source ~/.bashrc # For Bash
source ~/.zshrc # For ZshCheck completion status:
ilha completion statusReinstall completion:
ilha completion uninstall
ilha completion installCreate custom environment files for specific worktrees:
# Create worktree with custom settings
ilha create feature-auth
# Edit environment file
vim worktrees/feature-auth/.ilha/env.ilha
# Start with custom configuration
ilha feature-auth up -d# Backup before major changes
ilha volumes backup feature-auth
# Check volume sizes
ilha volumes size
# Clean up old volumes
ilha volumes clean feature-authAuto-Detection (recommended):
# Automatically chooses correct mode based on current directory
ilha packages import myapp-feature-auth.tar.gzStandalone Import (create new project):
# Create new project from package
mkdir ~/new-project && cd ~/new-project
ilha packages import myapp-feature-auth.tar.gz --standalone
# Or specify target directory
ilha packages import myapp-feature-auth.tar.gz --standalone --target-dir ./myprojectNormal Import (add to existing project):
# Import to existing project as new worktree
cd /path/to/existing-project
ilha packages import myapp-feature-auth.tar.gzImport Options:
# Import without data (configuration and code only)
ilha packages import myapp-feature-auth.tar.gz --no-data
# Import to specific branch name
ilha packages import myapp-feature-auth.tar.gz --target-branch new-branch-name
# Domain/IP overrides for deployment
ilha packages import myapp-feature-auth.tar.gz --standalone --domain myapp.example.com # HTTPS via Caddy
ilha packages import myapp-feature-auth.tar.gz --standalone --ip 203.0.113.10 # HTTP-only (no TLS)Export Options:
# Export complete environment as shareable package (includes code by default)
ilha packages export feature-auth
# Export environment without code (smaller package)
ilha packages export feature-auth --no-code
# Export to specific directory
ilha packages export feature-auth --output-dir ./exportsPackage Management:
# List available packages
ilha packages list
# Validate package integrity
ilha packages validate myapp-feature-auth.tar.gz# Remove all worktrees (keeps git branches)
ilha remove-all
# Delete all worktrees and branches (destructive)
ilha delete-all --force
# Clean up prunable worktrees
ilha prune# Remove all test branches (keeps git branches)
ilha remove test-*
# Delete all feature branches and their worktrees
ilha delete feature-*
# Remove branches matching pattern with confirmation
ilha remove bugfix-*
# Output: Found 3 matching branch(es): bugfix-auth, bugfix-payment, bugfix-ui
# Remove 3 worktree(s) (keep branches)? [Y/n]:
# Delete with force flag (skips confirmation)
ilha delete temp-* --forceAgent skills: This repo includes an Agent Skill at ilha/skills/ilha/ (inside the ilha package). Symlinks from .agents/skills/ilha and .claude/skills/ilha point to it so Cursor, Claude Code, and Codex discover the skill when the repo is open; no installation needed.
- 📖 Full Documentation Site - Complete documentation with search and navigation
- Project Setup Guide - Best practices for configuring your project for ilha
- Architecture Guide - Detailed system architecture and component relationships
- Troubleshooting Guide - Common issues and solutions
- Design Documents - Original design and implementation plans
- Test Documentation - Comprehensive test suite and coverage
- Configuration Reference - All configuration files and options
The ilha CLI follows a modular architecture designed for easy extension:
- Core Infrastructure:
core/- Docker, Git, and environment management - Command Layer:
commands/- Business logic for each command - Utilities:
utils/- Shared functionality and validation - Configuration:
config/- Settings and Docker Compose files
- Create command class in
commands/ - Add CLI interface in
cli.py - Add tests in
tests/unit/ - Update documentation
- New Volume Types: Modify
EnvironmentManager.get_volume_names() - Custom Networks: Update
DockerManager.create_network() - Additional Validation: Extend
utils/validation.py
- Worktree Creation: ~30 seconds
- Volume Copying: ~60 seconds (depending on data size)
- Container Startup: ~45 seconds
- Memory Usage: ~1GB per worktree
- Disk Usage: ~5GB per worktree (varies with data)
Production Ready: All tests passing, 100% functional compatibility with original bash script.
Key Features:
- ✅ Complete environment isolation
- ✅ Dynamic routing with Caddy
- ✅ Volume management and backup
- ✅ Git worktree integration
- ✅ Docker Compose commands
- ✅ Comprehensive error handling
- ✅ Rich console output
- ✅ Type safety throughout
- ✅ Project-agnostic setup
- ✅ User guide in each .ilha directory
- ✅ JSON output mode for programmatic access
- ✅ Slim deploy via rsync + remote docker compose (no Python/git/ilha on server)
# Create a DigitalOcean droplet and deploy via rsync + remote docker compose
# Branch name auto-detected from current directory
ilha droplet deploy --domain app.example.com --ssh-keys anders
# With explicit branch and custom sizing
ilha droplet deploy staging \
--domain staging.example.com \
--ssh-keys anders \
--region syd1 --size s-1vcpu-2gb
# Create droplet only, deploy later
ilha droplet deploy staging --create-only --ssh-keys anders# Skip droplet creation, deploy to a server you already have
ilha droplet deploy staging --no-create --domain app.example.com
ilha droplet deploy staging --no-create --scp-target root@1.2.3.4:/rootRecommended flows
- First deploy to a new droplet (domain + TLS + published images):
ilha image publish <branch>then
ilha droplet deploy <branch> --domain <fqdn> --ssh-keys <key> --region syd1 --size s-1vcpu-2gb
Pass--buildonly when there is no registry manifest.--restore-volumesis the deploy default so a fresh droplet gets local DB/media; use--no-restore-volumesfor empty volumes. - Update an existing droplet (never copies the laptop DB unless asked):
ilha droplet dump-db <branch> --domain <fqdn>
ilha image publish <branch>
ilha droplet put <branch> --domain <fqdn>
ilha droplet status <branch>
SOPSsecrets.default_profileis applied automatically;--no-secretsskips it.--restore-volumesreplaces remote volumes (requires--i-mean-itor a prompt when a--domainis set). - Fast redeploy after server prep and images exist:
ilha droplet deploy <branch> --domain <fqdn> --no-create --no-prepare --skip-volume-backup --no-restore-volumes - Ship local Postgres/Redis volumes to the server (explicit):
ilha droplet put <branch> --domain <fqdn> --restore-volumes --i-mean-it
After restore, ilha checks that Postgres acceptsPOSTGRES_PASSWORDfrom the merged.envbefore starting the app. - Reinstall the CLI after pulling this repo:
uv tool install --force --editable .(ilha --versionmust match git).
# Push code + config only (default; does not copy the laptop database)
ilha droplet put staging --domain app.example.com
# Dump remote Postgres first, then update
ilha droplet dump-db staging --domain app.example.com
ilha droplet put staging --domain app.example.com
# Replace remote volumes with the laptop copies (destructive)
ilha droplet put staging --domain app.example.com --restore-volumes --i-mean-it# Tail all service logs (default: 100 lines)
ilha droplet logs staging
# Follow logs in real-time (Ctrl+C to stop)
ilha droplet logs staging -f
# Specific services
ilha droplet logs staging -s web -s db
# More lines
ilha droplet logs staging -n 500
# With explicit target
ilha droplet logs --domain app.example.com -s web -fLocal worktree diagnostics: ilha doctor staging. ilha droplet doctor was removed.
ilha supports deploying multiple independent projects to the same server, each with their own unique URL. This enables efficient resource usage while maintaining complete isolation between projects.
Global Caddy Proxy:
- A single global Caddy container (
ilha_caddy_proxy) runs on the server - Monitors all Docker containers on the host via
/var/run/docker.sock - Automatically discovers containers with
caddy.proxylabels from any project - Routes traffic based on domain/IP specified in container labels
Container Discovery:
- Caddy dynamically scans all containers on the Docker host
- Finds containers with
caddy.proxylabels regardless of which project directory they belong to - Each project's containers are automatically discovered and routed
Shared Network:
- All project containers join the
ilha_caddy_proxynetwork (external) - This enables the global Caddy to route to containers from any project
- Projects remain isolated but share the routing infrastructure
# Deploy Project 1 (e.g., Django app) to server
cd /path/to/project1
ilha droplet deploy --no-create --domain app1.example.com
# Deploy Project 2 (e.g., Flask API) to same server
cd /path/to/project2
ilha droplet deploy --no-create --domain api.example.com
# Deploy Project 3 (e.g., Node.js app) to same server
cd /path/to/project3
ilha droplet deploy --no-create --domain app3.example.comAll three projects will:
- ✅ Run independently with isolated containers and volumes
- ✅ Be accessible via their own unique domains
- ✅ Share the same global Caddy proxy for routing
- ✅ Not interfere with each other
-
Global Caddy Must Be Running: Ensure
ilha_caddy_proxycontainer is running on the server# On the server, start global Caddy if not already running ilha start-proxy -
Unique Domains/IPs: Each project must have a unique domain or IP address
- ✅
app1.example.com,app2.example.com,api.example.com(all different) - ❌ Cannot use the same domain for multiple projects
- ✅
-
Project Isolation: Projects can be in completely different directories
- Standalone mode works perfectly for multi-project deployments
- Each project maintains its own
.ilha/configuration
-
Automatic Label Configuration: ilha automatically adds
caddy.proxylabels to containers- Labels are set based on
--domainor--ipflags during deploy - No manual configuration needed
- Labels are set based on
- Resource Efficiency: One reverse proxy handles all projects
- Complete Isolation: Each project has its own containers, volumes, and networks
- Simple Management: Deploy each project independently
- Flexible: Mix domain-based HTTPS and IP-based HTTP deployments
- Scalable: Add new projects without affecting existing ones
┌─────────────────────────────────────────────────────────┐
│ Global Caddy Proxy │
│ (ilha_caddy_proxy) │
│ Ports: 80, 443, 2019 │
│ │
│ Monitors ALL containers via Docker socket │
│ Routes based on caddy.proxy labels │
└─────────────────────────────────────────────────────────┘
│
┌───────────┼───────────┐
│ │ │
┌───────▼──────┐ ┌──▼──────┐ ┌──▼──────┐
│ Project 1 │ │Project 2│ │Project 3│
│ │ │ │ │ │
│ app1.example │ │api.exam │ │app3.exam│
│ .com │ │ ple.com│ │ ple.com│
│ │ │ │ │ │
│ • PostgreSQL │ │• MongoDB│ │• Redis │
│ • Redis │ │• Web │ │• Web │
│ • Web App │ │• API │ │• App │
└──────────────┘ └─────────┘ └─────────┘
ilha can automatically manage DNS records via Digital Ocean DNS API:
- Automatic Domain Creation: If a subdomain doesn't exist, ilha will automatically create it when
--domainis provided - Domain Validation: Checks if DNS records already exist and point to the correct server
- Supported Provider: Digital Ocean DNS
- API Token Configuration: Multiple options supported (priority order):
- CLI flag:
--dns-token <token> - Shell environment:
export DIGITALOCEAN_API_TOKEN=token .envfile: AddDIGITALOCEAN_API_TOKEN=tokento project root.envfile- Global config: Add
DIGITALOCEAN_API_TOKEN=tokento~/.ilha/env.ilhafile
- CLI flag:
- Generate a personal access token from https://cloud.digitalocean.com/account/api/tokens
- Configure token using one of these methods:
- Shell environment (recommended for CI/CD):
export DIGITALOCEAN_API_TOKEN=your_token - Project
.envfile (recommended for project-specific tokens): AddDIGITALOCEAN_API_TOKEN=your_tokento your project root.envfile - Global config (recommended for personal tokens): Add
DIGITALOCEAN_API_TOKEN=your_tokento~/.ilha/env.ilhafile - CLI flag: Use
--dns-token <token>when pushing
- Shell environment (recommended for CI/CD):
When ilha creates a DNS record, it is immediately available on Digital Ocean's authoritative nameservers, but it takes time to propagate to all DNS resolvers worldwide.
What is DNS Propagation? DNS propagation is the time it takes for DNS record changes to spread across all DNS servers on the internet. When you create a new DNS record, it's immediately available on the authoritative nameservers (Digital Ocean's in this case), but other DNS resolvers (like Google's 8.8.8.8 or Cloudflare's 1.1.1.1) cache DNS records and may take time to update.
Expected Propagation Times:
- Authoritative nameservers: Immediate (Digital Ocean nameservers)
- Public resolvers: 5-60 minutes typically
- Global propagation: Up to 48 hours in rare cases
Verifying DNS Records:
You can verify DNS records in several ways:
-
Check on Digital Ocean nameservers (immediate):
dig your-domain.com A @ns1.digitalocean.com dig your-domain.com A @ns2.digitalocean.com dig your-domain.com A @ns3.digitalocean.com
-
Check on public resolvers (may take time):
dig your-domain.com A @8.8.8.8 # Google DNS dig your-domain.com A @1.1.1.1 # Cloudflare DNS
-
Check from your local machine:
dig your-domain.com A nslookup your-domain.com
Troubleshooting:
- If the record exists on Digital Ocean nameservers but not on public resolvers, wait a few more minutes
- If the record doesn't exist on any nameserver, check that the DNS record was created successfully
- Browser DNS caches may need to be cleared or wait for TTL expiration
ilha supports VPC (Virtual Private Cloud) deployments for secure private networking between droplets. For ephemeral RQ workers that connect to a central droplet over private IP, see Ephemeral Worker Pool above.
Central and Worker Droplet Setup:
# 1. Deploy central server with db/redis services
ilha droplet deploy test \
--domain central.example.com \
--ssh-keys anders
# 2. Deploy worker in same VPC (reuses VPC UUID from central)
ilha droplet deploy test \
--central-droplet-name central \
--ssh-keys andersVPC Features:
- Automatic VPC Detection: Extracts private IP addresses and VPC UUID from droplets
- VPC UUID Reuse: Use
--central-droplet-nameto automatically reuse VPC UUID from central droplet - Port Binding: Configure
.ilha/config.ymlto automatically bind ports for VPC-accessible services:vpc: auto_bind_ports: true # Enable automatic port binding (default: false) bind_to_private_ip: true # Bind to private IP instead of 0.0.0.0 (default: true, recommended for security)
- Security: Redis and database ports are automatically bound to private IP (not public) when available, preventing public internet exposure
- Firewall Configuration: Optional automatic UFW firewall rules to restrict access to VPC network only
- Worker Environment Configuration: When deploying workers with
--exclude-deps, environment variables are automatically configured to point to central server's private IP - VPC Metadata: Package metadata includes VPC deployment information for automatic configuration
VPC Configuration Options:
--vpc-uuid <uuid>- Explicitly specify VPC UUID--central-droplet-name <name>- Reuse VPC UUID from central droplet (for worker deployments)--exclude-deps <services>- Exclude services from dependency resolution (indicates worker deployment)
Security Best Practices for VPC Deployments:
When deploying with --central-server and VPC networking, ilha implements multiple security layers:
-
Private IP Binding (Default: Enabled)
- Redis and database ports bind to private IP address (e.g.,
10.x.x.x:6379) instead of0.0.0.0 - Prevents public internet exposure while maintaining VPC accessibility
- Configure via
.ilha/config.yml:vpc: bind_to_private_ip: true # Default: true
- Redis and database ports bind to private IP address (e.g.,
-
Firewall Rules (Optional: Opt-in)
- Automatically configures UFW firewall rules to restrict access to VPC network (10.0.0.0/8)
- Only allows connections from within the VPC
- Configure via
.ilha/config.yml:vpc: auto_configure_firewall: true # Default: false (opt-in)
-
Verification After deployment, verify security:
# On central server, check port bindings docker ps --format "{{.Names}}\t{{.Ports}}" | grep -E "(redis|db)" # Should show: 10.x.x.x:6379->6379/tcp (not 0.0.0.0:6379) # Verify public access is blocked telnet <public-ip> 6379 # Should fail # Verify VPC access works telnet <private-ip> 6379 # Should succeed from worker droplet
- When using
--domain, ilha automatically enables HTTPS via Caddy's Let's Encrypt integration - When using
--ip, deployments are HTTP-only. Certificate authorities do not issue certificates for IP addresses; use a domain for HTTPS. - DNS records are automatically created when
--domainis provided (use--skip-dns-checkto skip DNS management) - You can add defaults in
.ilha/config.yml(optional):
deployment:
default_server: user@server:/var/ilha/packages
default_domain: myapp.example.com
default_ip: 203.0.113.10
ssh_key: ~/.ssh/deploy_key
vpc:
auto_bind_ports: true # Enable automatic port binding for VPC-accessible services
bind_to_private_ip: true # Bind to private IP only (default: true, recommended for security)
auto_configure_firewall: false # Auto-configure UFW rules (default: false, opt-in)For detailed architecture information, see documentation/ARCHITECTURE.md
Additional Push Details:
- Auto-import uses a robust remote script with strict mode and consistent quoting to avoid empty variables and broken chains.
- The remote script resolves the ilha binary (prefers
/opt/ilha-venv/bin/ilha, falls back toilha). - Import runs with
--non-interactive; older ilha versions safely ignore unknown flags. - If an existing ilha project is detected on the server, a normal import is used; otherwise standalone import is performed.