-
-
Notifications
You must be signed in to change notification settings - Fork 41
feat: containerd/nerdctl engine backend #445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 34 commits
f1820f7
9d73794
18ecd05
c0bbb00
cad27e7
40000b1
c623d21
1952476
6c05b2d
270c369
9eff2a6
c8c5a14
36923bd
980f6bd
b064ca0
05d105d
da24125
9dca7db
5c38568
6ed6c1a
abd2ef0
440b97e
e6d3afe
7729c16
63516bb
1b395e6
09b9a7e
854cb9f
1be7349
22f5771
4e2e99e
5784ca6
678de49
9f53c2d
a0e6931
73a5b28
f87eb45
82cfe1b
b41d2b6
bc4b1d4
a73dfd5
525fb9a
bcca717
513f08c
abf6ce8
31c853c
5f2f969
7de2358
c1dc204
c376f7c
07014b3
5df6efa
c41b5c0
e43c728
7a68b80
269033f
c08f532
0a2d87c
afb01b9
2444814
d8ccba7
e4fffb7
d78dd06
da922de
e1639d3
47ef985
4a56845
f774732
a1ce102
232ad8f
1eaad1b
af2f1e8
34db526
af1c4e3
4d63559
d4112de
fa7043c
dab5b91
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,6 +5,7 @@ on: | |
|
|
||
| jobs: | ||
| leia-tests: | ||
| name: ${{ matrix.leia-test }} (${{ matrix.engine }}) | ||
| runs-on: ${{ matrix.os }} | ||
| env: | ||
| TERM: xterm | ||
|
|
@@ -62,6 +63,9 @@ jobs: | |
| - update | ||
| - version | ||
| - yaml | ||
| engine: | ||
| - docker | ||
| - containerd | ||
| node-version: | ||
| - "20" | ||
| os: | ||
|
|
@@ -106,13 +110,14 @@ jobs: | |
| pkg: "@yao-pkg/pkg@5.16.1" | ||
| - name: Install full deps | ||
| run: npm clean-install --prefer-offline --frozen-lockfile | ||
| - name: Setup lando ${{ steps.pkg-action.outputs.file }} | ||
| - name: Setup lando ${{ steps.pkg-action.outputs.file }} (${{ matrix.engine }}) | ||
| uses: lando/setup-lando@v3 | ||
| with: | ||
| lando-version: ${{ steps.pkg-action.outputs.file }} | ||
| telemetry: false | ||
| config: | | ||
| setup.skipCommonPlugins=true | ||
| engine=${{ matrix.engine }} | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Config YAML indentation breaks
|
||
| - name: Run Leia Tests | ||
| uses: lando/run-leia-action@v2 | ||
| env: | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -6,7 +6,7 @@ const _ = require('lodash'); | |
| /* | ||
| * Helper to get core proxy service | ||
| */ | ||
| const getProxy = ({proxyCommand, proxyPassThru, proxyDomain, userConfRoot, version = 'unknown'} = {}) => { | ||
| const getProxy = ({proxyCommand, proxyPassThru, proxyDomain, userConfRoot, dockerSocket, version = 'unknown'} = {}) => { | ||
| return { | ||
| services: { | ||
| proxy: { | ||
|
|
@@ -21,7 +21,7 @@ const getProxy = ({proxyCommand, proxyPassThru, proxyDomain, userConfRoot, versi | |
| }, | ||
| networks: ['edge'], | ||
| volumes: [ | ||
| '/var/run/docker.sock:/var/run/docker.sock', | ||
| `${dockerSocket || '/var/run/docker.sock'}:/var/run/docker.sock`, | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Proxy builder never receives
|
||
| `${userConfRoot}/scripts/proxy-certs.sh:/scripts/100-proxy-certs`, | ||
| 'proxy_config:/proxy_config', | ||
| ], | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| --- | ||
| title: Engine | ||
| description: How to configure the Lando container engine backend (Docker or containerd) | ||
| --- | ||
|
|
||
| # Engine | ||
|
|
||
| Lando supports multiple container engine backends. By default it uses [Docker](https://www.docker.com/) but can also use [containerd](https://containerd.io/) with [nerdctl](https://github.com/containerd/nerdctl) as an alternative runtime. | ||
|
|
||
| The engine backend can be configured via the `engine` key in the [global config](global.md) or per-project in your `.lando.yml`. | ||
|
|
||
| ## Available Values | ||
|
|
||
| | Value | Description | | ||
| |---|---| | ||
| | `auto` | **(default)** Auto-detects the best available backend. Prefers containerd if all binaries are found, otherwise falls back to Docker. | | ||
| | `docker` | Always use the Docker daemon and Docker Compose. This is the traditional Lando behavior. | | ||
| | `containerd` | Use Lando's own isolated containerd + buildkitd + nerdctl stack. | | ||
|
|
||
| ## Configuration | ||
|
|
||
| **Global config (~/.lando/config.yml)** | ||
|
|
||
| ```yaml | ||
| # use auto-detection (default) | ||
| engine: auto | ||
|
|
||
| # force Docker | ||
| engine: docker | ||
|
|
||
| # force containerd | ||
| engine: containerd | ||
| ``` | ||
|
|
||
| **Per-project (.lando.yml)** | ||
|
|
||
| ```yaml | ||
| name: my-app | ||
| engine: containerd | ||
| services: | ||
| web: | ||
| type: php:8.2 | ||
| via: nginx | ||
| ``` | ||
|
|
||
| ## Auto-Detection | ||
|
|
||
| When `engine` is set to `auto` (the default), Lando checks for the presence of three binaries inside `~/.lando/bin/`: | ||
|
|
||
| 1. `containerd` — the container runtime daemon | ||
| 2. `nerdctl` — the Docker-compatible CLI for containerd | ||
| 3. `buildkitd` — the image build daemon | ||
|
|
||
| If **all three** binaries exist, Lando uses the containerd backend. If any are missing, it falls back to Docker. | ||
|
|
||
| ::: tip | ||
| The containerd binaries are installed automatically by `lando setup` when containerd support is enabled. You don't need to install them manually. | ||
| ::: | ||
|
|
||
| ## Overriding Binary Paths | ||
|
|
||
| If your containerd stack binaries are installed in a non-standard location, you can override each path individually in the [global config](global.md): | ||
|
|
||
| ```yaml | ||
| # Override individual binary paths | ||
| containerdBin: /usr/local/bin/containerd | ||
| nerdctlBin: /usr/local/bin/nerdctl | ||
| buildkitdBin: /usr/local/bin/buildkitd | ||
|
|
||
| # Override the containerd socket path | ||
| containerdSocket: /run/containerd/containerd.sock | ||
| ``` | ||
|
|
||
| By default, Lando looks for binaries in `~/.lando/bin/` and manages its own isolated containerd socket at `~/.lando/run/containerd.sock`. | ||
|
|
||
| ## How It Works | ||
|
|
||
| When using the containerd backend, Lando: | ||
|
|
||
| 1. Starts its **own isolated** containerd and buildkitd daemons (separate from any system containerd) | ||
| 2. Uses `nerdctl compose` instead of `docker compose` for service orchestration | ||
| 3. Uses `nerdctl` instead of `docker` for container inspection, listing, and management | ||
| 4. Manages all state in `~/.lando/` to avoid interfering with system containers | ||
|
|
||
| The containerd backend is fully compatible with existing Lando apps and compose files — no changes to your `.lando.yml` services are required. | ||
|
|
||
| ::: warning EXPERIMENTAL | ||
| The containerd engine backend is experimental. While it is designed to be a drop-in replacement for the Docker backend, some edge cases may behave differently. Please report any issues you encounter. | ||
| ::: |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| # containerd Engine Performance | ||
|
|
||
| This document covers performance characteristics of the containerd backend compared to the Docker backend, and how to benchmark them. | ||
|
|
||
| ## Benchmark Script | ||
|
|
||
| The benchmark script at `scripts/benchmark-engines.sh` measures three core operations: | ||
|
|
||
| | Operation | What It Measures | | ||
| |-----------|-----------------| | ||
| | **Image pull** | Downloads `alpine:latest` from a registry. Measures registry I/O and image unpacking speed. The image is removed before each pull to ensure a fresh download. | | ||
| | **Container run** | Runs `echo hello` in a fresh container and removes it (`--rm`). Measures container creation, execution, and teardown overhead. | | ||
| | **Container list** | Runs `ps` to list containers. Measures daemon response time for metadata queries. | | ||
|
|
||
| ### Usage | ||
|
|
||
| ```bash | ||
| # Compare both engines (3 runs each, default) | ||
| ./scripts/benchmark-engines.sh | ||
|
|
||
| # Benchmark only containerd with 5 runs | ||
| ./scripts/benchmark-engines.sh --engine containerd --runs 5 | ||
|
|
||
| # Benchmark only Docker, output to a specific file | ||
| ./scripts/benchmark-engines.sh --engine docker --runs 3 --output ./results.md | ||
| ``` | ||
|
|
||
| Results are written as a markdown table to `/tmp/lando-benchmark-<timestamp>.md` by default. | ||
|
|
||
| ### Environment Variables | ||
|
|
||
| | Variable | Default | Description | | ||
| |----------|---------|-------------| | ||
| | `DOCKER_BIN` | `docker` | Path to the Docker CLI binary | | ||
| | `NERDCTL_BIN` | `~/.lando/bin/nerdctl` | Path to the nerdctl binary | | ||
| | `CONTAINERD_SOCK` | `~/.lando/run/containerd.sock` | Path to the containerd socket | | ||
| | `LANDO_DIR` | `~/.lando` | Lando configuration root | | ||
|
|
||
| ## Known Performance Characteristics | ||
|
|
||
| ### Linux: containerd avoids Docker Desktop overhead | ||
|
|
||
| On Linux, Lando's containerd backend runs natively — there is no Docker Desktop layer, no VM, and no Docker daemon multiplexing. This eliminates several sources of overhead: | ||
|
|
||
| - **No Docker daemon**: containerd is a minimal container runtime. Docker adds an additional daemon layer (dockerd) on top of containerd that handles API translation, logging drivers, networking plugins, and more. Bypassing this layer reduces per-operation latency. | ||
| - **No Docker Desktop VM**: On macOS and Windows, Docker Desktop runs containers inside a Linux VM (HyperKit / WSL2). On Linux with containerd, containers run directly on the host kernel. | ||
| - **Direct socket communication**: nerdctl talks to containerd's gRPC API directly, without the Docker API translation layer. | ||
|
|
||
| ### Container startup | ||
|
|
||
| Container startup time is primarily bounded by: | ||
|
|
||
| 1. Image layer unpacking (first run only — cached thereafter) | ||
| 2. Namespace and cgroup setup (kernel overhead, similar for both engines) | ||
| 3. Network namespace creation (Lando uses CNI with containerd vs. Docker's libnetwork) | ||
|
|
||
| In practice, the difference for container startup is small (tens of milliseconds) because both engines ultimately call the same Linux kernel primitives. | ||
|
|
||
| ### Image operations | ||
|
|
||
| Image pull performance is dominated by network I/O and registry latency. The containerd backend uses the same OCI registries and the same content-addressable storage model. Differences are typically negligible. | ||
|
|
||
| ### BuildKit Cache Optimization | ||
|
|
||
| The containerd backend uses BuildKit directly (not via Docker's BuildKit integration). The BuildKit configuration (see Task 24) includes GC policies that manage the build cache: | ||
|
|
||
| ```toml | ||
| [worker.containerd] | ||
| gc = true | ||
| gckeepstorage = 10000 # ~10 GB | ||
|
|
||
| [[worker.containerd.gcpolicy]] | ||
| keepBytes = 1073741824 # 1 GB reserved | ||
| keepDuration = 604800 # 7 days | ||
| all = true | ||
|
|
||
| [[worker.containerd.gcpolicy]] | ||
| keepBytes = 5368709120 # 5 GB reserved | ||
| all = false | ||
| ``` | ||
|
|
||
| These GC policies ensure the build cache doesn't grow unbounded while retaining frequently-used layers. This is particularly beneficial for iterative development where the same base images and dependency layers are rebuilt frequently. | ||
|
|
||
| ### Performance Logging | ||
|
|
||
| The containerd daemon includes built-in performance timers (via `utils/perf-timer.js`) that log elapsed time for key operations when debug mode is enabled: | ||
|
|
||
| - `up()` — total engine startup time | ||
| - `_startContainerd()` — containerd daemon spawn + socket ready | ||
| - `_startBuildkitd()` — buildkitd daemon spawn + socket ready | ||
|
|
||
| Enable debug logging with `DEBUG=@lando/*` or by setting `debug: true` in your Lando config to see these timings. | ||
|
|
||
| ## Benchmark Results | ||
|
|
||
| <!-- TODO: Populate with actual benchmark data from CI or local runs --> | ||
|
|
||
| _No benchmark results recorded yet. Run `./scripts/benchmark-engines.sh` and paste the output here._ | ||
|
|
||
| ### Example Output | ||
|
|
||
| ```markdown | ||
| # Lando Engine Benchmark Results | ||
|
|
||
| - **Date**: 2026-03-14 00:00:00 UTC | ||
| - **Host**: Linux 6.x.x x86_64 | ||
| - **Runs per operation**: 3 | ||
|
|
||
| ## Docker | ||
|
|
||
| | Operation | Mean (ms) | Median (ms) | Raw (ms) | | ||
| |-----------|-----------|-------------|----------| | ||
| | Image pull (`alpine:latest`) | — | — | — | | ||
| | Container run (`echo hello`) | — | — | — | | ||
| | Container list (`ps`) | — | — | — | | ||
|
|
||
| ## containerd (nerdctl) | ||
|
|
||
| | Operation | Mean (ms) | Median (ms) | Raw (ms) | | ||
| |-----------|-----------|-------------|----------| | ||
| | Image pull (`alpine:latest`) | — | — | — | | ||
| | Container run (`echo hello`) | — | — | — | | ||
| | Container list (`ps`) | — | — | — | | ||
| ``` | ||
|
|
||
| ## Future Work | ||
|
|
||
| - **CI integration**: Run benchmarks automatically on tagged releases to track regressions. | ||
| - **Application-level benchmarks**: Measure `lando start` / `lando rebuild` end-to-end with a sample app. | ||
| - **Memory profiling**: Compare RSS of containerd + buildkitd vs. dockerd + containerd + buildkitd. | ||
| - **macOS Lima benchmarks**: Compare containerd-in-Lima vs. Docker Desktop performance on macOS. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,84 @@ | ||
| 'use strict'; | ||
|
|
||
| const _ = require('lodash'); | ||
|
|
||
| module.exports = async (app, lando) => { | ||
| // Skip if not using the containerd backend | ||
| const backend = _.get(lando, 'engine.engineBackend', _.get(lando, 'config.engine', 'auto')); | ||
| if (backend !== 'containerd') return; | ||
|
|
||
| _.forEach(_(lando.versions).filter(version => version && !version.dockerVersion).value(), thing => { | ||
| // handle generic unsupported or untested notices | ||
| if (!thing.satisfied) app.addMessage(require('../messages/unsupported-version-warning')({ | ||
|
Check failure on line 12 in hooks/app-check-containerd-compat.js
|
||
| ...thing, | ||
| name: thing.name, | ||
| })); | ||
| if (thing.untested) app.addMessage(require('../messages/untested-version-notice')(thing)); | ||
|
|
||
| // handle nerdctl (compose equivalent) recommend update | ||
| if (thing.name === 'nerdctl' && thing.rupdate) { | ||
| app.addMessage(require('../messages/update-nerdctl-warning')(thing)); | ||
| } | ||
| }); | ||
|
|
||
| // Run live containerd-specific health checks | ||
| try { | ||
| const daemon = lando.engine.daemon; | ||
|
|
||
| // Verify containerd daemon is running | ||
| const isUp = await daemon.isUp(); | ||
| if (!isUp) { | ||
| app.addMessage({ | ||
| type: 'warning', | ||
| title: 'Containerd daemon is not running', | ||
| detail: [ | ||
| 'The containerd daemon does not appear to be running.', | ||
| 'Lando needs containerd to manage containers. Try running "lando start"', | ||
| 'which will attempt to start the daemon automatically.', | ||
| ], | ||
| }); | ||
| } | ||
|
|
||
| // Verify nerdctl compose is functional | ||
| if (isUp) { | ||
| try { | ||
| const runCommand = require('../utils/run-command'); | ||
| await runCommand(daemon.nerdctlBin, ['compose', 'version'], { | ||
| debug: daemon.debug, | ||
| ignoreReturnCode: false, | ||
| }); | ||
| } catch (err) { | ||
| app.addMessage({ | ||
| type: 'warning', | ||
| title: 'nerdctl compose is not functional', | ||
| detail: [ | ||
| 'Could not run "nerdctl compose version" successfully.', | ||
| 'nerdctl compose is required for service orchestration.', | ||
| `Error: ${err.message}`, | ||
| ], | ||
| url: 'https://github.com/containerd/nerdctl/releases', | ||
| }); | ||
| } | ||
|
|
||
| // Verify buildkitd is running | ||
| const buildkitRunning = daemon._isProcessRunning | ||
| ? daemon._isProcessRunning(daemon.buildkitdPidFile) | ||
| : false; | ||
|
|
||
| if (!buildkitRunning) { | ||
| app.addMessage({ | ||
| type: 'warning', | ||
| title: 'BuildKit daemon is not running', | ||
| detail: [ | ||
| 'The BuildKit daemon (buildkitd) does not appear to be running.', | ||
| 'BuildKit is required for building container images with the containerd backend.', | ||
| 'Try running "lando start" which will attempt to start buildkitd automatically.', | ||
| ], | ||
| url: 'https://github.com/moby/buildkit/releases', | ||
| }); | ||
| } | ||
| } | ||
| } catch (err) { | ||
| lando.log.debug('containerd health check encountered an error: %s', err.message); | ||
| } | ||
| }; | ||


There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CI matrix doubles all tests including Docker-only ones
Medium Severity
Adding
engine: [docker, containerd]to the matrix without any exclusions doubles every single test job. Since the PR description states "Leia integration tests do not run against containerd in CI" and containerd isn't set up in CI, all thecontainerdmatrix variants will likely fail. This matrix expansion needsexcluderules or a separate job rather than a full cross-product.