Add support for envoy gateway by smatting · Pull Request #5150 · wireapp/wire-server

smatting · 2026-03-23T13:52:27Z

This PR:

Introduces a new Helm chart wire-ingress that targets Envoy Gateway. It is intended as a replacement for the nginx-ingress-services chart, which uses ingress-nginx. The wire-ingress chart is not production-ready yet.
Changes the integration test suite: all tests now run against the wire-ingress chart. The ingress solution can be selected via the WIRE_INGRESS_MODE environment variable. The federation domains change in Envoy mode — see comments in the code. Changes to the federator and integration charts are made to accommodate both variants for
testing. As a consequence, any changes to nginx-ingress-services will be untested once this PR is merged. I've added a check integration-setup-federation.sh that prevents any changes to nginx-ingress-services to avoid this being overlooked.
Changes the temporary filenames used in the integration test suite. This fixes issues with filenames that were too long for nginz to handle.
Deletes the unused file hack/helmfile-federation-v0.yaml.gotmpl.
Add a post-upgrade to all objects needed for testing. This makes running tests on the cluster manually more convenient

Checklist

Revert .envrc before merging !!!!
Add a new entry in an appropriate subdirectory of changelog.d
Read and follow the PR guidelines

wartraxx51 · 2026-03-24T09:30:19Z

Naming thing that bugs me across the whole PR, the chart is wire-ingress, the file is ingress-envoy.yaml the route is nginz-websockets. None of thes have any Ingress in them. My point of view, name it like the replacement wire-gateway , gateway-envoy.yaml etc... not blocking when I open a file called ingress-envoy.yaml and the first thing I see is kind: gateway. I have to stop and double check.

wartraxx51 · 2026-03-24T09:48:35Z

A BackendTrafficPolicy is missing for WebSockets, the default timeouts will terminate long-lived connections, I didn't check the default value, https://gateway.envoyproxy.io/latest/api/extension_types/#backendtrafficpolicy

smatting · 2026-03-25T14:24:49Z

Naming thing that bugs me across the whole PR, the chart is wire-ingress, the file is ingress-envoy.yaml the route is nginz-websockets. None of thes have any Ingress in them. My point of view, name it like the replacement wire-gateway , gateway-envoy.yaml etc... not blocking when I open a file called ingress-envoy.yaml and the first thing I see is kind: gateway. I have to stop and double check.

I changed the ingress-envoy.yaml filename. But I couldn't find other occurences. Everywhere the filenames should be <kind>...
Can you please more concrete?

smatting · 2026-03-25T14:25:37Z

A BackendTrafficPolicy is missing for WebSockets, the default timeouts will terminate long-lived connections, I didn't check the default value, https://gateway.envoyproxy.io/latest/api/extension_types/#backendtrafficpolicy

Thank you! Added the policy. We need to run QA tests against this on staging to see if it really works

wartraxx51 · 2026-04-17T11:45:20Z

+    - type: Inline
+      inline: |
+        function envoy_on_request(request_handle)
+          local ssl = request_handle:connection():ssl()


Something feels off here, we are trusting a header that the client may be able to forge themselves. Since we do not remove it first, it looks spoofable. Is that really expected on the Federator side?

Good catch! Addressed in 300c32e

In addition to this it would've been nice to also enforce client certificates, but this would change the protocol (change error codes). I left a comment why client certs are configured optionally

wartraxx51 · 2026-04-17T11:47:41Z

+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: EnvoyExtensionPolicy
+metadata:
+  name: {{ include "wire-ingress.fullname" . }}-federator-cert-header


Other policies in this chart set namespace: {{ .Release.Namespace }} ?

Addressed in d9dac75

wartraxx51 · 2026-04-17T11:49:39Z

+    - type: Inline
+      inline: |
+        function envoy_on_request(request_handle)
+          local ssl = request_handle:connection():ssl()


same headers():add issue

addressed in adc4668

wartraxx51 · 2026-04-17T12:00:40Z

+      name: {{ printf "%s/%s/https" .Release.Namespace $gatewayName | quote }}
+      operation:
+        op: add
+        path: "/filter_chains/0/filters/0/typed_config/strip_trailing_host_dot"


Why don't you use the same approach las in charts/wire-ingress/templates/envoypatchpolicy-federator.yaml:36-41

ah, this is a leftover from a previous solution. I switched to the new solution in 05e23ad

wartraxx51 · 2026-04-17T12:02:31Z

@@ -0,0 +1,154 @@
+{{- if .Values.envoy.enabled }}


None of the five kinds here carry chart/release label

addressed in 196c682

wartraxx51 · 2026-04-17T12:53:43Z

+apiVersion: v1
+kind: Service
+metadata:
+  name: account-pages-http


no label, no namespace

Address in 196c682

wartraxx51 · 2026-04-17T12:53:49Z

+apiVersion: v1
+kind: Service
+metadata:
+  name: team-settings-http


no label, no namespace

wartraxx51 · 2026-04-17T12:54:02Z

+kind: Service
+metadata:
+  name: {{ .Release.Namespace }}-fed
+  namespace: {{ .Values.gateway.controllerNamespace }}


wartraxx51 · 2026-04-17T12:54:07Z

+apiVersion: v1
+kind: Service
+metadata:
+  name: webapp-http


no label, no namespace

wartraxx51 · 2026-04-17T13:02:00Z

+apiVersion: v1
+kind: Service
+metadata:
+  name: {{ .Release.Namespace }}-fed


The -fed suffix here is also hardcoded in integration/_helpers.tpl .Both need to stay aligned renaming one side breaks federation discovery with no obvious error.
Ideally a shared helper ? otherwise at least a cross-reference comment in both files pointing to each other ? that would also explain why we don't use wire-ingress.fullname here like everywhere else

I explored the the shared helper idea again, but it would require passing 4 values consistently in 3 helm releases to avoid this hidden assumption. I went with commenting in both places instead (7f220cc)

wartraxx51 · 2026-04-20T08:53:04Z

+#     -----END PRIVATE KEY-----
+#
+# When federator.enabled is true, a ConfigMap named `federator-ca` with a `ca.crt` key
+# must exist in the release namespace. It is created by the wire-server chart


This ConfigMap is created by the federator chart, not wire-server ?

Users don't install the federator only directly, but via requirement of the wire-server chart. (federator is a subchart)

wartraxx51 · 2026-04-20T08:56:56Z

+  controllerName: gateway.envoyproxy.io/gatewayclass-controller
+```
+
+You need to refer to this object in the `gateway.className` paramter.


wartraxx51 · 2026-04-20T08:57:59Z

+| `config.dns.base` | Only used for CSP header rendering, which is a multi-ingress feature |
+| `tls.verify_depth` | Envoy Gateway `ClientTrafficPolicy` does not expose a direct verify-depth knob; the CA chain itself controls this |
+| `tls.enabled` | Removed — had no effect; all routes are always TLS-terminated |
+| `secrets.tlsClientCA` | No longer supplied via values. The `federator-ca` ConfigMap is created by the wire-server chart and referenced directly. |


is created by federator chart ?

Same as previous comment

wartraxx51 · 2026-04-20T10:12:28Z

@@ -0,0 +1,75 @@
+{{/* vim: set filetype=mustache: */}}


getCertificateSecretName, getCustomSolversSecretName, getIssuerName, getGatewayName
I would use it without get, because in Helm charts you usually use short names focused on the result but that's just my opinion

yeah good point. I removed the prefix in b76200c

only

mastaab · 2026-05-04T12:30:06Z

I think we should be agnostic of product names here in general, especially given the current situation with minio

Make sense. I generalized it in fab3f52

fisx

i've read all the non-helm code (3 files), LGTM!

fisx · 2026-05-07T16:00:32Z

 CHARTS_DIR="${TOP_LEVEL}/.local/charts"
 HELM_PARALLELISM=${HELM_PARALLELISM:-1}

+changed_files=$(git --no-pager diff-tree --no-commit-id -r --name-only HEAD)


so this guard only works if i don't commit my changes first, before running tests. i guess that's a lot better than no guard. you could compute a sha256sum from all of nginx-ingress-services i guess, and compare against a stored copy of the hash. if there is a mismatch, the error will instruct you to commit the changes and update the hash in the script.

i'm not saying you should do that, this is just my brain having ideas.

fisx · 2026-05-07T16:02:51Z

-prepareNginzK8sRuntimeFiles :: String -> ServiceMap -> IO (FilePath, FilePath, FilePath)
-prepareNginzK8sRuntimeFiles domain sm = do
-  tmpDir <- createTempDirectory "/tmp" ("nginz" <> "-" <> domain)
+prepareNginzK8sRuntimeFiles :: ServiceMap -> IO (FilePath, FilePath, FilePath)
+prepareNginzK8sRuntimeFiles sm = do
+  tmpDir <- createTempDirectory "/tmp" "nginz-"


are you merely changing the name of the temp directory here? why? (just curious.)

smatting force-pushed the WPB-23903-routes-1 branch from c4a1b87 to 28aad07 Compare March 23, 2026 13:59

zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Mar 23, 2026

smatting force-pushed the WPB-23903-routes-1 branch from 8e81e3f to 83b86e7 Compare March 25, 2026 14:02

smatting marked this pull request as ready for review March 25, 2026 14:26

smatting requested review from a team as code owners March 25, 2026 14:26

wartraxx51 reviewed Apr 17, 2026

View reviewed changes

wartraxx51 reviewed Apr 20, 2026

View reviewed changes

smatting added 16 commits April 29, 2026 10:39

wip

3c13e5e

checkboxes

0e32863

mention nginx-ingress-services in the backwards compatbility section

9487700

only

scaffolding + helpers

466b51b

modify .envrc for development

c3fb370

update plan

aa46978

phase 2-3

9404385

Allow for externally created secret

7c6c5d6

nginz routes

357edd5

more routes

ad89047

Adjust plan: routes together with services

f2f4e3c

team settings and account pages

2af357a

fake-s3 route

7e27e06

complete federator phase

b2a50c4

use different secret name for federators certificate

c2ba573

udpate plan notes for federation helper

0fb6404

smatting added 12 commits April 29, 2026 11:40

small corrections

caf3618

revert to normal

e6d9479

remove unused file

b57c883

revert file

dba41cc

update comment

d5a92ef

rename file ingress-envoy.aml to envoy-gateway.yaml

fb06b53

add missing backentrafficpolicy to support websockets

6fce96e

remove CLAUDE.md

a51ed9e

moved todo to ticket

f5b4422

prevent any untested changes to the nginx-ingress-services chart

d27cf75

remove PLAN.md

1206c78

add changelog entry

a59a6c7

smatting force-pushed the WPB-23903-routes-1 branch from f8ead45 to a59a6c7 Compare April 29, 2026 09:40

smatting added 8 commits April 29, 2026 11:51

Prevent spoofing of the X-SSL-Certificate header

300c32e

fix: remove unused arg

249e3b5

integration tests: Prevent spoofing of the X-SSL-Certificate header

adc4668

Set namespace consitently

d9dac75

apply labels consitenly

196c682

leave comments to keep service names in-sync

7f220cc

Fix spelling

0377f36

integration tests: solve trailing dot same way as rest

05e23ad

mastaab requested changes May 4, 2026

View reviewed changes

smatting added 7 commits May 5, 2026 13:50

Update error message for keeping ingress solutions in sync

707c7a3

generalize the s3 httproute from minio

fab3f52

Document dependency on wire-server chart

e395316

remove duplicated default in template

24630f1

Add additional guard to secret creation

599b57d

remove yaml separator

c52c64d

refactor: strip unnecessary "get" prefix from helpers

b76200c

fisx approved these changes May 7, 2026

View reviewed changes

Conversation

smatting commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

wartraxx51 commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wartraxx51 commented Mar 24, 2026

Uh oh!

smatting commented Mar 25, 2026

Uh oh!

smatting commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smatting Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smatting Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smatting May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smatting commented Mar 23, 2026 •

edited

Loading

wartraxx51 commented Mar 24, 2026 •

edited

Loading

smatting commented Mar 25, 2026 •

edited

Loading

smatting Apr 29, 2026 •

edited

Loading

smatting Apr 29, 2026 •

edited

Loading

smatting May 5, 2026 •

edited

Loading