Skip to content

fix: restore Linux Go plugin telemetry and outbound TLS (godebug tlsmlkem=0) - #1778

Merged
suyadav1 merged 3 commits into
ci_prodfrom
fix/linux-go-plugin-telemetry-regression
Sep 3, 2026
Merged

fix: restore Linux Go plugin telemetry and outbound TLS (godebug tlsmlkem=0)#1778
suyadav1 merged 3 commits into
ci_prodfrom
fix/linux-go-plugin-telemetry-regression

Conversation

@suyadav1

@suyadav1 suyadav1 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Problem

Starting with 3.6.1-ci-prod-08-26-2026-9dcdee18, every Go-plugin-emitted telemetry signal from Linux agents stopped reaching App Insights, fleet-wide. All Ruby-plugin signals stayed healthy, and Windows was unaffected — which is what made this look like a version-format or pipeline problem rather than an outage.

Bisected via production telemetry to #1770 (Go 1.26.7 → 1.27.0).

Root cause

Microsoft Go is built with systemcrypto, so TLS crypto is delegated to the runtime image's OpenSSL. Go 1.27 enables X25519MLKEM768 in the ClientHello by default.

The OpenSSL in the Azure Linux 3.0 base image has a hardcoded allowlist in EVP_PKEY_Q_keygen that rejects provider-supplied key names. ML-KEM-768 can therefore be fetched via the SymCrypt provider but not generated. Key generation fails while building the ClientHello, so 100% of outbound Go TLS fails before a connection is ever established.

Verified in the shipped ciprod image:

  • Go 1.27.0 and 1.27.1 both fail identically
  • OpenSSL 3.3.5 passes, 3.3.7 fails — so this is not simply "older than 3.5"
  • InsecureSkipVerify and GOFIPS140=off do not help; tlsmlkem=0 does
  • Windows is structurally immune: it builds with upstream Go (go.dev/dl), not the MS fork, so there is no systemcrypto path

Upstream tracking: microsoft/go#2472, microsoft/azurelinux#18546.

Fix

Set godebug tlsmlkem=0 in both Go module files. This disables only the ML-KEM key exchange and falls back to X25519; everything else about the TLS stack is unchanged.

This is deliberately a forward fix. Pinning Go back to 1.26.7 was rejected because it reintroduces the CVEs that 1.27 fixed.

Impact while broken

Customer log/metric data continued to flow — the ingestion path to mdsd is a local socket and does not use Go TLS. The customer-visible gap was limited to kube-apiserver-based image enrichment (Image / ImageID on ContainerLog v1 when enrichment is enabled), which measured at ~846 clusters. Agent-side telemetry and diagnostics were fully dark for the duration.

Removal criteria

Remove godebug tlsmlkem=0 once the base image ships openssl >= 3.3.7-5.azl3. That fix (microsoft/azurelinux#18630) is merged into 3.0-dev but has not been cherry-picked to the 3.0 release branch yet, so it is not in any published base image today.

Validation

  • Reproduced the failure and confirmed the fix inside the real ciprod image for both Go 1.27.0 and 1.27.1
  • Confirmed an unknown godebug key fails the build, so the directive is genuinely applied
  • go mod edit -json reports GoDebug: [{tlsmlkem 0}] in both modules
  • No go.sum drift

@suyadav1
suyadav1 requested a review from a team as a code owner September 3, 2026 21:33
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@suyadav1
suyadav1 changed the base branch from suyadav/remove-public-rollout-dependency to ci_prod September 3, 2026 21:35
@suyadav1

suyadav1 commented Sep 3, 2026

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

@zanejohnson-azure

Copy link
Copy Markdown
Contributor

the change tlsmlkem=0 will be also applied to windows build. so we need test both win and linux after this change.

@suyadav1
suyadav1 enabled auto-merge (squash) September 3, 2026 23:04
@suyadav1
suyadav1 merged commit fd42f68 into ci_prod Sep 3, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants