fix: restore Linux Go plugin telemetry and outbound TLS (godebug tlsmlkem=0) - #1778
Merged
Merged
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
suyadav1
changed the base branch from
suyadav/remove-public-rollout-dependency
to
ci_prod
September 3, 2026 21:35
Contributor
Author
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. |
Contributor
|
the change tlsmlkem=0 will be also applied to windows build. so we need test both win and linux after this change. |
zanejohnson-azure
approved these changes
Sep 3, 2026
suyadav1
enabled auto-merge (squash)
September 3, 2026 23:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Starting with
3.6.1-ci-prod-08-26-2026-9dcdee18, every Go-plugin-emitted telemetry signal from Linux agents stopped reaching App Insights, fleet-wide. All Ruby-plugin signals stayed healthy, and Windows was unaffected — which is what made this look like a version-format or pipeline problem rather than an outage.Bisected via production telemetry to #1770 (Go 1.26.7 → 1.27.0).
Root cause
Microsoft Go is built with
systemcrypto, so TLS crypto is delegated to the runtime image's OpenSSL. Go 1.27 enablesX25519MLKEM768in the ClientHello by default.The OpenSSL in the Azure Linux 3.0 base image has a hardcoded allowlist in
EVP_PKEY_Q_keygenthat rejects provider-supplied key names. ML-KEM-768 can therefore be fetched via the SymCrypt provider but not generated. Key generation fails while building the ClientHello, so 100% of outbound Go TLS fails before a connection is ever established.Verified in the shipped ciprod image:
InsecureSkipVerifyandGOFIPS140=offdo not help;tlsmlkem=0doesgo.dev/dl), not the MS fork, so there is nosystemcryptopathUpstream tracking: microsoft/go#2472, microsoft/azurelinux#18546.
Fix
Set
godebug tlsmlkem=0in both Go module files. This disables only the ML-KEM key exchange and falls back to X25519; everything else about the TLS stack is unchanged.This is deliberately a forward fix. Pinning Go back to 1.26.7 was rejected because it reintroduces the CVEs that 1.27 fixed.
Impact while broken
Customer log/metric data continued to flow — the ingestion path to mdsd is a local socket and does not use Go TLS. The customer-visible gap was limited to
kube-apiserver-based image enrichment (Image/ImageIDonContainerLogv1 when enrichment is enabled), which measured at ~846 clusters. Agent-side telemetry and diagnostics were fully dark for the duration.Removal criteria
Remove
godebug tlsmlkem=0once the base image shipsopenssl >= 3.3.7-5.azl3. That fix (microsoft/azurelinux#18630) is merged into3.0-devbut has not been cherry-picked to the3.0release branch yet, so it is not in any published base image today.Validation
godebugkey fails the build, so the directive is genuinely appliedgo mod edit -jsonreportsGoDebug: [{tlsmlkem 0}]in both modulesgo.sumdrift