fix(windows): start telegraf on Windows Server 2025 hosts

[zanejohnson-azure]

Problem

Test-FluentbitTcpListener detects the fluent-bit listener by scanning netstat -an for a LISTENING row on port 25229.

Inside the ama-logs container on a Windows Server 2025 host, both netstat -an and Get-NetTCPConnection return zero connection rows — even though the listener is present and accepting connections. TCP itself works fine; only enumeration is blind.

So the check never succeeds on WS2025, Start-Telegraf always takes the failure path, and the agent logs:

Test-FluentbitTcpListener: Retry attempt 1/45 ... 45/45
Test-FluentbitTcpListener: Failed to detect TCP listener after 45 attempts. Exiting script.
Telegraf not started since Fluentbit tcp listener is not up and running on port 25229

Because --service install runs unconditionally, the telegraf service exists but is never started. No custom Prometheus metrics are collected from WS2025 nodes at all.

Measured inside the agent container on both OS versions:

Probe	WS2022	WS2025
netstat -an LISTENING rows	38-39	0
Get-NetTCPConnection -State Listen	35	0
Bind port 25999 -> enumerate	visible	invisible
Bind port 25999 -> TCP connect	succeeds	succeeds

Fix

Detect the listener by connecting to it instead of enumerating sockets. A TcpClient.BeginConnect/WaitOne probe tests exactly what the caller needs to know, and works on both WS2022 and WS2025.

Also bound the retry loop by wall-clock time. A failed connect can consume its full timeout, so counting attempts would have stretched the configured wait (e.g. 45s) to roughly 3x that. The deadline keeps the configured window accurate.

Validation

Run on an AKS cluster (K8s 1.35.7) with both WS2022 and WS2025 node pools, agent ciprod:win-3.6.0:

• Shipped function executed inside the real WS2025 container: old netstat method False, new connect method True, closed-port control False (no false positive).
• End-to-end: WS2025 produced zero InsightsMetrics rows across the entire ~12h cluster lifetime (verified with firstSeen=min(TimeGenerated)). After the listener was detected and telegraf started, scraped samples began flowing to the workspace within ~20 seconds and sustained ~18-20 per 5-minute bin.
• WS2022 unchanged: listener still detected (attempt 3), telegraf runs.
• Retry-loop timing: negative case returns in 6.1s for a 6s window (previously ~18s when attempt-counted); positive case returns in 0.01s on the first attempt.
• main.ps1 parses with 0 syntax errors.

Independently reproduced on a second cluster

The same defect was confirmed on a different cluster, in a different region, running a newer agent image (win-3.8.0-ci-prod-09-03-2026-fd42f68c): netstat and Get-NetTCPConnection both returned zero rows, a real connect to 127.0.0.1:25229 succeeded, a closed port (39999) was correctly refused, and the telegraf service was left Stopped with no process. Line 60 still uses netstat in 3.8.0, so this fix is still required on current images.

Data-side impact on that cluster: the WS2022 node ingests custom Prometheus metrics normally while the WS2025 node produces none at all.

Important for sequencing - this fix alone is not sufficient for WS2025

Starting telegraf on WS2025 also exposes it to a separate, pre-existing defect that affects WS2022 today, caused by the old telegraf version shipped on Windows (1.24.2, verified by --version inside the container).

The defect: the kubelet credential is captured once and never refreshed

In telegraf v1.24.2, plugins/inputs/prometheus/kubernetes.go passes the in-cluster bearer token straight through, with no re-read:

case <-time.After(time.Second):
	if p.isNodeScrapeScope {
		err = p.cAdvisor(ctx, config.BearerToken)   // v1.24.2: snapshot, never refreshed

and cAdvisor builds the request once and reuses it:

// The request will be the same each time
req.Header.Set("Authorization", "Bearer "+bearerToken)

So Kubernetes service-account token rotation (~hourly) is never picked up, and roughly an hour after telegraf starts all pod discovery fails permanently:

E! [inputs.prometheus] Unable to monitor pods with node scrape scope:
   error updating pod list: error when making request for pod list with status 401 Unauthorized

Measured by manually starting telegraf on a WS2025 node to simulate post-fix behaviour: process start 16:26:49Z, cached token expiry 17:14:36Z, first 401 at 17:16:51Z, then ~57 errors/min indefinitely. A fixed WS2025 node therefore gets roughly 48 minutes of working discovery before failing permanently.

The failure is silent: telegraf keeps scraping its cached pod list, so throughput never drops while newly created pods become permanently undiscoverable. Confirmed on WS2022: 13 pods running, only 11 scraped, frozen for 10 hours.

Root cause was proven rather than inferred: from inside the same container while telegraf returned 401, the current on-disk token returned HTTP 200 from the kubelet /pods endpoint and an anonymous call returned 401 — eliminating RBAC, reachability, and cluster configuration.

This is already fixed upstream - the existing chart comment is correct

From v1.28.5 onward (verified in v1.28.5 and v1.39.1), the outer loop re-reads the token file before every cAdvisor call:

bearerToken := config.BearerToken
if config.BearerTokenFile != "" {
	bearerTokenBytes, err := os.ReadFile(config.BearerTokenFile)
	...
	bearerToken = string(bearerTokenBytes)
}
err = p.cAdvisor(ctx, bearerToken)

Because cAdvisor returns on its first failed poll, the outer loop re-reads the current token ~1s later and rebuilds the request — so the failure is self-healing rather than permanent.

This validates the existing comment in charts/azuremonitor-containers/templates/ama-logs-daemonset-windows.yaml:

#  Uncomment when telegraf upgraded to 1.28.5 or higher
# - name: kube-api-access

Recommendation: upgrade Windows telegraf from 1.24.2 to >= 1.28.5 (Linux already ships 1.39.1, which is why Linux is unaffected). Note the fix calls os.ReadFile unconditionally rather than checking file metadata — which matters, because on Windows the projected token file's LastWriteTimeUtc does not advance when the token rotates (observed content dated Sept 15 with an mtime of Sept 11). A metadata-based cache-invalidation approach would not work here.

A second, still-unfixed upstream bug worth reporting separately

Present in v1.24.2, v1.28.5 and v1.39.1 — the response body is leaked on any non-200:

	if resp.StatusCode != 200 {
		return fmt.Errorf("error when making request for pod list with status %s", resp.Status)
	}

	defer resp.Body.Close()      // placed AFTER the early return

The connection is never released and retains its bound ephemeral port. On a canary cluster running 1.24.2 this was measured as a sustained leak of one bound port per failed request (handles/error = 1.0010, distinct bound ports/error = 1.0000, R² = 0.9999994 across 8 points, no plateau) while thread count stayed flat.

Upgrading to >= 1.28.5 makes this largely moot in practice, since the 401 storm that drives it stops recurring — but the leak remains latent for any other non-200 (kubelet restart, transient failure) and is worth an upstream issue.

Secondary change

test/scenario/log-gen-all-nodes-daemonset.yaml — dropped the kubernetes.azure.com/os-sku: Windows2022 selector so the Windows log generator schedules on all Windows nodes. The ltsc2022 image runs with process isolation on both WS2022 and WS2025 hosts, so no second DaemonSet is needed. This is what made WS2025 testable. (WS2019 hosts cannot run this image and would need a separate one.)

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.