fix: Honor stderrthreshold when logtostderr is enabled by pierluigilenoci · Pull Request #2016 · kubernetes-sigs/secrets-store-csi-driver

pierluigilenoci · 2026-03-23T18:20:53Z

Summary

Update k8s.io/klog/v2 from v2.120.1 to v2.140.0 which includes the fix for kubernetes/klog#212 (stderrthreshold not honored when logtostderr is set, fixed by kubernetes/klog#432)
Opt into the new klog behavior by setting -legacy_stderr_threshold_behavior=false after klog.InitFlags(nil) in main.go
Set -stderrthreshold=INFO to preserve backward-compatible behavior (all logs still appear on stderr by default)
Users can now override -stderrthreshold to WARNING or ERROR to reduce stderr noise, which was previously impossible with -logtostderr=true
Expose optional stderrThreshold Helm value for chart-based configuration

Motivation

With the default klog configuration (-logtostderr=true), the -stderrthreshold flag was completely ignored and all log messages of every severity were written to stderr. This caused excessive log noise and increased storage costs for log aggregation systems (see kubernetes/klog#212, Azure/secrets-store-csi-driver-provider-azure#387).

klog v2.140.0 introduced a new -legacy_stderr_threshold_behavior flag that, when set to false, makes -stderrthreshold work correctly even with -logtostderr=true.

Changes

File	Change
`go.mod` / `go.sum`	klog v2.120.1 → v2.140.0
`cmd/secrets-store-csi-driver/main.go`	Set `legacy_stderr_threshold_behavior=false` and `stderrthreshold=INFO` after `klog.InitFlags(nil)`
`charts/.../values.yaml`	Added optional `stderrThreshold` value (commented out)
`charts/.../secrets-store-csi-driver.yaml`	Render `-stderrthreshold` if configured
`charts/.../secrets-store-csi-driver-windows.yaml`	Same for Windows DaemonSet

Backward Compatibility

This change is fully backward compatible:

Default behavior is preserved: stderrthreshold=INFO means all logs still go to stderr
Users who want to reduce noise can now set -stderrthreshold=WARNING or -stderrthreshold=ERROR
The legacy_stderr_threshold_behavior flag can still be overridden via command line

Test plan

go build ./... passes
Pre-existing unit tests pass (failures in pkg/secrets-store are pre-existing macOS socket path issues)
Deploy with default configuration and verify all logs appear on stderr (same as before)
Deploy with -stderrthreshold=ERROR and verify only ERROR+ messages appear on stderr
Deploy with Helm stderrThreshold: WARNING and verify only WARNING+ messages appear on stderr

k8s-ci-robot · 2026-03-23T18:20:56Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

k8s-ci-robot · 2026-03-23T18:20:57Z

@pierluigilenoci: The label(s) area/logging cannot be applied, because the repository doesn't have them.

Details

In response to this:

Summary

Update k8s.io/klog/v2 from v2.120.1 to v2.140.0 which includes the fix for kubernetes/klog#212 (stderrthreshold not honored when logtostderr is set, fixed by kubernetes/klog#432)

Opt into the new klog behavior by setting -legacy_stderr_threshold_behavior=false after klog.InitFlags(nil) in main.go

Set -stderrthreshold=INFO to preserve backward-compatible behavior (all logs still appear on stderr by default)

Users can now override -stderrthreshold to WARNING or ERROR to reduce stderr noise, which was previously impossible with -logtostderr=true

Expose optional stderrThreshold Helm value for chart-based configuration

Motivation

With the default klog configuration (-logtostderr=true), the -stderrthreshold flag was completely ignored and all log messages of every severity were written to stderr. This caused excessive log noise and increased storage costs for log aggregation systems (see kubernetes/klog#212, Azure/secrets-store-csi-driver-provider-azure#387).

klog v2.140.0 introduced a new -legacy_stderr_threshold_behavior flag that, when set to false, makes -stderrthreshold work correctly even with -logtostderr=true.

Changes

File Change

go.mod / go.sum klog v2.120.1 → v2.140.0

cmd/secrets-store-csi-driver/main.go Set legacy_stderr_threshold_behavior=false and stderrthreshold=INFO after klog.InitFlags(nil)

charts/.../values.yaml Added optional stderrThreshold value (commented out)

charts/.../secrets-store-csi-driver.yaml Render -stderrthreshold if configured

charts/.../secrets-store-csi-driver-windows.yaml Same for Windows DaemonSet

Backward Compatibility

This change is fully backward compatible:

Default behavior is preserved: stderrthreshold=INFO means all logs still go to stderr

Users who want to reduce noise can now set -stderrthreshold=WARNING or -stderrthreshold=ERROR

The legacy_stderr_threshold_behavior flag can still be overridden via command line

Test plan

go build ./... passes

Pre-existing unit tests pass (failures in pkg/secrets-store are pre-existing macOS socket path issues)

Deploy with default configuration and verify all logs appear on stderr (same as before)

Deploy with -stderrthreshold=ERROR and verify only ERROR+ messages appear on stderr

Deploy with Helm stderrThreshold: WARNING and verify only WARNING+ messages appear on stderr

/kind feature
/area logging

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2026-03-23T18:21:01Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pierluigilenoci
Once this PR has been reviewed and has the lgtm label, please assign aramase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

codecov-commenter · 2026-03-23T18:33:37Z

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 22.04%. Comparing base (fd0c075) to head (4200ec4).
⚠️ Report is 29 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/secrets-store-csi-driver/main.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2016      +/-   ##
==========================================
- Coverage   22.07%   22.04%   -0.03%     
==========================================
  Files          57       57              
  Lines        3198     3202       +4     
==========================================
  Hits          706      706              
- Misses       2400     2404       +4     
  Partials       92       92

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

pierluigilenoci · 2026-03-23T23:03:21Z

/retest

pierluigilenoci · 2026-03-23T23:21:12Z

/retest

pierluigilenoci · 2026-03-24T16:29:12Z

/retest

pierluigilenoci · 2026-03-24T17:06:41Z

cc @aramase @ritazh — would you be able to review this when you get a chance? All CI checks are green. This opts into the klog fix for kubernetes/klog#212 so that -stderrthreshold is honored even when -logtostderr=true.

pierluigilenoci · 2026-03-24T17:16:40Z

/retest release-secrets-store-csi-driver-e2e-aws

pierluigilenoci · 2026-03-24T17:16:59Z

/test release-secrets-store-csi-driver-e2e-aws

pierluigilenoci · 2026-03-24T20:20:36Z

/test release-secrets-store-csi-driver-e2e-aws

k8s-ci-robot · 2026-03-24T20:43:19Z

@pierluigilenoci: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
release-secrets-store-csi-driver-e2e-aws	`2df9bff`	link	false	`/test release-secrets-store-csi-driver-e2e-aws`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

pierluigilenoci · 2026-03-25T18:20:56Z

/retest release-secrets-store-csi-driver-e2e-aws

k8s-ci-robot · 2026-03-25T18:21:00Z

@pierluigilenoci: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test pull-secrets-store-csi-driver-build

/test pull-secrets-store-csi-driver-e2e-azure

/test pull-secrets-store-csi-driver-e2e-gcp

/test pull-secrets-store-csi-driver-e2e-provider-k8s-1-32-8

/test pull-secrets-store-csi-driver-e2e-provider-k8s-1-33-4

/test pull-secrets-store-csi-driver-e2e-provider-k8s-1-34-0

/test pull-secrets-store-csi-driver-e2e-provider-k8s-1-35-0

/test pull-secrets-store-csi-driver-e2e-vault

/test pull-secrets-store-csi-driver-e2e-windows

/test pull-secrets-store-csi-driver-lint

/test pull-secrets-store-csi-driver-sanity

/test pull-secrets-store-csi-driver-unit

/test release-secrets-store-csi-driver-e2e-azure

/test release-secrets-store-csi-driver-e2e-gcp

/test release-secrets-store-csi-driver-e2e-vault

The following commands are available to trigger optional jobs:

/test pull-secrets-store-csi-driver-e2e-aws

/test pull-secrets-store-csi-driver-e2e-conjur

/test pull-secrets-store-csi-driver-e2e-deploy-manifest-e2e-provider

/test pull-secrets-store-csi-driver-e2e-fortanix

/test pull-secrets-store-csi-driver-e2e-openbao

/test pull-secrets-store-csi-driver-image-scan

/test release-secrets-store-csi-driver-e2e-aws

Use /test all to run the following jobs that were automatically triggered:

pull-secrets-store-csi-driver-build

pull-secrets-store-csi-driver-e2e-aws

pull-secrets-store-csi-driver-e2e-azure

pull-secrets-store-csi-driver-e2e-deploy-manifest-e2e-provider

pull-secrets-store-csi-driver-e2e-gcp

pull-secrets-store-csi-driver-e2e-provider-k8s-1-32-8

pull-secrets-store-csi-driver-e2e-provider-k8s-1-33-4

pull-secrets-store-csi-driver-e2e-provider-k8s-1-34-0

pull-secrets-store-csi-driver-e2e-provider-k8s-1-35-0

pull-secrets-store-csi-driver-e2e-vault

pull-secrets-store-csi-driver-e2e-windows

pull-secrets-store-csi-driver-image-scan

pull-secrets-store-csi-driver-lint

pull-secrets-store-csi-driver-sanity

pull-secrets-store-csi-driver-unit

release-secrets-store-csi-driver-e2e-aws

release-secrets-store-csi-driver-e2e-azure

release-secrets-store-csi-driver-e2e-gcp

release-secrets-store-csi-driver-e2e-vault

Details

In response to this:

/retest release-secrets-store-csi-driver-e2e-aws

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

aramase

First pass.

aramase · 2026-03-24T22:12:19Z

Chart changes should only be made in the manifest_staging directory. Revert changes to charts/ and make them in manifest_staging/charts/secrets-store-csi-driver/ instead.

aramase · 2026-03-25T20:59:19Z

+	flag.Set("legacy_stderr_threshold_behavior", "false") //nolint:errcheck
+	flag.Set("stderrthreshold", "INFO")                   //nolint:errcheck


//nolint:errcheck is not used anywhere else in this repo. flag.Set returns an error if the flag doesn't exist — which would happen if the klog dep bump is reverted but this code stays. Handle the errors:

Suggested change

flag.Set("legacy_stderr_threshold_behavior", "false") //nolint:errcheck

flag.Set("stderrthreshold", "INFO") //nolint:errcheck

if err := flag.Set("legacy_stderr_threshold_behavior", "false"); err != nil {

klog.ErrorS(err, "failed to set legacy_stderr_threshold_behavior flag")

}

if err := flag.Set("stderrthreshold", "INFO"); err != nil {

klog.ErrorS(err, "failed to set stderrthreshold flag")

}

aramase · 2026-03-25T20:59:48Z

+# stderr when -logtostderr=true (the default). Requires klog v2.140.0+ with
+# -legacy_stderr_threshold_behavior=false which is set by the driver.
+# Valid values: INFO, WARNING, ERROR, FATAL (default: INFO).
+# stderrThreshold: ""


If adding stderrThreshold to values.yaml, the README.md table in manifest_staging/charts/secrets-store-csi-driver/ needs a corresponding entry (see the logVerbosity and logFormatJSON rows for the pattern).

aramase · 2026-03-25T21:00:32Z

/retitle fix: Honor stderrthreshold when logtostderr is enabled
/triage accepted

pierluigilenoci · 2026-03-26T21:28:40Z

Hi @aramase — I've addressed all your review feedback in the latest commit:

✅ Moved chart changes to manifest_staging/charts/
✅ Replaced //nolint:errcheck with proper error handling via klog.ErrorS
✅ Added stderrThreshold row to the chart's README values table

The only remaining CI failure is govulncheck, which appears to be an org-wide issue — the golang/govulncheck-action uses tag-based internal references that conflict with the new policy requiring all actions pinned to SHA. This affects all PRs in the repo, not just this one.

Thank you for the thorough review!

pierluigilenoci · 2026-03-27T20:49:13Z

Hi @aramase — I've addressed all three points from your review:

Retitled the PR as requested
Accepted the triage label
Pushed the code fixes

Could you take another look when you get a chance? Thank you!

pierluigilenoci · 2026-04-10T14:31:12Z

Hi @aramase — friendly ping. All review feedback has been addressed (chart path, error handling, README update) and CI is fully green except for the govulncheck action-pinning issue that affects all PRs repo-wide (not specific to this change).

Could you take another look when you have a moment? Thank you!

Update klog from v2.120.1 to v2.140.0 which includes the fix for kubernetes/klog#212 (stderrthreshold not honored when logtostderr is set). Opt into the new behavior by setting -legacy_stderr_threshold_behavior=false after klog.InitFlags(nil). Also set -stderrthreshold=INFO to preserve backward-compatible behavior (all logs still appear on stderr by default). Users can now override -stderrthreshold to WARNING or ERROR to reduce stderr noise, which was previously impossible with -logtostderr=true. The Helm chart exposes a new optional stderrThreshold value to configure this via the chart. Ref: kubernetes/klog#212, kubernetes/klog#432 Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>

The e2eprovider module has a replace directive pointing to the main module. Run go mod tidy to sync the klog dependency bump. Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>

- Move Helm chart changes from charts/ to manifest_staging/ - Handle flag.Set errors with klog.ErrorS instead of nolint:errcheck - Add stderrThreshold documentation to Helm chart README Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>

pierluigilenoci · 2026-04-21T10:07:26Z

Hi @aramase — friendly follow-up. All 3 review items have been addressed (charts moved to manifest_staging/, nolint replaced with proper error handling, stderrThreshold added to README). CI is fully green. Would you be able to take another look when you get a chance? Thanks!

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. kind/feature Categorizes issue or PR as related to a new feature. labels Mar 23, 2026

k8s-ci-robot requested review from aramase and ritazh March 23, 2026 18:21

k8s-ci-robot added needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 23, 2026

pierluigilenoci marked this pull request as ready for review March 23, 2026 18:25

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 23, 2026

pierluigilenoci force-pushed the fix/honor-stderrthreshold branch from 5d0743d to 2df9bff Compare March 23, 2026 22:49

enj added this to SIG Auth Mar 24, 2026

enj moved this to Subprojects - Needs Triage in SIG Auth Mar 24, 2026

pierluigilenoci mentioned this pull request Mar 24, 2026

feat: fix stderrthreshold not honored when logtostderr is set (#212) + two new flags kubernetes/klog#432

Merged

aramase requested changes Mar 25, 2026

View reviewed changes

github-project-automation Bot moved this from Subprojects - Needs Triage to Changes Requested in SIG Auth Mar 25, 2026

k8s-ci-robot changed the title ~~Honor stderrthreshold when logtostderr is enabled~~ fix: Honor stderrthreshold when logtostderr is enabled Mar 25, 2026

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Mar 25, 2026

pierluigilenoci added 3 commits April 10, 2026 18:44

Update klog in test/e2eprovider module to v2.140.0

1b5ed8e

The e2eprovider module has a replace directive pointing to the main module. Run go mod tidy to sync the klog dependency bump. Signed-off-by: Pierluigi Lenoci <pierluigi.lenoci@gmail.com>

pierluigilenoci force-pushed the fix/honor-stderrthreshold branch from 95f2027 to 4200ec4 Compare April 10, 2026 16:56

		flag.Set("legacy_stderr_threshold_behavior", "false") //nolint:errcheck
		flag.Set("stderrthreshold", "INFO") //nolint:errcheck

-	flag.Set("legacy_stderr_threshold_behavior", "false") //nolint:errcheck
-	flag.Set("stderrthreshold", "INFO")                   //nolint:errcheck
+      if err := flag.Set("legacy_stderr_threshold_behavior", "false"); err != nil {
+              klog.ErrorS(err, "failed to set legacy_stderr_threshold_behavior flag")
+      }
+      if err := flag.Set("stderrthreshold", "INFO"); err != nil {
+              klog.ErrorS(err, "failed to set stderrthreshold flag")
+      }

Conversation

pierluigilenoci commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Motivation

Changes

Backward Compatibility

Test plan

Uh oh!

k8s-ci-robot commented Mar 23, 2026

Uh oh!

k8s-ci-robot commented Mar 23, 2026

Summary

Motivation

Changes

Backward Compatibility

Test plan

Uh oh!

k8s-ci-robot commented Mar 23, 2026

Uh oh!

codecov-commenter commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

pierluigilenoci commented Mar 23, 2026

Uh oh!

pierluigilenoci commented Mar 23, 2026

Uh oh!

pierluigilenoci commented Mar 24, 2026

Uh oh!

pierluigilenoci commented Mar 24, 2026

Uh oh!

pierluigilenoci commented Mar 24, 2026

Uh oh!

pierluigilenoci commented Mar 24, 2026

Uh oh!

pierluigilenoci commented Mar 24, 2026

Uh oh!

k8s-ci-robot commented Mar 24, 2026

Uh oh!

pierluigilenoci commented Mar 25, 2026

Uh oh!

k8s-ci-robot commented Mar 25, 2026

Uh oh!

aramase left a comment

Choose a reason for hiding this comment

Uh oh!

aramase Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

aramase Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

aramase Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

aramase commented Mar 25, 2026

Uh oh!

pierluigilenoci commented Mar 26, 2026

Uh oh!

pierluigilenoci commented Mar 27, 2026

Uh oh!

pierluigilenoci commented Apr 10, 2026

Uh oh!

pierluigilenoci commented Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pierluigilenoci commented Mar 23, 2026 •

edited

Loading

codecov-commenter commented Mar 23, 2026 •

edited

Loading