Skip to content

[shiftstack] Bound soft-transient oc exec retries during pod polling - #4192

Merged
Valkyrie00 merged 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-soft-transient-budget
Oct 6, 2026
Merged

Valkyrie00 merged 1 commit into
openstack-k8s-operators:mainfrom
tusharjadhav3302:fix/shiftstack-exec-soft-transient-budget

Conversation

@tusharjadhav3302

Copy link
Copy Markdown
Contributor

Summary

  • Add a short consecutive-failure budget for soft-transient oc exec errors (connection refused, i/o timeout) while polling long-running shiftstack tests in shiftstackclient.
  • Keeps the PR Retry transient oc exec errors during shiftstack pod polling #4150 behavior for hard-transient errors (SCC / InternalError / stream reset): those still retry for the full poll window.
  • Avoids failing on a brief management-API VIP blip (observed on serval70 ocp4.23 when cifmw-dnsmasq restarted mid-test) without spinning for ~8h when the API is truly unreachable.

Why

On shiftstack-rhel9-rhoso18.0-ocp4.23 (serval70, 2026-09-17) the outer poller failed immediately on:

The connection to the server api.ocp.openstack.lab:6443 was refused

The background osp_verification_4.23_nightly run continued and completed afterward. Re-adding connection refused to the unlimited hard-transient list was deliberately rejected in #4150 review; this change uses a bounded soft budget instead (default 12 × 5s ≈ 60s).

Test plan

  • ci-framework checks / molecule shiftstack
  • Confirm a single connection refused during polling no longer fails the testconfig
  • Confirm 12 consecutive soft-transient failures still fail promptly with the new message
  • Confirm SCCExecRestrictions / Internal error still retry for the full window

Made with Cursor

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Hi @tusharjadhav3302. Thanks for your PR.

I'm waiting for a openstack-k8s-operators member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@brjackma

Copy link
Copy Markdown
Contributor

/ok-to-test

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-soft-transient-budget branch from 84b7935 to 7139f5d Compare September 18, 2026 14:26
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/7752626782184cd6a2414e6e8400a27f

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 56s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 26s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 51s
❌ cifmw-crc-podified-edpm-baremetal-minor-update NODE_FAILURE Node(set) request 099-0000207282 failed in 0s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 20s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 59m 52s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 36m 26s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 26s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 31s
✔️ cifmw-molecule-shiftstack SUCCESS in 24m 19s

@brjackma

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/b3eb6f60dd1e4faa809e39fc556a34f0

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 07s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 25s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 32m 24s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 12s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 26s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 38m 02s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 25m 52s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 33s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 11s
✔️ cifmw-molecule-shiftstack SUCCESS in 24m 12s

brjackma
brjackma previously approved these changes Sep 21, 2026
@Valkyrie00

Copy link
Copy Markdown
Contributor

/agentic_review

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can add REVIEW.md to your repo root and Qodo follows it on every PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread roles/shiftstack/tasks/exec_command_in_pod.yml Outdated
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/c143093e3d5942cfa9ca003865de0c68

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 18m 25s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 30m 17s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 41s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 56m 28s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 24s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 52m 51s
❌ cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000209264 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 18s
✔️ cifmw-pod-pre-commit SUCCESS in 9m 06s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 42s

@brjackma

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/8f3852d6ac024178817879b2fca5964f

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 49m 18s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 25m 07s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 37s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 02m 14s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 25s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 4h 10m 10s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 37m 27s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 20s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 23s
✔️ cifmw-molecule-shiftstack SUCCESS in 25m 02s

@michburk michburk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One concern, otherwise looks much cleaner. Thanks!

Comment thread roles/shiftstack/tasks/exec_command_in_pod.yml Outdated
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-soft-transient-budget branch from 1fb42bd to a2bc174 Compare September 28, 2026 04:45
@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-soft-transient-budget branch from a2bc174 to 92c0704 Compare September 28, 2026 04:46
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/a4fdb3aed1c047f58de5409df63f2ab1

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 30s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 53s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 35m 27s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 54m 53s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 28s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 04m 09s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 29m 34s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 31s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 21s
✔️ cifmw-molecule-shiftstack SUCCESS in 18m 28s

@tusharjadhav3302
tusharjadhav3302 force-pushed the fix/shiftstack-exec-soft-transient-budget branch from 8fba0ac to 47ed24d Compare September 29, 2026 07:05
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/6384819739574b8d84424ec0cfa4a50a

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 39m 10s
❌ podified-multinode-edpm-deployment-crc FAILURE in 21m 05s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 26m 42s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 55s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 55m 12s
✔️ cifmw-pod-zuul-files SUCCESS in 5m 05s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 12m 20s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 28m 23s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 12s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 13s
✔️ cifmw-molecule-shiftstack SUCCESS in 18m 12s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/847a3b83ba004e32a1ed52fc87349df6

❌ openstack-k8s-operators-content-provider FAILURE in 5m 13s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ podified-multinode-edpm-deployment-crc-centos-10 SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 14s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 5m 20s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 30s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 29s
✔️ cifmw-molecule-shiftstack SUCCESS in 16m 15s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/72c5b0ae9413403f850253137da5a3d9

❌ openstack-k8s-operators-content-provider FAILURE in 5m 18s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ podified-multinode-edpm-deployment-crc-centos-10 SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 12s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 5m 25s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 10s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 20s
✔️ cifmw-molecule-shiftstack SUCCESS in 17m 07s

@tusharjadhav3302

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/9c372bf3b2f74dbfbd5dfa267ca92166

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 22m 50s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 20s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 20m 59s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 33m 40s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 02m 26s
✔️ cifmw-pod-zuul-files SUCCESS in 7m 41s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 12m 43s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 32m 38s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 12m 19s
✔️ cifmw-pod-pre-commit SUCCESS in 11m 42s
✔️ cifmw-molecule-shiftstack SUCCESS in 14m 33s

@michburk michburk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@brjackma

brjackma commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

/approve

@openshift-ci

openshift-ci Bot commented Oct 5, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brjackma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved label Oct 5, 2026
@Valkyrie00

Copy link
Copy Markdown
Contributor

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/97f0ef1799024c3d9c1336822b3505e5

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 14m 04s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 14s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 24m 59s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 39s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 15m 59s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 25s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 29m 55s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 32m 17s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 28s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 29s
✔️ cifmw-molecule-shiftstack SUCCESS in 15m 33s

@Valkyrie00

Copy link
Copy Markdown
Contributor

/override rdoproject.org/github-check

@openshift-ci

openshift-ci Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

@Valkyrie00: Overrode contexts on behalf of Valkyrie00: rdoproject.org/github-check

Details

In response to this:

/override rdoproject.org/github-check

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Ride through brief API VIP blips (connection refused / i/o timeout)
with a consecutive failure budget instead of failing on the first poll
or spinning for the full 8h window when the API is truly unreachable.
Hard-transient patterns from openstack-k8s-operators#4150 keep the full poll timeout. Move the
poller into roles/shiftstack/files/shiftstack_cmd_poll.sh and invoke it
with ansible.builtin.script so it is copied from the controller. Match
kubectl's "connection ... was refused" via connection.*refused.

Signed-off-by: tusharjadhav3302 <tjadhav@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@Valkyrie00
Valkyrie00 force-pushed the fix/shiftstack-exec-soft-transient-budget branch from 47ed24d to b89fdb3 Compare October 6, 2026 07:56
@openshift-ci openshift-ci Bot removed the lgtm label Oct 6, 2026
@openshift-ci

openshift-ci Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

@Valkyrie00 Valkyrie00 added the lgtm label Oct 6, 2026
@Valkyrie00
Valkyrie00 merged commit 32250c1 into openstack-k8s-operators:main Oct 6, 2026
7 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants