Skip to content

pcp_metrics: tolerate corrupted PCP archives during gather - #4296

Open
lmiccini wants to merge 1 commit into
openstack-k8s-operators:mainfrom
lmiccini:pcp-metrics-tolerate-corrupted-archive
Open

lmiccini wants to merge 1 commit into
openstack-k8s-operators:mainfrom
lmiccini:pcp-metrics-tolerate-corrupted-archive

Conversation

@lmiccini

Copy link
Copy Markdown
Contributor

A corrupted record in any single host's PCP archive caused pmrep to exit non-zero, which failed the "Collect the metrics from host" task and, since the gather play runs across all hosts at once, aborted the whole pcp-metrics hook even when every other host collected cleanly.

pmrep still writes the valid rows to stdout before aborting on a corrupted record, so treat that specific failure as non-fatal: keep the partial output, emit a warning naming the affected host, and only skip the save when there is no output at all. Other non-zero exits (missing pmrep, bad metricspec, etc.) still fail the task as before.

@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@openshift-ci

openshift-ci Bot commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign evallesp for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

A corrupted record in any single host's PCP archive caused pmrep to exit
non-zero, which failed the "Collect the metrics from host" task and, since
the gather play runs across all hosts at once, aborted the whole
pcp-metrics hook even when every other host collected cleanly.

pmrep still writes the valid rows to stdout before aborting on a corrupted
record, so treat that specific failure as non-fatal: keep the partial
output, emit a warning naming the affected host, and only skip the save
when there is no output at all. Other non-zero exits (missing pmrep, bad
metricspec, etc.) still fail the task as before.

Signed-off-by: Luca Miccini <lmiccini@redhat.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@lmiccini
lmiccini force-pushed the pcp-metrics-tolerate-corrupted-archive branch from 7f69c96 to b544201 Compare October 26, 2026 13:10
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/32e6d20485a44610afc51cdea40316ab

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 53m 07s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 19m 54s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 23m 46s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 40s
❌ cifmw-crc-podified-edpm-baremetal-minor-update FAILURE in 2h 03m 21s (non-voting)
✔️ periodic-podified-edpm-baremetal-minor-update-bootc-master-ocp-crc SUCCESS in 1h 53m 06s (non-voting)
✔️ cifmw-pod-zuul-files SUCCESS in 4m 22s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 10m 06s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc (non-voting)
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 29s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 03s
✔️ cifmw-molecule-pcp_metrics SUCCESS in 2m 25s

Warning:
File .kube/http-cache/e7f0e9675b534f5990130519704a787f found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-zuul-files. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/8ee34c336110486332966f0ed55eea72 found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-zuul-files. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/0d0c215b9e6fccbbac173d38078fd700 found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-zuul-files. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/cache/discovery/38.102.83.49_9023/servergroups.json found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-zuul-files. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/cache/discovery/38.102.83.49_9023/v1/serverresources.json found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-zuul-files. Add to preserve-home-paths if safe, or otherwise remove.
File .ssh/config found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/pod-jobs.yml for job cifmw-pod-zuul-files. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/e7f0e9675b534f5990130519704a787f found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-pre-commit. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/8ee34c336110486332966f0ed55eea72 found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-pre-commit. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/0d0c215b9e6fccbbac173d38078fd700 found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-pre-commit. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/cache/discovery/38.102.83.49_9023/servergroups.json found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-pre-commit. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/cache/discovery/38.102.83.49_9023/v1/serverresources.json found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-pre-commit. Add to preserve-home-paths if safe, or otherwise remove.
File .ssh/config found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/pod-jobs.yml for job cifmw-pod-pre-commit. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/bdd7b2cb6584e62d442216995e51db5f found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-ansible-test. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/3418855402ab8666a195458ea6d5d4a2 found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-ansible-test. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/http-cache/a0a4de7d63733803eeececaa48e4969f found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-ansible-test. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/cache/discovery/38.102.83.139_9023/servergroups.json found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-ansible-test. Add to preserve-home-paths if safe, or otherwise remove.
File .kube/cache/discovery/38.102.83.139_9023/v1/serverresources.json found in build home directory during playbook review.rdoproject.org/rdo-jobs/playbooks/base/pre.yaml for job cifmw-pod-ansible-test. Add to preserve-home-paths if safe, or otherwise remove.
File .ssh/config found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/pod-jobs.yml for job cifmw-pod-ansible-test. Add to preserve-home-paths if safe, or otherwise remove.
File .ssh/config found in build home directory during playbook opendev.org/zuul/zuul-jobs/playbooks/unittests/pre.yaml for job cifmw-molecule-pcp_metrics. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/6ed07a52647d4999a353c3be85a616f7 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job periodic-podified-edpm-baremetal-minor-update-bootc-master-ocp-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/a7561e1789f242da9a09c878b8d44acd found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job periodic-podified-edpm-baremetal-minor-update-bootc-master-ocp-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/e93feccd008345008ae65699c551bc10 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job periodic-podified-edpm-baremetal-minor-update-bootc-master-ocp-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/2cb3522aecd9451dbdef864e7add30ec found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job periodic-podified-edpm-baremetal-minor-update-bootc-master-ocp-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/6ed07a52647d4999a353c3be85a616f7 found in build home directory during playbook review.rdoproject.org/config/playbooks/ci-framework-rdo-base/dlrn/dlrn-report.yaml for job periodic-podified-edpm-baremetal-minor-update-bootc-master-ocp-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/bac5c2e8f60d47718b8b3aa8dfb8c102 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job podified-multinode-edpm-deployment-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/cfa09a2ef9764103ada22f45bf6e9bb7 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job podified-multinode-edpm-deployment-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/bac5c2e8f60d47718b8b3aa8dfb8c102 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml for job podified-multinode-edpm-deployment-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/cfa09a2ef9764103ada22f45bf6e9bb7 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml for job podified-multinode-edpm-deployment-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/bac5c2e8f60d47718b8b3aa8dfb8c102 found in build home directory during playbook review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml for job podified-multinode-edpm-deployment-crc. Add to preserve-home-paths if safe, or otherwise remove.
File .ssh/known_hosts.old found in build home directory during playbook review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml for job podified-multinode-edpm-deployment-crc. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/d25d39a109994033afa90eb2bf8eff7e found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job podified-multinode-edpm-deployment-crc-centos-10. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/877421855b994295b189db9b2dc8a2e6 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job podified-multinode-edpm-deployment-crc-centos-10. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/d25d39a109994033afa90eb2bf8eff7e found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml for job podified-multinode-edpm-deployment-crc-centos-10. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/877421855b994295b189db9b2dc8a2e6 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/multinode-autohold.yml for job podified-multinode-edpm-deployment-crc-centos-10. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/d25d39a109994033afa90eb2bf8eff7e found in build home directory during playbook review.rdoproject.org/config/playbooks/crc/print-machineconfigpool.yaml for job podified-multinode-edpm-deployment-crc-centos-10. Add to preserve-home-paths if safe, or otherwise remove.
File .ssh/known_hosts.old found in build home directory during playbook review.rdoproject.org/config/playbooks/base-minimal/post-logs.yaml for job podified-multinode-edpm-deployment-crc-centos-10. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/cd7eeb2589bc4487a9e62056ebc99de3 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/047d214b2a2a47b2bb39cf4d05aa3ec9 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/39a935f8e9f04f7da42fa91c15222cd7 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/3d050e73a0c341e3830044e025ab0c02 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/ed7284722b814bfe8a0cc4c7a007bdf7 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/123df771556b431b8d64481bb282705a found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal-minor-update. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/12f8d4fc386f476fb3dc58777164fc96 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal-minor-update. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/cde29690fa88498096faa3a3edefabe3 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal-minor-update. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/225af8b840444633beaf3d3da266296e found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal-minor-update. Add to preserve-home-paths if safe, or otherwise remove.
File ansible/tmp/4896aa406bc94e91b5ac717fbf408014 found in build home directory during playbook github.com/openstack-k8s-operators/ci-framework/ci/playbooks/collect-logs.yml for job cifmw-crc-podified-edpm-baremetal-minor-update. Add to preserve-home-paths if safe, or otherwise remove.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant