Skip to content

[update] Restore Ceph after EDPM reboot - #4191

Open
srac0 wants to merge 2 commits into
openstack-k8s-operators:mainfrom
srac0:update/remove-ceph-stray-daemon
Open

srac0 wants to merge 2 commits into
openstack-k8s-operators:mainfrom
srac0:update/remove-ceph-stray-daemon

Conversation

@srac0

@srac0 srac0 commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@openshift-ci

openshift-ci Bot commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ec14b75026ff4a23b210b1507d005b3b

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 40m 31s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 06s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 37m 59s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 05m 14s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 09s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 1h 19m 03s
❌ cifmw-crc-podified-edpm-baremetal-bootc NODE_FAILURE Node(set) request 099-0000207011 failed in 0s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 01s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 39s
✔️ cifmw-molecule-update SUCCESS in 5m 55s

@srac0
srac0 force-pushed the update/remove-ceph-stray-daemon branch from 11ad381 to 0f63525 Compare September 18, 2026 17:03
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/5e6a656b788e4c30bde86cb159974720

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 24m 31s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 22m 39s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 28m 19s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 02m 28s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 25s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 07m 12s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 28m 36s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 30s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 16s
✔️ cifmw-molecule-update SUCCESS in 5m 54s

@srac0
srac0 force-pushed the update/remove-ceph-stray-daemon branch from 0f63525 to 1e26e8b Compare September 20, 2026 11:16
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/ecae8aacc91a4eeba2e66a8db79c4c9c

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 15s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 25m 40s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 29m 45s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 55m 10s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 33s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 05m 00s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 29m 49s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 21s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 58s
✔️ cifmw-molecule-update SUCCESS in 5m 37s

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/e33c3233867e4006a2fb880b44792f8e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 16m 30s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 27s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 38m 32s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 03m 43s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 47s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 18m 23s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 43s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 27s
✔️ cifmw-molecule-update SUCCESS in 5m 44s

@srac0
srac0 force-pushed the update/remove-ceph-stray-daemon branch from 358be02 to 8855f0e Compare September 29, 2026 07:51
@srac0
srac0 marked this pull request as ready for review September 29, 2026 07:51
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@srac0 srac0 changed the title [update] Remove stray Ceph mgr daemon after hypervisor reboot [update] Restore Ceph after EDPM reboot Sep 29, 2026
In adoption HCI scenarios where Ceph MON IPs are assigned to compute nodes, these IPs
are not persisted in NetworkManager profiles. After a compute node reboot, the
IPs must be manually re-applied to the storage VLAN interface to restore Ceph
connectivity. This change adds an optional feature to automate that step.

Signed-off-by: Samuel Rac <srac@redhat.com>
@srac0
srac0 force-pushed the update/remove-ceph-stray-daemon branch from 8855f0e to 7f6b00e Compare September 29, 2026 08:06
A leftover ceph controller manager can become active after adoption and
report no capacity. Detect that exact daemon and fail it so a another
standby controller takes over.

Signed-off-by: Samuel Rac <srac@redhat.com>
@srac0
srac0 force-pushed the update/remove-ceph-stray-daemon branch from 7f6b00e to 1ef09af Compare September 29, 2026 08:22
@openshift-ci

openshift-ci Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign dasm for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/144fc463a5d44938a980e1e05120779d

✔️ openstack-k8s-operators-content-provider SUCCESS in 5h 15m 50s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 24m 50s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 27m 41s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 33m 33s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 04m 17s
❌ cifmw-pod-zuul-files FAILURE in 2m 47s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 5m 38s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
❌ cifmw-pod-ansible-test FAILURE in 2m 46s
❌ cifmw-pod-pre-commit FAILURE in 2m 59s
❌ cifmw-molecule-update FAILURE in 6m 31s

@michburk

Copy link
Copy Markdown
Contributor

recheck

@michburk

Copy link
Copy Markdown
Contributor

/retest-required

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/fce1bd53efd3439ea59b4b2ee4abe8e0

❌ openstack-k8s-operators-content-provider FAILURE in 5m 42s
⚠️ podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ podified-multinode-edpm-deployment-crc-centos-10 SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
⚠️ cifmw-crc-podified-edpm-baremetal-minor-update SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider
✔️ cifmw-pod-zuul-files SUCCESS in 4m 33s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 5m 26s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 32s
✔️ cifmw-pod-pre-commit SUCCESS in 7m 58s
✔️ cifmw-molecule-update SUCCESS in 5m 37s

@srac0

srac0 commented Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@ciecierski
ciecierski requested review from fmount and fultonj October 1, 2026 09:31
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/bce5d37ff4cb4c02bb3000f3a3c02b0a

✔️ openstack-k8s-operators-content-provider SUCCESS in 16h 38m 09s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 21m 54s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 23m 08s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 34m 16s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 08m 41s
✔️ cifmw-pod-zuul-files SUCCESS in 9m 36s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 31m 42s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 22m 54s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 10m 14s
✔️ cifmw-pod-pre-commit SUCCESS in 12m 36s
✔️ cifmw-molecule-update SUCCESS in 10m 28s

- name: Re-apply Ceph MON IPs after compute reboot
when: cifmw_update_reapply_ceph_mon_ips | default(false)
block:
- name: Extract Ceph storage-net IPs from os-net-config

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if there's a special reason why we use shell + ssh to grep the ip address instead of a delegate_to and loop through the target nodes that are supposed to be part of the ceph cluster.

@ciecierski Maybe you're using an inventory that has localhost only?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an inventory, but the update role does not have a mapping from each compute to its migrated MON IP. That address is recorded in the compute’s updated /etc/os-net-config/config.yaml, which is why the task reads the file on the target host.

ssh -i {{ cifmw_update_ansible_ssh_private_key_file }}
-o BatchMode=yes -o ConnectTimeout=10 -o StrictHostKeyChecking=no
{{ cifmw_update_hypervisor_short_name }}
"sudo ip addr add {{ item }} dev vlan{{ cifmw_update_ceph_storage_vlan_id }}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wondering if this config should be persisted in os-net-config [1] and use it instead of ip a a but I definitely miss context here.

[1] https://github.com/openstack-k8s-operators/data-plane-adoption/blob/main/tests/roles/ceph_migrate/tasks/network.yaml#L26

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried applying config.yaml with os-net-config on an adopted compute. The call failed because it selected nmstate, while NetworkManager marks br-ex as strictly unmanaged. During Ceph MON migration (in data-plane-adoption), manual_migration is set to true so it uses ip addr add instead of os-net-config. For this rolling CI reboot, I propose restoring only the MON address with ip addr add. Persistent profile configuration can be addressed in the adoption networking flow.

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/0bf67331eee645fa9ec8f25be9e6de42

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 19m 58s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 17m 25s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 34m 27s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 31m 47s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 1h 59m 41s
✔️ cifmw-pod-zuul-files SUCCESS in 3m 56s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 2h 19m 45s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 29m 40s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 7m 46s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 05s
✔️ cifmw-molecule-update SUCCESS in 5m 47s

@srac0
srac0 force-pushed the update/remove-ceph-stray-daemon branch from f97e1ea to 1ef09af Compare October 9, 2026 08:04
@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/611c988beb3b4c68be10144b39e37a7e

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 41m 25s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 20m 53s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 18m 03s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 28m 21s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 01m 52s
✔️ cifmw-pod-zuul-files SUCCESS in 7m 59s
❌ openstack-k8s-operators-content-provider-bootc FAILURE in 17m 04s
⚠️ cifmw-crc-podified-edpm-baremetal-bootc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider-bootc
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 12m 37s
✔️ cifmw-pod-pre-commit SUCCESS in 10m 51s
✔️ cifmw-molecule-update SUCCESS in 11m 23s

@srac0

srac0 commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

recheck

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build succeeded (check pipeline).
https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/4de13f0322fb4ead80640baa58a04cef

✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 33m 11s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 23m 40s
✔️ podified-multinode-edpm-deployment-crc-centos-10 SUCCESS in 1h 18m 36s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 25m 44s
✔️ cifmw-crc-podified-edpm-baremetal-minor-update SUCCESS in 2h 03m 13s
✔️ cifmw-pod-zuul-files SUCCESS in 4m 34s
✔️ openstack-k8s-operators-content-provider-bootc SUCCESS in 3h 28m 17s
✔️ cifmw-crc-podified-edpm-baremetal-bootc SUCCESS in 1h 33m 43s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 08s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 41s
✔️ cifmw-molecule-update SUCCESS in 5m 51s

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants