Skip to content

Support pod remediation for unreachable local PVC #534

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dciabrin wants to merge 1 commit into
base: main
Choose a base branch
from
+1,404 −18
Open

Support pod remediation for unreachable local PVC #534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions api/bases/mariadb.openstack.org_galeras.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,36 @@ spec:
default: false
description: Does the galera cluster requires to be stopped globally
type: boolean
unavailablePods:
additionalProperties:
description: |-
UnavailablePodStatus describes why a specific pod is unavailable or excluded
from operator decisions. It is a generic, reusable structure that is not tied
to any single remediation mechanism.
properties:
node:
description: |-
Node is the Kubernetes node name relevant to the unavailability condition
(e.g. the node a stuck PVC is bound to). Empty when not applicable.
type: string
reason:
description: Reason describes why this pod is considered unavailable.
type: string
remediation:
default: "false"
description: |-
Remediation indicates that the operator has decided to actively remediate
this pod. At most one pod in UnavailablePods has Remediation set to "true"
at any time; others wait until the active remediation completes.
enum:
- "true"
- "false"
type: string
type: object
description: |-
UnavailablePods tracks pods that are unavailable or excluded from operator
decisions, keyed by pod name. Nil when all pods are healthy and managed.
type: object
required:
- bootstrapped
- stopRequired
Expand Down
2 changes: 1 addition & 1 deletion api/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ require (
github.com/go-logr/logr v1.4.4
github.com/onsi/ginkgo/v2 v2.33.1
github.com/onsi/gomega v1.44.0
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20261029162903-5860bf3fd6a5
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20261103144031-385d97c7bf10
github.com/openstack-k8s-operators/lib-common/modules/common v0.6.1-0.20261029132259-9d025389145b
k8s.io/api v0.33.13
k8s.io/apimachinery v0.33.13
Expand Down
4 changes: 2 additions & 2 deletions api/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,8 +94,8 @@ github.com/onsi/gomega v1.44.0 h1:eAiGl3Pw5jz5GQdDff0BcxYpAX1JxW8xD7mFUuwNfZQ=
github.com/onsi/gomega v1.44.0/go.mod h1:e/C2HwaZ1DhvjzXXuFhcR7hY7Sh9pl7MmoWKEjzwcdA=
github.com/openshift/api v0.0.0-20260710141509-36dec0bfafe4 h1:YVY7Mb0AdJBbNgC/Zz9R/ZwhixOmZNA8P+6ZPaDW/0M=
github.com/openshift/api v0.0.0-20260710141509-36dec0bfafe4/go.mod h1:SPLf21TYPipzCO67BURkCfK6dcIIxx0oNRVWaOyRcXM=
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20261029162903-5860bf3fd6a5 h1:FO9tXZxz+toJZliVW9wgc4ds3yTkOGHe1+Y6lD+AlQY=
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20261029162903-5860bf3fd6a5/go.mod h1:q/ndoGVc4EDClWuXx3/KsKhMuLBayxwIpfxrbxdD+9M=
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20261103144031-385d97c7bf10 h1:/PTU8pkJJLvh0JdPHymlr7xZRtNOh9PDjC81lOLNkXQ=
github.com/openstack-k8s-operators/infra-operator/apis v0.6.1-0.20261103144031-385d97c7bf10/go.mod h1:s8UKU0sWYxpNCUQd3pPic1JO0t3iM6bn3UFpfd1ucks=
github.com/openstack-k8s-operators/lib-common/modules/common v0.6.1-0.20261029132259-9d025389145b h1:3eq3qihAhvKwcjOsI0s0stZ8z/+KKMyWBx2Y/bdfnsg=
github.com/openstack-k8s-operators/lib-common/modules/common v0.6.1-0.20261029132259-9d025389145b/go.mod h1:eqG33ZqMiKk2LI4lsIav6J6wI5XM158/N24vRwyil3o=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
Expand Down
55 changes: 55 additions & 0 deletions api/v1beta1/galera_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,45 @@ type GaleraOverrideSpec struct {
Probes probes.OverrideSpec `json:"probes,omitempty"`
}

// PodUnavailabilityReason describes why a Galera pod is considered unavailable
// or excluded from operator decisions.
type PodUnavailabilityReason string

const (
// PodUnavailabilityReasonPVCStuck means the pod's PVC is bound to an
// unresponsive OCP worker node and cannot be recovered by normal scheduling.
PodUnavailabilityReasonPVCStuck PodUnavailabilityReason = "PVCStuckOnNode"
)

// PodRemediationStatus is the string-encoded boolean used for the Remediation
// field in UnavailablePodStatus. CRD validation rules forbid bare bool types
// in status schemas, so "true"/"false" strings are used instead.
type PodRemediationStatus = string

const (
PodRemediationActive PodRemediationStatus = "true"
PodRemediationNone PodRemediationStatus = "false"
)

// UnavailablePodStatus describes why a specific pod is unavailable or excluded
// from operator decisions. It is a generic, reusable structure that is not tied
// to any single remediation mechanism.
type UnavailablePodStatus struct {
// Reason describes why this pod is considered unavailable.
// +optional
Reason PodUnavailabilityReason `json:"reason,omitempty"`
// Node is the Kubernetes node name relevant to the unavailability condition
// (e.g. the node a stuck PVC is bound to). Empty when not applicable.
// +optional
Node string `json:"node,omitempty"`
// Remediation indicates that the operator has decided to actively remediate
// this pod. At most one pod in UnavailablePods has Remediation set to "true"
// at any time; others wait until the active remediation completes.
// +kubebuilder:validation:Enum="true";"false"
// +kubebuilder:default="false"
Remediation PodRemediationStatus `json:"remediation,omitempty"`
}

// GaleraAttributes holds startup information for a Galera host.
// This struct is read-only from the operator's perspective — all fields
// are pushed by the galera pods via the report_local_galera_state script.
Expand Down Expand Up @@ -178,6 +217,10 @@ type GaleraStatus struct {
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
// LastAppliedTopology - the last applied Topology
LastAppliedTopology *topologyv1.TopoRef `json:"lastAppliedTopology,omitempty"`
// UnavailablePods tracks pods that are unavailable or excluded from operator
// decisions, keyed by pod name. Nil when all pods are healthy and managed.
// +kubebuilder:validation:Optional
UnavailablePods map[string]UnavailablePodStatus `json:"unavailablePods,omitempty"`
}

// +kubebuilder:object:root=true
Expand Down Expand Up @@ -207,6 +250,18 @@ func init() {
SchemeBuilder.Register(&Galera{}, &GaleraList{})
}

// RemediationInProgress returns the name of the pod currently being remediated
// and true if any pod in UnavailablePods has Remediation set to true.
// Returns ("", false) when no remediation is active.
func (s *GaleraStatus) RemediationInProgress() (string, bool) {
for podName, up := range s.UnavailablePods {
if up.Remediation == PodRemediationActive {
return podName, true
}
}
return "", false
}

// IsReady - returns true if service is ready to serve requests
func (instance Galera) IsReady() bool {
return instance.Status.Conditions.IsTrue(condition.DeploymentReadyCondition)
Expand Down
22 changes: 22 additions & 0 deletions api/v1beta1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 30 additions & 0 deletions config/crd/bases/mariadb.openstack.org_galeras.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -484,6 +484,36 @@ spec:
default: false
description: Does the galera cluster requires to be stopped globally
type: boolean
unavailablePods:
additionalProperties:
description: |-
UnavailablePodStatus describes why a specific pod is unavailable or excluded
from operator decisions. It is a generic, reusable structure that is not tied
to any single remediation mechanism.
properties:
node:
description: |-
Node is the Kubernetes node name relevant to the unavailability condition
(e.g. the node a stuck PVC is bound to). Empty when not applicable.
type: string
reason:
description: Reason describes why this pod is considered unavailable.
type: string
remediation:
default: "false"
description: |-
Remediation indicates that the operator has decided to actively remediate
this pod. At most one pod in UnavailablePods has Remediation set to "true"
at any time; others wait until the active remediation completes.
enum:
- "true"
- "false"
type: string
type: object
description: |-
UnavailablePods tracks pods that are unavailable or excluded from operator
decisions, keyed by pod name. Nil when all pods are healthy and managed.
type: object
required:
- bootstrapped
- stopRequired
Expand Down
Loading
Loading