Skip to content

Allow set extra labels/annotations #2791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andriishestakov wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Allow set extra labels/annotations #2791

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions api/v1/slurmcluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,16 @@ type SlurmClusterSpec struct {
// +kubebuilder:validation:Required
SlurmNodes SlurmNodes `json:"slurmNodes"`

// ExtraLabels are custom K8s labels added to every Pod and the spool PVC of this cluster.

@Uburro Uburro Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spool PVC doc comment doesn't match the behavior. Both CRD fields say "added to every Pod and the spool PVC of this cluster", but nothing in the operator labels a PVC: common/volume.go:46-60 (the VolumeClaimTemplates, including the worker spool) is untouched. The only PVC labeling is chart-side in pvc.yaml, which iterates .Values.volumeSources with createPVC — jail and friends, not spool, and driven by chart values rather than the CR field. If the use case is billing attribution, unlabeled operator-created PVCs are probably the most valuable remaining gap; either way the comment should be corrected.

@Uburro Uburro Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ActiveCheck can use spool PVC. It needs to be rewritten.

//
// +kubebuilder:validation:Optional
ExtraLabels map[string]string `json:"extraLabels,omitempty"`

// ExtraAnnotations are custom K8s annotations added to every Pod and the spool PVC of this cluster.
//
// +kubebuilder:validation:Optional
ExtraAnnotations map[string]string `json:"extraAnnotations,omitempty"`

// PartitionConfiguration define partition configuration of slurm worker nodes
// https://slurm.schedmd.com/slurm.conf.html#SECTION_PARTITION-CONFIGURATION
// +kubebuilder:validation:Optional
Expand Down
14 changes: 14 additions & 0 deletions api/v1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions api/v1alpha1/nodeset_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -299,6 +299,11 @@ type NodeSetSpec struct {
// +kubebuilder:validation:Optional
WorkerAnnotations map[string]string `json:"workerAnnotations,omitempty"`

// WorkerLabels represent K8S labels that should be added to the worker pods.
//
// +kubebuilder:validation:Optional
WorkerLabels map[string]string `json:"workerLabels,omitempty"`

// CustomInitContainers represent additional init containers which will be added to worker pods.
//
// +kubebuilder:validation:Optional
Expand Down
7 changes: 7 additions & 0 deletions api/v1alpha1/zz_generated.deepcopy.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions config/crd/bases/slurm.nebius.ai_nodesets.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12134,6 +12134,12 @@ spec:
format: int32
minimum: 0
type: integer
workerLabels:
additionalProperties:
type: string
description: WorkerLabels represent K8S labels that should be added
to the worker pods.
type: object
required:
- munge
- slurmd
Expand Down
12 changes: 12 additions & 0 deletions config/crd/bases/slurm.nebius.ai_slurmclusters.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,18 @@ spec:
Soperator does not guarantee the validity of the raw configuration.
Raw config is merged with existing SlurmConfig values.
type: string
extraAnnotations:
additionalProperties:
type: string
description: ExtraAnnotations are custom K8s annotations added to
every Pod and the spool PVC of this cluster.
type: object
extraLabels:
additionalProperties:
type: string
description: ExtraLabels are custom K8s labels added to every Pod
and the spool PVC of this cluster.
type: object
healthCheckConfig:
description: HealthCheckConfig defines Slurm health check configuration.
properties:
Expand Down
5 changes: 5 additions & 0 deletions helm/nodesets/templates/nodeset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,11 @@ spec:
{{- toYaml . | nindent 4 }}
{{- end }}

{{- with (.workerLabels | default dict) }}
workerLabels:
{{- toYaml . | nindent 4 }}
{{- end }}

{{- include "nodesets.customInitContainers" (dict "root" $ "customInitContainers" .customInitContainers "nodeExporter" .nodeExporter) | nindent 2 }}
{{- end }}
{{- end }}
3 changes: 3 additions & 0 deletions helm/nodesets/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,9 @@ nodesets:
workerAnnotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
# Additional labels to be added to the worker pods
# Optional, defaults to empty dict
workerLabels: {}
# A list of custom init containers for worker pods
# Each item must be a corev1.Container spec
# Optional, defaults to empty list
Expand Down
8 changes: 8 additions & 0 deletions helm/slurm-cluster/templates/pvc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,14 @@ kind: PersistentVolumeClaim
metadata:
namespace: {{ $.Release.Namespace }}
name: {{ required "Claim name must be provided." $volume.persistentVolumeClaim.claimName | quote }}
{{- with ($.Values.extraLabels | default dict) }}
labels:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with ($.Values.extraAnnotations | default dict) }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
accessModes:
- ReadWriteMany
Expand Down
8 changes: 8 additions & 0 deletions helm/slurm-cluster/templates/slurm-cluster-cr.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ spec:
maintenance: {{ default "none" .Values.maintenance | quote }}
crVersion: {{ .Chart.Version }}
useDefaultAppArmorProfile: {{ .Values.useDefaultAppArmorProfile }}
{{- with (.Values.extraLabels | default dict) }}
extraLabels:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- with (.Values.extraAnnotations | default dict) }}
extraAnnotations:
{{- toYaml . | nindent 4 }}
{{- end }}
partitionConfiguration:
configType: {{ (default "default" .Values.partitionConfiguration.configType) }}
{{- if and .Values.partitionConfiguration.rawConfig (eq .Values.partitionConfiguration.configType "custom") }}
Expand Down
3 changes: 3 additions & 0 deletions helm/slurm-cluster/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
clusterName: "slurm1"
# Additional annotations for the cluster
annotations: {}
# Custom labels/annotations applied to every Pod and the spool PVC
extraLabels: {}
extraAnnotations: {}
# Add appArmor profile to the cluster
useDefaultAppArmorProfile: false
# Maintenance defines the maintenance window for the cluster.
Expand Down
18 changes: 18 additions & 0 deletions helm/soperator-crds/templates/slurmcluster-crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27312,6 +27312,12 @@ spec:
format: int32
minimum: 0
type: integer
workerLabels:
additionalProperties:
type: string
description: WorkerLabels represent K8S labels that should be added
to the worker pods.
type: object
required:
- munge
- slurmd
Expand Down Expand Up @@ -27502,6 +27508,18 @@ spec:
Soperator does not guarantee the validity of the raw configuration.
Raw config is merged with existing SlurmConfig values.
type: string
extraAnnotations:
additionalProperties:
type: string
description: ExtraAnnotations are custom K8s annotations added to
every Pod and the spool PVC of this cluster.
type: object
extraLabels:
additionalProperties:
type: string
description: ExtraLabels are custom K8s labels added to every Pod
and the spool PVC of this cluster.
type: object
healthCheckConfig:
description: HealthCheckConfig defines Slurm health check configuration.
properties:
Expand Down
18 changes: 18 additions & 0 deletions helm/soperator/crds/slurmcluster-crd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27312,6 +27312,12 @@ spec:
format: int32
minimum: 0
type: integer
workerLabels:
additionalProperties:
type: string
description: WorkerLabels represent K8S labels that should be added
to the worker pods.
type: object
required:
- munge
- slurmd
Expand Down Expand Up @@ -27502,6 +27508,18 @@ spec:
Soperator does not guarantee the validity of the raw configuration.
Raw config is merged with existing SlurmConfig values.
type: string
extraAnnotations:
additionalProperties:
type: string
description: ExtraAnnotations are custom K8s annotations added to
every Pod and the spool PVC of this cluster.
type: object
extraLabels:
additionalProperties:
type: string
description: ExtraLabels are custom K8s labels added to every Pod
and the spool PVC of this cluster.
type: object
healthCheckConfig:
description: HealthCheckConfig defines Slurm health check configuration.
properties:
Expand Down
2 changes: 2 additions & 0 deletions internal/controller/nodesetcontroller/reconcile.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,8 @@ func (r *NodeSetReconciler) reconcile(ctx context.Context, nodeSet *slurmv1alpha
cluster.Name,
cluster.Spec.Maintenance,
cluster.Spec.UseDefaultAppArmorProfile,
cluster.Spec.ExtraLabels,
cluster.Spec.ExtraAnnotations,
)

nodeSets, err := resourcegetter.ListNodeSetsByClusterRef(ctx, r.Client, client.ObjectKeyFromObject(cluster))
Expand Down
4 changes: 4 additions & 0 deletions internal/render/accounting/mariadb.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,10 @@ func RenderMariaDb(
SecurityContext: mariaDb.SecurityContext,
},
PodTemplate: mariadbv1alpha1.PodTemplate{
PodMetadata: &mariadbv1alpha1.Metadata{
Labels: accounting.Labels,
Annotations: accounting.Annotations,
},
NodeSelector: nodeFilter.NodeSelector,
Affinity: affinityConfig,
Tolerations: nodeFilter.Tolerations,
Expand Down
31 changes: 31 additions & 0 deletions internal/render/accounting/mariadb_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@ package accounting
import (
"testing"

mariadbv1alpha1 "github.com/mariadb-operator/mariadb-operator/v25/api/v1alpha1"
"github.com/stretchr/testify/assert"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/api/resource"
"k8s.io/utils/ptr"

slurmv1 "nebius.ai/slurm-operator/api/v1"
"nebius.ai/slurm-operator/internal/consts"
Expand Down Expand Up @@ -116,3 +119,31 @@ func Test_RenderMariaDb(t *testing.T) {
assert.Equal(t, consts.MariaDbSecretRootName, mariaDb.Spec.RootPasswordSecretKeyRef.SecretKeySelector.Name)
assert.Equal(t, consts.MariaDbPasswordKey, mariaDb.Spec.RootPasswordSecretKeyRef.SecretKeySelector.Key)
}

func Test_RenderMariaDb_CustomLabelsAndAnnotations(t *testing.T) {
acc := &values.SlurmAccounting{
SlurmNode: slurmv1.SlurmNode{
K8sNodeFilterName: "test-filter",
},
Labels: map[string]string{"gcore.com/project-id": "123"},
Annotations: map[string]string{"gcore.com/note": "abc"},
MariaDb: slurmv1.MariaDbOperator{
Enabled: true,
NodeContainer: slurmv1.NodeContainer{
Image: "mariadb:10.5",
},
Storage: mariadbv1alpha1.Storage{
Size: ptr.To(resource.MustParse("1Gi")),
},
},
}

nodeFilters := []slurmv1.K8sNodeFilter{{Name: "test-filter"}}

result, err := RenderMariaDb("test-namespace", "test-cluster", acc, nodeFilters)
assert.NoError(t, err)

assert.NotNil(t, result.Spec.PodTemplate.PodMetadata)
assert.Equal(t, "123", result.Spec.PodTemplate.PodMetadata.Labels["gcore.com/project-id"])
assert.Equal(t, "abc", result.Spec.PodTemplate.PodMetadata.Annotations["gcore.com/note"])
}
11 changes: 9 additions & 2 deletions internal/render/accounting/pod.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package accounting

import (
"maps"
"slices"

corev1 "k8s.io/api/core/v1"
Expand Down Expand Up @@ -65,10 +66,16 @@ func BasePodTemplateSpec(
common.RenderContainerMunge(&accounting.ContainerMunge),
)

labels := maps.Clone(matchLabels)
maps.Copy(labels, accounting.Labels)

annotations := common.RenderDefaultContainerAnnotation(consts.ContainerNameAccounting)
maps.Copy(annotations, accounting.Annotations)

res := &corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: matchLabels,
Annotations: common.RenderDefaultContainerAnnotation(consts.ContainerNameAccounting),
Labels: labels,
Annotations: annotations,
},
Spec: corev1.PodSpec{
HostUsers: accounting.HostUsers,
Expand Down
15 changes: 15 additions & 0 deletions internal/render/accounting/pod_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,3 +115,18 @@ func Test_BasePodTemplateSpec_PriorityClass(t *testing.T) {
})
}
}

func Test_BasePodTemplateSpec_CustomLabelsAndAnnotations(t *testing.T) {
testAcc := *acc
testAcc.Labels = map[string]string{"gcore.com/project-id": "123"}
testAcc.Annotations = map[string]string{"gcore.com/note": "abc"}

result, err := accounting.BasePodTemplateSpec(
defaultNameCluster, &testAcc, defaultNodeFilter, defaultVolumeSources, matchLabels,
)
assert.NoError(t, err)

assert.Equal(t, "123", result.Labels["gcore.com/project-id"])
assert.Equal(t, "value", result.Labels["key"]) // matchLabels preserved
assert.Equal(t, "abc", result.Annotations["gcore.com/note"])
}
13 changes: 9 additions & 4 deletions internal/render/controller/statefulset.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ package controller

import (
"fmt"
"maps"
"slices"

appspub "github.com/openkruise/kruise-api/apps/pub"
Expand Down Expand Up @@ -34,6 +35,12 @@ func RenderStatefulSet(

labels[consts.LabelControllerType] = consts.LabelControllerTypeMain
matchLabels[consts.LabelControllerType] = consts.LabelControllerTypeMain
maps.Copy(labels, controller.Labels)

annotations := map[string]string{
consts.AnnotationDefaultContainerName: consts.ContainerNameSlurmctld,
}
maps.Copy(annotations, controller.Annotations)

nodeFilter := sliceutils.MustGetBy(
nodeFilters,
Expand Down Expand Up @@ -100,10 +107,8 @@ func RenderStatefulSet(
},
Template: corev1.PodTemplateSpec{
ObjectMeta: metav1.ObjectMeta{
Labels: labels,
Annotations: map[string]string{
consts.AnnotationDefaultContainerName: consts.ContainerNameSlurmctld,
},
Labels: labels,
Annotations: annotations,
},
Spec: corev1.PodSpec{
ReadinessGates: []corev1.PodReadinessGate{
Expand Down
Loading
Loading