[FEATURES] Allow CacheRuntime DataLoad (DataOperationSpec) to mount ConfigMaps/Secrets and reuse config generation

[cheyang]

What feature you'd like to add:

Enhance the CacheRuntime DataOperationSpec (used by dataOperationSpecs, e.g. DataLoad) so that a data operation Pod can declare dependencies the same way topology components do — at minimum the ability to mount ConfigMaps and Secrets, and ideally to reuse the same config-generation mechanism (init container + extraResources ConfigMap template) that master/worker/client already use.

Today DataOperationSpec only carries name, image, command, and args:

fluid/api/v1alpha1/cacheruntimeclass_types.go

Lines 188 to 202 in bf04633

	type DataOperationSpec struct {
	// Name is the data operation name like DataLoad, DataBackup, DataMigrate etc.
	// +kubebuilder:validation:Enum=DataLoad;DataBackup;DataMigrate;DataProcess
	Name string `json:"name"`
	// Image the image for data operation, if not existed, use the runtime/runtimeclass defined worker image.
	// +optional
	Image string `json:"image,omitempty"`
	// Command for data operation Pod container
	Command []string `json:"command,omitempty"`
	// Args for data operation Pod container
	Args []string `json:"args,omitempty"`
	}

type DataOperationSpec struct {
	// +kubebuilder:validation:Enum=DataLoad;DataBackup;DataMigrate;DataProcess
	Name string `json:"name"`
	// +optional
	Image string `json:"image,omitempty"`
	Command []string `json:"command,omitempty"`
	Args []string `json:"args,omitempty"`
}

And the DataLoad Pod values are built only from those fields plus a single injected env (FLUID_RUNTIME_CONFIG_PATH) — there is no way to attach a ConfigMap/Secret volume, an init container, or a full pod template:

fluid/pkg/ddc/cache/engine/dataload.go

Lines 113 to 124 in ccd52dc

	dataloadInfo.Command = opSpec.Command
	dataloadInfo.Args = opSpec.Args
	if len(dataloadInfo.Command) == 0 && len(dataloadInfo.Args) == 0 {
	ctx.Recorder.Eventf(dataload, corev1.EventTypeWarning, common.DataOperationExecutionFailed, "dataLoad command and args defined in cache runtime class can not be both empty")
	return nil, errors.New("dataLoad command and args defined in cache runtime class can not be both empty")
	}
	// DataOperationSpecs image takes precedence; falls back to worker image if empty.
	dataloadInfo.Image = opSpec.Image
	if len(dataloadInfo.Image) == 0 {
	dataloadInfo.Image, err = e.getDataOperationImage(runtime, runtimeClass)
	if err != nil {

dataloadInfo.Command = opSpec.Command
dataloadInfo.Args = opSpec.Args
// ...
// DataOperationSpecs image takes precedence; falls back to worker image if empty.
dataloadInfo.Image = opSpec.Image

fluid/pkg/ddc/cache/engine/dataload.go

Lines 168 to 176 in ccd52dc

	Name: "FLUID_RUNTIME_CONFIG_PATH",
	Value: e.getRuntimeConfigPath(),
	},
	// FLUID_DATALOAD_DATA_PATH and FLUID_DATALOAD_PATH_REPLICAS is generated and set in the helm job yaml.
	}
	dataLoadValue := &cdataload.DataLoadValue{
	Name: dataload.Name,
	OwnerDatasetId: utils.GetDatasetId(targetDataset.Namespace, targetDataset.Name, string(targetDataset.UID)),

dataloadInfo.Envs = []cdataload.Env{
	{
		Name:  "FLUID_RUNTIME_CONFIG_PATH",
		Value: e.getRuntimeConfigPath(),
	},
	// FLUID_DATALOAD_DATA_PATH and FLUID_DATALOAD_PATH_REPLICAS is generated and set in the helm job yaml.
}

By contrast, topology components (RuntimeComponentDefinition) already support a full template (PodTemplateSpec), dependencies (extraResources + secretMount), and executionEntries, which is exactly why master/worker/client can mount the curvine-config ConfigMap, run a gomplate init container to render curvine.toml, and mount encrypt-option Secrets:

fluid/api/v1alpha1/cacheruntimeclass_types.go

Lines 41 to 97 in bf04633

	// RuntimeComponentDefinition defines the configuration for a CacheRuntime component
	type RuntimeComponentDefinition struct {
	// Options is a set of key-value pairs that provide additional configuration for the component
	// +optional
	Options map[string]string `json:"options,omitempty"`
	// Template describes the pods that will be created.
	// The template follows the standard PodTemplateSpec from Kubernetes core.
	// +optional
	Template corev1.PodTemplateSpec `json:"template,omitempty"`
	// Service is the service configuration for the component
	// +optional
	Service RuntimeComponentService `json:"service,omitempty"`
	// Dependencies specifies the dependencies required by the component
	// +optional
	Dependencies RuntimeComponentDependencies `json:"dependencies,omitempty"`
	// ExecutionEntries entries to support out-of-tree integration.
	// +optional
	ExecutionEntries *ExecutionEntries `json:"executionEntries,omitempty"`
	}
	type ExecutionEntries struct {
	// MountUFS defines the operations for mounting UFS. The command's stdout must be JSON matching CacheRuntimeMountUfsOutput.
	MountUFS *ExecutionCommonEntry `json:"mountUFS,omitempty"`
	// ReportSummary it defines the operation how to get cache status like capacity, hit ratio etc.
	ReportSummary *ExecutionCommonEntry `json:"reportSummary,omitempty"`
	}
	type ExecutionCommonEntry struct {
	Command []string `json:"command"`
	// TimeoutSeconds is the timeout(seconds) for the execution entry, at least(default) 20 seconds.
	TimeoutSeconds int32 `json:"timeout,omitempty"`
	}
	// ExtraResourcesComponentDependency defines the extra resources configuration for component dependencies
	type ExtraResourcesComponentDependency struct {
	// ConfigMaps is a list of ConfigMaps in the same namespace to mount into the component
	// +optional
	ConfigMaps []ConfigMapDependencyConfig `json:"configMaps,omitempty"`
	}
	// RuntimeComponentDependencies defines the dependencies required by a CacheRuntime component
	type RuntimeComponentDependencies struct {
	// SecretMount controls whether dataset encrypt-option secrets are mounted into this component pod.
	// Defaults to true for Master/Worker, false for Client unless explicitly enabled.
	// +optional
	SecretMount *SecretMountComponentDependency `json:"secretMount,omitempty"`
	// ExtraResources specifies the usage of extra resources such as ConfigMaps
	// +optional
	ExtraResources *ExtraResourcesComponentDependency `json:"extraResources,omitempty"`
	}

Why is this feature needed:

Because DataLoad cannot mount ConfigMaps/Secrets or run an init container, the e2e Curvine integration has to hardcode the journal address inside the DataLoad script, which the author explicitly called out as a workaround:

fluid/test/gha-e2e/curvine/cacheruntimeclass.yaml

Lines 37 to 55 in bf04633

	dataOperationSpecs:
	- name: DataLoad
	command:
	- "/bin/bash"
	- "-c"
	args:
	# Actually, the cache runtime image should use $(FLUID_RUNTIME_CONFIG_PATH) to generate the config file, and
	# use $(FLUID_DATALOAD_DATA_PATH) to execute data load.
	- |
	# currently we have no customized image supporting dataload for curvine, so we write the curvine.toml with fixed journal address for test case.
	echo -e '[journal]\njournal_addrs = [\n{id=1, hostname="curvine-demo-master-0.svc-curvine-demo-master"}\n]' > /etc/curvine.toml
	IFS=: read -ra paths <<< "$FLUID_DATALOAD_DATA_PATH"
	for p in "${paths[@]}"; do
	/app/curvine/bin/cv load "$p" --watch --conf /etc/curvine.toml || {
	echo "Error: load $p failed."
	exit 1
	}
	done

dataOperationSpecs:
  - name: DataLoad
    command: ["/bin/bash", "-c"]
    args:
      - |
        # currently we have no customized image supporting dataload for curvine, so we write the curvine.toml with fixed journal address for test case.
        echo -e '[journal]\njournal_addrs = [\n{id=1, hostname="curvine-demo-master-0.svc-curvine-demo-master"}\n]' > /etc/curvine.toml

        IFS=: read -ra paths <<< "$FLUID_DATALOAD_DATA_PATH"
        for p in "${paths[@]}"; do
          /app/curvine/bin/cv load "$p" --watch --conf /etc/curvine.toml || {
            echo "Error: load $p failed."
            exit 1
          }
        done

This hardcoding is a symptom; the missing capability in DataOperationSpec is the root cause. Concrete consequences:

1. Hardcoded / non-generic config. The journal hostname curvine-demo-master-0.svc-curvine-demo-master is pinned to a single master and to the fixed runtime name curvine-demo. It breaks if the runtime name changes or if master.replicas > 1, because the cluster.toml template renders journal addresses dynamically from master.replicas while DataLoad assumes a single static journal.

2. No access to UFS credentials. The mountUFS.sh path can read encryptOptions Secret files, but DataLoad has no way to mount those Secrets — so DataLoad cannot work against authenticated UFS (e.g. S3 access/secret keys). The two code paths have inconsistent capabilities.

3. Cannot reuse the existing config-generation pattern. Master/worker/client use an init-curvine (gomplate) container + the curvine-config ConfigMap to generate curvine.toml. DataLoad cannot reuse this, forcing every integrator to re-implement config generation inline in the script.

Proposed direction (for discussion):

• Extend DataOperationSpec to support dependencies consistent with RuntimeComponentDefinition — at minimum extraResources.configMaps (mount ConfigMap templates) and secretMount (mount dataset encrypt-option Secrets).
• Optionally allow a richer pod spec for data operations (e.g. initContainers / volumes / a template) so the same gomplate-based config generation used by topology components can be reused, instead of hardcoding config inside args.
• Plumb the new fields through genDataLoadValue in pkg/ddc/cache/engine/dataload.go and the dataloader helm chart (DataLoadInfo / DataLoadValue in pkg/dataload/value.go) so the rendered Job mounts the requested ConfigMaps/Secrets.

Additional context / follow-ups (not the core ask):

• The --watch flag on cv load in the same fixture is likely a separate functional bug (it is a blocking call and would prevent the DataLoad Job from completing). Worth tracking separately.
• Once DataLoad can mount the curvine-config ConfigMap + run the gomplate init container, the hardcoded journal address in test/gha-e2e/curvine/cacheruntimeclass.yaml can be removed and the fixture made generic.

[ayushi-work]

Hi @cheyang
The root issue is that DataOperationSpec only exposes the basic pod fields (image, command, args, etc.), so DataLoad pods can't mount ConfigMaps or Secrets. That makes authenticated UFS access and config-generation workflows (like the existing runtime components use) difficult to support.
My plan is to:

• Add Dependencies *RuntimeComponentDependencies and Template *corev1.PodTemplateSpec to DataOperationSpec
• Reuse RuntimeComponentDependencies instead of introducing new types
• Mount ConfigMaps defined in CacheRuntimeClass.ExtraResources and support encrypt-option secret mounts through Dependencies.SecretMount
• Use Template for InitContainers, additional Volumes, and VolumeMounts (keeping the main container managed by the DataLoad spec)
• Add InitContainers to DataLoadInfo, wire it through genDataLoadValue, and render it in the Job/CronJob templates
• Regenerate the deepcopy and OpenAPI files

Volumes and VolumeMounts already flow through the existing values path into the Helm templates, so only InitContainers need chart changes.

[ayushi-work]

Hi @cheyang can I work on this?