[BUG]CacheRuntime: editing `tieredStore` updates the ConfigMap but not the worker Pod, and nothing reports the split

[btxu-db]

What is your environment(Kubernetes version, Fluid version, etc.)

• Kubernetes v1.30.0 (kind v0.23.0)
• Fluid CacheRuntime controller built from ce7ef73d (master plus fix(cache): propagate runtime-level podMetadata and imagePullSecrets to component pods #6188, which does not touch this code path)
• Runtime: CacheRuntime

Describe the bug

Changing spec.worker.tieredStore on an existing CacheRuntime updates the runtime ConfigMap but not the worker Pod. The quota ends up recorded as two different values, and no event, condition or log line says so.

A tiered store quota is recorded in three places, written at different times:

Where	Written
tmpfs sizeLimit on the AdvancedStatefulSet	at creation only
container memory request/limit	at creation only
quotas in the runtime ConfigMap	on every sync, recomputed from the spec

The first two come from TransformRuntimeTieredStore, which runs only in Setup, and Setup stops running once IsSetupDone is true. The ConfigMap is rebuilt from the spec on every sync: generateRuntimeConfigData (pkg/ddc/cache/engine/cm.go:102) never reads the existing ConfigMap, and syncRuntimeValueConfigMap replaces its Data wholesale. So the three agree until the spec is edited, and after that they never converge again.

With the fixture below, changing the quota from 8Gi to 16Gi:

                   after create   ~1 min after the edit
spec quota         8Gi            16Gi
ConfigMap quota    8Gi            16Gi      (changed within 5s)
tmpfs sizeLimit    8Gi            8Gi
memory limit       12Gi           12Gi
ASTS generation    1              1
Warning events     0              0

When the damage shows up depends on the cache system. The Mooncake sample reads its quota from the config file at start-up, so a running worker is unaffected until it restarts for some unrelated reason (eviction, OOM, a scale-out, a rolling update) and comes up told it has the new quota on top of the old tmpfs. A process that watches the file, which the integration guide suggests, would pick up the new value straight away. With Mooncake (2Gi → 4Gi), the restarted worker started with segment_size=4GB against a 2.0G /dev/shm, and a 3072MiB write stopped at 2048MiB. Checked again two days later, the same runtime still showed the split, with generation at 1.

By then the failure is detached in time from the edit that caused it, and the config file inside the Pod agrees with the spec, which sends whoever is debugging in the wrong direction.

What you expect to happen:

tieredStore cannot be updated in place: SyncComponentSpec never patches volumes, and a tmpfs sizeLimit cannot change without recreating the Pod. A validating webhook that rejects such edits is proposed in #6193. That does not cover every deployment: webhook.enabled can be turned off (charts/fluid/fluid/values.yaml:357), and a cluster where the quota was already edited before the webhook existed will not send another request for it to reject.

For those cases the controller should at least make the split visible:

1. Emit a Warning event on the CacheRuntime when a tiered store level in the spec differs from what the worker workload was created with. Emitting it once is not enough, since events expire after an hour by default and this failure usually surfaces days later. Emitting it on every sync is not right either: the recorder's spam filter is keyed on the involved object alone (a burst of 25, then one every five minutes), so a drift warning on every sync would crowd out every other warning for the same CacheRuntime. Re-emitting it while the split persists, at an interval shorter than the event TTL, keeps it visible without that; the engine already throttles this way with timeOfLastSync. A condition would stay visible without any of this, if one is preferred.
2. Document that tieredStore cannot be changed after creation, and what happens if it is. Neither cacheruntime_spec_update.md nor cache_runtime_tieredstore.md says anything about this today; the former does not mention tieredStore at all.

How to reproduce it

Same busybox fixture as #6166, under different names.

apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntimeClass
metadata:
  name: repro-tsedit
fileSystemType: reprofs
topology:
  master:
    service: { headless: {} }
    template:
      spec:
        restartPolicy: Always
        containers:
          - name: master
            image: busybox:1.36
            imagePullPolicy: IfNotPresent
            command: ["sh", "-c", "while true; do nc -l -p 50051; done"]
            readinessProbe:
              tcpSocket: { port: 50051 }
              initialDelaySeconds: 3
              periodSeconds: 5
            ports: [{ containerPort: 50051, name: rpc }]
  worker:
    service: { headless: {} }
    template:
      spec:
        restartPolicy: Always
        containers:
          - name: worker
            image: busybox:1.36
            imagePullPolicy: IfNotPresent
            command: ["sh", "-c", "while true; do nc -l -p 50052; done"]
            readinessProbe:
              tcpSocket: { port: 50052 }
              initialDelaySeconds: 3
              periodSeconds: 5
            ports: [{ containerPort: 50052, name: data }]
---
apiVersion: v1
kind: Namespace
metadata: { name: repro-tsedit }
---
apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata: { name: repro-tsedit, namespace: repro-tsedit }
spec:
  placement: Shared
  accessModes: ["ReadWriteMany"]
  mounts: [{ name: r, mountPoint: "reprofs:///" }]
---
apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntime
metadata: { name: repro-tsedit, namespace: repro-tsedit }
spec:
  runtimeClassName: repro-tsedit
  master: { replicas: 1 }
  worker:
    replicas: 1
    resources:
      limits:
        memory: "4Gi"
    tieredStore:
      levels:
        - processMemory: { quota: 8Gi }
          high: "0.8"
          low: "0.5"

snap() {
  NS=repro-tsedit; RT=repro-tsedit
  echo "spec      $(kubectl get cacheruntime $RT -n $NS -o jsonpath='{.spec.worker.tieredStore.levels[0].processMemory.quota}')"
  echo "configmap $(kubectl get cm fluid-runtime-config-$RT -n $NS -o jsonpath='{.data.runtime\.json}' | jq -r '.worker.tieredStoreLevels[0].quotas[0]')"
  echo "tmpfs     $(kubectl get advancedstatefulset $RT-worker -n $NS -o jsonpath='{.spec.template.spec.volumes[?(@.emptyDir.medium=="Memory")].emptyDir.sizeLimit}')"
  echo "memory    $(kubectl get advancedstatefulset $RT-worker -n $NS -o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')"
}

kubectl apply -f repro.yaml
# wait for the Dataset to be Bound, then:
snap
kubectl patch cacheruntime repro-tsedit -n repro-tsedit --type=merge \
  -p '{"spec":{"worker":{"tieredStore":{"levels":[{"processMemory":{"quota":"16Gi"},"high":"0.8","low":"0.5"}]}}}}'
sleep 60
snap
kubectl get events -n repro-tsedit --field-selector type=Warning

Additional Information

syncRuntimeSpec already reads the worker Pod spec (sync.go:224), so the check needs no change to cm.go and no new data source. It should compare level by level, with Quantity.Cmp rather than as strings: each processMemory and emptyDir quota against the sizeLimit of the volume the creation path generated for that level, whose name is deterministic (getMemoryTieredStoreVolumeName, getTieredStoreVolumeName). chargedTieredStoreMemoryQuota (sync.go:229) walks the same volumes but sums only the memory-backed ones, so a comparison against it would miss a disk-backed emptyDir, whose sizeLimit is set the same way. hostPath levels cannot split like this, since their quotas are only passed through to the cache system. One more gap: that block sits inside if workerResources != nil, so when neither the CacheRuntime nor the CacheRuntimeClass declares resources it is not reached, and the check needs its own read of the Pod spec.

spec.client.tieredStore should behave the same way, going by the code: generateRuntimeConfigData rebuilds the client's levels from the spec on every sync (cm.go:184), while the client DaemonSet is never updated after creation (#6154). I have only reproduced the worker case.

The alternative is to freeze the ConfigMap, keeping the creation-time tieredStoreLevels on sync. That removes the split, but it needs a new source for the old value, either parsing the existing ConfigMap or reading it back from the workload, and once #6191 lands the quota is carried in both runtime.json and runtime.sh. The event is much cheaper, works with or without the webhook, and does not depend on any decision about it, so I would start there.