CacheRuntime Spec Field Update Capabilities
1. Overviewâ
This document describes which spec fields of the CacheRuntime's Master and Worker components (backed by AdvancedStatefulSet) can be updated in-place.
The current version supports in-place updates for the following three fields:
- Container image (
runtimeVersion) - Resource limits (
resources) - Replica count (
replicas)
Modifications to other fields will not be propagated to the AdvancedStatefulSet and require redeployment to take effect.
2. Supported Componentsâ
| Component | Workload Type | Field Update Support |
|---|---|---|
| Master | AdvancedStatefulSet | â
Supports runtimeVersion, resources and replicas |
| Worker | AdvancedStatefulSet | â
Supports runtimeVersion, resources and replicas |
| Client | DaemonSet | â Not supported (any changes require redeployment) |
Notes:
- Modifying
runtimeVersion,resourcesandreplicason Master and Worker components automatically propagates to the underlying AdvancedStatefulSet. - The Client component uses a DaemonSet and does not support dynamic updates.
3. Supported Update Fieldsâ
3.1 Container Image (runtimeVersion)â
Field path: spec.{master,worker}.runtimeVersion
Supported sub-fields:
image: Image nameimageTag: Image tag
Example:
spec:
worker:
runtimeVersion:
image: fluid-cache
imageTag: v1.1.0
Resolution: on creation and on every reconcile, the image of the component's first container in the CacheRuntimeClass template is the baseline, and the image and imageTag set in the CacheRuntime override it.
Completion rules:
imageandimageTagmay be set on their own. The missing half is taken from the template image. With a template image offluid-cache:v1.0.0:- Only
imageTag: v1.1.0: the template's image name is kept, givingfluid-cache:v1.1.0 - Only
image: registry.example.com/fluid-cache: the template's tag is kept (notlatest), givingregistry.example.com/fluid-cache:v1.0.0 - Neither: the template image
fluid-cache:v1.0.0is used
- Only
- A
runtimeVersionsetting bothimageandimageTagis used as is, even when the template image is pinned by digest. - When only one half is set and the other cannot be completed (only
imageis set and the template image has no tag, or the template image orimageis pinned by digest (@sha256:...)), the component uses the template image on creation; on update the edit has no effect and the component keeps its current image. In both cases the controller emits aRuntimeVersionIncompleteWarning event on the CacheRuntime and logs the edit; the client, which is not updated in place, reports it on creation only.runtimeVersioncannot express a digest reference; to pin a digest, put it in the CacheRuntimeClass template. - Data operations that name no image of their own (such as DataLoad) use the same completed image as the worker. When the worker's version cannot be completed they use the template image, so they can differ from a worker that kept its current image, for example after the template image is changed to one pinned by digest.
Limitations:
- â ī¸ In cgroupv1 environments, this field cannot be updated simultaneously with
resources(perform step-by-step; see the resource field section below). - â ī¸ When setting only
image, make sure the template's tag exists in the new repository. Otherwise the pods fail withImagePullBackOff; on an in-place update the component is unavailable until the new image can be pulled. The same holds the other way round: an instance that sets onlyimageTagmoves to that tag in the new repository when the template's image repository changes, so make sure the tag exists there too. - â ī¸ Image edits in the CacheRuntimeClass template reach existing instances: after the template image changes, each enabled master and worker of a CacheRuntime using that CacheRuntimeClass that does not fully set
runtimeVersionis re-resolved against the new template image, and rolls only if the resolved image changes (one that sets onlyimageTagkeeps its tag, one that sets onlyimagekeeps its image name; an instance that cannot be resolved to aname:tagimage keeps its current image, for example when the new template image is pinned by digest). The controller does not watch CacheRuntimeClass, so the change takes effect on each CacheRuntime's next reconcile rather than immediately. The client is not updated. - â ī¸ Removing
runtimeVersionfrom a CacheRuntime returns the component to the template image, except when the template image has no tag or is pinned by digest: the component then keeps its current image, and you need to set a fullruntimeVersionor recreate the CacheRuntime.
3.2 Resource Limits (resources)â
Field path: spec.{master,worker}.resources
Supported sub-fields:
requests.cpu: CPU requestrequests.memory: Memory requestlimits.cpu: CPU limitlimits.memory: Memory limitclaims: Dynamic Resource Allocation (DRA) claims; they can only reference pod-levelresourceClaimsthe template declares, see the limitations below
Example:
spec:
worker:
resources:
requests:
cpu: "4"
memory: 8Gi
limits:
cpu: "8"
memory: 16Gi
Resolution order:
resources is resolved as follows on every reconcile:
- the
resourcesdeclared by the CacheRuntimeClass template form the baseline; - the
resourcesset on the CacheRuntime are overlaid on that baseline key by key ârequestsandlimitsmerge by resource name,claimsby claim name; - a key neither declares is cleared from the workload.
The CacheRuntime therefore expresses only the differences it wants from the template. Naming
limits.memory alone moves that one value and leaves the template's requests.cpu,
requests.memory and limits.cpu in place.
Removing a key from the CacheRuntime works the same way: it falls back to the template's value,
or is cleared from the workload when the template does not declare it. An empty list such as
limits: {} names no key, so it reads the same as leaving limits out.
Limitations:
- â ī¸ Cannot exceed the node's available resources.
- â ī¸ A key the CacheRuntimeClass template declares can be overridden but not removed: omitting it from the CacheRuntime leaves the template's value in place rather than leaving the component unconstrained. To adjust resource requirements, explicitly set the desired values in the CacheRuntime.
- â ī¸ How CacheRuntimeClass edits affect existing CacheRuntimes is tracked in #6213; don't rely on them reaching existing instances.
- â ī¸ The CacheRuntime and its template are the source of truth: resources edited directly on the master or worker AdvancedStatefulSet are reverted on the next reconcile.
- â ī¸ A request must not exceed its limit after the merge: the template and the CacheRuntime
can each be valid and still merge into a request above its limit, for example a template
requests.cpu: 1with a CacheRuntimelimits.cpu: 500m. Huge pages (hugepages-*) and extended resources (such asnvidia.com/gpu) are stricter: their request must equal their limit, and a request needs a limit, so a template settingnvidia.com/gputo 1 for both with a CacheRuntime raising only the limit to 2 conflicts as well. The controller then does not sync that component'sresources, keeps the workload's current values and emits aRuntimeResourcesConflictWarning event on the CacheRuntime naming the conflicting keys; a new CacheRuntime with such a pair fails to set up. Set the matching request on the CacheRuntime too (equal to the limit for huge pages and extended resources), or change the template. - â ī¸
claimscan only reference pod-levelresourceClaimsthe workload declares: the CacheRuntime sets the container'sresources.claimsbut cannot add pod-levelspec.resourceClaims, and the sync does not carry pod-levelresourceClaimsadded to the CacheRuntimeClass template after creation to existing workloads either. On creation, a claim name the template does not declare makes the API server reject every master or worker Pod (must be one of the names in pod.spec.resourceClaims), and the component stays NotReady; clusters withoutDynamicResourceAllocationenabled (off by default before K8s 1.34) dropclaimssilently instead. On update, a claim that the workload neither declares nor already carries, whether it comes from the CacheRuntime or from a template edit, is not synced: the controller leaves it out, still syncs the rest ofresources, and emits aRuntimeResourcesConflictWarning event on the CacheRuntime naming the claim. To use a new claim, recreate the CacheRuntime. - â ī¸ Kubernetes version requirement: K8s >= 1.27 with the
InPlacePodVerticalScalingFeature Gate enabled.# Check if the Feature Gate is enabled
kubectl get nodes -o jsonpath='{.items[0].status.config.kubeletConfig.featureGates.InPlacePodVerticalScaling}' - â ī¸ Cgroupv1 limitation: Cannot be updated simultaneously with the
runtimeVersionfield.- Reason: Cgroupv1 does not support updating container image and resource limits in the same operation.
- Solution: Perform step-by-step updates â update resources first, wait for completion, then update the image.
See Kubernetes Issue #127356.# Step 1: Update resources
kubectl patch cacheruntime my-cache --type='merge' -p '{"spec":{"worker":{"resources":{"requests":{"cpu":"4","memory":"8Gi"}}}}}'
kubectl wait pod -l fluid.io/cache-runtime-name=my-cache --for=condition=InPlaceUpdateReady --timeout=300s
# Step 2: Update image
kubectl patch cacheruntime my-cache --type='merge' -p '{"spec":{"worker":{"runtimeVersion":{"imageTag":"v1.1.0"}}}}'
3.3 Replica Count (replicas)â
Field path: spec.{master,worker}.replicas
Example:
spec:
worker:
replicas: 3
kubectl patch cacheruntime my-cache --type='merge' -p '{"spec":{"worker":{"replicas":3}}}'
The new value is synced to the AdvancedStatefulSet's spec.replicas, which then performs the scaling.
Limitations:
- â ī¸ Scaling in discards cached data: the Worker Pods removed are deleted outright, and the data they had cached is not migrated to the remaining Workers â it has to be loaded from the underlying storage again. Scaling out leaves the existing cache untouched.
- â ī¸ Scaling writes no RuntimeCondition and emits no Kubernetes Event. Progress can be observed through
status.{master,worker}.{readyReplicas,desiredReplicas}or the controller logs:# Current replica status
kubectl get cacheruntime my-cache -o jsonpath='{.status.worker.readyReplicas}/{.status.worker.desiredReplicas}'
# Or from the controller logs
kubectl -n fluid-system logs deploy/cacheruntime-controller | grep "replicas changed" - â ī¸ If the replica count exceeds the number of schedulable nodes, the surplus Worker Pods stay
Pending.
4. Unsupported Update Fieldsâ
Any field not listed in section 3 cannot be updated in place; changes to it are not propagated to the AdvancedStatefulSet.
Notes:
- Such a field takes effect only after the CacheRuntime is redeployed.
- The system does not automatically detect or propagate these changes.
5. Summaryâ
| Field | Supported | Update Method |
|---|---|---|
runtimeVersion | â Yes | Automatically synced to AdvancedStatefulSet |
resources | â Yes | Automatically synced to AdvancedStatefulSet (requires K8s >= 1.27) |
replicas | â Yes | Automatically synced to AdvancedStatefulSet (scaling in discards that Worker's cache) |
| All other fields | â No | Not synced; redeployment required |
Key Takeaways:
- The current version supports dynamic updates for
runtimeVersion,resourcesandreplicas. - In cgroupv1 environments,
runtimeVersionandresourcesmust be updated separately (step-by-step);replicasis not subject to this limitation. - Modifications to other fields will not take effect and require redeployment.