跳到主要内容
版本:v1.1

CacheRuntime Integration Guide

Installation​

  • Install Fluid v1.1.0 or later, which supports CacheRuntime. The CacheRuntime controller is disabled by default, so enable it with --set runtime.cacheruntime.enabled=true. See the Installation Guide for more installation options.
helm repo add fluid https://fluid-cloudnative.github.io/charts

helm repo update

helm install fluid fluid/fluid -n fluid-system --create-namespace --set runtime.cacheruntime.enabled=true

Integration​

Step 1. Plan Cluster Topology​

First, you need to plan a cluster topology:

  • Determine the topology type and which components are included:

  • MasterSlave: Master/Worker/Client

  • P2P/DHT: Worker/Client

  • ClientOnly: Client

  • Determine the form and configuration of each component:

  • Stateful/Stateless - Determines the workload type

  • Standalone/Active-Standby/Cluster

The table below shows basic information examples for deploying several major cache topology types.

  • MasterSlave: CubeFS/Alluxio
TopologySettings
Master* workLoadType: apps/v1/StatefulSet

* Image configuration

* Startup command

* UFS mount command

* HeadlessService needs to be created

* Authentication keys need to be mounted
Worker: Used for single worker role definition* workLoadType: apps/v1/StatefulSet

* Image configuration

* Startup command

* HeadlessService needs to be created

* Authentication keys do NOT need to be mounted

* TieredStore needs to be configured
ClientFuse* Role: Posix client

* workLoadType: apps/v1/DaemonSet

* Image configuration

* Startup command

* Authentication parameters do NOT need to be mounted

* TieredStore is NOT supported
  • P2P Worker: JuiceFS
TopologySettings
Worker: Used for single worker role definition* workLoadType: apps/v1/StatefulSet

* Image configuration

* Startup command

* HeadlessService

* Authentication parameters need to be mounted

* TieredStore is supported
Client* Role: Fuse client

* workLoadType: apps/v1/DaemonSet

* Image configuration

* Startup command

* Service is NOT required

* Authentication parameters need to be mounted

* TieredStore is supported

Step 2. Prepare Cache System Template​

A cache system template in Fluid contains the following parts:

├── Name # runtimeClassName is specified in CacheRuntime
├── FileSystemType # File system type, used for mount readiness verification
├── Topology
│ ├── Master[component]
│ ├── Worker[component]
│ └── client[component]
└── ExtraResources
└── ConfigMaps

The component in Topology mainly contains the following content:

ContentDescriptionRecommendation
WorkloadTypeThe workload type of this componentFor stateful applications like Master/Worker, StatefulSet is the most common choice, as it can more easily cooperate with formatted DNS domain names provided by Headless Service for access
If Client is a Fuse client responsible for providing Posix access capability for pods on nodes, DaemonSet is generally used
If Client is an SDK proxy as a centralized stateless application, Deployment with ClusterIP type Service is generally used
OptionsDefault options, will be overridden by user settings
TemplatePodTemplateSpec native field
ServiceCurrently only supports Headless
DependenciesExtraResourcesWhether this component needs to mount additional ConfigMaps (the dependent ConfigMap information is defined in the ExtraResources field of CacheRuntimeClass).
ExecutionEntriesMountUFSFor Master-Worker architecture, when Master is Ready, the underlying file system mount operation needs to be executed. The MountUFS script must output JSON in CacheRuntimeMountUfsOutput struct format, containing the list of mounted UFS paths. See Step 2.7 for details.
ExecutionEntriesReportSummaryDefines operations for the cache system to obtain cache information metrics, the command must output JSON in CacheRuntimeReportSummary struct format.

Step 2.1 Prepare K8s-adapted Native Images and Define Component workloadType and PodTemplate​

You can first use native images, configure component workloadType and PodTemplate, manually start a fixed cache system in the K8s cluster, manually start the cache system in the pod, and make it locally accessible. This step is mainly used to clarify what K8s resources are needed and to prepare base images.

Step 2.2 Clarify What Configurations CacheRuntime Should Provide for Components​

Mainly clarify the following settings:

  • Service

  • Dependencies

Step 2.3 Confirm Default ENV Provided by Fluid CacheRuntime for Components, Applicable by Scripts Inside Containers​

ENVDescription
FLUID_DATASET_NAMEDataset name, generally used for isolation between groups in cache group concepts
FLUID_DATASET_NAMESPACENamespace where the dataset is located
FLUID_RUNTIME_CONFIG_PATHRuntime configuration path provided by fluid
FLUID_RUNTIME_MOUNT_PATHOften used by Client, the target path where client performs mount action
FLUID_RUNTIME_COMPONENT_TYPEIndicates whether the current component is master, worker, or client
FLUID_RUNTIME_COMPONENT_SVC_NAMEIf the component defines a service, this value is the service name

Step 2.4 Create RuntimeClass Example and Field Description:​

apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntimeClass
metadata:
name: demofs
fileSystemType: $fsType
topology:
master:
service: # Need to create Headless Service for master
headless: {}
executionEntries:
mountUFS: # mount ufs in master pod, see section 2.7
command:
- bash
- -c
- /mountUfs.sh
timeout: 120
reportSummary: # get cache states in master pod, see section 2.8
command:
- bash
- -c
- /reportSummary.sh
timeout: 30
template:
spec:
restartPolicy: Always
containers:
- name: master
image: $image
args:
- /bin/sh
- -c
- custom-endpoint.sh
imagePullPolicy: IfNotPresent
worker:
service: # create Headless Service for worker
headless: {}
template:
spec:
restartPolicy: Always
containers:
- name: worker
image: $image
args:
- /bin/sh
- -c
- custom-endpoint.sh
imagePullPolicy: IfNotPresent
client:
template:
spec:
restartPolicy: Always
containers:
- name: client
image: $image
securityContext: # Usually client needs to configure privileged for operating fuse device
privileged: true
runAsUser: 0
args:
- /bin/sh
- -c
- custom-endpoint.sh
imagePullPolicy: IfNotPresent
Cache Systems Without a Client Component​

The master, worker and client components under topology are all optional in the API. You may omit the client component if the underlying cache system has both of the following characteristics:

  1. No POSIX mount semantics: the cache system itself provides no FUSE-like mount capability.
  2. Applications connect directly: applications read and write against the master/worker through the cache system's own SDK or RPC client, without relying on a mount point.

In that case the Master and Worker components start as usual, the Dataset reaches the Bound phase as usual, and cache status is still reported through the ReportSummary script.

Note that Fluid still creates a PVC/PV for such a Dataset and reports them as Bound, but an application pod that mounts the PVC stays in ContainerCreating forever (reporting timeout waiting for FUSE mount point). For this kind of cache system, applications should talk to the cache service directly instead of going through the Dataset → PVC → volumeMounts path. For a complete configuration example, see Deploy Mooncake with CacheRuntime.

Step 2.5 User Creates Runtime​

apiVersion: data.fluid.io/v1alpha1
kind: Dataset
metadata:
name: demofs
namespace: default
spec:
placement: Shared
accessModes:
- ReadWriteMany
mounts:
- name: demo
mountPoint: "demofs:///"
options:
key1: value1
key2: value2
encryptOptions:
- name: token
valueFrom:
secretKeyRef:
name: jfs-secret
key: token
- name: access-key
valueFrom:
secretKeyRef:
name: jfs-secret
key: access-key
- name: secret-key
valueFrom:
secretKeyRef:
name: jfs-secret
key: secret-key
---
apiVersion: data.fluid.io/v1alpha1
kind: CacheRuntime
metadata:
name: demofs
namespace: default
spec:
runtimeClassName: demofs
master:
options: # master option
key1: value1
key2: value2
replicas: 2 # master replica count
worker:
options: # worker option
key1: value1
key2: value2
replicas: 2 # worker
tieredStore:
levels: # worker cache configuration
- emptyDir:
quota: 1Gi
high: "0.8"
low: "0.5"
client:
options:
key1: value1
key2: value2
volumeMounts: # Can configure volumes and corresponding volumeMounts
- name: demo
mountPath: /mnt
volumes:
- name: demo
persistentVolumeClaim:
claimName: test

For detailed configuration instructions on tiered storage, please refer to CacheRuntime TieredStore Configuration Example.

Step 2.6 Confirm RuntimeConfig Provided by Fluid CacheRuntime for Components, Parse Parameters to Start Containers​

You can modify the entryPoint script based on the native image, first parse RuntimeConfig, generate corresponding configuration files, and then start the container. You can refer to the integration example in test/gha-e2e/curvine in the official repository.

In cacheruntime, all control plane processes are handled by Fluid. However, as a data caching engine, when providing services, the entire cache system requires topology, data source, authentication, and cache information. Fluid will provide this information to components through configuration files based on different Component roles. The component's internal process is responsible for parsing this configuration to perform environment variable configuration, data engine configuration file generation, and other operations. After preparation is complete, the data engine process can be started. For specific parsing details, please refer to the table below:

  • Taking the above resources as an example, the Config examples mounted by Master/Worker/Client and maintained by Fluid are as follows: the mounts, accessModes, and targetPath fields in the JSON are all derived from the Dataset's Spec definition.
{
"mounts": [
{
"mountPoint": "s3://test",
"options": {
"access": "minioadmin",
"endpoint_url": "http://minio:9000",
"path_style": "true",
"region_name": "us-east-1",
"secret": "minioadmin"
},
"encryptOptions": {
"access-key": "/etc/fluid/secrets/minio-secret/access-key",
"secret-key": "/etc/fluid/secrets/minio-secret/secret-key"
},
"name": "minio",
"path": "/minio"
}
],
"accessModes": [
"ReadWriteMany"
],
"targetPath": "/runtime-mnt/cache/default/curvine-demo/cache-fuse",
"master": {
"enabled": true,
"name": "curvine-demo-master",
"options": {
"key1": "master-value1"
},
"replicas": 1,
"service": {
"name": "svc-curvine-demo-master"
}
},
"worker": {
"enabled": true,
"name": "curvine-demo-worker",
"options": {
"key1": "worker-value1"
},
"replicas": 1,
"service": {
"name": "svc-curvine-demo-worker"
},
"tieredStoreLevels": [
{
"mountPaths": [
"/etc/fluid/mount/tiered-store/level-0-index-0-emptydir"
],
"mediumType": "HDD",
"quotas": [
"1Gi"
],
"high": "0.8",
"low": "0.5"
}
]
},
"client": {
"enabled": true,
"name": "curvine-demo-client",
"options": {
"key1": "value1"
},
"service": {
"name": ""
}
}
}

Step 2.7 MountUFS Script Output Format Requirements​

For cache systems with Master-Worker architecture, after the Master component is Ready, Fluid will execute the MountUFS script to mount the underlying file system (UFS). The MountUFS script must output JSON in CacheRuntimeMountUfsOutput struct format, so that Fluid can correctly parse the mounted paths and synchronize the Dataset status.

Optional Configuration Note​

If the underlying cache system's image has the following capabilities, you can omit the MountUFS configuration:

  1. Automatic RuntimeConfig Monitoring: The process inside the container can monitor changes to the "$FLUID_RUNTIME_CONFIG_PATH" file
  2. Automatic Mount Execution: When changes to the mounts configuration are detected, automatically execute the underlying file system mount operations
  3. Ready State Control: Ensure that the Master component's Pod starting status only becomes Ready after all UFS mounts are completed

In this case, Fluid will confirm whether the Master component is ready through Kubernetes' Ready probe, without needing to execute the MountUFS script additionally.

Use Cases​

The output of the MountUFS script is mainly used for the following scenarios:

  1. Mount Status Synchronization: Fluid parses the script output to confirm which UFS paths have been successfully mounted
  2. Dataset Status Update: Updates the Dataset's mount status and Phase based on the mount results
  3. Dynamic Mount Management: When the Dataset's mounts configuration changes, Fluid re-executes the MountUFS script and verifies whether the mount operation completed successfully through the output
  4. Remount Detection: In scenarios such as Master Pod restarts, Fluid determines whether remounting is necessary based on the output
Output Format Specification​

The standard output of the MountUFS script must be in the following JSON format:

{
"mounted": ["/path1", "/path2", "/path3"]
}

Where:

  • mounted: A string array containing the list of all successfully mounted UFS paths
  • If no paths are mounted, output: {"mounted": []}
Go Struct Definition​
type CacheRuntimeMountUfsOutput struct {
// Mounted are the ufs paths that have been mounted.
Mounted []string `json:"mounted,omitempty"`
}
Important Notes​
  1. Must output to standard output (stdout): Fluid reads JSON data from the script's standard output
  2. Error messages to standard error (stderr): Use >&2 to output error messages to stderr to avoid polluting stdout
  3. JSON format must strictly comply with requirements: Otherwise, Fluid cannot parse it, leading to mount failure

Step 2.8 ReportSummary Script Output Format Requirements​

The ReportSummary script is used to obtain runtime metrics of the cache system. It executes commands within the component's Pod, and the results are updated to the DataSet's Status field. The ReportSummary script must output JSON in CacheRuntimeReportSummary struct format, so that Fluid can correctly parse cache system metrics.

Output Format Specification​

The standard output of the ReportSummary script must be in the following JSON format:

{
"cached": "0.00B",
"cachedPercentage": "0",
"cacheCapacity": "4.00GiB",
"cacheHitRatio": "0",
"fileNum": "400",
"ufsTotal": "100GB"
}

Where:

  • cached: the amount of data cached, in bytes
  • cachedPercentage: percentage of cached data relative to cache capacity, 0-100
  • cacheCapacity: total cache data capacity (in bytes)
  • cacheHitRatio: cache hit ratio, 0-100
  • fileNum: number of files in the Dataset
  • ufsTotal: total size of the Dataset (in GB)
Important Notes​
  1. Must output to standard output (stdout): Fluid reads JSON data from the script's standard output
  2. Error messages to standard error (stderr): Use >&2 to output error messages to stderr to avoid polluting stdout
  3. JSON format must strictly comply with requirements: Otherwise, Fluid cannot parse it
  4. If the execution time of command is long, such as for the statistics of fileNum and ufsTotal, the script on the caching side should not obtain this information in real time