Use Curvine as CacheRuntime for Data Caching
This example demonstrates how to use Curvine as a CacheRuntime in Fluid to accelerate data access from object storage (e.g., S3/MinIO). Curvine is a high-performance, cloud-native distributed file system that integrates seamlessly with Fluid's CacheRuntime abstraction.
Prerequisites
Before running this example, make sure:
- Fluid with CacheRuntime support is installed on your Kubernetes cluster. Refer to the Installation Guide. The CacheRuntime controller is enabled with
--set runtime.cacheruntime.enabled=truewhen installing the Helm chart. - Your cluster has the AdvancedStatefulSet controller installed (Curvine uses AdvancedStatefulSet for its worker nodes).
- You have a MinIO or S3-compatible object storage service available.
Overview
The complete workflow consists of these steps:
- Deploy MinIO as the underlying object storage
- Create a Secret for MinIO credentials
- Define a
Datasetpointing to the S3 bucket - Define a
CacheRuntimeClasswith Curvine topology and components - Create a
CacheRuntimeto instantiate the Curvine cluster - Write data into the cache (via a job that writes to the mounted PVC)
- Run a DataLoad to preload data into Curvine
- Read data through the cached layer (faster access)
- (Optional) Use a reference Dataset for cross-namespace data sharing
For detailed documentation on how Curvine integrates with Fluid, including how Curvine parameters are parsed and used, refer to the Curvine Fluid Integration Guide.
Step 1: Deploy MinIO
Create a MinIO deployment to serve as the S3-compatible backend:
$ cat<<EOF >minio.yaml
apiVersion: v1
kind: Secret
metadata:
name: curvine-secret
stringData:
access-key: minioadmin
secret-key: minioadmin
---
apiVersion: v1
kind: Service
metadata:
name: minio
spec:
type: ClusterIP
ports:
- port: 9000
targetPort: 9000
protocol: TCP
selector:
app: minio
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: minio
spec:
selector:
matchLabels:
app: minio
strategy:
type: Recreate
template:
metadata:
labels:
app: minio
spec:
containers:
- name: minio
image: registry-cn-hongkong.ack.aliyuncs.com/acs/minio:RELEASE.2022-10-24T18-35-07Z-update
imagePullPolicy: IfNotPresent
resources:
limits:
memory: "512Mi"
args:
- server
- /data
env:
- name: MINIO_ROOT_USER
value: "minioadmin"
- name: MINIO_ROOT_PASSWORD
value: "minioadmin"
ports:
- containerPort: 9000
hostPort: 9000
EOF
$ kubectl create -f minio.yaml
Step 2: Create a MinIO Bucket
Curvine requires the target bucket to exist before mounting. Run a one-time job to create it:
$ cat<<EOF >minio_create_bucket.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: minio-bucket-create
spec:
template:
spec:
containers:
- name: create-bucket
image: curlimages/curl:8.22.0
imagePullPolicy: IfNotPresent
resources:
limits:
memory: "512Mi"
command:
- /bin/sh
- -c
- "curl -sSf -X PUT --aws-sigv4 aws:amz:us-east-1:s3 --user \$MINIO_ROOT_USER:\$MINIO_ROOT_PASSWORD http://minio:9000/test"
env:
- name: MINIO_ROOT_USER
value: "minioadmin"
- name: MINIO_ROOT_PASSWORD
value: "minioadmin"
restartPolicy: OnFailure
backoffLimit: 4
EOF
$ kubectl create -f minio_create_bucket.yaml
$ kubectl wait --for=condition=complete job/minio-bucket-create --timeout=120s