risingwave-operator

README

RisingWave Operator

Introduction

The RisingWave operator is a deployment and management system of the RisingWave streaming database that runs on top of Kubernetes. It provides functionalities like provisioning, upgrading, scaling and destroying the RisingWave instances inside the Kubernetes cluster. It models the deployment and management progress with the concepts provided in Kubernetes and organizes them in a way called Operator Pattern. Thus, we can just declare what kind of RisingWave instances we want and create them as objects in the Kubernetes. The RisingWave operator will always make sure that they are finally there.

The operator also contains several custom resources. Refer to the API docs for more details.

Quick Start

Installation

First, you need to install the cert-manager in the cluster before installing the risingwave-operator.

The default static configuration cert-manager can be installed as follows:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

More information on the installation of the cert-manager can be found here.

Afterward, you can install the risingwave-operator with the following command:

kubectl apply --server-side -f https://github.com/risingwavelabs/risingwave-operator/releases/latest/download/risingwave-operator.yaml

To check if the installation is successful, you can run the following commands to check if the Pods are all running.

kubectl -n cert-manager get pods
kubectl -n risingwave-operator-system get pods

NOTE: Don't worry if you encountered the following errors:

Error from server (InternalError): error when creating "https://github.com/risingwavelabs/risingwave-operator/releases/latest/download/risingwave-operator.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.111.35.75:443: connect: connection refused
Error from server (InternalError): error when creating "https://github.com/risingwavelabs/risingwave-operator/releases/latest/download/risingwave-operator.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": Post "https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s": dial tcp 10.111.35.75:443: connect: connection refused

The errors are because of the initializing cert-manager. Just hold tight and wait for another minute, and re-apply the risingwave-operator.yaml above.

First RisingWave Instance

Now you can deploy a RisingWave instance with in-memory storage with the following command (be careful about the node arch):

kubectl apply -f https://raw.githubusercontent.com/risingwavelabs/risingwave-operator/main/docs/manifests/risingwave/risingwave-in-memory.yaml

Check the running status of RisingWave with the following command:

kubectl get risingwave

The expected output is like this:

NAME                    RUNNING   STORAGE(META)   STORAGE(OBJECT)   AGE
risingwave-in-memory    True      Memory          Memory            30s

If you find the risingwave-in-memory RUNNING filed is false, please type kubectl get pods. If you find some pods status is ImagePullBackOff rather than Running, the problem may result from your network.

Connect & Query

ClusterIP

By default, the operator will create a service for the frontend component, with the type of ClusterIP if not specified. It is not accessible from the outside. So we will create a standalone Pod of PostgreSQL inside the Kubernetes, which runs an infinite loop so that we can attach to it.

You can create one by following the commands below, or you can just do it yourself:

kubectl apply -f docs/manifests/psql/psql-console.yaml

And then you will find a Pod named psql-console running in the Kubernetes, and you can attach to it to execute commands inside the container with the following command:

kubectl exec -it psql-console bash

Finally, we can get access to the RisingWave with the psql command inside the Pod:

psql -h risingwave-in-memory-frontend -p 4567 -d dev -U root

NodePort

If you want to connect to the RisingWave from the nodes (e.g., EC2) in the Kubernetes, you can set the service type to NodePort, and run the following commands on the node:

export RISINGWAVE_NAME=risingwave-in-memory
export RISINGWAVE_NAMESPACE=default
export RISINGWAVE_HOST=`kubectl -n ${RISINGWAVE_NAMESPACE} get node -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'`
export RISINGWAVE_PORT=`kubectl -n ${RISINGWAVE_NAMESPACE} get svc -l risingwave/name=${RISINGWAVE_NAME},risingwave/component=frontend -o jsonpath='{.items[0].spec.ports[0].nodePort}'`

psql -h ${RISINGWAVE_HOST} -p ${RISINGWAVE_PORT} -d dev -U root

# ...
spec:
  global:
    serviceType: NodePort
# ...

LoadBalancer

For EKS/GKE and some other Kubernetes services provided by cloud vendors, we can expose the Service to the public network with a load balancer on the cloud. We can simply achieve this by setting the service type to LoadBalancer, by setting the following field:

# ...
spec:
  global:
    serviceType: LoadBalancer
# ...

And then you can connect to the RisingWave with the following command:

export RISINGWAVE_NAME=risingwave-in-memory
export RISINGWAVE_NAMESPACE=default
export RISINGWAVE_HOST=`kubectl -n ${RISINGWAVE_NAMESPACE} get svc -l risingwave/name=${RISINGWAVE_NAME},risingwave/component=frontend -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}'`
export RISINGWAVE_PORT=`kubectl -n ${RISINGWAVE_NAMESPACE} get svc -l risingwave/name=${RISINGWAVE_NAME},risingwave/component=frontend -o jsonpath='{.items[0].spec.ports[0].port}'`

psql -h ${RISINGWAVE_HOST} -p ${RISINGWAVE_PORT} -d dev -U root

Storages

Memory

Currently, memory storage is supported for test usage only. We highly discourage you use the memory storage for other purposes. For now, you can enable the memory metadata and object storage with the following configs:

#...
spec:
  storages:
    meta:
      memory: true
    object:
      memory: true
#...

etcd (meta)

We recommend using the etcd to store the metadata. You can specify the connection information of the etcd you'd like to use like the following:

#...
spec:
  storages:
    meta:
      etcd: 
        endpoint: risingwave-etcd:2388
        secret: etcd-credentials      # optional, empty means no authentication 
#...

Check the docs/manifests/risingwave/risingwave-etcd-minio.yaml for how to provision a simple RisingWave with an etcd instance as the metadata storage.

MinIO

We support using MinIO as the object storage. Check the docs/manifests/risingwave/risingwave-etcd-minio.yaml for details. The YAML structure is like the following:

#...
spec:
  storages:
    object:
      minio:
        secret: minio-credentials
        endpoint: minio-endpoint:2388
        bucket: hummock001
#...

S3

We support using AWS S3 as the object storage. Follow the steps below and check the docs/manifests/risingwave/risingwave-etcd-s3.yaml for details:

First, you need to create a Secret with the name s3-credentials:

kubectl create secret generic s3-credentials --from-literal AccessKeyID=${ACCESS_KEY} --from-literal SecretAccessKey=${SECRET_ACCESS_KEY} --from-literal Region=${AWS_REGION}

Then, you need to create a bucket on the console, e.g., hummock001.

Finally, you can specify S3 as the object storage in YAML, like the following:

#...
spec:
  storages:
    object:
      s3:
        secret: s3-credentials
        bucket: hummock001
#...

Monitoring

For monitoring the RisingWaves running in Kubernetes, please refer to the Monitoring Guidance for more details.

License

The RisingWave operator is developed under the Apache License 2.0. Please refer to LICENSE for more information.

Contributing

Thanks for your interest in contributing to the project! Please refer to the Contribution and Development Guidelines for more information.