Bookkeeper Operator
Project status: alpha
The project is currently alpha. While no breaking API changes are currently planned, we reserve the right to address bugs and change the API before the project is declared stable.
Table of Contents
Overview
Bookkeeper A scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads.
The Bookkeeper Operator manages Bookkeeper clusters deployed to Kubernetes and automates tasks related to operating a Bookkeeper cluster.
- Create and destroy a Bookkeeper cluster
- Resize cluster
- Rolling upgrades
Requirements
- Kubernetes 1.9+
- Helm 2.10+
- An existing Apache Zookeeper 3.5 cluster. This can be easily deployed using our Zookeeper operator
Quickstart
Install the Operator
Note: If you are running on Google Kubernetes Engine (GKE), please check this first.
Use Helm to quickly deploy a Bookkeeper operator with the release name pravega-bk
.
$ helm install charts/bookkeeper-operator --name pr
Verify that the Bookkeeper Operator is running.
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
pr-bookkeeper-operator 1 1 1 1 17s
Install the Operator in Test Mode
The Operator can be run in test mode
if we want to deploy the Bookkeeper Cluster on minikube or on a cluster with very limited resources by setting testmode: true
in values.yaml
file. Operator running in test mode skips the minimum replica requirement checks. Test mode provides a bare minimum setup and is not recommended to be used in production environments.
Install a sample Bookkeeper cluster
Note that the Bookkeeper cluster must be installed in the same namespace as the Zookeeper cluster.
If the Bookkeeper cluster is expected to work with Pravega, we need to create a ConfigMap which needs to have the following values
KEY |
VALUE |
PRAVEGA_CLUSTER_NAME |
Name of Pravega Cluster using this BookKeeper Cluster |
WAIT_FOR |
Zookeeper URL |
The name of this ConfigMap needs to be mentioned in the field envVars
present in the BookKeeper Spec. For more details about this ConfigMap refer to this.
Helm can be used to install a sample Bookkeeper cluster.
$ helm install charts/bookkeeper --name pravega-bk --set zookeeperUri=[ZOOKEEPER_HOST]
where:
[ZOOKEEPER_HOST]
is the Zookeeper service endpoint of your Zookeeper deployment (e.g. zookeeper-client:2181
). It expects the zookeeper service URL in the given format <service-name>:<port-number>
Check out the Bookkeeper Helm Chart for more a complete list of installation parameters.
Verify that the cluster instances and its components are being created.
$ kubectl get bk
NAME VERSION DESIRED MEMBERS READY MEMBERS AGE
pravega-bk 0.7.0 3 1 25s
After a couple of minutes, all cluster members should become ready.
$ kubectl get bk
NAME VERSION DESIRED MEMBERS READY MEMBERS AGE
pravega-bk 0.7.0 3 3 2m
$ kubectl get all -l bookkeeper_cluster=pravega-bk
NAME READY STATUS RESTARTS AGE
pod/pravega-bk-bookie-0 1/1 Running 0 2m
pod/pravega-bk-bookie-1 1/1 Running 0 2m
pod/pravega-bk-bookie-2 1/1 Running 0 2m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/pravega-bk-bookie-headless ClusterIP None <none> 3181/TCP 2m
NAME DESIRED CURRENT AGE
statefulset.apps/pravega-bk-bookie 3 3 2m
By default, a BookkeeperCluster
is reachable using this kind of headless service URL for each pod:
http://pravega-bk-bookie-0.pravega-bk-bookie-headless.pravega-bk-bookie:3181
Scale a Bookkeeper cluster
You can scale Bookkeeper cluster by updating the replicas
field in the BookkeeperCluster Spec.
Example of patching the Bookkeeper Cluster resource to scale the server instances to 4.
kubectl patch bk pravega-bk --type='json' -p='[{"op": "replace", "path": "/spec/replicas", "value": 4}]'
Upgrade a Bookkeeper cluster
Check out the upgrade guide.
Uninstall the Bookkeeper cluster
$ helm delete pravega-bk --purge
Once the Bookkeeper cluster has been deleted, make sure to check that the zookeeper metadata has been cleaned up before proceeding with the deletion of the operator. This can be confirmed with the presence of the following log message in the operator logs.
zookeeper metadata deleted
However, if the operator fails to delete this metadata from zookeeper, you will instead find the following log message in the operator logs.
failed to cleanup <bookkeeper-cluster-name> metadata from zookeeper (znode path: /pravega/<pravega-cluster-name>): <error-msg>
The operator additionally sends out a ZKMETA_CLEANUP_ERROR
event to notify the user about this failure. The user can check this event by doing kubectl get events
. The following is the sample describe output of the event that is generated by the operator in such a case
Name: ZKMETA_CLEANUP_ERROR-nn6sd
Namespace: default
Labels: app=bookkeeper-cluster
bookkeeper_cluster=pravega-bk
Annotations: <none>
API Version: v1
Event Time: <nil>
First Timestamp: 2020-04-27T16:53:34Z
Involved Object:
API Version: app.k8s.io/v1beta1
Kind: Application
Name: bookkeeper-cluster
Namespace: default
Kind: Event
Last Timestamp: 2020-04-27T16:53:34Z
Message: failed to cleanup pravega-bk metadata from zookeeper (znode path: /pravega/pravega): failed to delete zookeeper znodes for (pravega-bk): failed to connect to zookeeper: lookup zookeeper-client on 10.100.200.2:53: no such host
Metadata:
Creation Timestamp: 2020-04-27T16:53:34Z
Generate Name: ZKMETA_CLEANUP_ERROR-
Resource Version: 864342
Self Link: /api/v1/namespaces/default/events/ZKMETA_CLEANUP_ERROR-nn6sd
UID: 5b4c3f80-36b5-43e6-b417-7992bc309218
Reason: ZK Metadata Cleanup Failed
Reporting Component: bookkeeper-operator
Reporting Instance: bookkeeper-operator-6769886978-xsjx6
Source:
Type: Error
Events: <none>
In case the operator fails to delete the zookeeper metadata, the user is expected to manually delete the metadata from zookeeper prior to reinstall.
Uninstall the Operator
Note that the Bookkeeper clusters managed by the Bookkeeper operator will NOT be deleted even if the operator is uninstalled.
$ helm delete pr --purge
Manual installation
You can also manually install/uninstall the operator and Bookkeeper with kubectl
commands. Check out the manual installation document for instructions.
Configuration
Check out the configuration document.
Development
Check out the development guide.
Releases
The latest Bookkeeper releases can be found on the Github Release project page.
Upgrade the Bookkeeper Operator
Bookkeeper operator can be upgraded by modifying the image tag using
$ kubectl edit <operator deployment name>