command module
v0.0.0-...-d70c95a Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jan 2, 2019 License: Apache-2.0 Imports: 14 Imported by: 0


= Crunchy Watch
Latest Release: 2.2.0 {docdate}


== Overview

Crunchy Watch is deprecated starting in January 2019.  For Kube/Openshift
HA/failover we recommend the Crunchy Postgres Operator project (

Crunchy Watch is an application that watches a PostgreSQL master
and looks for a failure, at which point it will perform
a failover scenario.

Failover scenarios are extensible.  Sample failover scenarios are
provided including:

 * trigger a failover on a random replica
 * trigger a failover on a replica using metadata labels
 * trigger a failover on a replica that is further ahead than others

Crunchy Watch is packaged into a Docker container which can execute in a pure
Docker 1.12, Kubernetes 1.9, and Openshift 3.11 environments.

You can also run Crunchy Watch outside of a container as a binary.

Crunchy provides a commercially supported version of this container built upon
RHEL 7 and the Crunchy supported PostgreSQL.  Contact Crunchy for more details
at link:

== Usage

Crunchy Watch is designed to operate on multiple platforms. Therefore, it is
necessary to specify the platform at startup.

`$> crunchy-watch <platform>`

Supported Platforms:

| Name       | Value
| Kubernetes | kube


`$> crunchy-watch kube`

=== Options

All options can be configured via a command-line flag or an environment variable.

Flag values will take precedence over values defined by an environment variable.

There are general options that apply across all platforms. As well, each
platform provides their own specific options. The details for each are provided

==== General

| Option | Environment Variable | Default | Description
| --primary
	| host of the primary PostgreSQL instance
| --primary-port
	| 5432
	| port of the primary PostreSQL instance
| --replica
	| host of the replica PostgreSQL instance
| --replica-port
	| 5432
	| port of the replica PostgreSQL instance
| --username
	| postgres
	| login user to connect to PostgreSQL
| --password
	| login user's password to connect to PostgreSQL
| --database
	| postgres
	| database to connect
| --target-type
        | pod
        | Failover tatget type can be POD or Deployment
| --timeout
	| 10s
	| connection timeout - valid time units are "ns", "us", "ms", "s", "m", "h"
| --max-failures
	| 0
	| maximum number of failures before performing a failover
| --healthcheck-interval
	| 10s
	| interval between healthchecks - valid time units are "ns", "us", "ms",
	"s", "m", "h"
| --failover-wait
	| 50s
	| time to wait for failover to process - valid time units are "ns", "us",
	"ms", "s", "m", "h"
| --pre-hook
	| failover hook to execute before processing failover
| --post-hook
	| failover hook to execute after processing failover
| --debug
	| when set to true, causes debug level messages to be output

==== Kubernetes

| Name | Environment Variable | Default | Description
| --kube-namespace
	| default
	| the kubernetes namespace
| --kube-failover-strategy
	| default
	| the kubernetes failover strategy

== Build

Building `crunchy-watch`, supporting plugin modules and docker image are
accomplished using `make` and the provide Makefile.

=== Requirements for Building from Source

 * Go 1.10 or greater
 * Docker 1.12 or greater

=== Centos Build Steps

These steps assume your normal userid is *someuser* and 
you are installing on a clean  minimal Centos7 install.

==== Install Docker

sudo yum -y install docker
sudo groupadd docker
sudo systemctl enable docker
sudo systemctl start docker
sudo usermod -a -G docker someuser
newgrp docker
docker ps

==== Install Build Dependencies

sudo yum -y install gettext git golang

==== Setup Project Settings and Structure

export GOPATH=$HOME/cdev
export PATH=$PATH:$GOPATH/bin
export CCP_IMAGE_PREFIX=crunchydata
export CCP_BASEOS=centos7
export CCP_VERSION=2.2.0
export WATCH_CLI=kubectl
export WATCH_ROOT=$GOPATH/src/
export WATCH_IMAGE_PREFIX=crunchydata
export WATCH_IMAGE_TAG=centos7-2.1.1

In the case of Openshift:
export WATCH_CLI=oc

Then, build the project structure as follows:
mkdir -p $GOPATH/src $GOPATH/bin $GOPATH/pkg
mkdir -p $GOPATH/src/
cd $GOPATH/src/
git clone
cd crunchy-watch
git checkout master

Configure storage for the Kube and Openshift examples by
setting the following environment variables:

For NFS:
export CCP_NFS_IP=
export CCP_STORAGE_MODE=ReadWriteMany
export CCP_SECURITY_CONTEXT='"supplementalGroups": [65534]'
export CCP_STORAGE_PATH=/nfsfileshare

For HostPath:
export CCP_STORAGE_MODE=ReadWriteMany
export CCP_STORAGE_PATH=/data

Create the demo namespace:
$ kubectl create -f $WATCH_ROOT/conf/demo-namespace.json
namespace "demo" created
$ kubectl get namespace demo
demo      Active    7s

Then set the namespace as the current location to avoid using the wrong namespace:
$ kubectl config set-context $(kubectl config current-context) --namespace=demo

==== Get Project Dependencies

make setup

==== Build from Source


==== Build the Docker Image

NOTE:  To build the RHEL based image, you will need the Crunchy
repo keys to be copied to the $GOPATH/src/ directory.   This is because the RHEL image is based on the 
Crunchy RPM packages.
cp CRUNCHY-GPG-KEY.public  $GOPATH/src/
cp crunchypg*.repo $GOPATH/src/

make docker-image

=== Targets

| Target | Description
| all
	| (*default*) calls `clean`, `resolve` and `build` targets
| build
	| builds `crunchy-watch` binary
| modules
	| builds all plugin modules
| kube-module
	| builds kubernetes plugin module
| clean
	| cleans all build related artifacts, including dependencies.
| resolve
	| resolves all build related dependencies
| docker-image
	| build docker image - *Note:* requires `CCP_BASEOS`, `CCP_PGVERSION`,
	`CCP_PG_FULLVERSION` and `CCP_VERSION` to be defined.
| setup
	| downloads required tools and docker image related dependencies

== Extending Crunchy Watch

Crunchy Watch is designed with extension of its function and supported
platforms in mind.

=== Extending by Plugin

Crunchy Watch makes use of the golang plugin package. Therefore it is possible
to build support for new platforms separate from each other.

To integrate with the plugin system the following interface must be met:

type FailoverHandler interface {
	Failover() error

`Failover()` is called to process the failover logic for the platform that the
plugin supports.

`SetFlags(*flag.FlagSet)` is called immediately after the plugin is loaded.
This allows for plugin to define options/flags that are unique to its

As well, it must be built with the `-buildmode=plugin` option. See an example
of this in the project link:Makefile[Makefile]

=== Extending by Hook

Crunchy Watch provides both a `pre` and `post` failover hook. These hooks will
be executed in a shell environment created by the `crunchy-watch` process.
Therefore they can be any executable or script that can be called by the user
running the `crunchy-watch` process.

To configure the execution of these hooks, a fully qualified path to the
executable or script must be provided by either the `--pre-hook` or
`--post-hook` flags.  Or by defining the `CRUNCHY_WATCH_PRE_HOOK` or
`CRUNCHY_WATCH_POST_HOOK` environment variables.


$> crunchy-watch kube --pre-hook=/tmp/watch-pre-hook


$> CRUNCHY_WATCH_PRE_HOOK=/tmp/watch-pre-hook crunchy-watch kube

== Examples

Crunchy-watch depends on an RBAC policy to be setup for the
service account it uses.  As a cluster-admin, you will need
to run the examples/ script a single time to create
the necessary service account with the correct RBAC roles.

. /home/some-normal-user/.bashrc
export PATH=$PATH:/home/some-normal-user/cdev/bin

Then as a normal user account, you can run the crunchy watch examples.

There are 2 primary examples for using crunchy-watch provided.  Both
examples work for both Kubernetes and Openshift environments.  Setting
the WATCH_CLI environment variable to *oc* for Openshift or *kubectl*
for Kubernetes is required to run the examples.

The first example has crunchy-watch watching 2 pods, a primary and
a replica pod.  Failover is performed on the primary pod.  

To run the pod example, first start up the sample pods:
cd examples/sample-pods

To run crunchy-watch for watching this set of pods, run:
cd examples/kube

To trigger a failover of the primary Pod to the replica Pod
enter the following:
$WATCH_CLI delete pod pr-primary
$WATCH_CLI logs watch --follow

To verify watch logs for the folowing:
ERRO[2018-09-06T13:38:50Z] Could not reach 'pr-primary' (Attempt: 1)
INFO[2018-09-06T13:38:50Z] Executing pre-hook: /hooks/watch-pre-hook
INFO[2018-09-06T13:38:50Z] Processing Failover: Strategy - latest
INFO[2018-09-06T13:38:50Z] Deleting existing primary...
INFO[2018-09-06T13:38:50Z] Deleted old primary
INFO[2018-09-06T13:38:50Z] Choosing failover replica...
INFO[2018-09-06T13:38:50Z] Chose failover target (pr-replica)
INFO[2018-09-06T13:38:50Z] Promoting failover replica...
DEBU[2018-09-06T13:38:50Z] executing cmd: [/opt/cpm/bin/] on pod pr-replica in namespace demo container: postgres
INFO[2018-09-06T13:38:50Z] Relabeling failover replica...
DEBU[2018-09-06T13:38:50Z] label: name
DEBU[2018-09-06T13:38:50Z] label: replicatype
INFO[2018-09-06T13:38:50Z] Executing post-hook: /hooks/watch-post-hook
INFO[2018-09-06T13:39:00Z] Health Checking: 'pr-primary'

To clean up the example:
cd $WATCH_ROOT/examples/sample-pods
cd $WATCH_ROOT/examples/kube

The 2nd example of crunchy-watch demonstrates failover of
a Deployment.  The sample Deployments used in the example
are started as follows:
cd $WATCH_ROOT/examples/sample-deployments

Run the crunchy-watch Deployment example as follows:
cd $WATCH_ROOT/examples/kube-deployments

To trigger a failover of the primary Deployment to the replica Deployment
enter the following:
$WATCH_CLI delete deploy watchprimary
$WATCH_CLI logs watch --follow

To verify watch the logs for:
INFO[2018-09-06T15:13:12Z] Health Checking: 'watchprimary'
ERRO[2018-09-06T15:13:22Z] dial tcp i/o timeout
ERRO[2018-09-06T15:13:22Z] Could not reach 'watchprimary' (Attempt: 1)
INFO[2018-09-06T15:13:22Z] Executing pre-hook: /hooks/watch-pre-hook
INFO[2018-09-06T15:13:22Z] Processing Failover: Strategy - latest
INFO[2018-09-06T15:13:22Z] Deleting existing primary...
INFO[2018-09-06T15:13:22Z] deleting deployment
WARN[2018-09-06T15:13:22Z] deployments.extensions "watchprimary" not found
INFO[2018-09-06T15:13:22Z] Deleted old primary
INFO[2018-09-06T15:13:22Z] Choosing failover replica...
INFO[2018-09-06T15:13:22Z] Chose failover target (watchreplica-56c48c7f4b-68fcb)
INFO[2018-09-06T15:13:22Z] Promoting failover replica...
DEBU[2018-09-06T15:13:22Z] executing cmd: [/opt/cpm/bin/] on pod watchreplica-56c48c7f4b-68fcb in namespace demo container: postgres
INFO[2018-09-06T15:13:22Z] Relabeling failover replica...
INFO[2018-09-06T15:14:28Z] Health Checking: 'watchprimary'
INFO[2018-09-06T15:14:28Z] Successfully reached 'watchprimary'

To clean up the example:
cd $WATCH_ROOT/examples/sample-deployments
cd $WATCH_ROOT/examples/kube-deployments

The examples on Openshift require the pg-watcher Service Account to 
have special priviledges, see the script for the 'oc adm' commands
required to grant those priviledges.  Customize this priviledge for your
local requirements.  


The Go Gopher

There is no documentation for this package.


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL