cbidxr

package module
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 2, 2020 License: Apache-2.0 Imports: 6 Imported by: 0

README

go-cbidxr

Couchbase index reconciliation tools for golang

Concept

This is, in effect, a state comparison library. You are expected to construct the desire state of indices, which are then compared against what is actually present in the connected Couchbase cluster, and differences are reconciled.

Basic order of operations:

  1. Define desired index state
  2. Connect to Couchbase cluster
  3. Perform action decision logic
  4. Execute non-noop actions

1. Define desire index state

Your desired state is defined by creating one or more IndexDefinition type(s). The models within support being defined from JSON or HCL.

As an example:

package main

import (
    "github.com/dcarbone/go-cbidxr"
)

const (
    indexPrefix = "my-app-"
    bucketName = "mybucket"
)

var (
    indexDefinitions = []*cbidxr.IndexDefinition{
        {
            Name:         indexPrefix + "idx1",
            KeyspaceID:   bucketName,
            IndexKey:     []string{"`field1`", "`field2`"},
            Condition:    "`_type` = \"sandwiches\"",
            Using:        cbidxr.IndexDefaultUsing,
            NumReplica:   0,
            DeferBuild:   true,
            ForceRebuild: false,
        },
        {
            Name:         indexPrefix + "idx2",
            KeyspaceID:   bucketName,
            IndexKey:     []string{"`field3`", "`field4`"},
            Condition:    "`_type` != \"sandwiches\"",
            Using:        cbidxr.IndexDefaultUsing,
            NumReplica:   0,
            DeferBuild:   true,
            ForceRebuild: false,
        },
    }
)

You can, obviously, specify as many indices as you need. Given the drastic improvement in n1ql performance, I suggest using quite a few.

2. Connect to Couchbase cluster

This library is built on top of gocb/v2.

Follow its documentation for how to connect to your cluster

3. Perform action decision logic

As part of constructing a Reconciler, you must first construct a ReconcilerConfig type.

This type specifies a few key fields:

IndexLocatorFunc

Called to build the list of indices currently within Couchbase that you are interested in comparing your desired state against.

Provided implementations:

DecisionFunc

Called per current index that is found.

Provided implementations:

  • DefaultDecisionFunc - Performs a basic comparison to determine if the current index must be dropped / created / is equivalent to the desired state.
IndexCreateFunc

Called when an index in the desired state is not present or as the 2nd operation in a "Recreate" action

Provided implementations:

  • DefaultIndexCreateFunc - Executes simple create n1ql query with support for build deferring and num_replica value setting.
IndexDropFunc

Called when an index currently exists within Couchbase that is not in the desired state or as the first task during a "Recreate" action.

Provided implementations:

ActionListFinalizerFunc

Called once initial list of action decisions have been made, allowing you to make any final modifications before they are acted upon

Provided implementations:

4. Execute non-noop actions

Once the list of actions have been determined, any decision with an IndexAction other than IndexActionNoop are acted upon.

Documentation

Index

Constants

View Source 
const (
	IndexDefaultUsing = "gsi"

	DefaultListQueryFormat   = "SELECT `indexes`.* FROM system:indexes WHERE `name` LIKE \"%s%%\""
	DefaultCreateQueryFormat = "CREATE INDEX `%s` ON `%s`(%s) WHERE (%s) USING %s WITH {\"num_replica\": %d, \"defer_build\": %t}"
)
View Source 
const (
	IndexActionReasonNoop = "Index definitions match"

	IndexActionReasonDropExcess = "Index is not in desired state map"

	IndexActionReasonRecreateForced             = "Index set to forcibly rebuild"
	IndexActionReasonRecreateKeyspaceIDMismatch = "Index has mismatched KeyspaceID: current=%q; expected=%q"
	IndexActionReasonRecreateConditionMismatch  = "Index has mismatched Condition: current=%q; expected=%q"
	IndexActionReasonRecreateIndexKeysMismatch  = "Index has mismatched IndexKey: current=%v; expected=%v"

	IndexActionReasonCreate = "Index is missing from Couchbase"
)

Variables

This section is empty.

Functions

func DefaultIndexCreateFunc 

func DefaultIndexCreateFunc(cluster *gocb.Cluster, def IndexDefinition) (*gocb.QueryResult, error)

DefaultIndexCreateFunc is a basic implementation of an IndexCreateFunc that accounts for configurable replicas and build deferring

func DefaultIndexDropFunc 

func DefaultIndexDropFunc(cluster *gocb.Cluster, def IndexDefinition) error

DefaultIndexDropFunc is a very basic implementation of an IndexDropFunc that merely calls the upstream gocb/v2 built-in DropIndex func

Types

type ActionListFinalizerFunc 

type ActionListFinalizerFunc func(actions []*IndexActionDecision, defMap *IndexDefinitionMap) ([]*IndexActionDecision, error)

ActionListFinalizerFunc is called after all decisions have been made to allow the user to make any final edits before execution

type DecisionFunc 

type DecisionFunc func(current *IndexDefinition, defMap *IndexDefinitionMap) (*IndexActionDecision, error)

DecisionFunc is called per index currently present in Couchbase to determine what will happen to it.

type IndexAction 

type IndexAction string
const (
	IndexActionNoop     IndexAction = "noop"
	IndexActionDrop     IndexAction = "drop"
	IndexActionRecreate IndexAction = "recreate"
	IndexActionCreate   IndexAction = "create"
)

type IndexActionDecision 

type IndexActionDecision struct {
	// IndexName is the name of the index being acted upon
	IndexName string `json:"name"`

	// Action describes what will happen to this index
	Action IndexAction `json:"action"`

	// ActionReason should contain a descriptive reason for this decision
	ActionReason string `json:"action_reason"`

	// CurrentDefinition will be populated if an index with this name already exists within Couchbase
	CurrentDefinition *IndexDefinition `json:"current_definition"`

	// NewDefinition will be populated if an index with this name is present in the new DefinitionMap provided to the
	// parent Reconciler
	NewDefinition *IndexDefinition `json:"new_definition"`
}

IndexActionDecision describes the outcome of the DecisionFunc

func DefaultActionListFinalizerFunc 

func DefaultActionListFinalizerFunc(actions []*IndexActionDecision, _ *IndexDefinitionMap) ([]*IndexActionDecision, error)

DefaultActionListFinalizerFunc is a basic implementation of an ActionListFinalizerFunc that merely accepts whatever the initial decisions were

func DefaultDecisionFunc 

func DefaultDecisionFunc(current *IndexDefinition, defMap *IndexDefinitionMap) (*IndexActionDecision, error)

DefaultDecisionFunc attempts to determine if a current index's definition is sane, returning an action appropriate to its current state

type IndexCreateFunc 

type IndexCreateFunc func(cluster *gocb.Cluster, def IndexDefinition) (*gocb.QueryResult, error)

IndexCreateFunc is called whenever an index is missing from Couchbase

type IndexDefinition 

type IndexDefinition struct {
	// Name [required] - The of the index to create / match
	Name string `json:"name" hcl:"name"`

	// KeyspaceID [required] - The name of the bucket to attach this index to
	KeyspaceID string `json:"keyspace_id" hcl:"keyspace_id"`

	// IndexKey [suggested] - List of keys within document objects to used to build this index.  You are
	// responsible for escaping, i.e. []string{"`field1`", "`obj1`.`field2`"}
	IndexKey []string `json:"index_key" hcl:"index_key"`

	// Condition [optional] - An optional WHERE clause for this index
	Condition string `json:"condition" hcl:"condition"`

	// Using [optional] - Generally probably want GSI
	Using string `json:"using" hcl:"using"`

	// NumReplica [optional] - Number of index replicas to create.  Only available to Couchbase Enterprise customers.
	NumReplica uint `json:"num_replica" hcl:"num_replica"`

	// DeferBuild [suggested] - This effectively speeds up execution as all indices will not be populated until the end
	DeferBuild bool `json:"defer_build" hcl:"defer_build"`

	// ForceRebuild [optional] - Set to true if you want the decision to automatically be "recreate" for indices that
	// currently exist
	ForceRebuild bool `json:"force_rebuild" hcl:"force_rebuild"`
}

IndexDefinition is the representation of both current and desired indices

type IndexDefinitionMap 

type IndexDefinitionMap struct {
	// contains filtered or unexported fields
}

func NewIndexDefinitionMap 

func NewIndexDefinitionMap(defs ...*IndexDefinition) (*IndexDefinitionMap, error)

func (*IndexDefinitionMap) Clear 

func (dm *IndexDefinitionMap) Clear()

Clear nils out the map of registered indices. It is intended to be used after the sync process has finished.

func (*IndexDefinitionMap) Get 

func (dm *IndexDefinitionMap) Get(name string) (IndexDefinition, bool)

Get attempts to return a specific definition based on name

func (*IndexDefinitionMap) Map 

func (dm *IndexDefinitionMap) Map() map[string]IndexDefinition

Map returns all currently registered index definitions

func (*IndexDefinitionMap) NameList 

func (dm *IndexDefinitionMap) NameList() []string

NameList returns a list of all currently registered index definition names

func (*IndexDefinitionMap) Register 

func (dm *IndexDefinitionMap) Register(defs ...*IndexDefinition) error

Register validates and then adds the provided definition(s) the map

func (*IndexDefinitionMap) Unregister 

func (dm *IndexDefinitionMap) Unregister(name string) bool

Unregister attempts to remove a definition from the map, returning true if something was removed.

type IndexDropFunc 

type IndexDropFunc func(cluster *gocb.Cluster, def IndexDefinition) error

IndexDropFunc is called whenever an index is present in Couchbase that is no longer present in the map provided to a Reconciler instance

type IndexLocatorFunc 

type IndexLocatorFunc func(*gocb.Cluster) ([]*IndexDefinition, error)

IndexLocatorFunc is used to retrieve the list of indices of interest from Couchbase

func PrefixIndexLocatorFunc 

func PrefixIndexLocatorFunc(idxPrefix string, opts *gocb.QueryOptions) IndexLocatorFunc

PrefixIndexLocatorFunc constructs a IndexLocatorFunc that queries for a list of indices based on a common prefix

type ReconcileResults 

type ReconcileResults struct {
	InitialCount int      `json:"initial_count"`
	InitialNames []string `json:"initial_names"`

	CreatedCount int      `json:"created_count"`
	CreatedNames []string `json:"created_names"`

	DroppedCount int      `json:"dropped_count"`
	DroppedNames []string `json:"dropped_names"`

	NoopCount int      `json:"noop_count"`
	NoopNames []string `json:"noop_names"`

	FinalCount int      `json:"final_count"`
	FinalNames []string `json:"final_names"`

	ActionList []*IndexActionDecision `json:"action_list"`

	Err error `json:"err"`
}

func (ReconcileResults) Error 

func (r ReconcileResults) Error() string

func (ReconcileResults) String 

func (r ReconcileResults) String() string

type Reconciler 

type Reconciler struct {
	// contains filtered or unexported fields
}

Reconciler is a Couchbase bucket index management utility that enables developers to control their state in code.

No concurrency protection is done, control yourself.

func NewReconciler 

func NewReconciler(conf *ReconcilerConfig) (*Reconciler, error)

NewReconciler creates a new Reconciler instance based on the provided config

func (*Reconciler) Definitions 

func (rc *Reconciler) Definitions() *IndexDefinitionMap

func (*Reconciler) Execute 

func (rc *Reconciler) Execute() (*ReconcileResults, error)

func (*Reconciler) RegisterDefinitions 

func (rc *Reconciler) RegisterDefinitions(defs ...*IndexDefinition) error

type ReconcilerConfig 

type ReconcilerConfig struct {
	// Cluster [required] - Up and valid couchbase cluster connection
	Cluster *gocb.Cluster

	// IndexDefinitionMap [optional] - Optional index map to provide at time of construction.  If left empty, a new
	// instance will be created as part of creating a new Reconciler
	IndexDefinitionMap *IndexDefinitionMap

	// IndexLocatorFunc [required] - Func used to retrieve list of current indices of interest within Couchbase
	IndexLocatorFunc IndexLocatorFunc

	// DecisionFunc [required] - Func used to determine the action to perform on an index currently within Couchbase
	DecisionFunc DecisionFunc

	// IndexCreateFunc [required] - Func used whenever an index is to be created in Couchbase
	IndexCreateFunc IndexCreateFunc

	// IndexDropFunc [required] - Func used whenever an index is to be dropped from Couchbase
	IndexDropFunc IndexDropFunc

	// ActionListFinalizerFunc [optional] - Optional func to make any final edits to the list of actions to perform on
	// Couchbase before they are executed
	ActionListFinalizerFunc ActionListFinalizerFunc

	// Logger [optional] - If you want logging, define this.
	Logger *log.Logger

	// Debug [optional] - Set to true for debug-level logging
	Debug bool
}

func DefaultReconcilerConfig 

func DefaultReconcilerConfig(cluster *gocb.Cluster, idxLocator IndexLocatorFunc) *ReconcilerConfig

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.