Back to

Package manipulate

Latest Go to latest

The latest major version is .

Published: Jul 14, 2020 | License: Apache-2.0 | Module:


Package manipulate provides everything needed to perform CRUD operations on an based data model.

The main interface is Manipulator. This interface provides various methods for creation, modification, retrieval and so on. TransactionalManipulator, which is an extension of the Manipulator add methods to manage transactions, like Commit and Abort.

A Manipulator works with some elemental.Identifiables.

The storage engine used by a Manipulator is abstracted. By default manipulate provides implementations for Rest API over HTTP or websocket, Mongo DB, Memory and a mock Manipulator for unit testing. You can of course create your own implementation.

Each method of a Manipulator is taking a manipulate.Context as argument. The context is used to pass additional informations like a Filter or some Parameters.

Example for creating an object:

// Create a User from a generated Elemental model.
user := models.NewUser()
user.FullName, user.Login := "Antoine Mercadal", "primalmotion"

// Create Mongo Manipulator.
m := manipmongo.NewMongoManipulator([]{""}, "test", "db-username", "db-password", "db-authsource", 512)

// Then create the User.
m.Create(nil, user)

Example for retreving an object:

// Create a Context with a filter.
ctx := manipulate.NewContextWithFilter(

// Retrieve the users matching the filter.
var users models.UserLists
m.RetrieveMany(ctx, models.UserIdentity, &users)


func IsCannotBuildQueryError

func IsCannotBuildQueryError(err error) bool

IsCannotBuildQueryError returns true if the given error is am ErrCannotBuildQuery.

func IsCannotCommitError

func IsCannotCommitError(err error) bool

IsCannotCommitError returns true if the given error is am ErrCannotCommit.

func IsCannotCommunicateError

func IsCannotCommunicateError(err error) bool

IsCannotCommunicateError returns true if the given error is am ErrCannotCommunicate.

func IsCannotExecuteQueryError

func IsCannotExecuteQueryError(err error) bool

IsCannotExecuteQueryError returns true if the given error is am ErrCannotExecuteQuery.

func IsCannotMarshalError

func IsCannotMarshalError(err error) bool

IsCannotMarshalError returns true if the given error is am ErrCannotMarshal.

func IsCannotUnmarshalError

func IsCannotUnmarshalError(err error) bool

IsCannotUnmarshalError returns true if the given error is am ErrCannotUnmarshal.

func IsConstraintViolationError

func IsConstraintViolationError(err error) bool

IsConstraintViolationError returns true if the given error is am ErrConstraintViolation.

func IsDisconnectedError

func IsDisconnectedError(err error) bool

IsDisconnectedError returns true if the given error is am ErrDisconnected.

func IsLockedError

func IsLockedError(err error) bool

IsLockedError returns true if the given error is am ErrLocked.

func IsMultipleObjectsFoundError

func IsMultipleObjectsFoundError(err error) bool

IsMultipleObjectsFoundError returns true if the given error is am ErrMultipleObjectsFound.

func IsNotImplementedError

func IsNotImplementedError(err error) bool

IsNotImplementedError returns true if the given error is am ErrNotImplemented.

func IsObjectNotFoundError

func IsObjectNotFoundError(err error) bool

IsObjectNotFoundError returns true if the given error is am ErrObjectNotFound.

func IsTLSError

func IsTLSError(err error) bool

IsTLSError returns true if the given error is am ErrTLS.

func IsTooManyRequestsError

func IsTooManyRequestsError(err error) bool

IsTooManyRequestsError returns true if the given error is am ErrTooManyRequests.

func IsTransactionNotFoundError

func IsTransactionNotFoundError(err error) bool

IsTransactionNotFoundError returns true if the given error is am ErrTransactionNotFound.

func Iter

func Iter(
	ctx context.Context,
	m Manipulator,
	mctx Context,
	identifiablesTemplate elemental.Identifiables,
	blockSize int,
) (elemental.Identifiables, error)

Iter is a helper function for IterFunc.

It will simply iterates on the object with identity of the given elemental.Identifiables. Not that this function cannot populate the data in the identifiable parameter. Instead It will return the destination.

Always pass an empty elemental.Identifiables to this function

For more information, please check IterFunc documentation.


dest, err := Iter(context.Background(), m, mctx, model.ThingsList{}, 100)

func IterFunc

func IterFunc(
	ctx context.Context,
	manipulator Manipulator,
	identifiablesTemplate elemental.Identifiables,
	mctx Context,
	iteratorFunc func(block elemental.Identifiables) error,
	blockSize int,
) error

IterFunc calls RetrieveMany on the given Manipulator, and will retrieve the data by block of the given blockSize.

IterFunc will naturally ends and return when there is no more data to pull.

For each retrieved block, the given func will be called with the current data block. If the function returns an error, the error is returned to the caller of IterFunc and the iteration stops.

The given context will be used if the underlying manipulator honors it. Be careful to NOT pass a filter matching objects then updating the objects to not match anynmore. This would shift pagination and will produce unexpected results. To do so, prefer using manipulate.IterUntilFunc

The given manipulate.Context will be used to retry any failed batch recovery.

The identifiablesTemplate parameter is must be an empty elemental.Identifiables that will be used to hold the data block. It is reset at every iteration. Do not rely on it to be filled once IterFunc is complete.

Finally, if the given blockSize is <= 0, then it will use the default that is 1000.

func IterUntilFunc

func IterUntilFunc(
	ctx context.Context,
	manipulator Manipulator,
	identifiablesTemplate elemental.Identifiables,
	mctx Context,
	iteratorFunc func(block elemental.Identifiables) error,
	blockSize int,
) error

IterUntilFunc works as IterFunc but pagination will not increase. It will always retrieve the first page with a size of given blockSize.

The goal of this function is to be used with a filter, then update (or delete) the objects that match until no more are matching.

func Retry

func Retry(ctx context.Context, manipulateFunc func() error, onRetryFunc func(int, error) error) error

Retry is deprecated and only calls manipulateFunc for backward compatibility.

type BufferedManipulator

type BufferedManipulator interface {

A BufferedManipulator is a Manipulator with a local cache

type Context

type Context interface {
	Count() int
	SetCount(count int)
	Filter() *elemental.Filter
	Finalizer() FinalizerFunc
	Version() int
	TransactionID() TransactionID
	Page() int
	PageSize() int
	After() string
	Limit() int
	Next() string
	Override() bool
	Recursive() bool
	Namespace() string
	Credentials() (string, string)
	Parameters() url.Values
	Parent() elemental.Identifiable
	ExternalTrackingID() string
	ExternalTrackingType() string
	Order() []string
	Context() context.Context
	Derive(...ContextOption) Context
	Fields() []string
	ReadConsistency() ReadConsistency
	WriteConsistency() WriteConsistency
	Messages() []string
	ClientIP() string
	RetryFunc() RetryFunc
	RetryRatio() int64


A Context holds all information regarding a particular manipulate operation.

func NewContext

func NewContext(ctx context.Context, options ...ContextOption) Context

NewContext creates a context with the given ContextOption.

type ContextOption

type ContextOption func(Context)

ContextOption represents an option can can be passed to NewContext.

func ContextOptionAfter

func ContextOptionAfter(from string, limit int) ContextOption

ContextOptionAfter sets the lazy pagination option of the context.

func ContextOptionClientIP

func ContextOptionClientIP(clientIP string) ContextOption

ContextOptionClientIP sets the optional headers for the request.

func ContextOptionCredentials

func ContextOptionCredentials(username, password string) ContextOption

ContextOptionCredentials sets user name and password for this context.

func ContextOptionFields

func ContextOptionFields(fields []string) ContextOption

ContextOptionFields sets the list of fields to include in the reply.

func ContextOptionFilter

func ContextOptionFilter(f *elemental.Filter) ContextOption

ContextOptionFilter sets the filter.

func ContextOptionFinalizer

func ContextOptionFinalizer(f FinalizerFunc) ContextOption

ContextOptionFinalizer sets the create finalizer option of the context.

func ContextOptionIdempotencyKey

func ContextOptionIdempotencyKey(key string) ContextOption

ContextOptionIdempotencyKey sets a custom idempotency key.

func ContextOptionNamespace

func ContextOptionNamespace(n string) ContextOption

ContextOptionNamespace sets the namespace.

func ContextOptionOpaque

func ContextOptionOpaque(o map[string]interface{}) ContextOption

ContextOptionOpaque sets a opaque data. Their interpretation depends on the manipulator implementation.

func ContextOptionOrder

func ContextOptionOrder(orders ...string) ContextOption

ContextOptionOrder sets the ordering option of the context.

func ContextOptionOverride

func ContextOptionOverride(o bool) ContextOption

ContextOptionOverride sets the override option of the context.

func ContextOptionPage

func ContextOptionPage(n, size int) ContextOption

ContextOptionPage sets the pagination option of the context.

func ContextOptionParameters

func ContextOptionParameters(p url.Values) ContextOption

ContextOptionParameters sets the parameters option of the context.

func ContextOptionParent

func ContextOptionParent(i elemental.Identifiable) ContextOption

ContextOptionParent sets the parent option of the context.

func ContextOptionReadConsistency

func ContextOptionReadConsistency(consistency ReadConsistency) ContextOption

ContextOptionReadConsistency sets the desired read consistency of the request.

func ContextOptionRecursive

func ContextOptionRecursive(r bool) ContextOption

ContextOptionRecursive sets the recursive option of the context.

func ContextOptionRetryFunc

func ContextOptionRetryFunc(f RetryFunc) ContextOption

ContextOptionRetryFunc sets the retry function. This function will be called on every communication error, and will be passed the try number and the error. If it itself return an error, retrying will stop and that error will be returned from the manipulator operation.

func ContextOptionRetryRatio

func ContextOptionRetryRatio(r int64) ContextOption

ContextOptionRetryRatio sets the retry ratio.

RetryRatio divides the remaining time unitl context deadline to perfrom single retry query down to a minimum defined by manipulator implementations (typically 20s). The default value is 4.

For example if the context has a timeout of 2m, each retry will use a sub context with a timeout of 30s.

func ContextOptionToken

func ContextOptionToken(token string) ContextOption

ContextOptionToken sets the token for this request.

func ContextOptionTracking

func ContextOptionTracking(identifier, typ string) ContextOption

ContextOptionTracking sets the opentracing tracking option of the context.

func ContextOptionTransactionID

func ContextOptionTransactionID(tid TransactionID) ContextOption

ContextOptionTransactionID sets the parameters option of the context.

func ContextOptionVersion

func ContextOptionVersion(v int) ContextOption

ContextOptionVersion sets the version option of the context.

func ContextOptionWriteConsistency

func ContextOptionWriteConsistency(consistency WriteConsistency) ContextOption

ContextOptionWriteConsistency sets the desired write consistency of the request.

type ErrCannotBuildQuery

type ErrCannotBuildQuery struct {
	// contains filtered or unexported fields

ErrCannotBuildQuery represents query building error.

func NewErrCannotBuildQuery

func NewErrCannotBuildQuery(message string) ErrCannotBuildQuery

NewErrCannotBuildQuery returns a new ErrCannotBuildQuery.

func (ErrCannotBuildQuery) Error

func (e ErrCannotBuildQuery) Error() string

type ErrCannotCommit

type ErrCannotCommit struct {
	// contains filtered or unexported fields

ErrCannotCommit represents commit execution error.

func NewErrCannotCommit

func NewErrCannotCommit(message string) ErrCannotCommit

NewErrCannotCommit returns a new ErrCannotCommit.

func (ErrCannotCommit) Error

func (e ErrCannotCommit) Error() string

type ErrCannotCommunicate

type ErrCannotCommunicate struct {
	// contains filtered or unexported fields

ErrCannotCommunicate represents a failure in backend communication.

func NewErrCannotCommunicate

func NewErrCannotCommunicate(message string) ErrCannotCommunicate

NewErrCannotCommunicate returns a new ErrCannotCommunicate.

func (ErrCannotCommunicate) Error

func (e ErrCannotCommunicate) Error() string

type ErrCannotExecuteQuery

type ErrCannotExecuteQuery struct {
	// contains filtered or unexported fields

ErrCannotExecuteQuery represents query execution error.

func NewErrCannotExecuteQuery

func NewErrCannotExecuteQuery(message string) ErrCannotExecuteQuery

NewErrCannotExecuteQuery returns a new ErrCannotExecuteQuery.

func (ErrCannotExecuteQuery) Error

func (e ErrCannotExecuteQuery) Error() string

type ErrCannotMarshal

type ErrCannotMarshal struct {
	// contains filtered or unexported fields

ErrCannotMarshal represents marshaling error.

func NewErrCannotMarshal

func NewErrCannotMarshal(message string) ErrCannotMarshal

NewErrCannotMarshal returns a new ErrCannotMarshal.

func (ErrCannotMarshal) Error

func (e ErrCannotMarshal) Error() string

type ErrCannotUnmarshal

type ErrCannotUnmarshal struct {
	// contains filtered or unexported fields

ErrCannotUnmarshal represents unmarshaling error.

func NewErrCannotUnmarshal

func NewErrCannotUnmarshal(message string) ErrCannotUnmarshal

NewErrCannotUnmarshal returns a new ErrCannotUnmarshal.

func (ErrCannotUnmarshal) Error

func (e ErrCannotUnmarshal) Error() string

type ErrConstraintViolation

type ErrConstraintViolation struct {
	// contains filtered or unexported fields

ErrConstraintViolation represents a failure to find a transaction.

func NewErrConstraintViolation

func NewErrConstraintViolation(message string) ErrConstraintViolation

NewErrConstraintViolation returns a new ErrConstraintViolation.

func (ErrConstraintViolation) Error

func (e ErrConstraintViolation) Error() string

type ErrDisconnected

type ErrDisconnected struct {
	// contains filtered or unexported fields

ErrDisconnected represents an error due user disconnection.

func NewErrDisconnected

func NewErrDisconnected(message string) ErrDisconnected

NewErrDisconnected returns a new ErrDisconnected.

func (ErrDisconnected) Error

func (e ErrDisconnected) Error() string

type ErrLocked

type ErrLocked struct {
	// contains filtered or unexported fields

ErrLocked represents the error returned when the server api is locked..

func NewErrLocked

func NewErrLocked(message string) ErrLocked

NewErrLocked returns a new ErrCannotCommunicate.

func (ErrLocked) Error

func (e ErrLocked) Error() string

type ErrMultipleObjectsFound

type ErrMultipleObjectsFound struct {
	// contains filtered or unexported fields

ErrMultipleObjectsFound represents too many object found error.

func NewErrMultipleObjectsFound

func NewErrMultipleObjectsFound(message string) ErrMultipleObjectsFound

NewErrMultipleObjectsFound returns a new ErrMultipleObjectsFound.

func (ErrMultipleObjectsFound) Error

func (e ErrMultipleObjectsFound) Error() string

type ErrNotImplemented

type ErrNotImplemented struct {
	// contains filtered or unexported fields

ErrNotImplemented represents a non implemented function.

func NewErrNotImplemented

func NewErrNotImplemented(message string) ErrNotImplemented

NewErrNotImplemented returns a new ErrNotImplemented.

func (ErrNotImplemented) Error

func (e ErrNotImplemented) Error() string

type ErrObjectNotFound

type ErrObjectNotFound struct {
	// contains filtered or unexported fields

ErrObjectNotFound represents object not found error.

func NewErrObjectNotFound

func NewErrObjectNotFound(message string) ErrObjectNotFound

NewErrObjectNotFound returns a new ErrObjectNotFound.

func (ErrObjectNotFound) Error

func (e ErrObjectNotFound) Error() string

type ErrTLS

type ErrTLS struct {
	// contains filtered or unexported fields

ErrTLS represents the error returned when there is a TLS error.

func NewErrTLS

func NewErrTLS(message string) ErrTLS

NewErrTLS returns a new ErrTLS.

func (ErrTLS) Error

func (e ErrTLS) Error() string

type ErrTooManyRequests

type ErrTooManyRequests struct {
	// contains filtered or unexported fields

ErrTooManyRequests represents the error returned when the server api is locked.

func NewErrTooManyRequests

func NewErrTooManyRequests(message string) ErrTooManyRequests

NewErrTooManyRequests returns a new ErrTooManyRequests.

func (ErrTooManyRequests) Error

func (e ErrTooManyRequests) Error() string

type ErrTransactionNotFound

type ErrTransactionNotFound struct {
	// contains filtered or unexported fields

ErrTransactionNotFound represents a failure to find a transaction.

func NewErrTransactionNotFound

func NewErrTransactionNotFound(message string) ErrTransactionNotFound

NewErrTransactionNotFound returns a new ErrTransactionNotFound.

func (ErrTransactionNotFound) Error

func (e ErrTransactionNotFound) Error() string

type Filter

type Filter = elemental.Filter

Filter is an alias of elemental.Filter

func NewFilter

func NewFilter() *Filter

NewFilter returns a new Filter using the aliased type.

func NewFilterFromString

func NewFilterFromString(filter string) (*Filter, error)

NewFilterFromString returns a new NewFilterFromString using the aliased type.

type FilterKeyComposer

type FilterKeyComposer = elemental.FilterKeyComposer

FilterKeyComposer is an alias of elemental.FilterKeyComposer

func NewFilterComposer

func NewFilterComposer() FilterKeyComposer

NewFilterComposer returns a new FilterKeyComposer using the aliased type.

type FilterParser

type FilterParser = elemental.FilterParser

FilterParser is an alias of elemental.FilterParser

func NewFilterParser

func NewFilterParser(input string) *FilterParser

NewFilterParser returns a new NewFilterParser using the aliased type.

type FinalizerFunc

type FinalizerFunc func(o elemental.Identifiable) error

A FinalizerFunc is the type of a function that can be used as a creation finalizer. This is only supported by manipulators that generate an ID to let a chance to the user. to now the intended ID before actually creating the object.

type FlushableManipulator

type FlushableManipulator interface {

	// Flush flushes and empties the cache.
	Flush(ctx context.Context) error

A FlushableManipulator is a manipulator that can flush its content to somewhere, like a file.

type Manipulator

type Manipulator interface {

	// RetrieveMany retrieves the a list of objects with the given elemental.Identity and put them in the given dest.
	RetrieveMany(mctx Context, dest elemental.Identifiables) error

	// Retrieve retrieves one or multiple elemental.Identifiables.
	// In order to be retrievable, the elemental.Identifiable needs to have their Identifier correctly set.
	Retrieve(mctx Context, object elemental.Identifiable) error

	// Create creates a the given elemental.Identifiables.
	Create(mctx Context, object elemental.Identifiable) error

	// Update updates one or multiple elemental.Identifiables.
	// In order to be updatable, the elemental.Identifiable needs to have their Identifier correctly set.
	Update(mctx Context, object elemental.Identifiable) error

	// Delete deletes one or multiple elemental.Identifiables.
	// In order to be deletable, the elemental.Identifiable needs to have their Identifier correctly set.
	Delete(mctx Context, object elemental.Identifiable) error

	// DeleteMany deletes all objects of with the given identity or
	// all the ones matching the filter in the given context.
	DeleteMany(mctx Context, identity elemental.Identity) error

	// Count returns the number of objects with the given identity.
	Count(mctx Context, identity elemental.Identity) (int, error)

Manipulator is the interface of a storage backend.

type ReadConsistency

type ReadConsistency string

ReadConsistency represents the desired consistency of the request. Not all driver may implement this.

const (
	ReadConsistencyDefault   ReadConsistency = "default"
	ReadConsistencyNearest   ReadConsistency = "nearest"
	ReadConsistencyEventual  ReadConsistency = "eventual"
	ReadConsistencyMonotonic ReadConsistency = "monotonic"
	ReadConsistencyStrong    ReadConsistency = "strong"

Various values for Consistency

type RetryFunc

type RetryFunc func(RetryInfo) error

A RetryFunc is a function that can be called during an auto retry. The current manipulate.Context is given, a Stringer interface containing, more info about the current request, the error that caused the retry and the try number. If this function returns an error, the retry procedure will be interupted and this error will be returns to the caller of the operation.

type RetryInfo

type RetryInfo interface {
	Err() error
	Context() Context
	Try() int

A RetryInfo is the interface that can be passed to RetryFunc that will contain retry information. Content will depend on the manipulator implementation.

type Subscriber

type Subscriber interface {

	// Start connects to the websocket and starts collecting events
	// until the given context is canceled or any non communication error is
	// received. The eventual error will be received in the Errors() channel.
	// If not nil, the given push config will be applied right away.
	Start(context.Context, *elemental.PushConfig)

	// UpdateFilter updates the current push config.

	// Events returns the events channel.
	Events() chan *elemental.Event

	// Errors returns the errors channel.
	Errors() chan error

	// Status returns the status channel.
	Status() chan SubscriberStatus

A Subscriber is the interface to control a push event subscription.

type SubscriberStatus

type SubscriberStatus int

SubscriberStatus is the type of a subscriber status.

const (
	SubscriberStatusInitialConnection SubscriberStatus = iota + 1

Various values of SubscriberEvent.

type TokenManager

type TokenManager interface {

	// Issues isses a new token.
	Issue(context.Context) (string, error)

	// Run runs the token renewal job and published the new token in the
	// given channel.
	Run(ctx context.Context, tokenCh chan string)

A TokenManager issues an renew tokens periodically.

type TransactionID

type TransactionID string

TransactionID is the type used to define a transcation ID of a store

func NewTransactionID

func NewTransactionID() TransactionID

NewTransactionID returns a new transaction ID.

type TransactionalManipulator

type TransactionalManipulator interface {

	// Commit commits the given TransactionID.
	Commit(id TransactionID) error

	// Abort aborts the give TransactionID. It returns true if
	// a transaction has been effectively aborted, otherwise it returns false.
	Abort(id TransactionID) bool


A TransactionalManipulator is a Manipulator that handles transactions.

type WriteConsistency

type WriteConsistency string

WriteConsistency represents the desired consistency of the request. Not all driver may implement this.

const (
	WriteConsistencyDefault   WriteConsistency = "default"
	WriteConsistencyNone      WriteConsistency = "none"
	WriteConsistencyStrong    WriteConsistency = "strong"
	WriteConsistencyStrongest WriteConsistency = "strongest"

Various values for Consistency

Package Files

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to identifier