mongodb

Documentation

Overview

Package mongodb implements a storage provider conforming to the storage interface in kms-go. It is compatible with MongoDB v4.0.0, v4.2.8, and v5.0.0. It is also compatible with Amazon DocumentDB 4.0.0. It may be compatible with other versions, but they haven't been tested.

Index

• func PrepareDataForBSONStorage(value interface{}) (map[string]interface{}, error)
• func PrepareFilter(expressions []string, isJSONQuery bool) (bson.D, error)
• type Iterator
• type Option
  • func WithDBPrefix(dbPrefix string) Option
  • func WithLogger(logger logger) Option
  • func WithMaxRetries(maxRetries uint64) Option
  • func WithTimeBetweenRetries(timeBetweenRetries time.Duration) Option
  • func WithTimeout(timeout time.Duration) Option
• type Provider
  • func NewProvider(connString string, opts ...Option) (*Provider, error)
  • func (p *Provider) Close() error
  • func (p *Provider) CreateCustomIndexes(storeName string, models ...mongo.IndexModel) error
  • func (p *Provider) GetOpenStores() []storage.Store
  • func (p *Provider) GetStoreConfig(name string) (storage.StoreConfiguration, error)
  • func (p *Provider) OpenStore(name string) (storage.Store, error)
  • func (p *Provider) Ping() error
  • func (p *Provider) SetStoreConfig(storeName string, config storage.StoreConfiguration) error
• type Store
  • func (s *Store) Batch(operations []storage.Operation) error
  • func (s *Store) BulkWrite(models []mongo.WriteModel, opts ...*mongooptions.BulkWriteOptions) error
  • func (s *Store) Close() error
  • func (s *Store) CreateMongoDBFindOptions(options []storage.QueryOption, isJSONQuery bool) *mongooptions.FindOptions
  • func (s *Store) Delete(key string) error
  • func (s *Store) Flush() error
  • func (s *Store) Get(key string) ([]byte, error)
  • func (s *Store) GetAsRawMap(id string) (map[string]interface{}, error)
  • func (s *Store) GetBulk(keys ...string) ([][]byte, error)
  • func (s *Store) GetBulkAsRawMap(keys ...string) ([]map[string]interface{}, error)
  • func (s *Store) GetTags(key string) ([]storage.Tag, error)
  • func (s *Store) Put(key string, value []byte, tags ...storage.Tag) error
  • func (s *Store) PutAsJSON(key string, value interface{}) error
  • func (s *Store) Query(expression string, options ...storage.QueryOption) (storage.Iterator, error)
  • func (s *Store) QueryCustom(filter interface{}, options ...*mongooptions.FindOptions) (Iterator, error)

Functions

func PrepareDataForBSONStorage

func PrepareDataForBSONStorage(value interface{}) (map[string]interface{}, error)

PrepareDataForBSONStorage takes the given value and converts it to the type expected by the MongoDB driver for inserting documents. The value must be a struct with exported fields and proper json tags or a map. To use the MongoDB primary key (_id), you must have an _id field in either the struct or map. Alternatively, add it to the map returned by this function. If no _id field is set, then MongoDB will generate one for you.

func PrepareFilter

func PrepareFilter(expressions []string, isJSONQuery bool) (bson.D, error)

PrepareFilter converts the expression into a MongoDB filter.

Types

type Iterator

type Iterator interface {
	storage.Iterator

	ValueAsRawMap() (map[string]interface{}, error)
}

Iterator represents a MongoDB/DocumentDB implementation of the storage.Iterator interface.

type Option

type Option func(opts *Provider)

Option represents an option for a MongoDB Provider.

func WithDBPrefix

func WithDBPrefix(dbPrefix string) Option

WithDBPrefix is an option for adding a prefix to all created database names. No prefix will be used by default.

func WithLogger

func WithLogger(logger logger) Option

WithLogger is an option for specifying a custom logger. The standard Golang logger will be used by default.

func WithMaxRetries

func WithMaxRetries(maxRetries uint64) Option

WithMaxRetries is an option for specifying how many retries are allowed when there are certain transient errors from MongoDB. These transient errors can happen in two situations:

1. An index conflict error when setting indexes via the SetStoreConfig method from multiple MongoDB Provider objects that look at the same stores (which might happen if you have multiple running instances of a service).
2. If you're using MongoDB 4.0.0 (or DocumentDB 4.0.0), a "dup key" type of error when calling Store.Put or Store.Batch from multiple MongoDB Provider objects that look at the same stores.

maxRetries must be > 0. If not set (or set to an invalid value), it will default to 3.

func WithTimeBetweenRetries

func WithTimeBetweenRetries(timeBetweenRetries time.Duration) Option

WithTimeBetweenRetries is an option for specifying how long to wait between retries when there are certain transient errors from MongoDB. These transient errors can happen in two situations:

1. An index conflict error when setting indexes via the SetStoreConfig method from multiple MongoDB Provider objects that look at the same stores (which might happen if you have multiple running instances of a service).
2. If you're using MongoDB 4.0.0 (or DocumentDB 4.0.0), a "dup key" type of error when calling Store.Put or Store.Batch multiple times in parallel on the same key.

Defaults to two seconds if not set.

func WithTimeout

func WithTimeout(timeout time.Duration) Option

WithTimeout is an option for specifying the timeout for all calls to MongoDB. The timeout is 10 seconds by default.

type Provider

type Provider struct {
	// contains filtered or unexported fields
}

Provider represents a MongoDB/DocumentDB implementation of the storage.Provider interface.

func NewProvider

func NewProvider(connString string, opts ...Option) (*Provider, error)

NewProvider instantiates a new MongoDB Provider. connString is a connection string as defined in https://docs.mongodb.com/manual/reference/connection-string/. Note that options supported by the Go Mongo driver (and the names of them) may differ from the documentation above. Check the Go Mongo driver (go.mongodb.org/mongo-driver/mongo) to make sure the options you're specifying are supported and will be captured correctly. If using DocumentDB, the retryWrites option must be set to false in the connection string (retryWrites=false) in order for it to work.

func (*Provider) Close

func (p *Provider) Close() error

Close closes all stores created under this Store provider.

func (*Provider) CreateCustomIndexes

func (p *Provider) CreateCustomIndexes(storeName string, models ...mongo.IndexModel) error

CreateCustomIndexes allows for any custom indexes to be created in MongoDB based on the given index models. Intended for use alongside the Store.PutAsJSON, Store.GetAsRawMap and Iterator.ValueAsRawMap methods.

func (*Provider) GetOpenStores

func (p *Provider) GetOpenStores() []storage.Store

GetOpenStores returns all Stores currently open in this Provider.

func (*Provider) GetStoreConfig

func (p *Provider) GetStoreConfig(name string) (storage.StoreConfiguration, error)

GetStoreConfig gets the current Store configuration. If the underlying database for the given name has never been created by a call to OpenStore at some point, then an error wrapping ErrStoreNotFound will be returned. This method will not open a Store in the Provider. If name is blank, then an error will be returned.

func (*Provider) OpenStore

func (p *Provider) OpenStore(name string) (storage.Store, error)

OpenStore opens a Store with the given name and returns a handle. If the underlying database for the given name has never been created before, then it is created. Store names are not case-sensitive. If name is blank, then an error will be returned.

func (*Provider) Ping

func (p *Provider) Ping() error

Ping verifies whether the MongoDB client can successfully connect to the deployment specified by the connection string used in the NewProvider call.

func (*Provider) SetStoreConfig

func (p *Provider) SetStoreConfig(storeName string, config storage.StoreConfiguration) error

SetStoreConfig sets the configuration on a Store. Indexes are created based on the tag names in config. This allows the Store.Query method to operate faster. Existing tag names/indexes in the Store that are not in the config passed in here will be removed. The Store must already be open in this provider from a prior call to OpenStore. The name parameter cannot be blank.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store represents a MongoDB/DocumentDB implementation of the storage.Store interface.

func (*Store) Batch

func (s *Store) Batch(operations []storage.Operation) error

Batch performs multiple Put and/or Delete operations in order. If storing a JSON value, then any key names (within the JSON) cannot contain "`" characters. This is because we use it as a replacement for "." characters, which are not valid in DocumentDB as JSON key names. Put operations can be sped up by making use of the storage.PutOptions.IsNewKey option for any keys that you know for sure do not already exist in the database. If this option is used and the key does exist, then this method will return an error.

func (*Store) BulkWrite

func (s *Store) BulkWrite(models []mongo.WriteModel, opts ...*mongooptions.BulkWriteOptions) error

BulkWrite executes the mongoDB BulkWrite command using the given WriteModels and BulkWriteOptions.

func (*Store) Close

func (s *Store) Close() error

Close removes this Store from the parent Provider's list of open stores. It does not close this Store's connection to the database, since it's shared across stores. To close the connection you must call Provider.Close.

func (*Store) CreateMongoDBFindOptions

func (s *Store) CreateMongoDBFindOptions(options []storage.QueryOption, isJSONQuery bool) *mongooptions.FindOptions

CreateMongoDBFindOptions converts the given storage options into MongoDB options.

func (*Store) Delete

func (s *Store) Delete(key string) error

Delete deletes the value (and all tags) associated with key.

func (*Store) Flush

func (s *Store) Flush() error

Flush doesn't do anything since this store type doesn't queue values.

func (*Store) Get

func (s *Store) Get(key string) ([]byte, error)

Get fetches the value associated with the given key. If key cannot be found, then an error wrapping ErrDataNotFound will be returned. If key is empty, then an error will be returned.

func (*Store) GetAsRawMap

func (s *Store) GetAsRawMap(id string) (map[string]interface{}, error)

GetAsRawMap fetches the full MongoDB JSON document stored with the given id (_id field in MongoDB). The document is returned as a map (which includes the _id field).

func (*Store) GetBulk

func (s *Store) GetBulk(keys ...string) ([][]byte, error)

GetBulk fetches the values associated with the given keys. If no data exists under a given key, then a nil []byte is returned for that value. It is not considered an error. Depending on the implementation, this method may be faster than calling Get for each key individually. If any of the given keys are empty, then an error will be returned.

func (*Store) GetBulkAsRawMap

func (s *Store) GetBulkAsRawMap(keys ...string) ([]map[string]interface{}, error)

GetBulkAsRawMap fetches the values associated with the given keys and returns the documents (as maps). If no data exists under a given key, then nil is returned for that value. It is not considered an error. Depending on the implementation, this method may be faster than calling Get for each key individually. If any of the given keys are empty, then an error will be returned.

func (*Store) GetTags

func (s *Store) GetTags(key string) ([]storage.Tag, error)

GetTags fetches all tags associated with the given key. If key cannot be found, then an error wrapping ErrDataNotFound will be returned. If key is empty, then an error will be returned.

func (*Store) Put

func (s *Store) Put(key string, value []byte, tags ...storage.Tag) error

Put stores the key + value pair along with the (optional) tags. If tag values are valid int32 or int64, they will be stored as integers in MongoDB, so we can sort numerically later. If storing a JSON value, then any key names (within the JSON) cannot contain "`" characters. This is because we use it as a replacement for "." characters, which are not valid in DocumentDB as JSON key names.

func (*Store) PutAsJSON

func (s *Store) PutAsJSON(key string, value interface{}) error

PutAsJSON stores the given key and value. Value must be a struct with exported fields and proper json tags or a map. It will get marshalled before being converted to the format needed by the MongoDB driver. Value is stored directly in a MongoDB document without wrapping, with key being used as the _id field. Data stored this way must be retrieved using the GetAsRawMap method. When querying for this data, use the QueryCustom method, and when retrieving from the iterator use the iterator.ValueAsRawMap method.

func (*Store) Query

func (s *Store) Query(expression string, options ...storage.QueryOption) (storage.Iterator, error)

Query does a query for data as defined by the documentation in storage.Store (the interface). This implementation also supports querying for data tagged with multiple tag name + value pairs (using AND logic). To do this, separate the tag name + value pairs using &&. You can still omit one or both of the tag values in order to indicate that you want any data tagged with the tag name, regardless of tag value. For example, TagName1:TagValue1&&TagName2:TagValue2:...:TagNameN:TagValueN will return only data that has been tagged with all pairs. See testQueryWithMultipleTags in store_test.go for more examples of querying using multiple tags. If the tag you're using has tag values that are integers, then you can use the <, <=, >, >= operators instead of : to get a range of matching data. For example, TagName>3 will return any data tagged with a tag named TagName that has a value greater than 3. It's recommended to set up an index using the Provider.SetStoreConfig method in order to speed up queries. TODO (#146) Investigate compound indexes and see if they may be useful for queries with sorts and/or for queries

with multiple tags.

func (*Store) QueryCustom

func (s *Store) QueryCustom(filter interface{}, options ...*mongooptions.FindOptions) (Iterator, error)

QueryCustom queries for data using the MongoDB find command. The given filter and options are passed directly to the driver. Intended for use alongside the Provider.CreateCustomIndexes, Store.PutAsJSON, and Iterator.ValueAsRawMap methods.