Documentation ¶
Overview ¶
Package store provides an extensible, high-performance configuration management library, specially optimized for hierarchical data.
The Store interface gives these APIs.
The `hedzr/store` (https://github.com/hedzr/store) accesses tree data with a dotted key path, which means you may point to a specified tree node and access it, monitor it or remove it.
conf := store.New() conf.Set("app.debug", false) conf.Set("app.verbose", true) conf.Set("app.dump", 3) conf.Set("app.logging.file", "/tmp/1.log") conf.Set("app.server.start", 5) ss := conf.WithPrefix("app.logging") ss.Set("rotate", 6) ss.Set("words", []any{"a", 1, false}) ss.Set("keys", map[any]any{"a": 3.13, 1.73: "zz", false: true}) conf.Set("app.bool", "[on,off, true]") conf.SetComment("app.bool", "a bool slice", "remarks here") conf.SetTag("app.bool", []any{"on", "off", true}) states.Env().SetNoColorMode(true) // to disable ansi escape sequences in dump output fmt.Println(conf.Dump()) data, found := conf.Get("app.logging.rotate") println(data, found) data := conf.MustGet("app.logging.rotate") println(data)
The `store` provides advanced APIs to extract typed data from node.
iData := conf.MustInt("app.logging.rotate") debugMode := conf.MustBool("app.debug") ...
The searching tool is also used to locate whether a key exists or not:
found := conf.Has("app.logging.rotate") node, isBranch, isPartialMatched, found := conf.Locate("app.logging.rotate") t.Logf("%v | %s | %v | | %v, %v, found: %v", node.Data(), node.Comment(), node.Tag(), isBranch, isPartialMatched, found)
The `store` provides many providers and codecs. A provider represents an external data source, such as file, environment, consul, etc. And a codec represents the data format, just like yaml, json, toml, etc.
So an app can [Store.Load] the external yaml files like the following way:
func TestStoreS_Load(t *testing.T) { conf := newBasicStore(WithWatchEnable(true)) defer conf.Close() ctx := context.Background() parser := yaml.New() _, err := conf.Load(ctx, store.WithStorePrefix("app.yaml"), store.WithCodec(parser), store.WithProvider(file.New("../../../testdata/2.yaml")), store.WithStoreFlattenSlice(true), // decode and flatten slice into tree structure instead treat it as a simple value ) assert.Equal(t, `-s`, s.MustGet("app.yaml.app.bgo.build.projects.000-default-group.items.001-bgo.ldflags.0")) assert.Equal(t, `-w`, s.MustGet("app.yaml.app.bgo.build.projects.000-default-group.items.001-bgo.ldflags.1")) m := map[string]any{ "m1.s1": "cool", "m1.s2": 9, "key2": map[any]any{ 9: 1, 8: false, }, "slice": []map[any]any{ {7.981: true, "cool": "maps"}, {"hello": "world"}, }, } _, err := conf.Load(ctx, WithProvider(maps.New(m, ".")), WithStoreFlattenSlice(true), WithStorePrefix("app.maps"), WithPosition(""), ) if ErrorIsNotFound(err) { t.Fail() } if err != nil { t.Fatalf("err: %v", err) } t.Logf("\nPath of 'conf' (delimeter=%v, prefix=%v)\n%v\n", conf.Delimiter(), conf.Prefix(), conf.Dump()) assertEqual(t, false, conf.MustBool("app.maps.key2.8")) assertEqual(t, 1, conf.MustInt("app.maps.key2.9", -1)) assertEqual(t, "cool", conf.MustString("app.maps.m1.s1")) assertEqual(t, 9, conf.MustInt("app.maps.m1.s2", -1)) }
For more information, browse these public sites:
Index ¶
- Constants
- Variables
- func ErrorIsNotFound(err error) bool
- func NewDummyStore() *dummyS
- func WithFilter[T any](filter radix.FilterFn[T]) radix.MOpt[T]
- func WithKeepPrefix[T any](b bool) radix.MOpt[T]
- func WithoutFlattenKeys[T any](b bool) radix.MOpt[T]
- type Change
- type Codec
- type CodecEx
- type Dumpable
- type FallbackProvider
- type LoadOpt
- type MinimalStoreT
- type OnChangeHandler
- type OnDeleteHandler
- type OnNewHandler
- type OnceProvider
- type Op
- type Opt
- func WithDelimiter(delimiter rune) Opt
- func WithFlattenSlice(b bool) Opt
- func WithOnChangeHandlers(handlers ...OnChangeHandler) Opt
- func WithOnDeleteHandlers(handlers ...OnDeleteHandler) Opt
- func WithOnNewHandlers(handlers ...OnNewHandler) Opt
- func WithPrefix(prefix string) Opt
- func WithWatchEnable(b bool) Opt
- type Peripheral
- type Provider
- type ProviderSupports
- type Reader
- type Store
- type StreamProvider
- type ValPkg
- type Watchable
- type Writeable
Constants ¶
const Version = "v1.0.7" // Version of libs.store
Variables ¶
var ErrNotImplemented = stderr.New("not implemented")
ErrNotImplemented is used to identify unimplemented API.
Functions ¶
func ErrorIsNotFound ¶
ErrorIsNotFound checks if TypedGetters returning a NotFound error.
_, err := trie.GetFloat64("app.dump.") println(store.ErrorIsNotFound(err)) # this should be 'true'
If you don't care about these errors, use MustXXX such as radix.Trie.MustFloat64.
func NewDummyStore ¶
func NewDummyStore() *dummyS
NewDummyStore returns an empty store with dummy abilities implemented.
func WithFilter ¶
WithFilter can be used in calling GetM(path, ...)
func WithKeepPrefix ¶
WithKeepPrefix can construct tree nodes hierarchy with the key prefix.
By default, the prefix will be stripped from a given key path.
For example, if a store has a prefix 'app.server', `store.Put("app.server.tls", map[string]any{ "certs": "some/where.pem" }` will produce the tree structure like:
app. Server. tls. certs => "some/where.pem"
But if you enable keep-prefix setting, the code can only be written as:
store.Put("tls", map[string]any{ "certs": "some/where.pem" }
We recommend using our default setting except that you knew what you want. By using the default setting, i.e., keepPrefix == false, we will strip the may-be-there prefix if necessary. So both "app.server.tls" and "tls" will work properly as you really want.
Types ¶
type Change ¶
type Change interface { Next() (key string, val any, ok bool) Path() string // specially for 'file' provider Op() Op // Has(op Op) bool Timestamp() time.Time Provider() Provider }
Change is an abstract interface for Watchable object.
type Codec ¶
type Codec interface { Marshal(m map[string]any) (data []byte, err error) Unmarshal(b []byte) (data map[string]any, err error) }
Codec is decoder and/or encoder for text format.
For example, a file can be encoded with JSON format. So you need a JSON codec parser here.
Well-known codec parsers can be JSON, YAML, TOML, ....
type CodecEx ¶
type CodecEx interface { MarshalEx(m map[string]ValPkg) (data []byte, err error) UnmarshalEx(b []byte) (data map[string]ValPkg, err error) }
CodecEx reserved.
type Dumpable ¶
type Dumpable interface {
Dump() string
}
Dumpable interface identify an object can be represented as a string for debugging.
type FallbackProvider ¶
type FallbackProvider interface { Reader() (r Reader, err error) // return ErrNotImplemented as an identifier if it wants to be skipped ProviderSupports }
FallbackProvider reserved for future.
type LoadOpt ¶
type LoadOpt func(*loadS) // options for loadS
func WithProvider ¶
WithProvider is commonly required. It specify what Provider will be [storeS.Load].
func WithStoreFlattenSlice ¶
WithStoreFlattenSlice can destruct slice/map as tree hierarchy instead of treating it as a node value.
func WithStorePrefix ¶
WithStorePrefix gives a prefix position, which is the store location that the external settings will be merged at.
type MinimalStoreT ¶ added in v1.0.0
type MinimalStoreT[T any] interface { MustGet(path string) (data T) Get(path string) (data T, found bool) Set(path string, data T) (node radix.Node[T], oldData any) Has(path string) (found bool) }
MinimalStoreT holds a minimal typed Store interface.
func NewStoreT ¶
func NewStoreT[T any]() MinimalStoreT[T]
NewStoreT allows reimplementing your own Store.
Any suggestions are welcome, please issue me.
type OnChangeHandler ¶
OnChangeHandler is called back when user setting key & value.
mergingMapOrLoading is true means that user is setting key recursively with a map (via [Store.Merge]), or a loader (re-)loading its source.
func (*OnChangeHandler) GobDecode ¶ added in v1.0.1
func (*OnChangeHandler) GobDecode([]byte) error
func (OnChangeHandler) GobEncode ¶ added in v1.0.1
func (OnChangeHandler) GobEncode() ([]byte, error)
type OnDeleteHandler ¶
type OnDeleteHandler func(path string, value any, mergingMapOrLoading bool) // when user deleting a key
func (*OnDeleteHandler) GobDecode ¶ added in v1.0.1
func (*OnDeleteHandler) GobDecode([]byte) error
func (OnDeleteHandler) GobEncode ¶ added in v1.0.1
func (OnDeleteHandler) GobEncode() ([]byte, error)
type OnNewHandler ¶
type OnNewHandler func(path string, value any, mergingMapOrLoading bool) // when user setting a new key
func (*OnNewHandler) GobDecode ¶ added in v1.0.1
func (*OnNewHandler) GobDecode([]byte) error
func (OnNewHandler) GobEncode ¶ added in v1.0.1
func (OnNewHandler) GobEncode() ([]byte, error)
type OnceProvider ¶
type OnceProvider interface { ReadBytes() (data []byte, err error) // return ErrNotImplemented as an identifier if it wants to be skipped Write(data []byte) (err error) // return ErrNotImplemented as an identifier if it wants to be skipped ProviderSupports }
OnceProvider is fit for a small-scale provider.
The kv data will be all loaded into memory.
type Op ¶
type Op uint32 // Op describes a set of file operations.
const ( // OpCreate is a new pathname was created. OpCreate Op = 1 << iota // OpWrite the pathname was written to; this does *not* mean the write has finished, // and a write can be followed by more writes. OpWrite // OpRemove the path was removed; any watches on it will be removed. Some "remove" // operations may trigger a Rename if the file is actually moved (for // example "remove to trash" is often a rename). OpRemove // OpRename the path was renamed to something else; any watched on it will be // removed. OpRename // OpChmod file attributes were changed. // // It's generally not recommended to take action on this event, as it may // get triggered very frequently by some software. For example, Spotlight // indexing on macOS, anti-virus software, backup software, etc. OpChmod OpNone = 0 )
The operations fsnotify can trigger; see the documentation on [Watcher] for a full description, and check them with [Event.Has].
func (*Op) MarshalText ¶
func (*Op) UnmarshalText ¶
type Opt ¶
type Opt func(s *storeS) // Opt(ions) for New Store
func WithDelimiter ¶
WithDelimiter sets the delimiter char.
A delimiter char is generally used for extracting the key-value pair via GetXXX, MustXXX, e.g., MustInt, MustStringSlice, ....
func WithFlattenSlice ¶
WithFlattenSlice sets a bool flag to tell Store the slice value should be treated as node leaf. The index of the slice would be part of node path. For example, you're loading a slice []string{"A","B"} into node path "app.slice", the WithFlattenSlice(true) causes the following structure:
app.slice.0 => "A" app.slice.1 => "B"
Also, WithFlattenSlice makes the map values to be flattened into a tree.
func WithOnChangeHandlers ¶
func WithOnChangeHandlers(handlers ...OnChangeHandler) Opt
WithOnChangeHandlers allows user's handlers can be callback once a node changed.
func WithOnDeleteHandlers ¶
func WithOnDeleteHandlers(handlers ...OnDeleteHandler) Opt
WithOnDeleteHandlers allows user's handlers can be callback once a node removed.
func WithOnNewHandlers ¶
func WithOnNewHandlers(handlers ...OnNewHandler) Opt
WithOnNewHandlers allows user's handlers can be callback if a new node has been creating.
func WithPrefix ¶
WithPrefix sets the associated prefix for the tree path.
func WithWatchEnable ¶
WithWatchEnable allows watching the external source if its provider supports Watchable ability.
type Provider ¶
type Provider interface { Read() (m map[string]ValPkg, err error) // return ErrNotImplemented as an identifier if it wants to be skipped ProviderSupports }
The Provider gives a minimal set of interface to identify a data source.
The typical data sources are: consul, etcd, file, OS environ, ....
The interfaces are split to several groups: Streamable, Reader, Read, ReadBytes and Write.
A provider can implement just one of the above groups. At this time, the other interfaces should return ErrNotImplemented.
The Streamable API includes these: Keys, Count, Has, Next, Value and "MustValue". If you are implementing it, Keys, Value and Next are Must-Have. Because our kernel uses Keys to confirm the provider is Streamable, and invokes Next to iterate the key one by one. Once a key got, Value to get its associated value.
If the dataset is not very large scale, implementing Read is recommended to you. Read returns hierarchical data set as a nested `map[string]any` at once. Our kernel (loader) likes its simple logics.
Some providers may support Watchable API.
All providers should always accept Codec and Position and store them. When a provider monitored changes, storeS will request a reload action and these two Properties shall be usable.
Implementing OnceProvider.Write allows the provider to support Write-back mechanism.
type ProviderSupports ¶
type ProviderSupports interface { GetCodec() (codec Codec) // return the bound codec decoder GetPosition() (pos string) // return a position pointed to a Trie-node path WithCodec(codec Codec) WithPosition(pos string) }
ProviderSupports means which ability is supported by a Provider.
type Reader ¶
type Reader interface { Len() int // Len returns the number of bytes of the unread portion of the slice. // Size returns the original length of the underlying byte slice. // Size is the number of bytes available for reading via ReadAt. // The result is unaffected by any method calls except Reset. Size() int64 // Read implements the io.Reader interface. Read(b []byte) (n int, err error) // ReadAt implements the io.ReaderAt interface. ReadAt(b []byte, off int64) (n int, err error) // ReadByte implements the io.ByteReader interface. ReadByte() (byte, error) // UnreadByte complements ReadByte in implementing the io.ByteScanner interface. UnreadByte() error // ReadRune implements the io.RuneReader interface. ReadRune() (ch rune, size int, err error) // UnreadRune complements ReadRune in implementing the io.RuneScanner interface. UnreadRune() error // Seek implements the io.Seeker interface. Seek(offset int64, whence int) (int64, error) // WriteTo implements the io.WriterTo interface. WriteTo(w io.Writer) (n int64, err error) // Reset resets the Reader to be reading from b. Reset(b []byte) }
Reader reserved for future purpose.
type Store ¶
type Store interface { // Close cleanup the internal resources. // See [basics.Peripheral] for more information. Close() // MustGet is the shortcut version of Get without // returning any error. MustGet(path string) (data any) // Get the value at path point 'path'. Get(path string) (data any, found bool) // Set sets key('path') and value pair into storeS. Set(path string, data any) (node radix.Node[any], oldData any) // Remove a key and its children Remove(path string) (removed bool) // Merge a map at path point 'pathAt'. Merge(pathAt string, data map[string]any) (err error) // Has tests if the given path exists Has(path string) (found bool) // Locate provides an advanced interface for locating a path. Locate(path string) (node radix.Node[any], branch, partialMatched, found bool) radix.TypedGetters[any] // getters SetComment(path, description, comment string) (ok bool) // set extra meta-info bound to a key SetTag(path string, tags any) (ok bool) // set extra notable data bound to a key // Dump prints internal data tree for debugging Dump() (text string) // Clone makes a clone copy for this store Clone() (newStore *storeS) // Dup is a native Clone tool. // // After Dup, a copy of the original store will be created, // but closers not. // Most of the closers are cleanup code fragments coming // from Load(WithProvider()), some of them needs to shut down the // remote connection such as what want to do by consul provider. // // At this scene, the parent store still holds the cleanup closers. Dup() (newStore *storeS) // Walk does iterate the whole Store. // // Walk("") walks from top-level root node. // Walk("app") walks from the parent of "app" node. // Walk("app.") walks from the "app." node. Walk(path string, cb func(path, fragment string, node radix.Node[any])) // WithPrefix makes a lightweight copy from current storeS. // // The new copy is enough light so that you can always use // it with quite a low price. // // WithPrefix appends an extra prefix at the end of the current // prefix. // // For example, on a store with old prefix "app", // WithPrefix("store") will return a new store 'NS' with prefix // "app.server". And NS.MustGet("type") retrieve value at key path // "app.server.type" now. // // conf := store.New() // s1 := conf.WithPrefix("app") // ns := s1.WithPrefix("server") // println(ns.MustGet("type")) # print conf["app.server.type"] // // It simplify biz-logic codes sometimes. // // A [Delimiter] will be inserted at jointing prefix and key. Also at // jointing old and new prefix. WithPrefix(prefix ...string) (newStore *storeS) // todo need a balance on returning *storeS or Store, for WithPrefix // WithPrefixReplaced is similar with WithPrefix, but it replaces old // prefix with new one instead of appending it. // // conf := store.New() // s1 := conf.WithPrefix("app") // ns := s1.WithPrefixReplaced("app.server") // println(ns.MustGet("type")) # print conf["app.server.type"] // // A [Delimiter] will be inserted at jointing prefix and key. // // todo need a balance on returning *storeS or Store, for WithPrefixReplaced. WithPrefixReplaced(newPrefix ...string) (newStore *storeS) // SetPrefix updates the prefix in current storeS. SetPrefix(newPrefix ...string) Prefix() string // return current prefix string Delimiter() rune // return current delimiter, generally it's dot ('.') SetDelimiter(delimiter rune) // setter. Change it in runtime doesn't update old delimiter inside tree nodes. // Load loads k-v pairs from external provider(s) with specified codec decoder(s). // // For those provider which run some service at background, such // as watching service, ctx gives a change to shut them down // gracefully. So you need pass a cancellable context into it. // // Or you know nothing or you don't care the terminating security, // simply passing context.TODO() is okay. Load(ctx context.Context, opts ...LoadOpt) (wr Writeable, err error) // WithinLoading executes a functor with loading state. // // About the Store's loading state: // If it's in loading, the k-v pairs will be put into store with a clean // modified flag. WithinLoading(fn func()) }
Store holds a standard Store interface.
type StreamProvider ¶
type StreamProvider interface { Keys() (keys []string, err error) // return ErrNotImplemented as an identifier if it wants to be skipped Count() int // count of keys and/or key-value pairs Has(key string) bool // test if the key exists Next() (key string, eol bool) // return next usable key Value(key string) (value any, ok bool) // return the associated value MustValue(key string) (value any) // return the value, or nil for a non-existence key ProviderSupports }
StreamProvider is fit for a large-scale provider and load data on-demand.
type ValPkg ¶
type ValPkg struct { Value any // node's value Desc string // description of a node Comment string // comment of a node Tag any // any extra data of a node }
ValPkg is a value pack, It will be inserted into trie-tree as a data field. A node is commentable by Desc and Comment field.
type Watchable ¶
type Watchable interface { // Watch accepts user's func and callback it when the external // data source is changing, creating or deleting. // // The supported oprations are specified in Op. // // Tne user's func checks 'event' for which operation was occurring. // For more info, see also storeS.Load, storeS.applyExternalChanges, // and loader.startWatch. Watch(ctx context.Context, cb func(event any, err error)) error // Close provides a closer to cleanup the peripheral gracefully Close() }
Watchable tips that a Provider can watch its external data source