README
¶
ferry/kv
Load configuration from a Consul-shaped key-value store into a Go struct, and write it back.
Experimental. Neither the Go API nor the way values are stored is settled yet, and either may change in a release that is not a new major version of this module. The reason is that this is Consul-shaped rather than Consul: the only
Clientin this repository is a test fake, so nothing here has been run against a real store, and the interface below is read off a specification rather than confirmed against a backend.
go get github.com/onhotpath/ferry/driver/kv
Bring your own client
There is no Consul, etcd or Redis dependency here. You implement four methods and the package works against whatever you have:
type Client interface {
Get(ctx context.Context, key string) (value []byte, found bool, err error)
List(ctx context.Context, prefix string) (map[string][]byte, error)
Put(ctx context.Context, key string, value []byte) error
Delete(ctx context.Context, key string) error
}
Delete is what makes a save a replacement rather than an addition, and it has to be idempotent: a key the store does not hold is nothing to remove and not a failure.
Loading
The example below implements those four over a plain Go map, so it runs anywhere.
It is Example in example_test.go, which go test compiles and runs.
type memory map[string][]byte
func (m memory) Get(_ context.Context, key string) ([]byte, bool, error) {
value, found := m[key]
return value, found, nil
}
func (m memory) List(_ context.Context, prefix string) (map[string][]byte, error) {
out := map[string][]byte{}
for key, value := range m {
if strings.HasPrefix(key, prefix) {
out[key] = value
}
}
return out, nil
}
func (m memory) Put(_ context.Context, key string, value []byte) error {
m[key] = value
return nil
}
func (m memory) Delete(_ context.Context, key string) error {
delete(m, key)
return nil
}
func Example() {
type DB struct {
Host string `ferry:"host"`
Port int `ferry:"port,default=5432"`
}
type Config struct {
Name string `ferry:"name"`
DB DB `ferry:"db"`
}
store := memory{
"app/name": []byte("checkout"),
"app/db/host": []byte("db.internal"),
}
src, err := kv.NewSource(store, kv.WithPrefix("app"))
if err != nil {
panic(err)
}
cfg, err := ferry.Load[Config](context.Background(), src)
if err != nil {
panic(err)
}
fmt.Printf("%s %s:%d\n", cfg.Name, cfg.DB.Host, cfg.DB.Port)
// Output:
// checkout db.internal:5432
}
Keys come from the tags, joined with /, so the nested db.host field reads db/host.
kv.NewSink is the other direction: ferry.Dump writes the same struct back to the same keys.
A save replaces what it wrote last time
A store holds whatever was put in it, so a list that loses an element and a map that loses a key would otherwise leave the previous save's keys behind, and the next load would read them back as though they were still configured.
They do not.
Before a save writes a list or a map, it tells the store to forget everything under that address, and at commit time the keys it did not write are removed.
It is ExampleSink_replace in example_test.go, over the same memory client as above:
func ExampleSink_replace() {
type Config struct {
Tags []string `ferry:"tags"`
}
store := memory{}
sink, err := kv.NewSink(store, kv.WithPrefix("app"))
if err != nil {
panic(err)
}
ctx := context.Background()
if err := ferry.Dump(ctx, Config{Tags: []string{"a", "b", "c"}}, sink); err != nil {
panic(err)
}
if err := ferry.Dump(ctx, Config{Tags: []string{"x"}}, sink); err != nil {
panic(err)
}
for _, key := range slices.Sorted(maps.Keys(store)) {
fmt.Printf("%s = %s\n", key, store[key])
}
// Output:
// app/tags/0 = x
}
Only a list or a map is replaced this way. A field your value omits is not written and is not removed either: silence never deletes anything.
Prefixes
kv.WithPrefix("app", "cfg") puts db/host at app/cfg/db/host.
Give it one argument per level rather than a path: WithPrefix("app/cfg") is rejected up front, so a prefix cannot smuggle in a level you did not mean.
One call or one call per key
By default each field is fetched as it is needed.
kv.WithBatch() fetches the whole prefix in one call instead, when the source is opened.
Pick by what your store costs you: batch is one round trip and a consistent snapshot, and per-key reads only what your struct actually names, which is cheaper when the prefix holds far more than you want.
Nothing else changes - the struct you get back is identical either way.
It is a load-time option, and NewSink rejects it rather than quietly ignoring it.
One load, several reads at once
A reader from this package tells ferry that it tolerates overlapping calls, so ferry.MaxConcurrency(n) overlaps a single load's reads:
cfg, err := ferry.Load[Config](ctx, src, ferry.MaxConcurrency(4))
It declares no bound of its own, because how much overlap your store will take is a fact about your store and your token, and you are the one holding both.
What it costs you is what Client already asks for: your client has to be safe for use from many goroutines at once.
Nothing else inside one open is shared - a batch open's snapshot is only ever read, and this package serialises the key function that turns an address into a store key.
A batch open has nothing left to overlap, since it already made its one call. The struct you get back is the same either way, under any budget, and ferry's conformance suite holds this driver to exactly that.
Everything is bytes
A store key carries no type, so 8080 in an int field and "8080" in a string field are both stored as 8080, and each field parses what it reads.
There is also no way to store "nothing", as distinct from an empty value. So four shapes cannot be saved here: a nil pointer, a nil slice, an empty map, and a non-nil pointer to a struct whose every field was omitted. Saving one fails and names the field rather than writing an empty string that would read back as something else. A struct with all four reports all four, and the store is left untouched.
The failures are two classes, and a caller matching on one of them misses the other.
A null at a leaf is this package's own refusal and carries ferry.ErrValue.
The other three are a container speaking at its own address, which this package supplies no way to write, so ferry refuses them for it and they carry ferry.ErrPlane.
Unless the store holds payloads
kv.Raw() says the values are bytes rather than text, so they cross ferry's boundary as the bytes the store holds:
store := memory{}
ctx := context.Background()
sink, err := kv.NewSink(store, kv.WithPrefix("app"), kv.Raw())
if err != nil {
panic(err)
}
if err := ferry.Dump(ctx, Certs{Cert: []byte{0x1f, 0x8b, 0x00}}, sink); err != nil {
panic(err)
}
src, err := kv.NewSource(store, kv.WithPrefix("app"), kv.Raw())
if err != nil {
panic(err)
}
cfg, err := ferry.Load[Certs](ctx, src)
if err != nil {
panic(err)
}
fmt.Printf("% x\n", cfg.Cert)
// Output:
// 1f 8b 00
It is symmetric, and the same spelling serves both directions, so a load and a save cannot drift apart.
It is a fact about the whole store, because a key carries no type for a driver to consult.
Once it is declared every value is a payload, so an int, a string or a time.Duration field over the same store is a value the field cannot take.
Declare it for a store whose values are payloads, and read the fields that are not through a source of their own.
Whether you can write is discovered when you open, not before
Implement ACL and the package asks your client before writing anything:
type ACL interface {
CanWrite(ctx context.Context, key string) error
}
A client that does not implement it is assumed writable.
A token with no write access fails when the writer opens, before a single key is written, rather than halfway through. A token that can write some paths and not others reports every key it was refused, not just the first, so you fix your ACL once instead of once per run.
More
The package documentation is the reference for everything above, and the design records behind it are in docs/adr/.
Documentation
¶
Overview ¶
Package kv loads configuration from a Consul-shaped key-value store into a Go struct, and writes it back.
Experimental ¶
Neither the Go API here nor the way values are stored is settled yet, and either may change in a release that is not a new major version of this module. The reason is that this is Consul-shaped rather than Consul: the only Client in this repository is a test fake, so nothing here has been run against a real store, and the interfaces below are read off a specification rather than confirmed against a backend.
Bring your own client ¶
There is no Consul, etcd or Redis dependency here. Implement Client's four methods over whatever store you have, and this package works against it.
src, err := kv.NewSource(store, kv.WithPrefix("app"))
cfg, err := ferry.Load[Config](ctx, src)
sink, err := kv.NewSink(store, kv.WithPrefix("app"))
err = ferry.Dump(ctx, cfg, sink)
Keys come from the tags, joined with "/", so a nested db.host field reads app/db/host under that prefix.
A save replaces what it wrote last time ¶
A store holds whatever was put in it, so a list that lost an element and a map that lost a key would otherwise leave the previous save's keys behind, and the next load would read them back as though they were still configured.
They do not. Before a save writes a list or a map, it tells the store to forget everything under that address, and at commit time the keys it did not write are removed. Saving a one-element Tags over a store holding app/tags/0, app/tags/1 and app/tags/2 leaves app/tags/0 and nothing else.
Only a list or a map is replaced this way. A field your value omits is not written and is not removed either, so silence never deletes anything.
Everything is bytes ¶
A store key carries no type, so 8080 in an int field and "8080" in a string field are stored alike and both read back as text, which each field parses with its own parser. Nothing is lost on the way into a Go value; what is lost is the store's own opinion about the value, which it never had.
There is also no way to store "nothing", as distinct from an empty value. So four shapes cannot be saved here: a nil pointer, a nil slice, an empty map, and a non-nil pointer to a struct whose every field was omitted. Saving one fails and names the field rather than writing an empty value that would read back as something else, a struct with all four reports all four, and the store is left untouched.
The failures are two classes and a caller matching on one of them misses the other. A null at a leaf is this package's own refusal and carries ferry.ErrValue. The other three are a container speaking at its own address, which this package supplies no way to write, so ferry refuses them for it and they carry ferry.ErrPlane. Match both, or match neither and read the error.
Or a store that holds payloads ¶
Raw says the store's values are bytes rather than text, so a value crosses ferry's boundary as the bytes the store holds and nothing in the middle turns them into a string:
src, err := kv.NewSource(store, kv.WithPrefix("app"), kv.Raw())
It is symmetric: a sink built with it writes a []byte field's bytes exactly as they are, through the same spelling, so the two directions cannot drift apart.
It is a fact about the whole store, because a key carries no type for a driver to consult. Once it is declared every value is a payload, so an int, a string or a duration field over the same store is a value the field cannot take: declare it for a store whose values are payloads, and read the fields that are not through a source of their own.
One call or one call per key ¶
By default each field is fetched as it is needed. WithBatch fetches the whole prefix in one call instead, when the load starts. Pick by what your store costs you: batch is one round trip and a consistent snapshot, and per-key reads only what your struct actually names, which is cheaper when the prefix holds far more than you want. The struct you get back is identical either way.
One load, several reads at once ¶
A reader from this package tells ferry that it tolerates overlapping calls, so a caller who sets ferry.MaxConcurrency has several of one load's reads in flight together. It declares no bound of its own: how much overlap a store will take is a fact about that store and that token, and the caller is the one holding both.
What it costs you is what Client already asks for, which is that your client is safe for use from many goroutines at once. Nothing else in one open is shared: a batch open's snapshot is read and never written, and this package serialises the key function that turns an address into a store key.
A batch open has nothing to overlap - it already made its one call - so the budget buys nothing there, and the struct you get back is the same either way, under any budget.
Whether you can write is discovered when the save starts ¶
Implement ACL and this package asks your client before writing anything. A client that does not implement it is assumed writable.
A token with no write access fails before a single key is written rather than halfway through, and a token that can write some paths and not others reports every key it was refused rather than only the first.
The design records behind these decisions are in docs/adr/.
Example ¶
Example loads an annotated struct out of a store, under a prefix.
package main
import (
"context"
"fmt"
"strings"
"github.com/onhotpath/ferry"
"github.com/onhotpath/ferry/driver/kv"
)
// memory is a [kv.Client] over an ordinary Go map, and the whole of what this
// driver needs from a backend: read one key, list a folder, write one key,
// remove one key.
//
// It is here so the example runs, and so a reader can see the shape a real
// adapter over consul/api, etcd or a two-column table has to fill.
type memory map[string][]byte
func (m memory) Get(_ context.Context, key string) ([]byte, bool, error) {
value, found := m[key]
return value, found, nil
}
func (m memory) List(_ context.Context, prefix string) (map[string][]byte, error) {
out := map[string][]byte{}
for key, value := range m {
if strings.HasPrefix(key, prefix) {
out[key] = value
}
}
return out, nil
}
func (m memory) Put(_ context.Context, key string, value []byte) error {
m[key] = value
return nil
}
func (m memory) Delete(_ context.Context, key string) error {
delete(m, key)
return nil
}
func main() {
type DB struct {
Host string `ferry:"host"`
Port int `ferry:"port,default=5432"`
}
type Config struct {
Name string `ferry:"name"`
DB DB `ferry:"db"`
}
store := memory{
"app/name": []byte("checkout"),
"app/db/host": []byte("db.internal"),
}
src, err := kv.NewSource(store, kv.WithPrefix("app"))
if err != nil {
panic(err)
}
cfg, err := ferry.Load[Config](context.Background(), src)
if err != nil {
panic(err)
}
fmt.Printf("%s %s:%d\n", cfg.Name, cfg.DB.Host, cfg.DB.Port)
}
Output: checkout db.internal:5432
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type ACL ¶
type ACL interface {
// CanWrite reports whether these credentials permit a write at key. The
// empty key asks about the store as a whole, which is what a sink with no
// prefix asks at open.
CanWrite(ctx context.Context, key string) error
}
ACL is implemented by a Client whose credentials can be asked, before anything is written, whether they permit a write.
It is optional. A Client that does not implement it is assumed writable everywhere, which is the honest answer for a store with no access control.
A save asks twice and about two different things. It asks about the whole prefix when it starts, which is where a token with no write access at all is refused, before a single key has been written. It then asks about each key before staging it, which is what lets a token that can write some paths and not others report every key it was refused rather than only the first.
A nil error means the write is permitted. Any other error is returned to the caller under ferry's own wrapper, so your own sentinel stays reachable through errors.Is.
type Client ¶
type Client interface {
// Get answers with the bytes stored at key, and with found false where the
// store does not hold it.
Get(ctx context.Context, key string) (value []byte, found bool, err error)
// List answers with every pair whose key begins with prefix, keyed by the
// whole key rather than by the part below the prefix. An empty prefix is
// the whole store.
//
// It is what one round trip means here: a load told to batch calls this
// once and answers every field out of what it got back.
List(ctx context.Context, prefix string) (map[string][]byte, error)
// Put stores value at key, creating it or replacing what is there.
Put(ctx context.Context, key string, value []byte) error
// Delete removes key, and reports nothing for a key the store does not
// hold.
//
// It is what makes a save a replacement rather than an addition: a list
// that lost its third element, or a map that lost a key, leaves the
// previous save's keys in the store without it, and the next load reads
// them back as though they were still configured.
Delete(ctx context.Context, key string) error
}
Client is the store this driver talks to, and the whole of what it needs from a backend: read one key, list what is under a folder, write one key, remove one key.
It is an interface rather than a dependency, so an adapter over consul/api, etcd, a Redis hash, a table with two columns or a test double is a few lines and this package never learns which of them it is talking to.
Four things an implementer owns.
Absence is a result and not an error. Get reports a key the store does not hold with found false and a nil error, so that a backend's own not-found stays distinguishable from a real failure. A zero-length value is a value the store holds, and it arrives as an empty string rather than as absence.
Delete is idempotent, and that is the same rule read from the write side: a key the store does not hold is nothing to remove and not a failure. A backend whose own delete reports not-found has that translated here rather than in this driver.
Cancellation is yours. The driver hands its caller's context to every call and adds no deadline of its own, so a client that ignores the context is the only thing standing between a cancelled load and a blocked one.
Safety for use from many goroutines at once is yours, and it is ordinary rather than exotic. A source or a sink is constructed once and a binding is held for the life of a process, so one client is reached from wherever a load or a save happens. A real backend's own client is usually safe already, and a test double over a plain map is usually not.
One load can reach it from many goroutines too. This package's reader tells ferry that it tolerates overlapping calls, so a caller who sets ferry.MaxConcurrency has several of a single load's reads in your client at once. That is the same obligation as the paragraph above rather than a new one, and it is why ferry.MaxConcurrency is the caller's to set: how much overlap your store will take is a fact about your store and your token.
type Option ¶
type Option func(*config) error
Option is a setting handed to NewSource or NewSink. The set is closed at four: WithPrefix, WithBatch, RootKey and Raw.
func Raw ¶
func Raw() Option
Raw says this store holds byte payloads, so that a value crosses ferry's boundary as the bytes the store holds and is never turned into text on the way.
src, err := kv.NewSource(store, kv.WithPrefix("app"), kv.Raw())
cfg, err := ferry.Load[Certs](ctx, src)
Without it a stored value arrives as text, which every field parses with its own parser: an int field reads the digits, a bool field reads true and false, and a []byte field takes the bytes of that text. With it a stored value arrives as bytes, which is the same []byte with one conversion fewer and no step in the middle that could have decided the bytes were a string.
The sharp edge is that this is a fact about the whole store and not about one field, because a key carries no type for a driver to consult. Once it is declared every value is a payload, so an int, a string or a duration field over the same store is then a value the field cannot take: declare it for a store whose values are payloads, and read the fields that are not through a source of their own.
It is symmetric. A Sink built with it writes a []byte field's bytes exactly as they are, which is what it already did, and takes them through the same spelling so that the two directions cannot drift apart.
Example ¶
ExampleRaw saves and loads a store whose values are byte payloads.
Without the Option a stored value crosses ferry's boundary as text, which every field parses for itself. With it a value crosses as the bytes the store holds, in both directions, and nothing in the middle turns them into a string.
It is a fact about the whole store: once it is declared, a field that is not a payload has no value it can take here.
package main
import (
"context"
"fmt"
"strings"
"github.com/onhotpath/ferry"
"github.com/onhotpath/ferry/driver/kv"
)
// memory is a [kv.Client] over an ordinary Go map, and the whole of what this
// driver needs from a backend: read one key, list a folder, write one key,
// remove one key.
//
// It is here so the example runs, and so a reader can see the shape a real
// adapter over consul/api, etcd or a two-column table has to fill.
type memory map[string][]byte
func (m memory) Get(_ context.Context, key string) ([]byte, bool, error) {
value, found := m[key]
return value, found, nil
}
func (m memory) List(_ context.Context, prefix string) (map[string][]byte, error) {
out := map[string][]byte{}
for key, value := range m {
if strings.HasPrefix(key, prefix) {
out[key] = value
}
}
return out, nil
}
func (m memory) Put(_ context.Context, key string, value []byte) error {
m[key] = value
return nil
}
func (m memory) Delete(_ context.Context, key string) error {
delete(m, key)
return nil
}
func main() {
type Certs struct {
Cert []byte `ferry:"cert"`
}
store := memory{}
ctx := context.Background()
sink, err := kv.NewSink(store, kv.WithPrefix("app"), kv.Raw())
if err != nil {
panic(err)
}
if err := ferry.Dump(ctx, Certs{Cert: []byte{0x1f, 0x8b, 0x00}}, sink); err != nil {
panic(err)
}
src, err := kv.NewSource(store, kv.WithPrefix("app"), kv.Raw())
if err != nil {
panic(err)
}
cfg, err := ferry.Load[Certs](ctx, src)
if err != nil {
panic(err)
}
fmt.Printf("% x\n", cfg.Cert)
}
Output: 1f 8b 00
func RootKey ¶
RootKey names the key a schema whose root is a single value is written at, which is this name under the prefix.
sink, err := kv.NewSink(store, kv.WithPrefix("app"), kv.RootKey("value"))
With WithPrefix("app") and RootKey("value") that root is the key "app/value", which is an ordinary key beside every other one this driver writes.
Without it such a schema is refused, in both directions. The store's own key at the prefix is the folder everything else is written under, so a value there would sit on an interior node, and a driver with no prefix would be asked to write at the empty key, which a real store rejects.
The name is one key and never a path: it may not be empty and may not contain "/". It says nothing about any other schema, since every address with a segment of its own is named by that segment as before.
func WithBatch ¶
func WithBatch() Option
WithBatch fetches the whole prefix in one call when the load starts, instead of one call per field as each is asked for.
src, err := kv.NewSource(store, kv.WithPrefix("app"), kv.WithBatch())
Pick by what your store costs you. Batch is one round trip and a snapshot that cannot change under the load; per-key reads only the fields your struct actually names, which is cheaper against a store whose prefix holds far more than you want. The struct you get back is identical either way.
It is a load-time Option. NewSink refuses it rather than ignoring it, because a save stages every write and commits them together, so there is no per-key half of it to choose against.
func WithPrefix ¶
WithPrefix places every key this driver reaches under these segments, so that a db.host field with WithPrefix("app", "cfg") reads the key "app/cfg/db/host".
It takes one argument per level and never a path. WithPrefix("app/cfg") is refused up front, so a prefix cannot smuggle in a level you did not mean, and there is no way to spell a prefix that runs into the first key without a separator between them.
It may be given once. Two prefixes are two key spaces and nothing in the call says which is meant, so a second one is refused rather than quietly winning.
type Sink ¶
type Sink struct {
// contains filtered or unexported fields
}
Sink is the write half of a key-value plane.
sink, err := kv.NewSink(store, kv.WithPrefix("app"))
err = ferry.Dump(ctx, cfg, sink)
It stages every write and performs them together at the end, so a save that fails leaves the store untouched. It also means one save reports every field it could not write, rather than stopping at the first.
Staging is not a transaction. The writes themselves go through your Client one key at a time, so a store that fails part way through the commit is left part way through it.
Example (Replace) ¶
ExampleSink_replace shows what a second save does to a list that lost an element: the keys the save did not write are removed, so a load afterwards reads the value that was saved rather than the union of both saves.
package main
import (
"context"
"fmt"
"maps"
"slices"
"strings"
"github.com/onhotpath/ferry"
"github.com/onhotpath/ferry/driver/kv"
)
// memory is a [kv.Client] over an ordinary Go map, and the whole of what this
// driver needs from a backend: read one key, list a folder, write one key,
// remove one key.
//
// It is here so the example runs, and so a reader can see the shape a real
// adapter over consul/api, etcd or a two-column table has to fill.
type memory map[string][]byte
func (m memory) Get(_ context.Context, key string) ([]byte, bool, error) {
value, found := m[key]
return value, found, nil
}
func (m memory) List(_ context.Context, prefix string) (map[string][]byte, error) {
out := map[string][]byte{}
for key, value := range m {
if strings.HasPrefix(key, prefix) {
out[key] = value
}
}
return out, nil
}
func (m memory) Put(_ context.Context, key string, value []byte) error {
m[key] = value
return nil
}
func (m memory) Delete(_ context.Context, key string) error {
delete(m, key)
return nil
}
func main() {
type Config struct {
Tags []string `ferry:"tags"`
}
store := memory{}
sink, err := kv.NewSink(store, kv.WithPrefix("app"))
if err != nil {
panic(err)
}
ctx := context.Background()
if err := ferry.Dump(ctx, Config{Tags: []string{"a", "b", "c"}}, sink); err != nil {
panic(err)
}
if err := ferry.Dump(ctx, Config{Tags: []string{"x"}}, sink); err != nil {
panic(err)
}
for _, key := range slices.Sorted(maps.Keys(store)) {
fmt.Printf("%s = %s\n", key, store[key])
}
}
Output: app/tags/0 = x
func NewSink ¶
NewSink builds a sink writing through client.
sink, err := kv.NewSink(consulClient, kv.WithPrefix("app"))
err = ferry.Dump(ctx, cfg, sink)
It refuses WithBatch rather than ignoring it: a save stages every write and commits them together, so there is no per-key half of that Option to choose against.
func (*Sink) Bind ¶
func (s *Sink) Bind(addrs *ferry.AddressSet) (ferry.OpenWriterFunc, error)
Bind computes this schema's store keys and checks them, exactly as Source.Bind does and for the same reasons.
It does no I/O, so a sink binds successfully against a store it may not be allowed to write to. See ACL for where that is discovered.
type Source ¶
type Source struct {
// contains filtered or unexported fields
}
Source is the read half of a key-value plane.
src, err := kv.NewSource(store, kv.WithPrefix("app"))
cfg, err := ferry.Load[Config](ctx, src)
It is a separate type from Sink, so a round trip names the store twice. The repetition buys the refusal being a compile error: code handed only a Source cannot save through it.
One source may be used by many loads at once, from many goroutines. Nothing mutable is shared between them.
func NewSource ¶
NewSource builds a source reading through client.
src, err := kv.NewSource(consulClient, kv.WithPrefix("app"), kv.WithBatch())
cfg, err := ferry.Load[Config](ctx, src)
It reports every Option that was wrong rather than only the first, and it refuses a nil client here rather than at the first read.
func (*Source) Bind ¶
Bind computes this schema's store keys and checks them, and does no I/O.
Two things are checked before anything is read: that every field has a store key at all, and that no two fields want the same key. A schema failing either is refused here, in one error naming every offending field.
It cannot fail for anything about the store itself. A store that is unreachable, or a token that has expired, is reported when the load starts.
Source Files