kvmapstruct

README

kvmapstruct

Package kvstruct exposes various utility functions to do conversions between: Consul KV pairs and native Go Struct or map[string]interface{}.

It also provides several utilities to convert directly:

• Nested map to flatten/kv map or Consul kv pairs
• Flatten/kv map to Go struct
• Kv map to nested map, etc.

There are some notions that are used in this package.

• Nested map: classic nested map[string]interface{}.
• Flatten map: map[string]interface{} represents key/value. It means that no nested map will be value. Value can be a normal type including slice.
• KV map: map[string]interface{} represents key/value but value can not be slice or map. A slice will be represented by keys suffixed by 0, 1, 2 etc.

This package only supports the following value types: int, bool, string, []int, []bool, []string and map[string]interface{}

Documentation

See the Godoc

Documentation

Overview

Package kvmapstruct exposes various utility functions to do conversions between: Consul KV pairs and native Go Struct or map[string]interface{}.

It also provides several utilities to convert directly: nested map to flatten/kv map or Consul kv pairs, flatten/kv map to Go struct, Kv map to nested map, etc.

There are some notions that are used in this package. Nested map: classic map[string]interface{}. Flatten map: map[string]interface{} represents key/value and value can be a normal type including slice or map KV map: map[string]interface{} represents key/value but value can not be slice or map. A slice will be represented by keys suffixed by 0, 1, 2 etc.

Only the following value types are supported: int, bool, string, []int, []bool, []string and map[string]interface{}

Index

• func FlattenMapToStruct(in map[string]interface{}, out interface{}) error
• func KVMapToMap(in map[string]interface{}, prefix string) (map[string]interface{}, error)
• func KVMapToStruct(in map[string]interface{}, prefix string, out interface{}) error
• func MapToFlattenMap(in map[string]interface{}, prefix string) map[string]interface{}
• func MapToKVMap(in map[string]interface{}, prefix string) map[string]interface{}
• type KVMapStruct
  • func NewKVMapStruct(url, token, path string) (*KVMapStruct, error)
  • func (kms *KVMapStruct) ConsulKVToMap() (map[string]interface{}, error)
  • func (kms *KVMapStruct) ConsulKVToStruct(out interface{}) error
  • func (kms *KVMapStruct) MapToConsulKV(input interface{}) error
  • func (kms *KVMapStruct) MapToKVPairs(in map[string]interface{}, prefix string) (consul.KVPairs, error)
  • func (kms *KVMapStruct) StructToConsulKV(input interface{}) error

Examples

• FlattenMapToStruct
• KVMapStruct.ConsulKVToMap
• KVMapStruct.ConsulKVToStruct (EmbeddedStruct)
• KVMapStruct.ConsulKVToStruct (NormalStruct)
• KVMapStruct.ConsulKVToStruct (PointerStruct)
• KVMapStruct.MapToConsulKV
• KVMapStruct.MapToKVPairs
• KVMapStruct.StructToConsulKV
• KVMapToMap
• KVMapToStruct
• KVMapToStruct (EmbeddedStruct)
• MapToFlattenMap
• MapToKVMap

Functions

func FlattenMapToStruct

func FlattenMapToStruct(in map[string]interface{}, out interface{}) error

FlattenMapToStruct converts a flatten map to a Go struct. Out argument must be a initialiezed pointer to a Go struct. Its substructs can be a pointer to a struct, embedded struct or struct. If it is a pointer, it must be initialized.

Example

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	Key43 *ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	Key4 *ExSTChildLevel1
}

input := map[string]interface{}{
	"Key1": "val1",
	"Key2": 2,
	"Key3": []int{1, 2, 3},
	"Key4": map[string]interface{}{
		"Key41": "val41",
		"Key42": map[string]interface{}{
			"Key421": "val421",
			"Key422": []string{"one", "two", "three"},
		},
		"Key43": map[string]interface{}{
			"Key431": map[string]interface{}{
				"Key4311": "val4311",
			},
		},
	},
}

st := &ExST{
	Key4: &ExSTChildLevel1{
		Key43: &ExSTChildLevel2{},
	},
}

err := FlattenMapToStruct(input, st)
if err != nil {
	return
}

fmt.Println(st)

func KVMapToMap

func KVMapToMap(in map[string]interface{}, prefix string) (map[string]interface{}, error)

KVMapToMap converts a KV map to nested map.

Example

input := map[string]interface{}{
	"test/key1":                "val1",
	"test/key2":                2,
	"test/key3/0":              1,
	"test/key3/1":              2,
	"test/key3/2":              3,
	"test/key4/key41":          "val41",
	"test/key4/key42/key421":   "val421",
	"test/key4/key42/key422/0": "one",
	"test/key4/key42/key422/1": "two",
	"test/key4/key42/key422/2": "three",
}

o, err := KVMapToMap(input, "test")
if err != nil {
	return
}

fmt.Println(o)

func KVMapToStruct

func KVMapToStruct(in map[string]interface{}, prefix string, out interface{}) error

KVMapToStruct converts a KV map to a Go struct. Out argument must be a initialiezed pointer to a Go struct. Its substructs can be a pointer to a struct, embedded struct or struct. If it is a pointer, it must be initialized.

Example

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	Key43 *ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	Key4 *ExSTChildLevel1
}

input := map[string]interface{}{
	"Key1":                      "val1",
	"Key2":                      2,
	"Key3/0":                    1,
	"Key3/1":                    2,
	"Key3/2":                    3,
	"Key4/Key41":                "val41",
	"Key4/Key42/Key421":         "val421",
	"Key4/Key42/Key422/0":       "one",
	"Key4/Key42/Key422/1":       "two",
	"Key4/Key42/Key422/2":       "three",
	"Key4/Key43/Key431/Key4311": "val4311",
}

st := &ExST{
	Key4: &ExSTChildLevel1{
		Key43: &ExSTChildLevel2{},
	},
}

err := KVMapToStruct(input, "", st)
if err != nil {
	return
}

fmt.Println(st)

Example (EmbeddedStruct)

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	ExSTChildLevel1
}

input := map[string]interface{}{
	"Key1":                           "val1",
	"Key2":                           2,
	"Key3/0":                         1,
	"Key3/1":                         2,
	"Key3/2":                         3,
	"ExSTChildLevel1/Key41":          "val41",
	"ExSTChildLevel1/Key42/Key421":   "val421",
	"ExSTChildLevel1/Key42/Key422/0": "one",
	"ExSTChildLevel1/Key42/Key422/1": "two",
	"ExSTChildLevel1/Key42/Key422/2": "three",
	"ExSTChildLevel1/ExSTChildLevel2/Key431/Key4311": "val4311",
}

st := &ExST{}

err := KVMapToStruct(input, "", st)
if err != nil {
	return
}

fmt.Printf("%++v\n", st)

Output:

&{Key1:val1 Key2:2 Key3:[1 2 3] ExSTChildLevel1:{Key41:val41 Key42:map[Key421:val421 Key422:[one two three]] ExSTChildLevel2:{Key431:map[Key4311:val4311]}}}

func MapToFlattenMap

func MapToFlattenMap(in map[string]interface{}, prefix string) map[string]interface{}

MapToFlattenMap converts a nested map to a flatten map.

Example

input := map[string]interface{}{
	"key1": "val1",
	"key2": 2,
	"key3": []int{1, 2, 3},
	"key4": map[string]interface{}{
		"key41": "val41",
		"key42": map[string]interface{}{
			"key421": "val421",
			"key422": []string{"one", "two", "three"},
		},
		"key43": map[string]interface{}{
			"key431": map[string]interface{}{
				"key4311": "val4311",
			},
		},
	},
}

o := MapToFlattenMap(input, "test")

keys := []string{}

for k := range o {
	keys = append(keys, k)
}

sort.Strings(keys)

for _, key := range keys {
	fmt.Println(key, ":", o[key])
}

Output:

test/key1 : val1
test/key2 : 2
test/key3 : [1 2 3]
test/key4/key41 : val41
test/key4/key42/key421 : val421
test/key4/key42/key422 : [one two three]
test/key4/key43/key431/key4311 : val4311

func MapToKVMap

func MapToKVMap(in map[string]interface{}, prefix string) map[string]interface{}

MapToKVMap convert a nested map to a KV map.

Example

input := map[string]interface{}{
	"key1": "val1",
	"key2": 2,
	"key3": []int{1, 2, 3},
	"key4": map[string]interface{}{
		"key41": "val41",
		"key42": map[string]interface{}{
			"key421": "val421",
			"key422": []string{"one", "two", "three"},
		},
	},
}

output := map[string]interface{}{
	"test/key1":                "val1",
	"test/key2":                2,
	"test/key3/0":              1,
	"test/key3/1":              2,
	"test/key3/2":              3,
	"test/key4/key41":          "val41",
	"test/key4/key42/key421":   "val421",
	"test/key4/key42/key422/0": "one",
	"test/key4/key42/key422/1": "two",
	"test/key4/key42/key422/2": "three",
}

o := MapToKVMap(input, "test")

// Compare result with expected output
fmt.Println(reflect.DeepEqual(o, output))

Output:

true

Types

type KVMapStruct

type KVMapStruct struct {
	// Path is consul key parent to store struct's fields
	Path string
	// Client is consul client
	Client *consul.Client
}

KVMapStruct contains consul informations.

func NewKVMapStruct

func NewKVMapStruct(url, token, path string) (*KVMapStruct, error)

NewKVMapStruct creates a new *KVMapStruct. URL format is ip:port.

func (*KVMapStruct) ConsulKVToMap

func (kms *KVMapStruct) ConsulKVToMap() (map[string]interface{}, error)

ConsulKVToMap gets list of all consul keys from kvmapstruct path and match them to a map[string]interface{}.

Example

input := map[string]interface{}{
	"test/Key1":                      "val1",
	"test/Key2":                      "2",
	"test/Key3/0":                    "1",
	"test/Key3/1":                    "2",
	"test/Key3/2":                    "3",
	"test/Key4/Key41":                "val41",
	"test/Key4/Key42/Key421":         "val421",
	"test/Key4/Key42/Key422/0":       "one",
	"test/Key4/Key42/Key422/1":       "two",
	"test/Key4/Key42/Key422/2":       "three",
	"test/Key4/Key43/Key431/Key4311": "val4311",
}

kms, err := NewKVMapStruct("localhost:8500", "adf4238a-882b-9ddc-4a9d-5b6758e4159e", "test")
if err != nil {
	return
}

kms.Path = "test"

// Insert data in to consul
for k, v := range input {
	kv := &consul.KVPair{
		Key:   k,
		Value: []byte(v.(string)),
	}

	_, err := kms.Client.KV().Put(kv, nil)
	if err != nil {
		return
	}
}

out, err := kms.ConsulKVToMap()
if err != nil {
	return
}

fmt.Println(out)

func (*KVMapStruct) ConsulKVToStruct

func (kms *KVMapStruct) ConsulKVToStruct(out interface{}) error

ConsulKVToStruct gets list of all consul keys from kvmapstruct path and match them to the given struct in argument. Out argument must be a initialiezed pointer to a Go struct. Its substructs can be a pointer to a struct, embedded struct or struct. If it is a pointer, it must be initialized.

Example (EmbeddedStruct)

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	ExSTChildLevel1
}

input := map[string]interface{}{
	"test/Key1":                           "val1",
	"test/Key2":                           "2",
	"test/Key3/0":                         "1",
	"test/Key3/1":                         "2",
	"test/Key3/2":                         "3",
	"test/ExSTChildLevel1/Key41":          "val41",
	"test/ExSTChildLevel1/Key42/Key421":   "val421",
	"test/ExSTChildLevel1/Key42/Key422/0": "one",
	"test/ExSTChildLevel1/Key42/Key422/1": "two",
	"test/ExSTChildLevel1/Key42/Key422/2": "three",
	"test/ExSTChildLevel1/ExSTChildLevel2/Key431/Key4311": "val4311",
}

kms, err := NewKVMapStruct("localhost:8500", "adf4238a-882b-9ddc-4a9d-5b6758e4159e", "test")
if err != nil {
	return
}

st := &ExST{}

kms.Path = "test"

// Insert data in to consul
for k, v := range input {
	kv := &consul.KVPair{
		Key:   k,
		Value: []byte(v.(string)),
	}

	_, err := kms.Client.KV().Put(kv, nil)
	if err != nil {
		return
	}
}

err = kms.ConsulKVToStruct(st)
if err != nil {
	return
}

kms.Client.KV().DeleteTree(kms.Path, nil)

fmt.Printf("%++v\n", st)

Output:

&{Key1:val1 Key2:2 Key3:[1 2 3] ExSTChildLevel1:{Key41:val41 Key42:map[Key421:val421 Key422:[one two three]] ExSTChildLevel2:{Key431:map[Key4311:val4311]}}}

Example (NormalStruct)

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	Key43 ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	Key4 ExSTChildLevel1
}

input := map[string]interface{}{
	"test/Key1":                      "val1",
	"test/Key2":                      "2",
	"test/Key3/0":                    "1",
	"test/Key3/1":                    "2",
	"test/Key3/2":                    "3",
	"test/Key4/Key41":                "val41",
	"test/Key4/Key42/Key421":         "val421",
	"test/Key4/Key42/Key422/0":       "one",
	"test/Key4/Key42/Key422/1":       "two",
	"test/Key4/Key42/Key422/2":       "three",
	"test/Key4/Key43/Key431/Key4311": "val4311",
}

kms, err := NewKVMapStruct("localhost:8500", "adf4238a-882b-9ddc-4a9d-5b6758e4159e", "test")
if err != nil {
	return
}

st := &ExST{}

kms.Path = "test"

// Insert data in to consul
for k, v := range input {
	kv := &consul.KVPair{
		Key:   k,
		Value: []byte(v.(string)),
	}

	_, err := kms.Client.KV().Put(kv, nil)
	if err != nil {
		return
	}
}

err = kms.ConsulKVToStruct(st)
if err != nil {
	return
}

kms.Client.KV().DeleteTree(kms.Path, nil)

fmt.Printf("%++v\n", st)

Output:

&{Key1:val1 Key2:2 Key3:[1 2 3] Key4:{Key41:val41 Key42:map[Key421:val421 Key422:[one two three]] Key43:{Key431:map[Key4311:val4311]}}}

Example (PointerStruct)

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	Key43 *ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	Key4 *ExSTChildLevel1
}

input := map[string]interface{}{
	"test/Key1":                      "val1",
	"test/Key2":                      "2",
	"test/Key3/0":                    "1",
	"test/Key3/1":                    "2",
	"test/Key3/2":                    "3",
	"test/Key4/Key41":                "val41",
	"test/Key4/Key42/Key421":         "val421",
	"test/Key4/Key42/Key422/0":       "one",
	"test/Key4/Key42/Key422/1":       "two",
	"test/Key4/Key42/Key422/2":       "three",
	"test/Key4/Key43/Key431/Key4311": "val4311",
}

kms, err := NewKVMapStruct("localhost:8500", "adf4238a-882b-9ddc-4a9d-5b6758e4159e", "test")
if err != nil {
	return
}

st := &ExST{
	Key4: &ExSTChildLevel1{
		Key43: &ExSTChildLevel2{},
	},
}

kms.Path = "test"

// Insert data in to consul
for k, v := range input {
	kv := &consul.KVPair{
		Key:   k,
		Value: []byte(v.(string)),
	}

	_, err := kms.Client.KV().Put(kv, nil)
	if err != nil {
		return
	}
}

err = kms.ConsulKVToStruct(st)
if err != nil {
	return
}

fmt.Println(st)

func (*KVMapStruct) MapToConsulKV

func (kms *KVMapStruct) MapToConsulKV(input interface{}) error

MapToConsulKV converts and saves the map to Consul KV store. input argument must be a map[string]interface{}.

Example

input := map[string]interface{}{
	"key1": "val1",
	"key2": 2,
	"key3": []int{1, 2, 3},
	"key4": map[string]interface{}{
		"key41": "val41",
		"key42": map[string]interface{}{
			"key421": "val421",
			"key422": []string{"one", "two", "three"},
		},
	},
}

kms, err := NewKVMapStruct("localhost:8500", "adf4238a-882b-9ddc-4a9d-5b6758e4159e", "test")
if err != nil {
	return
}

out := make(map[string]interface{})
keys := []string{}

kms.Path = "nestedmap"
err = kms.MapToConsulKV(input)
if err != nil {
	return
}

pairs, _, err := kms.Client.KV().List(kms.Path, nil)
if err != nil {
	return
}

for _, kv := range pairs {
	out[kv.Key] = string(kv.Value)
	keys = append(keys, kv.Key)
}

sort.Strings(keys)

// Compare result with expected output
for _, key := range keys {
	fmt.Println(key, ":", out[key])
}

Output:

nestedmap/key1 : val1
nestedmap/key2 : 2
nestedmap/key3/0 : 1
nestedmap/key3/1 : 2
nestedmap/key3/2 : 3
nestedmap/key4/key41 : val41
nestedmap/key4/key42/key421 : val421
nestedmap/key4/key42/key422/0 : one
nestedmap/key4/key42/key422/1 : two
nestedmap/key4/key42/key422/2 : three

func (*KVMapStruct) MapToKVPairs

func (kms *KVMapStruct) MapToKVPairs(in map[string]interface{}, prefix string) (consul.KVPairs, error)

MapToKVPairs convert a nested map to an array of Consul KV pairs

Example

input := map[string]interface{}{
	"key1": "val1",
	"key2": 2,
	"key3": []int{1, 2, 3},
	"key4": map[string]interface{}{
		"key41": "val41",
		"key42": map[string]interface{}{
			"key421": "val421",
			"key422": []string{"one", "two", "three"},
		},
	},
}

kms, err := NewKVMapStruct("", "", "test")
if err != nil {
	fmt.Println(err)
	return
}

o, err := kms.MapToKVPairs(input, kms.Path)
if err != nil {
	fmt.Println(err)
	return
}

keys := []string{}
out := make(map[string]interface{})

for _, kv := range o {
	out[kv.Key] = string(kv.Value)
	keys = append(keys, kv.Key)
}

sort.Strings(keys)

// Compare result with expected output
for _, key := range keys {
	fmt.Println(key, ":", out[key])
}

Output:

test/key1 : val1
test/key2 : 2
test/key3/0 : 1
test/key3/1 : 2
test/key3/2 : 3
test/key4/key41 : val41
test/key4/key42/key421 : val421
test/key4/key42/key422/0 : one
test/key4/key42/key422/1 : two
test/key4/key42/key422/2 : three

func (*KVMapStruct) StructToConsulKV

func (kms *KVMapStruct) StructToConsulKV(input interface{}) error

StructToConsulKV converts and saves the struct to Consul KV store input argument must be a Go struct.

Example

type ExSTChildLevel2 struct {
	Key431 map[string]interface{}
}

type ExSTChildLevel1 struct {
	Key41 string
	Key42 map[string]interface{}
	Key43 *ExSTChildLevel2
}

type ExST struct {
	Key1 string
	Key2 int
	Key3 []int
	Key4 *ExSTChildLevel1
}

input := ExST{
	Key1: "val1",
	Key2: 2,
	Key3: []int{1, 2, 3},
	Key4: &ExSTChildLevel1{
		Key41: "val41",
		Key42: map[string]interface{}{
			"Key421": "val421",
			"Key422": []string{"one", "two", "three"},
		},
		Key43: &ExSTChildLevel2{
			Key431: map[string]interface{}{
				"Key4311": "val4311",
			},
		},
	},
}

kms, err := NewKVMapStruct("localhost:8500", "adf4238a-882b-9ddc-4a9d-5b6758e4159e", "test")
if err != nil {
	return
}

out := make(map[string]interface{})
keys := []string{}

kms.Path = "nestedstructmap"

err = kms.StructToConsulKV(input)
if err != nil {
	fmt.Println(err)
	return
}

pairs, _, err := kms.Client.KV().List(kms.Path, nil)
if err != nil {
	fmt.Println(err)
	return
}

for _, kv := range pairs {
	out[kv.Key] = string(kv.Value)
	keys = append(keys, kv.Key)
}

sort.Strings(keys)

for _, key := range keys {
	fmt.Println(key, ":", out[key])
}

Output:

nestedstructmap/Key1 : val1
nestedstructmap/Key2 : 2
nestedstructmap/Key3/0 : 1
nestedstructmap/Key3/1 : 2
nestedstructmap/Key3/2 : 3
nestedstructmap/Key4/Key41 : val41
nestedstructmap/Key4/Key42/Key421 : val421
nestedstructmap/Key4/Key42/Key422/0 : one
nestedstructmap/Key4/Key42/Key422/1 : two
nestedstructmap/Key4/Key42/Key422/2 : three
nestedstructmap/Key4/Key43/Key431/Key4311 : val4311