flatten

README

flatten

English | 中文

flatten is a Go package for working with hierarchical key-value data structures, primarily for handling nested data in formats like JSON, YAML, or TOML.

Features

1. Data Flattening

• Converts nested maps, slices, and arrays into a flat map[string]string.
• Uses dot notation for maps (e.g., db.hosts).
• Uses index notation for arrays/slices (e.g., hosts[0]).
• Example: {"db": {"hosts": ["a", "b"]}} becomes {"db.hosts[0]": "a", "db.hosts[1]": "b"}.

2. Path Handling

• Defines a Path abstraction, representing hierarchical keys as a sequence of typed segments (map keys or array indices).
• Supports parsing string paths (e.g., "foo.bar[0]") into Path objects.
• Supports converting Path objects back into string paths.

3. Storage Management

• The Storage type manages a collection of flattened key-value pairs.
• Internally builds and maintains a hierarchical tree structure to prevent key conflicts.
• Associates values with their source files, supporting multi-file merging and source tracking.

4. Querying

• Provides helper methods for retrieving values.
• Checks for the existence of keys.
• Enumerates subkeys.
• Iterates in a deterministic order.

Typical Use Cases

1. Standardizing configuration files from multiple sources into a flat key-value map for comparison, merging, or diffing.
2. Querying nested data with simple string paths, avoiding direct reflection or manual traversal of nested maps.
3. Building tools that unify structured data from multiple files while preserving source information and preventing conflicts.

Example

package main

import (
	"fmt"
	"github.com/go-spring/stdlib/flatten"
)

func main() {
	// Create a nested data structure
	data := map[string]interface{}{
		"database": map[string]interface{}{
			"host": "localhost",
			"port": 5432,
			"credentials": map[string]interface{}{
				"username": "admin",
				"password": "secret",
			},
		},
		"features": []interface{}{
			"feature1",
			"feature2",
			map[string]interface{}{
				"name":    "feature3",
				"enabled": true,
			},
		},
	}

	// Flatten the data
	flat := flatten.Flatten(data)

	// Print flattened results
	for key, value := range flat {
		fmt.Printf("%s: %s\n", key, value)
	}

	// Use Storage to manage data
	storage := flatten.NewStorage()
	fileID := storage.AddFile("config.yaml")

	// Set values
	storage.Set("database.host", "localhost", fileID)
	storage.Set("database.port", "5432", fileID)

	// Retrieve values
	host := storage.Get("database.host")
	fmt.Printf("Database host: %s\n", host)

	// Check if a key exists
	if storage.Has("database.host") {
		fmt.Println("Database host exists")
	}
}

License

Apache License 2.0

Documentation

Overview

Package flatten provides utilities for handling hierarchical key/value data structures that commonly appear in configuration formats such as JSON, YAML, or TOML. It is designed to transform nested data into a flat representation, while preserving enough metadata to reconstruct paths, detect conflicts, and manage data from multiple sources.

Key features include:

• Flattening: Nested maps, slices, and arrays can be converted into a flat map[string]string using dot notation for maps and index notation for arrays/slices. For example, {"db": {"hosts": ["a", "b"]}} becomes {"db.hosts[0]": "a", "db.hosts[1]": "b"}.

• Path handling: The package defines a Path abstraction that represents hierarchical keys as a sequence of typed segments (map keys or array indices). Paths can be split from strings like "foo.bar[0]" or joined back into their string form.

• Storage: A Storage type manages a collection of flattened key/value pairs. It builds and maintains a hierarchical tree internally to prevent property conflicts (e.g., treating the same key as both a map and a value). Storage also associates values with the files they originated from, which allows multi-file merging and provenance tracking.

• Querying: The Storage type provides helper methods for retrieving values, checking for the existence of keys, enumerating subkeys, and iterating in a deterministic order.

Typical use cases:

• Normalizing configuration files from different sources into a flat key/value map for comparison, merging, or diffing.
• Querying nested data using simple string paths without dealing with reflection or nested map structures directly.
• Building tools that need to unify structured data from multiple files while preserving provenance information and preventing conflicts.

Overall, flatten acts as a bridge between deeply nested structured data and flat, queryable representations that are easier to work with in configuration management, testing, or data transformation pipelines.

Index

• func Flatten(m map[string]any) map[string]string
• func JoinPath(path []Path) string
• type Path
  • func SplitPath(key string) (_ []Path, err error)
• type PathType
• type Storage
  • func NewStorage() *Storage
  • func (s *Storage) AddFile(file string) int8
  • func (s *Storage) Data() map[string]string
  • func (s *Storage) Get(key string, def ...string) string
  • func (s *Storage) Has(key string) bool
  • func (s *Storage) Keys() []string
  • func (s *Storage) Lookup(key string) (string, bool)
  • func (s *Storage) Merge(p *Storage) error
  • func (s *Storage) MergeMap(data map[string]any, file string) error
  • func (s *Storage) RawData() map[string]ValueInfo
  • func (s *Storage) RawFile() map[string]int8
  • func (s *Storage) Set(key string, val string, file int8) error
  • func (s *Storage) SubKeys(key string) (_ []string, err error)
  • func (s *Storage) SubMap(key string) (map[string]string, error)
• type ValueInfo

Functions

func Flatten

func Flatten(m map[string]any) map[string]string

Flatten flattens a nested map[string]any into a map[string]string.

This function is intended for data produced by encoding/json.Unmarshal, where values are limited to the following kinds:

• map[string]any
• []any
• primitive JSON types (bool, number, string, nil)

Structs, custom types, and non-string map keys are explicitly out of scope.

Flattening rules:

• Nested maps are expanded using dot notation: {"a": {"b": 1}} -> "a.b" = "1"

• Slices (and arrays, although arrays do not originate from json.Unmarshal) are expanded using index notation: {"a": [1, 2]} -> "a[0]" = "1", "a[1]" = "2"

• Nil values (both untyped nil and typed nil) are represented as "<nil>".

• Empty (zero-length but non-nil) maps are represented as "{}".

• Empty (zero-length but non-nil) slices are represented as "[]".

• Primitive values are converted to strings using deterministic, Go-native formatting (strconv).

The resulting map is intended for display-oriented use cases such as logging, diffing, diagnostics, or inspection. The output is not reversible and must not be treated as a lossless serialization format.

func JoinPath

func JoinPath(path []Path) string

JoinPath converts a slice of Path objects into a string representation. Keys are joined with dots, and array indices are wrapped in square brackets. Example: [key, index(0), key] => "key[0].key".

Types

type Path

type Path struct {
	// Whether the element is a key or an index.
	Type PathType

	// Actual key or index value as a string.
	// For PathTypeKey, it's the key string;
	// for PathTypeIndex, it's the index number as a string.
	Elem string
}

Path represents a single segment in a parsed key path. A path is composed of multiple Path elements that can be joined or split. For example, "foo.bar[0]" parses into:

[{Type: PathTypeKey, Elem: "foo"},
 {Type: PathTypeKey, Elem: "bar"},
 {Type: PathTypeIndex, Elem: "0"}].

func SplitPath

func SplitPath(key string) (_ []Path, err error)

SplitPath parses a hierarchical key string into a slice of Path objects. It supports dot-notation for maps and bracket-notation for arrays. Examples:

"foo.bar[0]" -> [{Key:"foo"}, {Key:"bar"}, {Index:"0"}]
"a[1][2]"    -> [{Key:"a"}, {Index:"1"}, {Index:"2"}]

Rules:

• Keys must be non-empty strings without spaces.
• Indices must be unsigned integers (no sign, no decimal).
• Empty maps/slices are not special-cased here.
• Returns an error if the key is malformed (e.g. unbalanced brackets, unexpected characters, or empty keys if disallowed).

type PathType

type PathType int8

PathType represents the type of a path element in a hierarchical key. A path element can either be a key (map field) or an index (array/slice element).

const (
	PathTypeKey   PathType = iota // A named key in a map.
	PathTypeIndex                 // A numeric index in a list.
)

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage manages hierarchical key/value data with structural validation. It provides:

• A hierarchical tree (root) for detecting structural conflicts.
• A flat map (data) for quick value lookups.
• An empty map (empty) for representing empty containers like "[]" or "{}".
• A file map for mapping file names to numeric indexes, allowing traceability.

Invariants:

• `root` stores only the tree structure (no leaf values).
• `data` stores leaf key-value pairs.
• `empty` stores leaf paths that represent empty arrays/maps or nil values.

func NewStorage

func NewStorage() *Storage

NewStorage creates a new Storage instance.

func (*Storage) AddFile

func (s *Storage) AddFile(file string) int8

AddFile registers a file name in the Storage and assigns it a unique int8 index if not already registered. Returns the index assigned to the given file.

func (*Storage) Data

func (s *Storage) Data() map[string]string

Data returns a simplified flattened key → string value mapping, omitting file index information.

func (*Storage) Get

func (s *Storage) Get(key string, def ...string) string

Get retrieves the value associated with the given flattened key. If the key is not found and a default value is provided, the default is returned instead. Only the first default value is considered.

func (*Storage) Has

func (s *Storage) Has(key string) bool

Has checks whether a given key (or path) exists in the Storage. Returns true if the key refers to either a stored value, an empty container, or a valid intermediate node in the hierarchy.

func (*Storage) Keys

func (s *Storage) Keys() []string

Keys returns all flattened keys currently stored in Storage, sorted lexicographically for consistent iteration.

func (*Storage) Lookup

func (s *Storage) Lookup(key string) (string, bool)

Lookup retrieves the value associated with the given flattened key. Returns the value and true if the key is found, or an empty string and false if the key is not found.

func (*Storage) Merge

func (s *Storage) Merge(p *Storage) error

Merge adds all key/value pairs from the given Storage to the Storage.

func (*Storage) MergeMap

func (s *Storage) MergeMap(data map[string]any, file string) error

MergeMap adds all key/value pairs from the given map to the Storage.

func (*Storage) RawData

func (s *Storage) RawData() map[string]ValueInfo

RawData exposes the internal flattened key → ValueInfo mapping, combining both data and empty containers if any exist.

WARNING: This method leaks internal state and should be used with caution (e.g., for debugging or low-level access).

func (*Storage) RawFile

func (s *Storage) RawFile() map[string]int8

RawFile exposes the internal file name → index mapping.

func (*Storage) Set

func (s *Storage) Set(key string, val string, file int8) error

Set inserts or updates a flattened key with the given value and the index of the file it originated from.

It validates the path to prevent structural conflicts:

• Cannot store a value where a container node already exists.
• Cannot change an array branch into a map branch or vice versa.

Returns an error if a structural conflict is detected.

func (*Storage) SubKeys

func (s *Storage) SubKeys(key string) (_ []string, err error)

SubKeys returns the immediate child keys under the given hierarchical path.

For example, if Storage contains keys:

a.b.c
a.b.d

then SubKeys("a.b") returns ["c", "d"].

If the path points to a leaf value or structural conflict, an error is returned. If the path does not exist, it returns nil.

func (*Storage) SubMap

func (s *Storage) SubMap(key string) (map[string]string, error)

SubMap returns a map of the immediate child keys under the given hierarchical path.

type ValueInfo

type ValueInfo struct {
	File  int8
	Value string
}

ValueInfo holds both the string value and the index of the file from which the value originated. This enables tracking of data provenance.