xload

package module
v0.9.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 10, 2024 License: MIT Imports: 12 Imported by: 4

README

xload - Load Anything

xload is a struct first data loader that simplifies the process of loading data from various sources into Go structs. It is designed to handle diverse use cases, from configuration loading to internationalization labels and even experiments.

Why xload?

We often encounter the need to load data from various sources, such as environment variables, command-line arguments, files, or remote configurations. This data may include configuration settings, experiment parameters, internationalization labels, design assets and more. Implementing these data loading methods can lead to boilerplate code, making the codebase hard to maintain and less readable.

xload is inspired by the popular go-envconfig library but extends it with custom data loaders and allows customizing struct tags, making it flexible for different use cases.

Installation

go get -u github.com/gojekfarm/xtools/xload

How xload Works

At a high level, xload takes a Go struct annotated with tags and populates it with data from any source. The sources can be anything that implements the Loader interface. By using xload, you can separate the data loading process from how and where the data is used.

Loading from Environment Variables

Let's take a look at a simple example of how to use xload to load data from environment variables into a struct:

type AppConfig struct {
    // your application config goes here
}

func DefaultAppConfig() AppConfig {
    return AppConfig{
        // set default values here
    }
}

func main() {
    ctx := context.Background()
    cfg := DefaultAppConfig()

    err := xload.Load(ctx, &cfg)
    if err != nil {
        panic(err)
    }

    // use cfg
}
Custom Types

With xload, you can also use custom types, such as time.Duration, to make the configuration more descriptive and easier to understand:

For example, timeouts need not be ambiguous int values, they can be typed time.Duration values:

type AppConfig struct {
    Timeout time.Duration `env:"TIMEOUT"` // TIMEOUT=5s will be parsed as 5 seconds
}

This helps you do away with naming conventions like *_TIMEOUT_MS or *_TIMEOUT_SECS.

Nested Structs

xload supports nested structs, slices, maps, and custom types making it easy to group, reuse and maintain complex configurations.

type HTTPConfig struct {
    Port int `env:"PORT"`
    Host string `env:"HOST"`
}

type AppConfig struct {
    Service1 HTTPConfig `env:",prefix=SERVICE1_"`
    Service2 HTTPConfig `env:",prefix=SERVICE2_"`
}

Inbuilt Loaders

xload comes with a few inbuilt loaders:

  • OSLoader - Loads data from environment variables
  • PrefixLoader - Prefixes keys before loading from another loader
  • SerialLoader - Loads data from multiple loaders, with last non-empty value winning

Custom Loaders

You can also write your own custom loaders by implementing the Loader interface:

type Loader interface {
    Load(ctx context.Context, key string) (string, error)
}

Documentation

Overview

Package xload is a struct first data loading library. xload provides a simple Loader interface to implement custom loaders and compose them in different ways.

Index

Examples

Constants

This section is empty.

Variables

View Source 
var (
	// ErrNotPointer is returned when the given config is not a pointer.
	ErrNotPointer = errors.New("xload: config must be a pointer")
	// ErrNotStruct is returned when the given config is not a struct.
	ErrNotStruct = errors.New("xload: config must be a struct")
	// ErrMissingKey is returned when the key is missing from the tag.
	ErrMissingKey = errors.New("xload: missing key on required field")
)
View Source 
var SkipCollisionDetection = &applier{f: func(o *options) { o.detectCollisions = false }}

SkipCollisionDetection disables detecting any key collisions while trying to load full keys.

Functions

func FlattenMap 

func FlattenMap(m map[string]interface{}, sep string) map[string]string

FlattenMap flattens a map[string]interface{} into a map[string]string. Nested maps are flattened using given separator.

func Load 

func Load(ctx context.Context, v any, opts ...Option) error

Load loads values into the given struct using the given options. "env" is used as the default tag name. xload.OSLoader is used as the default loader.

Example (ArrayDelimiter) 
package main

import (
	"context"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		// value will be split by |, instead of ,
		// e.g. HOSTS=host1|host2|host3
		Hosts []string `env:"HOSTS,delimiter=|"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (ConcurrentLoading) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	loader := xload.LoaderFunc(func(ctx context.Context, key string) (string, error) {
		// lookup value from a remote service

		// NOTE: this function is called for each key concurrently
		// so make sure it is thread-safe.
		// Using a pooled client is recommended.
		return "", nil
	})

	err := xload.Load(
		context.Background(),
		&conf,
		xload.Concurrency(3), // load 3 keys concurrently
		xload.FieldTagName("env"),
		loader,
	)
	if err != nil {
		panic(err)
	}
}

Output:
Example (CustomDecoder) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

type Host string

func (h *Host) Decode(val string) error {

	return nil
}

func main() {
	// Custom decoder can be used for any type that
	// implements the Decoder interface.

	type AppConf struct {
		Host    Host          `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (CustomLoader) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	loader := xload.LoaderFunc(func(ctx context.Context, key string) (string, error) {
		// lookup value from somewhere
		return "", nil
	})

	err := xload.Load(
		context.Background(),
		&conf,
		xload.FieldTagName("env"),
		loader,
	)
	if err != nil {
		panic(err)
	}
}

Output:
Example (CustomTagNames) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `custom:"HOST"`
		Debug   bool          `custom:"DEBUG"`
		Timeout time.Duration `custom:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf, xload.FieldTagName("custom"))
	if err != nil {
		panic(err)
	}
}

Output:
Example (DecodingJSONValue) 
package main

import (
	"context"
	"encoding/json"

	"github.com/gojekfarm/xtools/xload"
)

type ServiceAccount struct {
	ProjectID string `json:"project_id"`
	ClientID  string `json:"client_id"`
}

func (sa *ServiceAccount) UnmarshalJSON(data []byte) error {
	type Alias ServiceAccount

	var alias Alias

	err := json.Unmarshal(data, &alias)
	if err != nil {
		return err
	}

	*sa = ServiceAccount(alias)

	return nil
}

func main() {
	// Decoding JSON value can be done by implementing
	// the json.Unmarshaler interface.
	//
	// If using json.Unmarshaler, use type alias to avoid
	// infinite recursion.

	type AppConf struct {
		ServiceAccount ServiceAccount `env:"SERVICE_ACCOUNT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (Default) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (ExtendingStructs) 
package main

import (
	"context"
	"net/url"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type Host struct {
		URL       url.URL `env:"URL"`
		Telemetry bool    `env:"TELEMETRY"`
	}

	type DB struct {
		Host
		Username string `env:"USERNAME"`
		Password string `env:"PASSWORD"`
	}

	type HTTP struct {
		Host
		Timeout time.Duration `env:"TIMEOUT"`
	}

	type AppConf struct {
		DB   DB   `env:",prefix=DB_"`
		HTTP HTTP `env:",prefix=HTTP_"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (MapSeparator) 
package main

import (
	"context"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		// key value pair will be split by :, instead of =
		// e.g. HOSTS=db:localhost,cache:localhost
		Hosts map[string]string `env:"HOSTS,separator=:"`
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (PrefixLoader) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(
		context.Background(),
		&conf,
		xload.PrefixLoader("MYAPP_", xload.OSLoader()),
	)
	if err != nil {
		panic(err)
	}
}

Output:
Example (Required) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"HOST,required"`
		Debug   bool          `env:"DEBUG"`
		Timeout time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	// if HOST is not set, Load will return ErrRequired
	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (SkipCollisionDetection) 
package main

import (
	"context"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host string `env:"HOST"`
		// Hostname is also loaded from HOST, and no error is returned
		Hostname string        `env:"HOST"`
		Timeout  time.Duration `env:"TIMEOUT"`
	}

	var conf AppConf

	err := xload.Load(
		context.Background(),
		&conf,
		xload.SkipCollisionDetection,
	)
	if err != nil {
		panic(err)
	}
}

Output:
Example (Structs) 
package main

import (
	"context"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type DBConf struct {
		Host string `env:"HOST"` // will be loaded from DB_HOST
		Port int    `env:"PORT"` // will be loaded from DB_PORT
	}

	type HTTPConf struct {
		Host string `env:"HTTP_HOST"` // will be loaded from HTTP_HOST
		Port int    `env:"HTTP_PORT"` // will be loaded from HTTP_PORT
	}

	type AppConf struct {
		DB   DBConf   `env:",prefix=DB_"` // example of prefix for nested struct
		HTTP HTTPConf // example of embedded struct
	}

	var conf AppConf

	err := xload.Load(context.Background(), &conf)
	if err != nil {
		panic(err)
	}
}

Output:
Example (TransformFieldName) 
package main

import (
	"context"
	"strings"
	"time"

	"github.com/gojekfarm/xtools/xload"
)

func main() {
	type AppConf struct {
		Host    string        `env:"MYAPP_HOST"`
		Debug   bool          `env:"MYAPP_DEBUG"`
		Timeout time.Duration `env:"MYAPP_TIMEOUT"`
	}

	var conf AppConf

	// transform converts key from MYAPP_HOST to myapp.host
	transform := func(next xload.Loader) xload.LoaderFunc {
		return func(ctx context.Context, key string) (string, error) {
			newKey := strings.ReplaceAll(key, "_", ".")
			newKey = strings.ToLower(newKey)

			return next.Load(ctx, newKey)
		}
	}

	err := xload.Load(
		context.Background(),
		&conf,
		transform(xload.OSLoader()),
	)
	if err != nil {
		panic(err)
	}
}

Output:

Types

type Concurrency added in v0.4.1 

type Concurrency int

Concurrency allows customising the number of goroutines to use. Default is 1.

type Decoder 

type Decoder interface {
	Decode(string) error
}

Decoder is the interface for custom decoders.

type ErrCast added in v0.8.1 

type ErrCast struct {
	// contains filtered or unexported fields
}

ErrCast wraps the actual error that occurred during casting value to go type.

func (ErrCast) Error added in v0.8.1 

func (e ErrCast) Error() string

func (ErrCast) Unwrap added in v0.8.1 

func (e ErrCast) Unwrap() error

type ErrCollision added in v0.8.0 

type ErrCollision struct {
	// contains filtered or unexported fields
}

ErrCollision is returned when key collisions are detected. Collision can happen when two or more fields have the same full key.

func (ErrCollision) Error added in v0.8.0 

func (e ErrCollision) Error() string

func (ErrCollision) Keys added in v0.8.0 

func (e ErrCollision) Keys() []string

Keys returns the collided keys.

type ErrDecode added in v0.8.1 

type ErrDecode struct {
	// contains filtered or unexported fields
}

ErrDecode wraps the actual error that occurred during decoding value to go type.

func (ErrDecode) Error added in v0.8.1 

func (e ErrDecode) Error() string

func (ErrDecode) Unwrap added in v0.8.1 

func (e ErrDecode) Unwrap() error

func (ErrDecode) Value added in v0.8.1 

func (e ErrDecode) Value() string

Value returns the raw value that was used to decode the key.

type ErrInvalidMapValue 

type ErrInvalidMapValue struct {
	// contains filtered or unexported fields
}

ErrInvalidMapValue is returned when the map value is invalid.

func (ErrInvalidMapValue) Error added in v0.6.0 

func (e ErrInvalidMapValue) Error() string

type ErrInvalidPrefix 

type ErrInvalidPrefix struct {
	// contains filtered or unexported fields
}

ErrInvalidPrefix is returned when the prefix option is used on a non-struct key.

func (ErrInvalidPrefix) Error added in v0.6.0 

func (e ErrInvalidPrefix) Error() string

type ErrInvalidPrefixAndKey added in v0.4.1 

type ErrInvalidPrefixAndKey struct {
	// contains filtered or unexported fields
}

ErrInvalidPrefixAndKey is returned when the prefix option is used with a key.

func (ErrInvalidPrefixAndKey) Error added in v0.6.0 

func (e ErrInvalidPrefixAndKey) Error() string

type ErrRequired 

type ErrRequired struct {
	// contains filtered or unexported fields
}

ErrRequired is returned when a required key is missing.

func (ErrRequired) Error added in v0.6.0 

func (e ErrRequired) Error() string

type ErrUnknownFieldType 

type ErrUnknownFieldType struct {
	// contains filtered or unexported fields
}

ErrUnknownFieldType is returned when the key type is not supported.

func (ErrUnknownFieldType) Error added in v0.6.0 

func (e ErrUnknownFieldType) Error() string

type ErrUnknownTagOption 

type ErrUnknownTagOption struct {
	// contains filtered or unexported fields
}

ErrUnknownTagOption is returned when an unknown tag option is used.

func (ErrUnknownTagOption) Error added in v0.6.0 

func (e ErrUnknownTagOption) Error() string

type FieldTagName 

type FieldTagName string

FieldTagName allows customising the struct tag name to use.

type Loader 

type Loader interface {
	Load(ctx context.Context, key string) (string, error)
}

Loader defines the interface for a config loader.

type LoaderFunc 

type LoaderFunc func(ctx context.Context, key string) (string, error)

LoaderFunc is a function that implements Loader.

func OSLoader 

func OSLoader() LoaderFunc

OSLoader loads values from the OS environment.

func PrefixLoader 

func PrefixLoader(prefix string, loader Loader) LoaderFunc

PrefixLoader wraps a loader and adds a prefix to all keys.

func SerialLoader 

func SerialLoader(loaders ...Loader) LoaderFunc

SerialLoader loads values from multiple loaders. Last non-empty value wins.

func (LoaderFunc) Load 

func (f LoaderFunc) Load(ctx context.Context, key string) (string, error)

Load returns the config value for the given key.

type MapLoader 

type MapLoader map[string]string

MapLoader loads values from a map.

Can be used with xload.FlattenMap as an intermediate format when loading from various sources.

func (MapLoader) Load 

func (m MapLoader) Load(_ context.Context, key string) (string, error)

Load fetches the value from the map.

type Option 

type Option interface {
	// contains filtered or unexported methods
}

Option configures the xload behaviour.

func WithLoader 

func WithLoader(loader Loader) Option

WithLoader allows customising the loader to use.

Source Files

View all Source files

Directories

Path Synopsis
providers
cached Module
viper Module
yaml Module
Package xloadtype contains commonly used types for working with xload.Loader.
 Package xloadtype contains commonly used types for working with xload.Loader.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.