lds3

README

LaunchDarkly Server-side SDK for Go - S3 data initializer

This repository is currently internal to LaunchDarkly. The library is not yet published or supported for external use.

This library provides an Amazon S3 data initializer for the LaunchDarkly Go SDK's data system. It reads one or more objects from an S3 bucket at SDK startup and uses their contents as the SDK's initial flag and segment data.

Initializers are one-shot: the SDK runs them once during startup to get flag values available as quickly as possible. Ongoing updates, if any, come from separately configured synchronizers. A typical use for this initializer is as a fallback tier so that an application can always start with a known-good flag snapshot, even when LaunchDarkly is unreachable.

Requirements

This library requires version 7.15.6 or higher of the LaunchDarkly Go SDK (the first release that includes the FDv2 data system's subsystems.DataInitializer interface), and Go 1.25 or higher.

Usage

Import this package (with an alias for readability):

import (
    lds3 "github.com/launchdarkly/go-server-sdk-s3"
)

Configure it as an initializer in the SDK's data system. For example, to try LaunchDarkly first and fall back to an S3 snapshot, with streaming for ongoing updates:

config := ld.Config{
    DataSystem: ldcomponents.DataSystem().Custom().
        Initializers(
            ldcomponents.PollingDataSourceV2().AsInitializer(),
            lds3.DataInitializer("my-bucket", "flags/snapshot.json"),
        ).
        Synchronizers(ldcomponents.StreamingDataSourceV2()),
}
client, err := ld.MakeCustomClient("sdk-key", config, 5*time.Second)

By default, the S3 client is created from the default AWS configuration sources (environment variables, shared config files, instance metadata). To customize it:

// With a specific AWS configuration:
lds3.DataInitializer("my-bucket", "snapshot.json").AWSConfig(myAWSConfig)

// Or with a preconfigured client, e.g. for an S3-compatible service:
lds3.DataInitializer("my-bucket", "snapshot.json").S3Client(myS3Client)

S3Client is the sanctioned way to use custom endpoints, path-style addressing, or local S3 stand-ins such as LocalStack or MinIO; the builder deliberately does not mirror the AWS SDK's endpoint options.

Two safety limits apply and can be adjusted:

// The whole fetch (all objects) must finish within this time; default 30s. The limit keeps a
// slow S3 request from consuming the SDK's entire initialization wait budget. Zero disables it.
lds3.DataInitializer("my-bucket", "snapshot.json").Timeout(10 * time.Second)

// Each object's content (after decompression, if compressed) may be at most this many bytes;
// default 100MB.
lds3.DataInitializer("my-bucket", "snapshot.json").MaxObjectSize(500 * 1024 * 1024)

Objects may be gzip-compressed if their Content-Encoding metadata is set to gzip. An object that contains gzip data without that metadata is rejected with an explanatory error.

Data format

Objects must contain flag data in the same JSON or YAML document format used by the SDK's file data sources, with any combination of flags, flagValues, and segments properties:

{
  "flagValues": {
    "my-boolean-flag-key": true,
    "my-string-flag-key": "value"
  }
}

See "Reading flags from a file" for a full description of the format. When multiple object keys are given, their contents are merged; by default a flag or segment key appearing in more than one object is an error, which can be changed with DuplicateKeysHandling.

Because this initializer's purpose is to provide a known-good fallback, it validates the data rather than applying whatever it finds:

• A snapshot that contains no flags and no segments is rejected. A corrupted, truncated, or misproduced object therefore causes a loud fallthrough to the next data source instead of silently turning off every flag. This also means a misspelled top-level property (for example "flagz") is caught.
• Documents must be valid UTF-8.
• A flag or segment whose embedded key property does not match the key it is defined under is rejected, as is a negative version.

Flags produced from flagValues entries have version 0, which is fine for a one-shot in-memory snapshot.

Prefer a single object per snapshot. Reads of multiple objects are sequential, so a publisher updating several objects concurrently can produce a mixed snapshot; if you use multiple objects, update them together and rely on the duplicate-keys check to catch overlap.

Initialization semantics

Data loaded from S3 is a snapshot: it has no FDv2 payload state, so the SDK treats it as "cached" data rather than "refreshed" (current) data. Flag evaluations serve the snapshot values immediately, but if data sources are configured, MakeCustomClient will keep waiting for a synchronizer to reach fresh data; if none can, it returns an initialization error even though the snapshot was applied. Applications using this initializer as an outage fallback should treat that error as a degraded-but-alive condition rather than a fatal one -- check client.Initialized(), which is true once the snapshot is applied.

Because of these semantics, place this initializer last in the initializer chain, after any initializers that can deliver current data from LaunchDarkly.

Out of scope

This library initializes once at startup. It does not watch S3 for changes or act as a synchronizer, it does not ask the SDK to persist the snapshot into a persistent store (Persist is always false), and it does not support the older FDv1 data system.

Contributing

This repository is maintained by the LaunchDarkly SDK team. While the repository is internal, changes follow the same pull request and review process as our other SDK repositories.

About LaunchDarkly

• LaunchDarkly is a continuous delivery platform that provides feature flags as a service and allows developers to iterate quickly and safely. We allow you to easily flag your features and manage them from the LaunchDarkly dashboard. With LaunchDarkly, you can:
  • Roll out a new feature to a subset of your users (like a group of users who opt-in to a beta tester group), gathering feedback and bug reports from real-world use cases.
  • Gradually roll out a feature to an increasing percentage of users, and track the effect that the feature has on key metrics (for instance, how likely is a user to complete a purchase if they have feature A versus feature B?).
  • Turn off a feature that you realize is causing performance problems in production, without needing to re-deploy, or even restart the application with a changed configuration file.
  • Grant access to certain features based on user attributes, like payment plan (eg: users on the 'gold' plan get access to more features than users in the 'silver' plan). Disable parts of your application to facilitate maintenance, without taking everything offline.
• LaunchDarkly provides feature flag SDKs for a wide variety of languages and technologies. Check out our documentation for a complete list.
• Explore LaunchDarkly
  • launchdarkly.com for more information
  • docs.launchdarkly.com for our documentation and SDK reference guides
  • apidocs.launchdarkly.com for our API documentation
  • blog.launchdarkly.com for the latest product updates

Documentation

Overview

Package lds3 provides an Amazon S3 data initializer for the LaunchDarkly Go SDK's data system.

The initializer reads one or more objects from an S3 bucket at startup and uses their contents as the SDK's initial set of flag and segment data. Initializers are one-shot: the SDK calls the initializer once during startup, and any ongoing updates must come from a separately configured synchronizer (or from nothing at all, if the S3 data is meant to be the only data source).

A typical use is a two-tier startup strategy: try LaunchDarkly first, and fall back to a known-good snapshot stored in S3 so that the application can always start, even during a network or service outage:

import (
    ld "github.com/launchdarkly/go-server-sdk/v7"
    "github.com/launchdarkly/go-server-sdk/v7/ldcomponents"
    lds3 "github.com/launchdarkly/go-server-sdk-s3"
)

config := ld.Config{
    DataSystem: ldcomponents.DataSystem().Custom().
        Initializers(
            ldcomponents.PollingDataSourceV2().AsInitializer(),
            lds3.DataInitializer("my-bucket", "flags/snapshot.json"),
        ).
        Synchronizers(ldcomponents.StreamingDataSourceV2()),
}
client, err := ld.MakeCustomClient("sdk-key", config, 5*time.Second)

The objects must be in the same JSON or YAML document format that the SDK's file data sources use, containing any combination of "flags", "flagValues", and "segments" properties. See the SDK's ldfiledata package documentation for a description of the format.

By default, the initializer creates an S3 client from the default AWS configuration sources (environment variables, shared config files, and instance metadata). Use the builder's methods to supply a custom aws.Config or a preconfigured client.

The initializer validates what it fetches: documents must be valid UTF-8, may be gzip-compressed when the object's Content-Encoding says so, are limited to a configurable maximum size, and a snapshot containing no flag or segment data is rejected rather than applied. The whole fetch is bounded by a configurable timeout so that a slow S3 request cannot consume the SDK's entire initialization wait budget.

Index

• Constants
• type DataInitializerBuilder
  • func DataInitializer(bucket string, objectKeys ...string) *DataInitializerBuilder
  • func (b *DataInitializerBuilder) AWSConfig(config aws.Config) *DataInitializerBuilder
  • func (b *DataInitializerBuilder) Build(clientContext subsystems.ClientContext) (subsystems.DataInitializer, error)
  • func (b *DataInitializerBuilder) DuplicateKeysHandling(duplicateKeysHandling DuplicateKeysHandling) *DataInitializerBuilder
  • func (b *DataInitializerBuilder) MaxObjectSize(limit int64) *DataInitializerBuilder
  • func (b *DataInitializerBuilder) ObjectKeys(objectKeys ...string) *DataInitializerBuilder
  • func (b *DataInitializerBuilder) S3Client(client S3Client) *DataInitializerBuilder
  • func (b *DataInitializerBuilder) Timeout(timeout time.Duration) *DataInitializerBuilder
• type DuplicateKeysHandling
• type S3Client

Constants

const (
	// DefaultMaxObjectSize is the default value for DataInitializerBuilder.MaxObjectSize: 100MB.
	DefaultMaxObjectSize int64 = 100 * 1024 * 1024

	// DefaultTimeout is the default value for DataInitializerBuilder.Timeout.
	DefaultTimeout = 30 * time.Second
)

Types

type DataInitializerBuilder

type DataInitializerBuilder struct {
	// contains filtered or unexported fields
}

DataInitializerBuilder is a builder for configuring the S3 data initializer.

Obtain an instance of this type by calling DataInitializer. After calling its methods to specify any desired custom settings, pass it to the Initializers method of github.com/launchdarkly/go-server-sdk/v7/ldcomponents.DataSystemConfigurationBuilder.

Builder calls can be chained, for example:

lds3.DataInitializer("my-bucket", "snapshot.json").AWSConfig(myConfig)

You do not need to call the builder's Build method yourself; that will be done by the SDK.

func DataInitializer

func DataInitializer(bucket string, objectKeys ...string) *DataInitializerBuilder

DataInitializer returns a configurable builder for an S3 data initializer.

The bucket is the name of the S3 bucket to read from, and objectKeys are the keys of one or more objects within it, each containing flag data in the SDK's file data source format. At least one object key is required; additional keys may also be added with DataInitializerBuilder.ObjectKeys.

func (*DataInitializerBuilder) AWSConfig

func (b *DataInitializerBuilder) AWSConfig(config aws.Config) *DataInitializerBuilder

AWSConfig specifies an AWS configuration to create the S3 client from, instead of the default configuration sources (environment variables, shared config files, and instance metadata).

This setting is ignored if a client is provided with DataInitializerBuilder.S3Client.

func (*DataInitializerBuilder) Build

func (b *DataInitializerBuilder) Build(
	clientContext subsystems.ClientContext,
) (subsystems.DataInitializer, error)

Build creates a DataInitializer instance that can be used by the SDK client.

func (*DataInitializerBuilder) DuplicateKeysHandling

func (b *DataInitializerBuilder) DuplicateKeysHandling(
	duplicateKeysHandling DuplicateKeysHandling,
) *DataInitializerBuilder

DuplicateKeysHandling specifies how to handle flag or segment keys that are duplicated across objects.

If this is not specified, or if you set it to an unrecognized value, the default is DuplicateKeysFail.

func (*DataInitializerBuilder) MaxObjectSize

func (b *DataInitializerBuilder) MaxObjectSize(limit int64) *DataInitializerBuilder

MaxObjectSize specifies the maximum size in bytes of a single object's flag data. The limit applies to each object separately, and to the decompressed content if an object is compressed. Reading stops with an error if an object exceeds it.

The limit exists so that a misconfigured object key cannot make the SDK buffer an arbitrarily large download during startup. If this is not specified, the default is DefaultMaxObjectSize. The value must be greater than zero.

func (*DataInitializerBuilder) ObjectKeys

func (b *DataInitializerBuilder) ObjectKeys(objectKeys ...string) *DataInitializerBuilder

ObjectKeys adds keys of objects to read from the bucket, after any keys that were previously specified.

func (*DataInitializerBuilder) S3Client

func (b *DataInitializerBuilder) S3Client(client S3Client) *DataInitializerBuilder

S3Client specifies an existing S3 client instance for the initializer to use, instead of creating one. Use this if you need full control of the client's configuration, for example to point it at an S3-compatible service.

func (*DataInitializerBuilder) Timeout

func (b *DataInitializerBuilder) Timeout(timeout time.Duration) *DataInitializerBuilder

Timeout specifies a time limit for the entire data-fetching operation, including reading every configured object. If the limit is reached, the initializer fails and the SDK moves on to the next initializer or synchronizer, instead of letting a slow S3 request consume the SDK's whole initialization wait budget.

If this is not specified, the default is DefaultTimeout. A value of zero disables the limit, leaving the operation bounded only by the SDK's initialization context. The value must not be negative.

type DuplicateKeysHandling

type DuplicateKeysHandling string

DuplicateKeysHandling is a parameter type used with DataInitializerBuilder.DuplicateKeysHandling.

const (
	// DuplicateKeysFail is an option for DataInitializerBuilder.DuplicateKeysHandling, meaning that
	// data loading should fail if keys are duplicated across objects. This is the default behavior.
	DuplicateKeysFail DuplicateKeysHandling = "fail"

	// DuplicateKeysKeepFirst is an option for DataInitializerBuilder.DuplicateKeysHandling,
	// meaning that if keys are duplicated across objects the first occurrence will be used, in the
	// order the object keys were given.
	DuplicateKeysKeepFirst DuplicateKeysHandling = "ignore"
)

type S3Client

type S3Client interface {
	GetObject(ctx context.Context, params *s3.GetObjectInput, optFns ...func(*s3.Options)) (*s3.GetObjectOutput, error)
}

S3Client is the subset of the AWS S3 client's operations that the data initializer uses. It is implemented by *s3.Client, and can be implemented by test fakes.