shadowbackend

README

Kraken Shadow Datastore

This backend is designed for transitioning Kraken to a new backend by allowing admins to specify an "active" and "shadow" backend. The term shadow is used because the other backend "shadows" the active backend. Writes are sent to both backends, but reads only occur from the active. This ensures data consistency between the backends, and allows the old, proven backed to act as a safety net in the case where the new backend fails or has to be taken offline due to some unforeseen problems.

Because all reads occur from the active backend, data needs to be manually migrated from the old backend to the new, before bringing the shadow backend online. This means there will be some downtime incurred during the transition.

Supported Backends

This currently supports SQL, HDFS, S3, and testfs backends.

Note that this backend is only enabled as a tag datastore for the build-index. However this could be used as the blob datastore for the origin servers, provided it is not configured to use the SQL backend in active or shadow modes.

Configuration

The config has two required items "active_backend" and "shadow_backend", each of which function the same as the standard "backend" in that the backend type and normal configuration is defined under each, depending on which role that backend will play. For example, here is a configuration that uses the SQL backend as the active, and the testfs backend as the shadow:

backends:
  - namespace: .*
    backend:
      shadow:
        active_backend:
          sql:
            dialect: mysql
            connection_string: "kraken:kraken@tcp(kraken-mysql:3306)/kraken?parseTime=True"
            debug_logging: true
        shadow_backend:
          testfs:
            addr: localhost:7357
            root: tags
            name_path: docker_tag

Documentation

Overview

Copyright (c) 2016-2020 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2020 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• type Client
  • func NewClient(config Config, masterAuthConfig backend.AuthConfig, stats tally.Scope) (*Client, error)
  • func (c *Client) Download(namespace string, name string, dst io.Writer) error
  • func (c *Client) List(prefix string, opts ...backend.ListOption) (*backend.ListResult, error)
  • func (c *Client) Stat(namespace string, name string) (*core.BlobInfo, error)
  • func (c *Client) Upload(namespace string, name string, src io.Reader) error
• type Config
• type Option

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client implements a backend.Client for shadow mode. See the README for full details on what shadow mode means.

func NewClient

func NewClient(config Config, masterAuthConfig backend.AuthConfig, stats tally.Scope) (*Client, error)

NewClient creates a new shadow Client

func (*Client) Download

func (c *Client) Download(namespace string, name string, dst io.Writer) error

Download gets the data from the backend and then writes it to the output writer.

func (*Client) List

func (c *Client) List(prefix string, opts ...backend.ListOption) (*backend.ListResult, error)

List lists names with start with prefix.

func (*Client) Stat

func (c *Client) Stat(namespace string, name string) (*core.BlobInfo, error)

Stat returns a non-nil core.BlobInfo struct if the data exists, an error otherwise.

func (*Client) Upload

func (c *Client) Upload(namespace string, name string, src io.Reader) error

Upload upserts the data into the backend.

type Config

type Config struct {
	ActiveClientConfig map[string]interface{} `yaml:"active_backend"`
	ShadowClientConfig map[string]interface{} `yaml:"shadow_backend"`
}

Config is used to initialize the Shadow client

type Option

type Option func(*Client)

Option allows setting optional Client parameters.