cluster

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2022 License: MPL-2.0 Imports: 19 Imported by: 0

Documentation

Overview

Package cluster contains EliasDB's clustering code.

The clustering code provides an abstraction layer to EliasDB's graphstorage.Storage. This means the actual storage of a cluster can be entirely memory based or use any other backend as long as it satisfies the graphstorage.Storage interface.

DistributedStorage wraps a graphstorage.Storage and has a manager.MemberManager object.

Members are identified by a unique name. Calling Start() on manager.MemberManager registers and starts the RPC server for the member. Cluster internal RPC requests are served by manager.Server. It is a singleton object which routes RPC calls to registered MemberManagers - this architecture makes it easy to unit test the cluster code. The manager.MemberManager has a manager.Client object which can be used to send messages to the cluster.

The integrity of the cluster is protected by a shared secret (string) among all members of the cluster. A new member can only join and communicate with the cluster if it has the secret string. The secret string is never transferred directly over the network - it is only used for generating a member specific token which can be verified by all other members.

The clustering code was inspired by Amazon DynamoDB http://www.allthingsdistributed.com/2012/01/amazon-dynamodb.html

Index

Constants

View Source 
const (
	RTGetMain RequestType = "GetMain"
	RTSetMain             = "SetMain"

	RTGetRoot = "GetRoot"
	RTSetRoot = "SetRoot"

	RTInsert = "Insert"

	RTUpdate = "Update"

	RTFree = "Free"

	RTExists = "Exists"

	RTFetch = "Fetch"

	RTRebalance = "Rebalance"
)

List of all possible request types

View Source 
const (
	RPStoreName DataRequestArg = "StoreName" // Name of the store
	RPLoc                      = "Loc"       // Location of data
	RPVer                      = "Ver"       // Version of data
	RPRoot                     = "Root"      // Root id
	RPSrc                      = "Src"       // Request source member
)

List of all possible data request parameters.

View Source 
const ClusterStoragePrefix = "cs_"

ClusterStoragePrefix is the prefix for cluster related storage managers

View Source 
const LocalStoragePrefix = "ls_"

LocalStoragePrefix is the prefix for local storage managers

View Source 
const MaxSizeRebalanceLists = 100

MaxSizeRebalanceLists is the maximum size for rebalancing lists within one rebalance request.

Variables

View Source 
var DSRetNew error

DSRetNew is the return value on successful creating a distributed storage (used for testing)

Functions

func ClearMSMap 

func ClearMSMap()

ClearMSMap clears the current map of known memory-only memberStorages.

func DumpMemoryClusterLayout 

func DumpMemoryClusterLayout(smname string) string

DumpMemoryClusterLayout returns the current storage layout in a memory-only cluster for a given storage manager (e.g. mainPerson.nodes for Person nodes of partition main).

func WaitForTransfer 

func WaitForTransfer()

WaitForTransfer waits for the datatransfer to happen.

Types

type DataRequest 

type DataRequest struct {
	RequestType RequestType                    // Type of request
	Args        map[DataRequestArg]interface{} // Request arguments
	Value       interface{}                    // Request value
	Transfer    bool                           // Flag for data transfer request
}

DataRequest data structure

type DataRequestArg 

type DataRequestArg string

DataRequestArg is a data request argument

type DistributedStorage 

type DistributedStorage struct {
	MemberManager *manager.MemberManager // Manager object
	// contains filtered or unexported fields
}

DistributedStorage data structure

func NewDistributedStorage 

func NewDistributedStorage(gs graphstorage.Storage, config map[string]interface{},
	stateInfo manager.StateInfo) (*DistributedStorage, error)

NewDistributedStorage creates a new cluster graph storage. The distributed storage wraps around a local graphstorage.Storage. The configuration of the distributed storage consists of two parts: A normal config map which defines static information like rpc port, secret string, etc and a StateInfo object which is used for dynamic information like cluster members, member status, etc. An empty StateInfo means that the cluster has only one member.

func (*DistributedStorage) Close 

func (ds *DistributedStorage) Close() error

Close closes the distributed storage.

func (*DistributedStorage) DistributionTable 

func (ds *DistributedStorage) DistributionTable() (*DistributionTable, error)

DistributionTable returns the current distribution table or an error if the storage is not available.

func (*DistributedStorage) FlushAll 

func (ds *DistributedStorage) FlushAll() error

FlushAll writes all pending local changes to the storage.

func (*DistributedStorage) FlushMain 

func (ds *DistributedStorage) FlushMain() error

FlushMain writes the main database to the storage.

func (*DistributedStorage) IsOperational 

func (ds *DistributedStorage) IsOperational() bool

IsOperational returns if this distribution storage is operational

func (*DistributedStorage) LocalName 

func (ds *DistributedStorage) LocalName() string

LocalName returns the local name of the wrapped DistributedStorage instance.

func (*DistributedStorage) MainDB 

func (ds *DistributedStorage) MainDB() map[string]string

MainDB returns the main database. The main database is a quick lookup map for meta data which is always kept in memory.

func (*DistributedStorage) Name 

func (ds *DistributedStorage) Name() string

Name returns the name of the cluster DistributedStorage instance.

func (*DistributedStorage) ReplicationFactor 

func (ds *DistributedStorage) ReplicationFactor() int

ReplicationFactor returns the replication factor of this cluster member. A value of 0 means the cluster is not operational in the moment.

func (*DistributedStorage) RollbackMain 

func (ds *DistributedStorage) RollbackMain() error

RollbackMain rollback the main database.

func (*DistributedStorage) SetDistributionTable 

func (ds *DistributedStorage) SetDistributionTable(dt *DistributionTable)

SetDistributionTable sets the distribution table and clears any error.

func (*DistributedStorage) SetDistributionTableError 

func (ds *DistributedStorage) SetDistributionTableError(err error)

SetDistributionTableError records an distribution table related error. This clears the current distribution table.

func (*DistributedStorage) Start 

func (ds *DistributedStorage) Start() error

Start starts the distributed storage.

func (*DistributedStorage) StorageManager 

func (ds *DistributedStorage) StorageManager(smname string, create bool) storage.Manager

StorageManager gets a storage manager with a certain name. A non-exisClusterting StorageManager is not created automatically if the create flag is set to false.

type DistributedStorageError 

type DistributedStorageError struct {
	// contains filtered or unexported fields
}

DistributedStorageError is an error related to the distribution storage. This error is returned when the data distribution fails for example when too many cluster members have failed.

func (*DistributedStorageError) Error 

func (dse *DistributedStorageError) Error() string

Error returns a string representation of a DistributedStorageError.

type DistributedStorageManager 

type DistributedStorageManager struct {
	// contains filtered or unexported fields
}

DistributedStorageManager is a storage.Manager which sends requests to the distributed storage.

func (*DistributedStorageManager) Close 

func (dsm *DistributedStorageManager) Close() error

Close is not implemented for a DistributedStorageManager. Only the local storage must be closed which is done when the DistributedStore is shut down.

func (*DistributedStorageManager) Exists 

func (dsm *DistributedStorageManager) Exists(loc uint64) (bool, error)

Exists checks if an object exists in a given storage location.

func (*DistributedStorageManager) Fetch 

func (dsm *DistributedStorageManager) Fetch(loc uint64, o interface{}) error

Fetch fetches an object from a given storage location and writes it to a given data container.

func (*DistributedStorageManager) FetchCached 

func (dsm *DistributedStorageManager) FetchCached(loc uint64) (interface{}, error)

FetchCached is not implemented for a DistributedStorageManager. Only defined to satisfy the StorageManager interface.

func (*DistributedStorageManager) Flush 

func (dsm *DistributedStorageManager) Flush() error

Flush is not implemented for a DistributedStorageManager. All changes are immediately written to disk in a cluster.

func (*DistributedStorageManager) Free 

func (dsm *DistributedStorageManager) Free(loc uint64) error

Free frees a storage location.

func (*DistributedStorageManager) Insert 

func (dsm *DistributedStorageManager) Insert(o interface{}) (uint64, error)

Insert inserts an object and return its storage location.

func (*DistributedStorageManager) Name 

func (dsm *DistributedStorageManager) Name() string

Name returns the name of the StorageManager instance.

func (*DistributedStorageManager) Rollback 

func (dsm *DistributedStorageManager) Rollback() error

Rollback is not implemented for a DistributedStorageManager. All changes are immediately written to disk in a cluster.

func (*DistributedStorageManager) Root 

func (dsm *DistributedStorageManager) Root(root int) uint64

Root returns a root value.

func (*DistributedStorageManager) SetRoot 

func (dsm *DistributedStorageManager) SetRoot(root int, val uint64)

SetRoot writes a root value.

func (*DistributedStorageManager) Update 

func (dsm *DistributedStorageManager) Update(loc uint64, o interface{}) error

Update updates a storage location.

type DistributionTable 

type DistributionTable struct {
	// contains filtered or unexported fields
}

DistributionTable is used to locate data in a cluster. The table contains all cluster members and can identify replication members for given data locations.

func NewDistributionTable 

func NewDistributionTable(members []string, repFac int) (*DistributionTable, error)

NewDistributionTable creates a new distribution table.

func (*DistributionTable) LocationHome 

func (dd *DistributionTable) LocationHome(loc uint64) (string, []string)

LocationHome return the member which is in charge of a given location and all its replicas.

func (*DistributionTable) MemberRange 

func (dd *DistributionTable) MemberRange(name string) (uint64, uint64)

MemberRange returns the location range of a given member.

func (*DistributionTable) Members 

func (dd *DistributionTable) Members() []string

Members returns all known cluster members.

func (*DistributionTable) OtherReplicationMembers 

func (dd *DistributionTable) OtherReplicationMembers(loc uint64, name string) []string

OtherReplicationMembers returns all members of a replication group (identified by a given locqtion) minus a given member.

func (*DistributionTable) Replicas 

func (dd *DistributionTable) Replicas(name string) []string

Replicas returns all replicas for a given member.

func (*DistributionTable) ReplicationRange 

func (dd *DistributionTable) ReplicationRange(name string) (uint64, uint64)

ReplicationRange return the location range which is replicated by a given member.

func (*DistributionTable) String 

func (dd *DistributionTable) String() string

String returns a string representation of this distribution table.

type RequestType 

type RequestType string

RequestType is the type of a request

Source Files

View all Source files

Directories

Path Synopsis
Package manager contains the management code for EliasDB's clustering feature.
 Package manager contains the management code for EliasDB's clustering feature.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.