ring

package
v1.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 30, 2018 License: Apache-2.0 Imports: 27 Imported by: 11

Documentation

Index

Constants

View Source 
const (
	NONE_DEV                 uint    = 65535
	MAX_BALANCE              float64 = 999.99
	MAX_BALANCE_GATHER_COUNT int     = 3
)

Variables

View Source 
var CacheMiss = fmt.Errorf("Server cache miss")

Functions

func AddDevice 

func AddDevice(builderPath string, id, region, zone int64, scheme, ip string, port int64, replicationIp string, replicationPort int64, device string, weight float64, debug bool) (int64, error)

AddDevice adds a device to the builder filer 

builderpath must include the filename of the builder file.
Returns the id of the device in the ring.

Note that no locking is done here, you should call LockBuilderPath first.

func CreateRing 

func CreateRing(builderPath string, partPower int, replicas float64, minPartHours int, debug bool) error

CreateRing creates a ring builder file. 

builderpath must include the filename of the the builder to create.
A backup folder will also be created in the back with a backup of the original builder.

Note that no locking is done here, you should call LockBuilderPath first.

func LockBuilderPath added in v1.3.0 

func LockBuilderPath(pth string) (*os.File, error)

func NewMemcacheRing 

func NewMemcacheRing(confPath string) (*memcacheRing, error)

func NewMemcacheRingFromConfig 

func NewMemcacheRingFromConfig(config conf.Config) (*memcacheRing, error)

func NewTracingMemcacheRing added in v1.4.0 

func NewTracingMemcacheRing(mc MemcacheRing, tracer opentracing.Tracer) *tracingMemcacheRing

func PretendMinPartHoursPassed added in v0.0.2 

func PretendMinPartHoursPassed(builderPath string) error

Note that no locking is done here, you should call LockBuilderPath first.

func Rebalance 

func Rebalance(builderPath string, debug bool, dryrun bool, quiet bool) (int, float64, int, error)

Rebalance attempts to rebalance the ring by reassigning partitions that haven't been recently reassigned. Note that no locking is done here, you should call LockBuilderPath first.

func RemoveDevs added in v0.0.2 

func RemoveDevs(builderPath string, devs []*RingBuilderDevice, purge bool) error

Note that no locking is done here, you should call LockBuilderPath first.

func SetInfo added in v0.0.2 

func SetInfo(builderPath string, devs []*RingBuilderDevice, newIp string, newPort int64, newRepIp string, newRepPort int64, newDevice, newMeta string, newScheme string) error

Note that no locking is done here, you should call LockBuilderPath first.

func SetWeight added in v0.0.2 

func SetWeight(builderPath string, devs []*RingBuilderDevice, weight float64) error

Note that no locking is done here, you should call LockBuilderPath first.

func Validate added in v0.0.2 

func Validate(builderPath string) error

Note that no locking is done here, you should call LockBuilderPath first.

func WriteRing added in v0.0.2 

func WriteRing(builderPath string) error

Note that no locking is done here, you should call LockBuilderPath first.

Types

type Device 

type Device struct {
	Id              int     `json:"id"`
	Device          string  `json:"device"`
	Scheme          string  `json:"scheme"`
	Ip              string  `json:"ip"`
	Meta            string  `json:"meta"`
	Port            int     `json:"port"`
	Region          int     `json:"region"`
	ReplicationIp   string  `json:"replication_ip"`
	ReplicationPort int     `json:"replication_port"`
	Weight          float64 `json:"weight"`
	Zone            int     `json:"zone"`
}

func (*Device) Active added in v1.2.0 

func (d *Device) Active() bool

func (*Device) String 

func (d *Device) String() string

type MemcacheRing 

type MemcacheRing interface {
	Decr(ctx context.Context, key string, delta int64, timeout int) (int64, error)
	Delete(ctx context.Context, key string) error
	Get(ctx context.Context, key string) (interface{}, error)
	GetStructured(ctx context.Context, key string, val interface{}) error
	GetMulti(ctx context.Context, serverKey string, keys []string) (map[string]interface{}, error)
	Incr(ctx context.Context, key string, delta int64, timeout int) (int64, error)
	Set(ctx context.Context, key string, value interface{}, timeout int) error
	SetMulti(ctx context.Context, serverKey string, values map[string]interface{}, timeout int) error
}

type MoreNodes 

type MoreNodes interface {
	Next() *Device
}

type Ring 

type Ring interface {
	GetNodes(partition uint64) (response []*Device)
	GetJobNodes(partition uint64, localDevice int) (response []*Device, handoff bool)
	GetPartition(account string, container string, object string) uint64
	LocalDevices(localPort int) (devs []*Device, err error)
	AllDevices() (devs []*Device)
	GetMoreNodes(partition uint64) MoreNodes
	ReplicaCount() (cnt uint64)
	PartitionCount() (cnt uint64)
	PartitionForHash(string) (uint64, error)
}

func GetRing 

func GetRing(ringType, prefix, suffix string, policy int) (Ring, error)

GetRing returns the current ring given the ring_type ("account", "container", "object"), hash path prefix, and hash path suffix. An error is returned if the requested ring does not exist.

func LoadRing 

func LoadRing(path string, prefix string, suffix string) (Ring, error)

type RingBuilder 

type RingBuilder struct {
	PartPower    int
	Replicas     float64
	MinPartHours int
	Parts        int
	Overload     float64
	Devs         []*RingBuilderDevice
	DevsChanged  bool
	Version      int
	Dispersion   float64

	Debug bool
	// contains filtered or unexported fields
}

func GetRingBuilder added in v1.3.0 

func GetRingBuilder(ringType string, policy int) (*RingBuilder, string, error)

GetRingBuilder will return the builder and builder-file-path or an error for the ring type and policy given. Note that no locking is done here, you should call LockBuilderPath and then call GetRingBuilder again.

func NewRingBuilder 

func NewRingBuilder(partPower int, replicas float64, minPartHours int, debug bool) (*RingBuilder, error)

func NewRingBuilderFromFile 

func NewRingBuilderFromFile(builderPath string, debug bool) (*RingBuilder, error)

Note that no locking is done here, you should call LockBuilderPath first.

func (*RingBuilder) AddDev 

func (b *RingBuilder) AddDev(dev *RingBuilderDevice) (int64, error)

AddDev adds a device to the ring

Note: This will not reblance the ring immediately as you may want to make multiple changes for a single rebalance

Returns the id of the device

func (*RingBuilder) ChangeMinPartHours 

func (b *RingBuilder) ChangeMinPartHours(minPartHours int)

ChangeMinPartHours changes the value used to decide if a given partition can be moved again. This restriction is to give the overall system enough time to settl a partition to its new location before moving it to yet another location. While no data would be lost if a partition is moved several times quickly, it could make the data unreachable for a short period of time.

This should be set to at least the average full partition replication time. Starting it at 24 hours and then lowering it to what the replicator reprots as the longest partition cycle is best.

func (*RingBuilder) GetBalance 

func (b *RingBuilder) GetBalance() float64

GetBalance gets the balance of the ring.

The balance value is the highest percentage of the desired amount of partitions a given device wants. For instance, if the "worst" device wants (based on tis weight relative to the sum of all devices' weights) 123 partitions and it has 124 partitions, the balance value would be 0.83 (1 extra / 123 wanted * 100)

func (*RingBuilder) GetRequiredOverload 

func (b *RingBuilder) GetRequiredOverload(weighted map[string]float64, wanted map[string]float64) (float64, error)

GetRequiredOverload returns the minimum overload value required to make the ring maximally dispersed.

The required overload is the largest percentage change of any single device from its weighted replicanth to its wanted replicanth (note: under weighted devices have a negative percentage of change) to achieve dispersion - that is to say a single device that must be overloaded by 5% is worse than 5 devices in a single tier overloaded by 1%.

func (*RingBuilder) GetRing 

func (b *RingBuilder) GetRing() *hashRing

func (*RingBuilder) MinPartSecondsLeft 

func (b *RingBuilder) MinPartSecondsLeft() int

MinPartSecondsLeft returns the total seconds until a reblanace can be performed.

func (*RingBuilder) PretendMinPartHoursPassed added in v0.0.2 

func (b *RingBuilder) PretendMinPartHoursPassed()

func (*RingBuilder) Rebalance 

func (b *RingBuilder) Rebalance() (int, float64, int, error)

Rebalance rebalances the ring.

This is the main work function of the builder, as it will assign and reassing partitions to devices in the ring based on weights, distinct zones, recent reassignments, etc.

The proces doesn't always perfectly assign partitions (that'd take a lot more analysis and therefore a lot more time. Because of this, it keeps rebalancing until the device skew (number of partitions a device wants compared to what it has) gets below 1% or doesn't change by more than 1% (only happens with a ring that can't be balanced no matter what).

func (*RingBuilder) RemoveDev added in v0.0.2 

func (b *RingBuilder) RemoveDev(devId int64, purge bool)

Remove a device from the ring.

func (*RingBuilder) Save 

func (b *RingBuilder) Save(builderPath string) error

Save serializes this RingBuilder instance to disk Note that no locking is done here, you should call LockBuilderPath first.

func (*RingBuilder) SearchDevs added in v0.0.2 

func (b *RingBuilder) SearchDevs(region, zone int64, ip string, port int64, repIp string, repPort int64, device string, weight float64, meta string, scheme string) []*RingBuilderDevice

func (*RingBuilder) SetDevWeight added in v0.0.2 

func (b *RingBuilder) SetDevWeight(devId int64, weight float64) error

Set the weight of a device. This should be called rather than just altering the weight directly, as the builder will need to rebuild some internal state to reflect the change.

func (*RingBuilder) SetReplicas 

func (b *RingBuilder) SetReplicas(newReplicaCount float64)

SetReplicas sets the number of replicas in this ring.

If the new replica count is sufficiently different that replica2Part2Dev will change size, sets devsChanged. This is so tools can know to write out the new ring rather than bailing out due to lack of balance change.

func (*RingBuilder) UpdateDevInfo added in v0.0.2 

func (b *RingBuilder) UpdateDevInfo(devId int64, newIp string, newPort int64, newRepIp string, newRepPort int64, newDevice, newMeta, newScheme string) error

func (*RingBuilder) Validate added in v0.0.2 

func (b *RingBuilder) Validate() error

validate validates the ring.

This is a safety method to try to catch any bugs in the building process. It ensures partitions have been assigned to real devices, aren't doubly assigned, etc.

func (*RingBuilder) WeightOfOnePart 

func (b *RingBuilder) WeightOfOnePart() float64

WeightOfOnePart returns the weight of each partition as calculated from the total weight of all the devices.

type RingBuilderDevice 

type RingBuilderDevice struct {
	ReplicationPort int64   `pickle:"replication_port"`
	Meta            string  `pickle:"meta"`
	PartsWanted     int64   `pickle:"parts_wanted"`
	Device          string  `pickle:"device"`
	Zone            int64   `pickle:"zone"`
	Weight          float64 `pickle:"weight"`
	Scheme          string  `pickle:"scheme"`
	Ip              string  `pickle:"ip"`
	Region          int64   `pickle:"region"`
	Port            int64   `pickle:"port"`
	ReplicationIp   string  `pickle:"replication_ip"`
	Parts           int64   `pickle:"parts"`
	Id              int64   `pickle:"id"`
	// contains filtered or unexported fields
}
func Search(builderPath string, region, zone int64, ip string, port int64, repIp string, repPort int64, device string, weight float64, meta string, scheme string) ([]*RingBuilderDevice, error)

Note that no locking is done here, you should call LockBuilderPath first.

type RingBuilderPickle 

type RingBuilderPickle struct {
	LastPartGatherStart int64                   `pickle:"_last_part_gather_start"`
	LastPartMovesEpoch  int64                   `pickle:"_last_part_moves_epoch"`
	PartPower           int64                   `pickle:"part_power"`
	DevsChanged         bool                    `pickle:"devs_changed"`
	Replicas            float64                 `pickle:"replicas"`
	MinPartHours        int64                   `pickle:"min_part_hours"`
	Parts               int64                   `pickle:"parts"`
	Overload            float64                 `pickle:"overload"`
	Dispersion          float64                 `pickle:"dispersion"`
	Version             int64                   `pickle:"version"`
	Devs                []*RingBuilderDevice    `pickle:"devs"`
	RemoveDevs          []*RingBuilderDevice    `pickle:"_remove_devs"`
	LastPartMoves       lastPartMovesArray      `pickle:"_last_part_moves"`
	Replica2Part2Dev    []replica2Part2DevArray `pickle:"_replica2part2dev"`
}

RingBuilderPickle is used for pickling/unpickling ringbuilder data

type RingMD5 added in v1.2.0 

type RingMD5 interface {
	Ring
	MD5() string
	DiskPath() string
	RingMatching(md5 string) RingMD5
	Reload() error
	AssignmentCount(devId int) int
}

func GetRingMD5 added in v1.2.0 

func GetRingMD5(ringType, prefix, suffix string, policy int) (RingMD5, error)

GetRingMD5 returns the current ring given the ring_type ("account", "container", "object"), hash path prefix, and hash path suffix. An error is returned if the requested ring does not exist. This differs from GetRing in that it returns a ring satisfying the RingMD5 interface and that it will compute the MD5 hash of the ring's persisted contents. Also, it will not automatically reload itself -- an explicit Reload is required.

func LoadRingMD5 added in v1.2.0 

func LoadRingMD5(path string, prefix string, suffix string) (RingMD5, error)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.