common

package
v0.0.0-...-96b211d Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 7, 2023 License: GPL-3.0 Imports: 18 Imported by: 15

Documentation

Index

Constants

View Source 
const FriendlyTimeFormat = "2006 Jan 02 15:04"
View Source 
const FriendlyTimeFormatWithSeconds = "2006 Jan 02 15:04:05"
View Source 
const MachineTimeFormatWithSeconds = "20060102150405"
View Source 
const NullSha512Hex = "0000000000000000000000000000000000000000000000000000000000000000"

Variables

This section is empty.

Functions

func Absolute 

func Absolute(basePath string, folderPath string) string

Absolute creates the absolute path definition base on basePath it accepts relative paths in folderPath

func CompareFileItems 

func CompareFileItems(fileItem1 SyncFileItem, fileItem2 SyncFileItem) bool

CompareFileItems compares two SyncFileItems to check if they are equal or not.

func CorrectPath 

func CorrectPath(folderPath string) string

CorrectPath fixes the path to the correct format base on dfs requirement

func CorrectPaths 

func CorrectPaths(paths []string) []string

CorrectPaths fix multiple paths to the correct format base on dfs requirement

func DivideParts 

func DivideParts(folderPath string) []string

DivideParts splits the full path to its path name counter parts

func EmptyChecksum 

func EmptyChecksum() string

EmptyChecksum checks if the checksum is empty

func Join 

func Join(inputs ...string) string

Join joins the paths with suitable path separator and creates a full path

func NewAnimation 

func NewAnimation(output terminal.Output, header string) *animation

func NewNotificationError 

func NewNotificationError(containerList NotificationContainerList, err error) error

func PathTree 

func PathTree(rootPath *string, folderPath string) []string

PathTree creates a path list from rootPath to folderPath every entry will contain the full path of path pointed in the tree

func Split 

func Split(path string) (string, string)

Split just splits the path to parent path and path name in the way of dfs required

func ValidatePath 

func ValidatePath(folderPath string) bool

ValidatePath checks if the path is a valid path for dfs

Types

type CacheFileItem 

type CacheFileItem struct {
	FileItem  SyncFileItem
	ClusterId string
	ExistsIn  CacheFileItemLocationMap
	ExpiresAt time.Time
}

func NewCacheFileItem 

func NewCacheFileItem(clusterId string, nodeId string, fileItem SyncFileItem) *CacheFileItem

func NewCacheFileItemFromMap 

func NewCacheFileItemFromMap(cache map[string]string) *CacheFileItem

func (*CacheFileItem) Expired 

func (c *CacheFileItem) Expired() bool

func (*CacheFileItem) Export 

func (c *CacheFileItem) Export() map[string]string

type CacheFileItemLocationMap 

type CacheFileItemLocationMap map[string]bool

type CacheFileItemMap 

type CacheFileItemMap map[string]*CacheFileItem

type Chunk 

type Chunk struct {
	Sequence uint16 `json:"sequence"`
	Index    uint64 `json:"index"`
	Size     uint32 `json:"size"`
}

Chunk struct is to hold the File particle information for the reservation operation Sequence is the order number of the particle Index is the particle starting point in the whole file Size is the length of the particle

func (Chunk) Ends 

func (c Chunk) Ends() uint64

Ends calculates the ending index of the particle in the whole file

func (Chunk) Starts 

func (c Chunk) Starts() uint64

Starts locates the starting index of the particle of the whole file

type Cluster 

type Cluster struct {
	Id           string       `json:"clusterId"`
	Size         uint64       `json:"size"`
	Used         uint64       `json:"used"`
	Nodes        NodeList     `json:"nodes"`
	Reservations Reservations `json:"reservations"`

	// If master node is unreachable and also unable to elect a new master in the cluster
	Paralyzed bool `json:"paralyzed"`

	// 0 = Online, 1 = Readonly, -1 = Offline
	// Readonly mode: Create and Delete file operations are forbidden.
	// Offline mode:  All file operations for the cluster is forbidden
	State States `json:"state"`

	// Cluster can be in any state in maintain mode but all administrative
	// operations are forbidden when the cluster is in this mode
	Maintain      bool   `json:"maintain"`
	MaintainTopic Topics `json:"maintainTopic"`

	Snapshots Snapshots `json:"snapshots"`
}

Cluster struct is to hold cluster details in dfs farm

func NewCluster 

func NewCluster(id string) *Cluster

NewCluster initialises a new Cluster struct

func (*Cluster) Available 

func (c *Cluster) Available() uint64

Available returns the available space in the cluster

func (*Cluster) CanSchedule 

func (c *Cluster) CanSchedule() bool

func (*Cluster) Commit 

func (c *Cluster) Commit(id string, size uint64)

Commit commits the reservation and the used space in the cluster and drops the reservation from the cluster

func (*Cluster) Delete 

func (c *Cluster) Delete(nodeId string, masterChangedHandler func(*Node) error) error

Delete deletes the node from the cluster. masterChangedHandler executed when the deleted node is the current master node and there are other nodes in the cluster

func (*Cluster) Discard 

func (c *Cluster) Discard(id string)

Discard discards the reservation from the cluster and free the reserved space

func (*Cluster) HighQualityMasterNodeCandidate 

func (c *Cluster) HighQualityMasterNodeCandidate() *Node

HighQualityMasterNodeCandidate returns the most responsive Node that can be evaluated as master node if there is no candidate, returns nil

func (*Cluster) Master 

func (c *Cluster) Master() *Node

Master returns the current master Node struct in the cluster

func (*Cluster) Node 

func (c *Cluster) Node(nodeId string) *Node

Node searches the node in the cluster by its id

func (*Cluster) Others 

func (c *Cluster) Others(nodeId string) NodeList

Others returns the nodes in the cluster other than the one provided in the nodeId if there is not any node in the cluster other than the one provided in the nodeId, returns nil

func (*Cluster) PrioritizedHighQualityNodes 

func (c *Cluster) PrioritizedHighQualityNodes(nodeIdsMap CacheFileItemLocationMap) PrioritizedHighQualityNodeList

PrioritizedHighQualityNodes returns the most responsive Nodes in the cluster ordered by their quality if there is not any node with a good quality, it returns nil

func (*Cluster) Reserve 

func (c *Cluster) Reserve(id string, size uint64)

Reserve adds the reservation id to the cluster to ensure that simultaneous write request will have enough space in the cluster

func (*Cluster) SetMaster 

func (c *Cluster) SetMaster(nodeId string) error

SetMaster changes the master in the cluster. If node is not exists, ErrNotFound will return

func (*Cluster) Slaves 

func (c *Cluster) Slaves() NodeList

Slaves returns only the slave nodes other than the master node

func (*Cluster) StateString 

func (c *Cluster) StateString() string

StateString returns the cluster current situation as string representative

func (*Cluster) Weight 

func (c *Cluster) Weight() float64

Weight calculates the load of the cluster as percent. MaxValue is 1 (theoretical)

type ClusterMap 

type ClusterMap struct {
	Id      string `json:"clusterId"`
	Address string `json:"address"`
	Chunk   Chunk  `json:"chunk"`
}

ClusterMap struct is to hold the Chunk information base on the cluster locations of the dfs farm this Chunk can be the whole File Chunk or the part of the File Chunk. Id is the cluster id where the File Chunk is located Address is the node address in the cluster that requester can reach and read the chunk Chunk is the information of chunk to read

type Clusters 

type Clusters []*Cluster

Clusters is the definition of the pointer array of Cluster struct

func (Clusters) Len 

func (c Clusters) Len() int

func (Clusters) Less 

func (c Clusters) Less(i, j int) bool

func (Clusters) Swap 

func (c Clusters) Swap(i, j int)

type CreationResult 

type CreationResult struct {
	Checksum string
	Chunks   DataChunks
}

CreationResult struct is to hold the file creation details in the dfs farm Checksum is the whole file checksum Chunks is the list of the whole File particles

func NewCreationResult 

func NewCreationResult(sha512Hex string, chunks DataChunks) *CreationResult

NewCreationResult initialises the new empty CreationResult struct

type DataChunk 

type DataChunk struct {
	Sequence uint16 `json:"sequence"`
	Size     uint32 `json:"size"`
	Hash     string `json:"hash"`
}

DataChunk struct is to hold the block File particle information

func NewDataChunk 

func NewDataChunk(sequence uint16, size uint32, sha512 string) *DataChunk

NewDataChunk initialises a new DataChunk using the given information

type DataChunks 

type DataChunks []*DataChunk

DataChunks is the definition of the pointer array of DataChunk struct

func (DataChunks) Len 

func (d DataChunks) Len() int

func (DataChunks) Less 

func (d DataChunks) Less(i, j int) bool

func (DataChunks) Swap 

func (d DataChunks) Swap(i, j int)

type DeletionResult 

type DeletionResult struct {
	Untouched []string
	Deleted   []string
	Missing   []string
}

DeletionResult struct is to hold the file deletion details applied to the dfs farm Untouched is the File Chunk Hash that wasn't got any action Deleted is the File Chunk Hash that is deleted from the dfs farm Missing is the File Chunk Hash that is not found or not indexed and may appear after the dfs farm sync/repair

func NewDeletionResult 

func NewDeletionResult() DeletionResult

NewDeletionResult initialises the new empty DeletionResult struct

type Error 

type Error struct {
	Code    int    `json:"code"`
	Message string `json:"message"`
}

Error struct is to hold and export/serialize the action result concluded with an error

func NewError 

func NewError(code int, message string) Error

NewError initialises the new Error base on the given details

func NewErrorFromReader 

func NewErrorFromReader(reader io.Reader) Error

NewErrorFromReader creates the Error struct by reading the data from stream source (ex: Http Request Result)

type File 

type File struct {
	Name     string     `json:"name"`
	Mime     string     `json:"mime"`
	Size     uint64     `json:"size"`
	Checksum string     `json:"checksum"`
	Created  time.Time  `json:"created"`
	Modified time.Time  `json:"modified"`
	Chunks   DataChunks `json:"chunks"`
	Missing  DataChunks `json:"missing"`
	Lock     *FileLock  `json:"lock"`
	Zombie   bool       `json:"zombie"`
}

File struct is to hold the actual file details in dfs cluster

func CreateJoinedFile 

func CreateJoinedFile(files Files) (*File, error)

CreateJoinedFile creates a file struct from multiple files and behave like a single file

func (*File) CanDie 

func (f *File) CanDie() bool

CanDie checks if the zombie file can be deleted

func (*File) CloneInto 

func (f *File) CloneInto(target *File)

CloneInto copies the file struct in to a target file struct

func (*File) IngestDeletion 

func (f *File) IngestDeletion(deletionResult DeletionResult)

IngestDeletion ingest the deletion information to file struct to be able to track the consistent deletion operation across the dfs clusters

func (*File) Locked 

func (f *File) Locked() bool

Locked checks if file is locked for any other operation

func (*File) Reset 

func (f *File) Reset(mime string, size uint64)

Reset resets the file struct fields to default values

func (*File) Resurrect 

func (f *File) Resurrect()

Resurrect resets the zombie situation of the file to able to repair

func (*File) ZombieCheck 

func (f *File) ZombieCheck() bool

ZombieCheck checks if the file is zombie in the dfs clustes

type FileLock 

type FileLock struct {
	Till time.Time `json:"till"`
}

FileLock struct is to hold the file locking details related to the file

func NewFileLock 

func NewFileLock(duration time.Duration) *FileLock

NewFileLock creates a lock for file

func NewFileLockForSize 

func NewFileLockForSize(size uint64) *FileLock

NewFileLockForSize creates a lock for the base of a file size and transfer speed calculations

func (*FileLock) Cancel 

func (f *FileLock) Cancel()

Cancel cancels or expires the file lock and release it

type Files 

type Files []*File

Files is the definition of the pointer array of File struct

func (Files) Len 

func (f Files) Len() int

func (Files) Less 

func (f Files) Less(i, j int) bool

func (Files) Swap 

func (f Files) Swap(i, j int)

type Folder 

type Folder struct {
	Full     string        `json:"full"`
	Name     string        `json:"name"`
	Created  time.Time     `json:"created"`
	Modified time.Time     `json:"modified"`
	Size     uint64        `json:"size" bson:"-"`
	Folders  FolderShadows `json:"folders"`
	Files    Files         `json:"files"`
	Hooks    hooks.Hooks   `json:"hooks,omitempty"`
}

Folder struct is to hold the virtual folder details associated in dfs cluster

func CreateJoinedFolder 

func CreateJoinedFolder(folders []*Folder) (*Folder, error)

CreateJoinedFolder merges the provided Folder array as a Folder

func NewFolder 

func NewFolder(folderPath string) *Folder

NewFolder creates a new empty Folder struct with folderPath

func (*Folder) CalculateUsage 

func (f *Folder) CalculateUsage(calculateUsageHandler func(FolderShadows))

CalculateUsage calculates the total size of the folder including the sub folders recursively calculateUsageHandler parameter is mandatory and should fill the FolderShadows size fields to have the correct size calculation of the Folder size

func (*Folder) CloneInto 

func (f *Folder) CloneInto(target *Folder)

CloneInto copies the details of the current Folder to the provided target Folder

func (*Folder) DeleteFile 

func (f *Folder) DeleteFile(name string, deleteFileHandler func(*File) error) error

DeleteFile searches the File by name and take action base on the result of deleteFileHandler if deleteFileHandler raises error, deletion will be canceled if File isn't being found, it will return ErrNotExists error

func (*Folder) DeleteFolder 

func (f *Folder) DeleteFolder(name string, deleteFolderHandler func(string) error) error

DeleteFolder searches the Folder by name and take action base on the result of deleteFolderHandler if deleteFolderHandler raises error, deletion will be canceled if Folder isn't being found, it will return ErrNotExists error

func (*Folder) File 

func (f *Folder) File(name string) *File

File searches the File by name and returns the File struct that points the file in dfs cluster

func (*Folder) Folder 

func (f *Folder) Folder(name string) *string

Folder searches the Folder by name and return the full folderPath

func (*Folder) Locked 

func (f *Folder) Locked() bool

Locked checks if the folder has any locked File

func (*Folder) NewFile 

func (f *Folder) NewFile(name string) (*File, error)

NewFile creates a File struct that points the actual file in the current folder

func (*Folder) NewFolder 

func (f *Folder) NewFolder(name string) (*Folder, error)

NewFolder creates an empty sub Folder in the current Folder

func (*Folder) ReplaceFile 

func (f *Folder) ReplaceFile(name string, file *File)

ReplaceFile searches the File by name and replace it with the provided File struct If File has been found and the File struct is null, it will remove the File from the Folder If File hasn't been found and File struct is not null, it will add the File to the Folder

type FolderShadow 

type FolderShadow struct {
	Full    string    `json:"full"`
	Name    string    `json:"name"`
	Created time.Time `json:"created"`
	Size    uint64    `json:"size" bson:"-"`
}

FolderShadow struct is to hold the virtual sub folder details

func NewFolderShadow 

func NewFolderShadow(folderPath string) *FolderShadow

NewFolderShadow creates an empty FolderShadow struct base on the provided folderPath

type FolderShadows 

type FolderShadows []*FolderShadow

FolderShadows is the definition of the pointer array of FolderShadow struct

func (FolderShadows) Len 

func (f FolderShadows) Len() int

func (FolderShadows) Less 

func (f FolderShadows) Less(i, j int) bool

func (FolderShadows) Swap 

func (f FolderShadows) Swap(i, j int)

type MapType 

type MapType int
const (
	MTRead   MapType = 1
	MTCreate MapType = 2
	MTDelete MapType = 3
)

type Node 

type Node struct {
	Id       string    `json:"nodeId"`
	Address  string    `json:"address"`
	Master   bool      `json:"master"`
	LeadTill time.Time `json:"leadTill"`
	Quality  int64     `json:"quality"`
}

Node struct is to hold the node details of the dfs cluster

func (*Node) LeadershipExpired 

func (n *Node) LeadershipExpired() bool

func (*Node) SetLeadDuration 

func (n *Node) SetLeadDuration()

type NodeList 

type NodeList []*Node

NodeList is the definition of the pointer array of Node struct

func (NodeList) Len 

func (n NodeList) Len() int

func (NodeList) Less 

func (n NodeList) Less(i, _ int) bool

func (NodeList) Swap 

func (n NodeList) Swap(i, j int)

type NotificationContainer 

type NotificationContainer struct {
	Create       bool         `json:"create"`
	FileItem     SyncFileItem `json:"fileItems"`
	ResponseChan chan bool    `json:"-"`
}

NotificationContainer struct is to hold and export/serialize file creation or deletion operation notification to the farm manager Create is the boolean and notify if the operation was the creation. otherwise deletion. FileItem is the File Chunk information that took place for the the operation ResponseChan is just used for the internal operation sync in the data node. Not exported.

type NotificationContainerList 

type NotificationContainerList []*NotificationContainer

NotificationContainerList is the definition of the pointer array of NotificationContainer struct

func (NotificationContainerList) ExportFileItemList 

func (n NotificationContainerList) ExportFileItemList() SyncFileItemList

ExportFileItemList exports the FileItem entries as the SyncFileItemList from the NotificationContainerList

type NotificationError 

type NotificationError struct {
	// contains filtered or unexported fields
}

func (*NotificationError) ContainerList 

func (n *NotificationError) ContainerList() NotificationContainerList

func (*NotificationError) Error 

func (n *NotificationError) Error() string

func (*NotificationError) Is 

func (n *NotificationError) Is(err error) bool

type PrioritizedHighQualityNodeList 

type PrioritizedHighQualityNodeList []*Node

PrioritizedHighQualityNodeList is the definition of the pointer array of Node struct

func (PrioritizedHighQualityNodeList) Len 

func (n PrioritizedHighQualityNodeList) Len() int

func (PrioritizedHighQualityNodeList) Less 

func (n PrioritizedHighQualityNodeList) Less(i, j int) bool

func (PrioritizedHighQualityNodeList) Swap 

func (n PrioritizedHighQualityNodeList) Swap(i, j int)

type ReadRange 

type ReadRange struct {
	Begins int64
	Ends   int64
}

func NewReadRange 

func NewReadRange(value string) (*ReadRange, error)

func (*ReadRange) HasRange 

func (r *ReadRange) HasRange() bool

type Reservation 

type Reservation struct {
	Size      uint64    `json:"size"`
	ExpiresAt time.Time `json:"expiresAt"`
}

Reservation struct is to hold the file creation reservation information specific to the cluster

type ReservationMap 

type ReservationMap struct {
	Id       string       `json:"reservationId"`
	Clusters []ClusterMap `json:"clusters"`
}

ReservationMap struct is to hold and export/serialize the file creation reservation details that will happen across the dfs farm

type Reservations 

type Reservations map[string]*Reservation

Reservations point the type declaration

func NewReservations 

func NewReservations() Reservations

NewReservations initialises a new Reservations struct

func (Reservations) CleanUp 

func (r Reservations) CleanUp()

type Snapshots 

type Snapshots []time.Time

func (Snapshots) Len 

func (s Snapshots) Len() int

func (Snapshots) Less 

func (s Snapshots) Less(i, j int) bool

func (Snapshots) Swap 

func (s Snapshots) Swap(i, j int)

type States 

type States int
const (
	StateOnline   States = 0
	StateReadonly States = 1
	StateOffline  States = -1
)

type SyncContainer 

type SyncContainer struct {
	FileItems SyncFileItemMap
	Snapshots Snapshots
}

SyncContainer struct holds the data node block file sync details of cluster

func NewSyncContainer 

func NewSyncContainer() *SyncContainer

NewSyncContainer creates an instance of SyncContainer

func (*SyncContainer) Sort 

func (c *SyncContainer) Sort()

Sort sorts the snapshots base on taken date

type SyncFileItem 

type SyncFileItem struct {
	Sha512Hex string `json:"sha512Hex"`
	Usage     uint16 `json:"usage"`
	Size      uint32 `json:"size"`
	Shadow    bool   `json:"shadow"`
}

SyncFileItem struct is to hold and export/serialize the file creation/deletion/sync operation across the dfs farm

type SyncFileItemList 

type SyncFileItemList []SyncFileItem

SyncFileItemList is the definition of the array of SyncFileItem struct

func (SyncFileItemList) PhysicalSize 

func (s SyncFileItemList) PhysicalSize() uint64

PhysicalSize calculates the total file size (ignoring the shadow entries) in the SyncFileItemList

func (SyncFileItemList) ShadowItems 

func (s SyncFileItemList) ShadowItems() SyncFileItemList

ShadowItems filters the map entries that are only marked as Shadow

type SyncFileItemMap 

type SyncFileItemMap map[string]SyncFileItem

SyncFileItemMap is the map definition of SyncFileItem struct base on Sha512Hex key for the O(1) reachability

func (SyncFileItemMap) ToList 

func (s SyncFileItemMap) ToList() SyncFileItemList

ToList exports the map entries to SyncFileItemList

type Topics 

type Topics string
const (
	TopicNone            Topics = ""
	TopicSynchronisation Topics = "Synchronisation"
	TopicBalance         Topics = "Balance"
	TopicMove            Topics = "Move"
	TopicRepair          Topics = "Repair"
	TopicUnregisterNode  Topics = "Unregister Node"
	TopicCreateSnapshot  Topics = "Create Snapshot"
	TopicDeleteSnapshot  Topics = "Delete Snapshot"
	TopicRestoreSnapshot Topics = "Restore Snapshot"
)

type Tree 

type Tree struct {
	// contains filtered or unexported fields
}

Tree organised the provided Folder array to create the Tree structure

func NewTree 

func NewTree() *Tree

NewTree creates an empty Tree

func (*Tree) CalculateUsage 

func (t *Tree) CalculateUsage()

CalculateUsage calculates the folder sizes of the each member of the tree

func (*Tree) Fill 

func (t *Tree) Fill(rootPath *string, folders []*Folder) error

Fill creates the Tree using the provided Folder array with the respect of rootPath entry this function will also fix / repair the missing part of the Tree if provided Folder array does not contain all the necessary parts of it

func (*Tree) Normalize 

func (t *Tree) Normalize() []*Folder

Normalize converts the Tree to Folder array

func (*Tree) Shadow 

func (t *Tree) Shadow() *TreeShadow

Shadow creates the TreeShadow object of the Tree by simplifying the information to make it serializable and exportable

type TreeShadow 

type TreeShadow struct {
	Full     string      `json:"full"`
	Name     string      `json:"name"`
	Created  time.Time   `json:"created"`
	Modified time.Time   `json:"modified"`
	Size     uint64      `json:"size"`
	Folders  TreeShadows `json:"folders"`
}

TreeShadow holds the vital folder details to create the tree output

func NewTreeShadowFromTree 

func NewTreeShadowFromTree(tree *Tree) *TreeShadow

NewTreeShadowFromTree creates a TreeShadow struct using the tree details.

type TreeShadows 

type TreeShadows []*TreeShadow

TreeShadows is the definition of the pointer array of TreeShadow struct

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.