dxfile

package
v1.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 1, 2020 License: Apache-2.0 Imports: 21 Imported by: 2

Documentation

Index

Constants

View Source 
const (

	// SectorSize is the size of a Sector, which is 4MiB
	SectorSize = uint64(1 << 22)

	// Version is the version of dxfile
	Version = "1.0.0"
)
View Source 
const (
	// RepairHealthThreshold is the threshold that file with smaller health is marked as Stuck and
	// to be repaired
	RepairHealthThreshold = 175

	// StuckThreshold is the threshold that defines the threshold between the stuck and unstuck segments
	StuckThreshold = 100

	// CompleteHealthThreshold is that segment upload all sectors
	CompleteHealthThreshold = 200
)
View Source 
const (
	// PageSize is the page size of persist Data
	PageSize = 4096
)

Variables

View Source 
var (
	// ErrUnknownFile is the error for opening a file that not exists on disk
	ErrUnknownFile = errors.New("file not known")

	// ErrFileExist is the error for creating a new file while there already exist a file
	// with the same name
	ErrFileExist = errors.New("file already exist")
)

Functions

func CmpRepairPriority 

func CmpRepairPriority(h1, h2 uint32) int

CmpRepairPriority compare two health. The compare result returns the priority the health related Segment should be fixed The priority is determined by the follows: When the file is not recoverable from contract (health 0~99), it has the highest property to recover from disk When the file is recoverable (health 100~199), it is then prioritized. When the file is totally health, there is no need to recover. Thus the priority of recovery is as follows: 175 == RepairHealthThreshold (175 ~ 200) < (0 ~ 99) < 174 < 100 In the following expression p(h) means the priority of the health If p(h1) == p(h2), return 0 If p(h1) < p(h2), return -1 If p(h1) > p(h2), return 1

Types

type CachedHealthMetadata 

type CachedHealthMetadata struct {
	Health      uint32
	Redundancy  uint32
	StuckHealth uint32
}

CachedHealthMetadata is a helper struct that contains the dxfile health metadata fields that are cached

type DxFile 

type DxFile struct {
	ID FileID
	// contains filtered or unexported fields
}

DxFile is the type of user uploaded DxFile

func New 

func New(filePath storage.SysPath, dxPath storage.DxPath, sourcePath storage.SysPath, wal *writeaheadlog.Wal, erasureCode erasurecode.ErasureCoder, cipherKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode) (*DxFile, error)

New creates a new dxfile. filePath is the file where DxFile locates, dxPath is the user input dxPath. sourcePath is the file of the original data. wal is the writeaheadlog. erasureCode is the erasure coder for encoding. cipherKey is the key for encryption. fileSize is the size of the original data file. fileMode is the file privilege mode (e.g. 0777)

func (*DxFile) AddSector 

func (df *DxFile) AddSector(address enode.ID, merkleRoot common.Hash, segmentIndex, sectorIndex int) error

AddSector add a Sector to DxFile to the location specified by segmentIndex and sectorIndex. The sector content is filled by address and merkleRoot.

func (*DxFile) ApplyCachedHealthMetadata 

func (df *DxFile) ApplyCachedHealthMetadata(metadata CachedHealthMetadata) error

ApplyCachedHealthMetadata apply the cachedHealthMetadata to the DxFile

func (*DxFile) CipherKey 

func (df *DxFile) CipherKey() (crypto.CipherKey, error)

CipherKey return the cipher key

func (*DxFile) Delete 

func (df *DxFile) Delete() error

Delete delete the DxFile. The function delete the DxFile on disk, and also mark df.deleted as true

func (*DxFile) Deleted 

func (df *DxFile) Deleted() bool

Deleted return the dxfile status of whether it is deleted

func (*DxFile) DxPath 

func (df *DxFile) DxPath() storage.DxPath

DxPath return dxfile.metadata.DxPath

func (*DxFile) ErasureCode 

func (df *DxFile) ErasureCode() (erasurecode.ErasureCoder, error)

ErasureCode return the erasure code

func (*DxFile) FileMode 

func (df *DxFile) FileMode() os.FileMode

FileMode return the os file mode of a dxfile

func (*DxFile) FilePath 

func (df *DxFile) FilePath() string

FilePath return the actual file path of the dxfile

func (*DxFile) FileSize 

func (df *DxFile) FileSize() uint64

FileSize return the file size of the dxfile

func (*DxFile) GetHealth 

func (df *DxFile) GetHealth() uint32

GetHealth return the health in the metadata

func (*DxFile) GetNumStuckSegments 

func (df *DxFile) GetNumStuckSegments() uint32

GetNumStuckSegments return NumStuckSegments in the metadata

func (*DxFile) GetRedundancy 

func (df *DxFile) GetRedundancy() uint32

GetRedundancy return the last redundancy in the metadata

func (*DxFile) GetStuckByIndex 

func (df *DxFile) GetStuckByIndex(index int) bool

GetStuckByIndex get the Stuck status of the indexed Segment

func (*DxFile) GetStuckHealth 

func (df *DxFile) GetStuckHealth() uint32

GetStuckHealth return the stuck health in the metadata

func (*DxFile) Health 

func (df *DxFile) Health(table storage.HostHealthInfoTable) (uint32, uint32, uint32)

Health return check for dxFile's segments and return the health, stuckHealth, and numStuckSegments Health 0~100: unrecoverable from contracts Health 100~200: recoverable Health 200: No fix needed

func (*DxFile) HostIDs 

func (df *DxFile) HostIDs() []enode.ID

HostIDs return all hosts in df.hostTable(including unused ones)

func (*DxFile) LastTimeRecentRepair 

func (df *DxFile) LastTimeRecentRepair() time.Time

LastTimeRecentRepair return df.metadata.LastTimeRecentRepair

func (*DxFile) LocalPath 

func (df *DxFile) LocalPath() storage.SysPath

LocalPath return the local path of a file

func (*DxFile) MarkAllHealthySegmentsAsUnstuck 

func (df *DxFile) MarkAllHealthySegmentsAsUnstuck(table storage.HostHealthInfoTable) error

MarkAllHealthySegmentsAsUnstuck mark all health > 100 segments as unstuck

func (*DxFile) MarkAllUnhealthySegmentsAsStuck 

func (df *DxFile) MarkAllUnhealthySegmentsAsStuck(table storage.HostHealthInfoTable) error

MarkAllUnhealthySegmentsAsStuck mark all unhealthy segments (health smaller than RepairHealthThreshold) as Stuck

func (*DxFile) NumSegments 

func (df *DxFile) NumSegments() int

NumSegments return the number of segments

func (*DxFile) NumStuckSegments 

func (df *DxFile) NumStuckSegments() int

NumStuckChunks returns the Number of Stuck Chunks recorded in the file's metadata

func (*DxFile) Redundancy 

func (df *DxFile) Redundancy(table storage.HostHealthInfoTable) uint32

Redundancy return the redundancy of a dxfile. The redundancy is goodSectors * 100 / minSectors.

func (*DxFile) Rename 

func (df *DxFile) Rename(newDxFile storage.DxPath, newDxFilename storage.SysPath) error

Rename rename the DxFile, remove the previous dxfile and create a new file

func (*DxFile) SectorSize 

func (df *DxFile) SectorSize() uint64

SectorSize return the Sector size of a dxfile

func (*DxFile) Sectors 

func (df *DxFile) Sectors(segmentIndex int) ([][]*Sector, error)

func (*DxFile) SectorsOfSegmentIndex 

func (df *DxFile) SectorsOfSegmentIndex(index int) ([][]*Sector, error)

SectorsOfSegmentIndex returns Sectors of a specific Segment with Index. Notice the returned sector is the deep copy of the sectors specified by index

func (*DxFile) SegmentHealth 

func (df *DxFile) SegmentHealth(segmentIndex int, table storage.HostHealthInfoTable) uint32

SegmentHealth return the health of a Segment based on information provided Health 0~100: unrecoverable from contracts Health 100~200: recoverable Health 200: No fix needed

func (*DxFile) SegmentSize 

func (df *DxFile) SegmentSize() uint64

SegmentSize return the size of a Segment for a DxFile.

func (*DxFile) SetFileMode 

func (df *DxFile) SetFileMode(mode os.FileMode) error

SetFileMode change the value of df.metadata.FileMode and save it to file

func (*DxFile) SetLocalPath 

func (df *DxFile) SetLocalPath(path storage.SysPath) error

SetLocalPath change the value of local path and save to disk

func (*DxFile) SetStuckByIndex 

func (df *DxFile) SetStuckByIndex(index int, stuck bool) (err error)

SetStuckByIndex set a Segment of Index to the value of Stuck.

func (*DxFile) SetTimeAccess 

func (df *DxFile) SetTimeAccess(t time.Time) error

SetTimeAccess set df.metadata.TimeAccess

func (*DxFile) SetTimeLastHealthCheck 

func (df *DxFile) SetTimeLastHealthCheck(t time.Time) error

SetTimeLastHealthCheck set and save df.metadata.TimeLastHealthCheck

func (*DxFile) SetTimeRecentRepair 

func (df *DxFile) SetTimeRecentRepair(t time.Time) error

SetTimeRecentRepair set and save df.metadata.TimeRecentRepair

func (*DxFile) Snapshot 

func (df *DxFile) Snapshot() (*Snapshot, error)

Snapshot creates the Snapshot of the DxFile

func (*DxFile) SnapshotReader 

func (df *DxFile) SnapshotReader() (*SnapshotReader, error)

SnapshotReader creates a reader that could be used to read DxFile content

func (*DxFile) TimeAccess 

func (df *DxFile) TimeAccess() time.Time

TimeAccess return the last access time of a DxFile

func (*DxFile) TimeCreate 

func (df *DxFile) TimeCreate() time.Time

TimeCreate returns the TimeCreate of a DxFile

func (*DxFile) TimeLastHealthCheck 

func (df *DxFile) TimeLastHealthCheck() time.Time

TimeLastHealthCheck return TimeHealthCheck

func (*DxFile) TimeModify 

func (df *DxFile) TimeModify() time.Time

TimeModify return the TimeModify of a DxFile

func (*DxFile) TimeRecentRepair 

func (df *DxFile) TimeRecentRepair() time.Time

func (*DxFile) TimeUpdate 

func (df *DxFile) TimeUpdate() time.Time

TimeUpdate return the last update time of a DxFile

func (*DxFile) UID 

func (df *DxFile) UID() FileID

UID return the id of the dxfile

func (*DxFile) UpdateUsedHosts 

func (df *DxFile) UpdateUsedHosts(used []enode.ID) error

UpdateUsedHosts update host table of the dxfile. hosts in df.hostTable exist in used slice are marked as used, rest are marked as unused

func (*DxFile) UploadProgress 

func (df *DxFile) UploadProgress() float64

UploadProgress return the upload process of a dxfile. The upload progress is calculated based on uploadedByte divided by total bytes to upload

func (*DxFile) UploadedBytes 

func (df *DxFile) UploadedBytes() uint64

UploadedBytes return the uploaded bytes. The uploaded bytes is calculated by number of sectors in df.segments

type FileID 

type FileID [fileIDSize]byte

FileID is the ID for a DxFile

type FileSet 

type FileSet struct {
	// contains filtered or unexported fields
}

FileSet is the set of DxFile

func NewFileSet 

func NewFileSet(rootDir storage.SysPath, wal *writeaheadlog.Wal) *FileSet

NewFileSet create a new DxFileSet with provided rootDir and wal.

func (*FileSet) Delete 

func (fs *FileSet) Delete(dxPath storage.DxPath) error

Delete delete a file with dxPath from the file set. Also the DxFile specified by dxPath on disk is also deleted

func (*FileSet) Exists 

func (fs *FileSet) Exists(dxPath storage.DxPath) bool

Exists is the public function that returns whether the dxPath exists (cached then on disk)

func (*FileSet) NewDxFile 

func (fs *FileSet) NewDxFile(dxPath storage.DxPath, sourcePath storage.SysPath, force bool, erasureCode erasurecode.ErasureCoder, cipherKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode) (*FileSetEntryWithID, error)

NewDxFile create a DxFile based on the params given. Return a FileSetEntryWithID that has been registered with threadID in FileSetEntry

func (*FileSet) NewRandomDxFile 

func (fs *FileSet) NewRandomDxFile(dxPath storage.DxPath, minSectors, numSectors uint32, ecCode uint8, ck crypto.CipherKey, fileSize uint64, missRate float32) (*FileSetEntryWithID, error)

NewRandomDxFile creates a new random DxFile with random segments data. missRate is the params indicate the possibility that a sector of a segment is missed. If not set, all sectors will be available.

func (*FileSet) Open 

func (fs *FileSet) Open(dxPath storage.DxPath) (*FileSetEntryWithID, error)

Open open a DxFile with dxPath, return the FileSetEntry, along with the threadID

func (*FileSet) Rename 

func (fs *FileSet) Rename(dxPath, newDxPath storage.DxPath) error

Rename rename the file with dxPath to newDxPath.

type FileSetEntryWithID 

type FileSetEntryWithID struct {
	// contains filtered or unexported fields
}

FileSetEntryWithID is a fileSetEntry with the threadID. FileSetEntryWithID extends DxFile

func (*FileSetEntryWithID) Close 

func (entry *FileSetEntryWithID) Close() error

Close close a FileSetEntryWithID

func (*FileSetEntryWithID) CopyEntry 

func (entry *FileSetEntryWithID) CopyEntry() *FileSetEntryWithID

CopyEntry copy the FileSetEntry. A new thread is created and registered in entry.threadMap

type Metadata 

type Metadata struct {
	ID FileID

	// storage related
	HostTableOffset uint64
	SegmentOffset   uint64

	// size related
	FileSize   uint64
	SectorSize uint64 // ShardSize is the size for one shard, which is by default 4MiB

	// path related
	LocalPath storage.SysPath // Local path is the on-disk location for uploaded files
	DxPath    storage.DxPath  // DxPath is the user specified dxpath

	// Encryption
	CipherKeyCode uint8  // cipher key code defined in cipher package
	CipherKey     []byte // Key used to encrypt pieces

	// Time fields. most of unix timestamp
	TimeModify uint64 // time of last content modification
	TimeUpdate uint64 // time of last Metadata update
	TimeAccess uint64 // time of last access
	TimeCreate uint64 // time of file creation

	// Repair loop fields
	Health              uint32 // Worst health of the file's unstuck segment
	StuckHealth         uint32 // Worst health of the file's Stuck segment
	TimeLastHealthCheck uint64 // Time of last health check happenning
	NumStuckSegments    uint32 // Number of Stuck segments
	TimeRecentRepair    uint64 // Timestamp of last segment repair
	LastRedundancy      uint32 // File redundancy from last check

	// File related
	FileMode os.FileMode // unix file mode

	// Erasure code field
	ErasureCodeType uint8  // the code for the specific erasure code
	MinSectors      uint32 // params for erasure coding. The number of slice raw Data split into.
	NumSectors      uint32 // params for erasure coding. The number of total Sectors
	ECExtra         []byte // extra parameters for erasure code

	// Version control for fork
	Version string
}

Metadata is the Metadata of a user uploaded file.

type Sector 

type Sector struct {
	MerkleRoot common.Hash
	HostID     enode.ID
}

Sector is the Data for a single Sector, which has Data of merkle root and related host address

func (*Sector) DecodeRLP 

func (s *Sector) DecodeRLP(st *rlp.Stream) error

DecodeRLP of Sector implements rlp decode rule

func (*Sector) EncodeRLP 

func (s *Sector) EncodeRLP(w io.Writer) error

EncodeRLP of Sector implements rlp encode rule

type Segment 

type Segment struct {
	Sectors [][]*Sector
	Index   uint64
	Stuck   bool
	// contains filtered or unexported fields
}

Segment is the Data for a Segment, which is composed of several Sectors

func (*Segment) DecodeRLP 

func (s *Segment) DecodeRLP(st *rlp.Stream) error

DecodeRLP of Segment implements rlp decode rule to decode the Sectors field

func (*Segment) EncodeRLP 

func (s *Segment) EncodeRLP(w io.Writer) error

EncodeRLP of Segment implements rlp encode rule to encode the Sectors field

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is the snapshot of a DxFile which contains necessary info for a DxFile. The data of Snapshot is deep copy of the Original DxFile and will not affect the Original DxFile

func (*Snapshot) CipherKey 

func (s *Snapshot) CipherKey() crypto.CipherKey

CipherKey return the cipher key

func (*Snapshot) DxPath 

func (s *Snapshot) DxPath() storage.DxPath

DxPath return the DxPath

func (*Snapshot) ErasureCode 

func (s *Snapshot) ErasureCode() erasurecode.ErasureCoder

ErasureCode return the erasure code

func (*Snapshot) FileMode 

func (s *Snapshot) FileMode() os.FileMode

FileMode return the file mode

func (*Snapshot) FileSize 

func (s *Snapshot) FileSize() uint64

FileSize return the file size

func (*Snapshot) NumSegments 

func (s *Snapshot) NumSegments() uint64

NumSegments return the number of segments for DxFile

func (*Snapshot) SectorSize 

func (s *Snapshot) SectorSize() uint64

SectorSize return the sectorSize

func (*Snapshot) Sectors 

func (s *Snapshot) Sectors(segmentIndex uint64) ([][]*Sector, error)

Sectors return the sectors of the segment index

func (*Snapshot) SegmentIndexByOffset 

func (s *Snapshot) SegmentIndexByOffset(offset uint64) (uint64, uint64)

SegmentIndexByOffset return the segment index and offset with the give offset of a file

func (*Snapshot) SegmentSize 

func (s *Snapshot) SegmentSize() uint64

SegmentSize return the size of a single segment

type SnapshotReader 

type SnapshotReader struct {
	// contains filtered or unexported fields
}

SnapshotReader is the structure that allow reading the raw DxFile content

func (*SnapshotReader) Close 

func (sr *SnapshotReader) Close() error

Close close the DxFile data file, and also release the lock of the DxFile

func (*SnapshotReader) Read 

func (sr *SnapshotReader) Read(b []byte) (int, error)

Read read the content from the DxFile data file

func (*SnapshotReader) Stat 

func (sr *SnapshotReader) Stat() (os.FileInfo, error)

Stat return the file info of the DxFile data file

type UpdateMetaData 

type UpdateMetaData struct {
	Health           uint32
	StuckHealth      uint32
	LastHealthCheck  time.Time
	NumStuckSegments uint64
	RecentRepairTime time.Time
	Redundancy       uint32
	Size             uint64
	TimeModify       time.Time
}

UpdateMetaData is the Metadata to be updated

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.