siafile

package
v1.5.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 22, 2021 License: MIT Imports: 21 Imported by: 0

README

SiaFile

The SiaFile contains all the information about an uploaded file that is required to download it plus additional metadata about the file. The SiaFile is split up into 4kib pages. The header of the SiaFile is located within the first page of the SiaFile. More pages will be allocated should the header outgrow the page. The metadata and host public key table are kept in memory for as long as the siafile is open, and the chunks are loaded and unloaded as they are accessed.

Since SiaFile's are rapidly accessed during downloads and repairs, the SiaFile was built with the requirement that all reads and writes must be able to happen in constant time, knowing only the offset of the logical data within the SiaFile. To achieve that, all the data is page-aligned which also improves disk performance. Overall the SiaFile package is designed to minimize disk I/O operations and to keep the memory footprint as small as possible without sacrificing performance.

Benchmarks

  • Writing to a random chunk of a SiaFile
    • i9-9900K with Intel SSDPEKNW010T8 -> 200 writes/second
  • Writing to a random chunk of a SiaFile (multithreaded)
    • i9-9900K with Intel SSDPEKNW010T8 -> 200 writes/second
  • Reading a random chunk of a SiaFile
    • i9-9900K with Intel SSDPEKNW010T8 -> 50,000 reads/second
  • Loading a a SiaFile's header into memory
    • i9-9900K with Intel SSDPEKNW010T8 -> 20,000 reads/second

Partial Uploads

This section contains information about how partial uploads are handled within the siafile package. "Partial Upload" refers to being able to upload a so-called partial chunk without padding it to the size of a full chunk and therefore not wasting money when uploading many small files or files with trailing partial chunks. This is achieved by combining multiple partial chunks of different SiaFiles into a combined chunk.

A SiaFile can contain at most a single partial chnk. This partial chunk can either be contained within a single combined chunk or spread across two combined chunks. If a SiaFile has a partial chunk, the HasPartialChunk field in the metadata will be set accordingly. Once it is clear which combined chunks the partial chunk is part of, SetPartialChunks will be called on the SiaFile to set the PartialChunks field in the Metadata. This field will contain one or two entries, depending on whether the partial chunk is split across two combined chunks or just one. These entries contain the required information to retrieve a partial chunk from a combined chunk and the status of the combined chunk to be able to determine whether to expect the combined chunk to be uploaded or not. Since multiple SiaFiles can reference the same combined chunks, a special type of SiaFile was introduced, called the "Partials Siafile" which also uses the SiaFile type but was a different file extension since it is never used directly.

Partials Siafiles

Partials siafiles are a special type of SiaFile. A partials siafile doesn't contain metadata about an individual file but rather contains metadata about so-called combined chunks which are referenced by the regular SiaFile type. A combined chunk is a chunk which contains multiple partial chunks which were combined into a combined chunk. As such, a SiaFile with a partial chunk contains a reference to a partials siafile and forwards calls to its exported methods to the partials siafile as necessary.

A partials siafile can't itself have partial chunks since that would require the partials siafile to reference another partials siafile. Instead it only contains combined chunks which are full chunks by definition. Since a combined chunk's size depends on its erasure code settings the same way that a regular full chunk's size does, we can only combined partial chunks with the same erasure code settings into a combined chunk which has the same settings as well. This means that for every new erasure code setting, a unique partials siafile will be created.

One implication of having a SiaFile point to a partials siafile is the fact that we don't know the corresponding partials siafile before loading the SiaFile unless we create a new SiaFile using New. That means when we load a SiaFile from a backup or from disk, we need to manually set the partials siafile afterwards using SetPartialsSiaFile.

Partial Upload Workflow

Upon the creation of a SiaFile we can determine if it contains a partial chunk by looking at the filesize. If the filesize is not a multiple of the chunk size of the file, we set the HasPartialChunk field in the metadata to 'true'. In this state, the reported Health and Redundancy of the partial chunk will be the worst possible value for both the repair code and users of the API since the chunk isn't downloadable. Once the repair code picks up the chunk, it will move the chunk into a combined chunk and call SetPartialChunks on the SiaFile, effectively moving the status of the partial chunk to CombinedChunkStatusIncomplete. At this point, the Health and Redundancy reported to users are the highest possible values while for the repair loop it is still the lowest. That way we guarantee that the repair loop periodically checks if the combined chunk is ready for uploading. Once it is, the status of the partial chunk will be moved to CombinedChunkStatusComplete and both Health and Redundancy will start reporting the actual values for the combined chunk.

Structure of the SiaFile:

Metadata

The metadata contains all the information about a SiaFile that is not specific to a single chunk of the file. This includes keys, timestamps, erasure coding etc. The definition of the Metadata type which contains all the persisted fields is located within metadata.go. The metadata is the only part of the SiaFile that is JSON encoded for easier compatibility and readability. The encoded metadata is written to the beginning of the header.

Host Public Key Table

The host public key table uses the ScPrime Binary Encoding and is written to the end of the header. As the table grows, it will grow towards the front of the header while the metadata grows towards the end. Should metadata and host public key table ever overlap, a new page will be allocated for the header. The host public key table is a table of all the hosts that contain pieces of the corresponding SiaFile.

Chunks

The chunks are written to disk starting at the first 4kib page after the header. For each chunk, the SiaFile reserves a full page on disk. That way the SiaFile always knows at which offset of the file to look for a chunk and can therefore read and write chunks in constant time. A chunk only consists of its pieces and each piece contains its merkle root and an offset which can be resolved to a host's public key using the host public key table. The chunk and piece types can be found in siafile.go.

Subsystems

The SiaFile is split up into the following subsystems.

Erasure Coding Subsystem

Key Files

File Format Subsystem

Key Files

The file format subsystem contains the type definitions for the SiaFile format and most of the exported methods of the package.

Persistence Subsystem

Key Files

The persistence subsystem handles all of the disk I/O and marshaling of datatypes. It provides helper functions to read the SiaFile from disk and atomically write to disk using the writeaheadlog package.

SiaFileSet Subsystem

Key Files

While a SiaFile object is threadsafe by itself, it's not safe to load a SiaFile into memory multiple times as this will cause corruptions on disk. Only one instance of a specific SiaFile can exist in memory at once. To ensure that, the siafileset was created as a pool for SiaFiles which is used by other packages to get access to SiaFileEntries which are wrappers for SiaFiles containing some extra information about how many threads are using it at a certain time. If a SiaFile was already loaded the siafileset will hand out the existing object, otherwise it will try to load it from disk.

Snapshot Subsystem

Key Files

The snapshot subsystem allows a user to create a readonly snapshot of a SiaFile. A snapshot contains most of the information a SiaFile does but can't be used to modify the underlying SiaFile directly. It is used to reduce locking contention within parts of the codebase where readonly access is good enough like the download code for example.

Partials Siafile Subsystem

Key Files

The partials siafile subsystem contains code which is exclusively used by partials siafiles or partial upload related helper functions. All other methods are shared by regular siafiles and partials siafiles.

Documentation

Index

Constants

View Source 
const (
	CombinedChunkStatusInvalid    = iota // status wasn't initialized
	CombinedChunkStatusInComplete        // partial chunk is included in an incomplete combined chunk.
	CombinedChunkStatusCompleted         // partial chunk is included in a completed combined chunk.
)

Constants to indicate which part of the partial upload the combined chunk is currently at.

Variables

View Source 
var (
	// ECReedSolomon is the marshaled type of the reed solomon coder.
	ECReedSolomon = modules.ErasureCoderType{0, 0, 0, 1}

	// ECReedSolomonSubShards64 is the marshaled type of the reed solomon coder
	// for files where every 64 bytes of an encoded piece can be decoded
	// separately.
	ECReedSolomonSubShards64 = modules.ErasureCoderType{0, 0, 0, 2}
)
View Source 
var (
	// ErrPathOverload is an error when a file already exists at that location
	ErrPathOverload = errors.New("a file already exists at that location")
	// ErrUnknownPath is an error when a file cannot be found with the given path
	ErrUnknownPath = errors.New("no file known with that path")
	// ErrUnknownThread is an error when a SiaFile is trying to be closed by a
	// thread that is not in the threadMap
	ErrUnknownThread = errors.New("thread should not be calling Close(), does not have control of the siafile")
	// ErrDeleted is returned when an operation failed due to the siafile being
	// deleted already.
	ErrDeleted = errors.New("files was deleted")
)

Functions

func ApplyUpdates 

func ApplyUpdates(updates ...writeaheadlog.Update) error

ApplyUpdates is a wrapper for applyUpdates that uses the production dependencies.

func CombinedChunkIndex 

func CombinedChunkIndex(numChunks, chunkIndex uint64, numCombinedChunks int) int

CombinedChunkIndex is a helper method which translates a chunk's index to the corresponding combined chunk index dependng on the number of combined chunks.

func ExtractSegment 

func ExtractSegment(pieces [][]byte, segmentIndex int, segmentSize uint64) [][]byte

ExtractSegment is a convenience method that extracts the data of the segment at segmentIndex from pieces.

func IsSiaFileUpdate 

func IsSiaFileUpdate(update writeaheadlog.Update) bool

IsSiaFileUpdate is a helper method that makes sure that a wal update belongs to the SiaFile package.

func LoadSiaFileFromReaderWithChunks 

func LoadSiaFileFromReaderWithChunks(r io.ReadSeeker, path string, wal *writeaheadlog.WAL) (*SiaFile, Chunks, error)

LoadSiaFileFromReaderWithChunks does not only read the header of the Siafile from disk but also the chunks which it returns separately. This is useful if the file is read from a buffer in-memory and the chunks can't be read from disk later.

func NewRSCode 

func NewRSCode(nData, nParity int) (modules.ErasureCoder, error)

NewRSCode creates a new Reed-Solomon encoder/decoder using the supplied parameters.

func NewRSSubCode 

func NewRSSubCode(nData, nParity int, segmentSize uint64) (modules.ErasureCoder, error)

NewRSSubCode creates a new Reed-Solomon encoder/decoder using the supplied parameters.

Types

type BubbledMetadata 

type BubbledMetadata struct {
	Health              float64
	LastHealthCheckTime time.Time
	ModTime             time.Time
	NumStuckChunks      uint64
	OnDisk              bool
	Redundancy          float64
	Size                uint64
	StuckHealth         float64
	UID                 SiafileUID
}

BubbledMetadata is the metadata of a siafile that gets bubbled

type Chunk 

type Chunk struct {
	Pieces [][]Piece
}

Chunk is an exported chunk. It contains exported pieces.

type Chunks 

type Chunks struct {
	// contains filtered or unexported fields
}

Chunks is an exported version of a chunk slice.. It exists for convenience to make sure the caller has an exported type to pass around.

type FileChunk 

type FileChunk struct {
	Pieces [][]Piece
}

FileChunk is a helper struct that contains data about a chunk.

type FileData 

type FileData struct {
	Name        string
	FileSize    uint64
	MasterKey   [crypto.EntropySize]byte
	ErasureCode modules.ErasureCoder
	RepairPath  string
	PieceSize   uint64
	Mode        os.FileMode
	Deleted     bool
	UID         SiafileUID
	Chunks      []FileChunk
}

FileData is a helper struct that contains all the relevant information of a file. It simplifies passing the necessary data between modules and keeps the interface clean.

type HostPublicKey 

type HostPublicKey struct {
	PublicKey types.SiaPublicKey // public key of host
	Used      bool               // indicates if we currently use this host
}

HostPublicKey is an entry in the HostPubKey table.

func (HostPublicKey) MarshalSia 

func (hpk HostPublicKey) MarshalSia(w io.Writer) error

MarshalSia implements the encoding.SiaMarshaler interface.

func (*HostPublicKey) UnmarshalSia 

func (hpk *HostPublicKey) UnmarshalSia(r io.Reader) error

UnmarshalSia implements the encoding.SiaUnmarshaler interface.

type Metadata 

type Metadata struct {
	UniqueID SiafileUID `json:"uniqueid"` // unique identifier for file

	StaticPagesPerChunk uint8    `json:"pagesperchunk"` // number of pages reserved for storing a chunk.
	StaticVersion       [16]byte `json:"version"`       // version of the sia file format used
	FileSize            int64    `json:"filesize"`      // total size of the file
	StaticPieceSize     uint64   `json:"piecesize"`     // size of a single piece of the file
	LocalPath           string   `json:"localpath"`     // file to the local copy of the file used for repairing

	// Fields for encryption
	StaticMasterKey      []byte            `json:"masterkey"` // masterkey used to encrypt pieces
	StaticMasterKeyType  crypto.CipherType `json:"masterkeytype"`
	StaticSharingKey     []byte            `json:"sharingkey"` // key used to encrypt shared pieces
	StaticSharingKeyType crypto.CipherType `json:"sharingkeytype"`

	// Fields for partial uploads
	DisablePartialChunk bool               `json:"disablepartialchunk"` // determines whether the file should be treated like legacy files
	PartialChunks       []PartialChunkInfo `json:"partialchunks"`       // information about the partial chunk.
	HasPartialChunk     bool               `json:"haspartialchunk"`     // indicates whether this file is supposed to have a partial chunk or not

	// The following fields are the usual unix timestamps of files.
	ModTime    time.Time `json:"modtime"`    // time of last content modification
	ChangeTime time.Time `json:"changetime"` // time of last metadata modification
	AccessTime time.Time `json:"accesstime"` // time of last access
	CreateTime time.Time `json:"createtime"` // time of file creation

	// Cached fields. These fields are cached fields and are only meant to be used
	// to create FileInfos for file related API endpoints. There is no guarantee
	// that these fields are up-to-date. Neither in memory nor on disk. Updates to
	// these fields aren't persisted immediately. Instead they will only be
	// persisted whenever another method persists the metadata or when the SiaFile
	// is closed.
	//
	// CachedRedundancy is the redundancy of the file on the network and is
	// updated within the 'Redundancy' method which is periodically called by the
	// repair code.
	//
	// CachedUserRedundancy is the redundancy of the file on the network as
	// visible to the user and is updated within the 'Redundancy' method which is
	// periodically called by the repair code.
	//
	// CachedHealth is the health of the file on the network and is also
	// periodically updated by the health check loop whenever 'Health' is called.
	//
	// CachedStuckHealth is the health of the stuck chunks of the file. It is
	// updated by the health check loop. CachedExpiration is the lowest height at
	// which any of the file's contracts will expire. Also updated periodically by
	// the health check loop whenever 'Health' is called.
	//
	// CachedUploadedBytes is the number of bytes of the file that have been
	// uploaded to the network so far. Is updated every time a piece is added to
	// the siafile.
	//
	// CachedUploadProgress is the upload progress of the file and is updated
	// every time a piece is added to the siafile.
	//
	CachedRedundancy     float64           `json:"cachedredundancy"`
	CachedUserRedundancy float64           `json:"cacheduserredundancy"`
	CachedHealth         float64           `json:"cachedhealth"`
	CachedStuckHealth    float64           `json:"cachedstuckhealth"`
	CachedExpiration     types.BlockHeight `json:"cachedexpiration"`
	CachedUploadedBytes  uint64            `json:"cacheduploadedbytes"`
	CachedUploadProgress float64           `json:"cacheduploadprogress"`

	// Repair loop fields
	//
	// Health is the worst health of the file's unstuck chunks and
	// represents the percent of redundancy missing
	//
	// LastHealthCheckTime is the timestamp of the last time the SiaFile's
	// health was checked by Health()
	//
	// NumStuckChunks is the number of all the SiaFile's chunks that have
	// been marked as stuck by the repair loop. This doesn't include a potential
	// partial chunk at the end of the file though. Use 'numStuckChunks()' for
	// that instead.
	//
	// Redundancy is the cached value of the last time the file's redundancy
	// was checked
	//
	// StuckHealth is the worst health of any of the file's stuck chunks
	//
	Health              float64   `json:"health"`
	LastHealthCheckTime time.Time `json:"lasthealthchecktime"`
	NumStuckChunks      uint64    `json:"numstuckchunks"`
	Redundancy          float64   `json:"redundancy"`
	StuckHealth         float64   `json:"stuckhealth"`

	// File ownership/permission fields.
	Mode    os.FileMode `json:"mode"`    // unix filemode of the sia file - uint32
	UserID  int32       `json:"userid"`  // id of the user who owns the file
	GroupID int32       `json:"groupid"` // id of the group that owns the file

	// The following fields are the offsets for data that is written to disk
	// after the pubKeyTable. We reserve a generous amount of space for the
	// table and extra fields, but we need to remember those offsets in case we
	// need to resize later on.
	//
	// chunkOffset is the offset of the first chunk, forced to be a factor of
	// 4096, default 4kib
	//
	// pubKeyTableOffset is the offset of the publicKeyTable within the
	// file.
	//
	ChunkOffset       int64 `json:"chunkoffset"`
	PubKeyTableOffset int64 `json:"pubkeytableoffset"`

	// erasure code settings.
	//
	// StaticErasureCodeType specifies the algorithm used for erasure coding
	// chunks. Available types are:
	//   0 - Invalid / Missing Code
	//   1 - Reed Solomon Code
	//
	// erasureCodeParams specifies possible parameters for a certain
	// StaticErasureCodeType. Currently params will be parsed as follows:
	//   Reed Solomon Code - 4 bytes dataPieces / 4 bytes parityPieces
	//
	StaticErasureCodeType   [4]byte `json:"erasurecodetype"`
	StaticErasureCodeParams [8]byte `json:"erasurecodeparams"`

	// Publink tracking. If this siafile is known to have sectors of any
	// pubfiles, those pubfiles will be listed here. It should be noted that
	// a single siafile can be responsible for tracking many pubfiles.
	Publinks []string `json:"publinks"`
	// contains filtered or unexported fields
}

Metadata is the metadata of a SiaFile and is JSON encoded. Note: Methods which update the metadata and can potentially fail after doing so and before persisting the change should use backup() and restore() to restore the metadata before returning the error. Also changes to Metadata require backup() and restore() to be updated as well.

func LoadSiaFileMetadata 

func LoadSiaFileMetadata(path string) (Metadata, error)

LoadSiaFileMetadata is a wrapper for loadSiaFileMetadata that uses the production dependencies.

type PartialChunkInfo 

type PartialChunkInfo struct {
	ID     modules.CombinedChunkID `json:"id"`     // ID of the combined chunk
	Index  uint64                  `json:"index"`  // Index of the combined chunk within partialsSiaFile
	Offset uint64                  `json:"offset"` // Offset of partial chunk within combined chunk
	Length uint64                  `json:"length"` // Length of partial chunk within combined chunk
	Status uint8                   `json:"status"` // Status of combined chunk
}

PartialChunkInfo contains all the essential information about a partial chunk relevant to SiaFiles. A SiaFile with a partial chunk may contain 1 or 2 PartialChunkInfos since the partial chunk might be split between 2 combined chunks.

type Piece 

type Piece struct {
	HostPubKey types.SiaPublicKey // public key of the host
	MerkleRoot crypto.Hash        // merkle root of the piece
}

Piece is an exported piece. It contains a resolved public key instead of the table offset.

type RSCode 

type RSCode struct {
	// contains filtered or unexported fields
}

RSCode is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface.

func (*RSCode) Encode 

func (rs *RSCode) Encode(data []byte) ([][]byte, error)

Encode splits data into equal-length pieces, some containing the original data and some containing parity data.

func (*RSCode) EncodeShards 

func (rs *RSCode) EncodeShards(pieces [][]byte) ([][]byte, error)

EncodeShards creates the parity shards for an already sharded input.

func (*RSCode) Identifier 

func (rs *RSCode) Identifier() modules.ErasureCoderIdentifier

Identifier returns an identifier for an erasure coder which can be used to identify erasure coders of the same type, dataPieces and parityPieces.

func (*RSCode) MinPieces 

func (rs *RSCode) MinPieces() int

MinPieces return the minimum number of pieces that must be present to recover the original data.

func (*RSCode) NumPieces 

func (rs *RSCode) NumPieces() int

NumPieces returns the number of pieces returned by Encode.

func (*RSCode) Reconstruct 

func (rs *RSCode) Reconstruct(pieces [][]byte) error

Reconstruct recovers the full set of encoded shards from the provided pieces, of which at least MinPieces must be non-nil.

func (*RSCode) Recover 

func (rs *RSCode) Recover(pieces [][]byte, n uint64, w io.Writer) error

Recover recovers the original data from pieces and writes it to w. pieces should be identical to the slice returned by Encode (length and order must be preserved), but with missing elements set to nil.

func (*RSCode) SupportsPartialEncoding 

func (rs *RSCode) SupportsPartialEncoding() bool

SupportsPartialEncoding returns false for the basic reed-solomon encoder.

func (*RSCode) Type 

func (rs *RSCode) Type() modules.ErasureCoderType

Type returns the erasure coders type identifier.

type RSSubCode 

type RSSubCode struct {
	RSCode
	// contains filtered or unexported fields
}

RSSubCode is a Reed-Solomon encoder/decoder. It implements the modules.ErasureCoder interface in a way that every crypto.SegmentSize bytes of encoded data can be recovered separately.

func (*RSSubCode) Encode 

func (rs *RSSubCode) Encode(data []byte) ([][]byte, error)

Encode splits data into equal-length pieces, some containing the original data and some containing parity data.

func (*RSSubCode) EncodeShards 

func (rs *RSSubCode) EncodeShards(pieces [][]byte) ([][]byte, error)

EncodeShards encodes data in a way that every segmentSize bytes of the encoded data can be decoded independently.

func (*RSSubCode) Identifier 

func (rs *RSSubCode) Identifier() modules.ErasureCoderIdentifier

Identifier returns an identifier for an erasure coder which can be used to identify erasure coders of the same type, dataPieces and parityPieces.

func (*RSSubCode) Reconstruct 

func (rs *RSSubCode) Reconstruct(pieces [][]byte) error

Reconstruct recovers the full set of encoded shards from the provided pieces, of which at least MinPieces must be non-nil.

func (*RSSubCode) Recover 

func (rs *RSSubCode) Recover(pieces [][]byte, n uint64, w io.Writer) error

Recover accepts encoded pieces and decodes the segment at segmentIndex. The size of the decoded data is segmentSize * dataPieces.

func (*RSSubCode) SupportsPartialEncoding 

func (rs *RSSubCode) SupportsPartialEncoding() bool

SupportsPartialEncoding returns true for the custom reed-solomon encoder.

func (*RSSubCode) Type 

func (rs *RSSubCode) Type() modules.ErasureCoderType

Type returns the erasure coders type identifier.

type SiaFile 

type SiaFile struct {
	// contains filtered or unexported fields
}

SiaFile is the disk format for files uploaded to the ScPrime network. It contains all the necessary information to recover a file from its hosts and allows for easy constant-time updates of the file without having to read or write the whole file.

func LoadSiaFile 

func LoadSiaFile(path string, wal *writeaheadlog.WAL) (*SiaFile, error)

LoadSiaFile is a wrapper for loadSiaFile that uses the production dependencies.

func LoadSiaFileFromReader 

func LoadSiaFileFromReader(r io.ReadSeeker, path string, wal *writeaheadlog.WAL) (*SiaFile, error)

LoadSiaFileFromReader allows loading a SiaFile from a different location that directly from disk as long as the source satisfies the SiaFileSource interface.

func New 

func New(siaFilePath, source string, wal *writeaheadlog.WAL, erasureCode modules.ErasureCoder, masterKey crypto.CipherKey, fileSize uint64, fileMode os.FileMode, partialsSiaFile *SiaFile, disablePartialUpload bool) (*SiaFile, error)

New create a new SiaFile.

func NewFromLegacyData 

func NewFromLegacyData(fd FileData, siaFilePath string, wal *writeaheadlog.WAL) (*SiaFile, error)

NewFromLegacyData creates a new SiaFile from data that was previously loaded from a legacy file.

func (*SiaFile) AccessTime 

func (sf *SiaFile) AccessTime() time.Time

AccessTime returns the AccessTime timestamp of the file.

func (*SiaFile) AddPiece 

func (sf *SiaFile) AddPiece(pk types.SiaPublicKey, chunkIndex, pieceIndex uint64, merkleRoot crypto.Hash) (err error)

AddPiece adds an uploaded piece to the file. It also updates the host table if the public key of the host is not already known. 

func (sf *SiaFile) AddPublink(s modules.Publink) (err error)

AddPublink will add a publink to the SiaFile.

func (*SiaFile) ChangeTime 

func (sf *SiaFile) ChangeTime() time.Time

ChangeTime returns the ChangeTime timestamp of the file.

func (*SiaFile) Chunk 

func (sf *SiaFile) Chunk(chunkIndex uint64) (chunk, error)

Chunk returns the chunk of a SiaFile at a given index.

func (*SiaFile) ChunkHealth 

func (sf *SiaFile) ChunkHealth(index int, offlineMap map[string]bool, goodForRenewMap map[string]bool) (float64, float64, error)

ChunkHealth returns the health of the chunk which is defined as the percent of parity pieces remaining.

func (*SiaFile) ChunkSize 

func (sf *SiaFile) ChunkSize() uint64

ChunkSize returns the size of a single chunk of the file.

func (*SiaFile) CreateTime 

func (sf *SiaFile) CreateTime() time.Time

CreateTime returns the CreateTime timestamp of the file.

func (*SiaFile) Delete 

func (sf *SiaFile) Delete() (err error)

Delete removes the file from disk and marks it as deleted. Once the file is deleted, certain methods should return an error.

func (*SiaFile) Deleted 

func (sf *SiaFile) Deleted() bool

Deleted indicates if this file has been deleted by the user.

func (*SiaFile) ErasureCode 

func (sf *SiaFile) ErasureCode() modules.ErasureCoder

ErasureCode returns the erasure coder used by the file.

func (*SiaFile) Expiration 

func (sf *SiaFile) Expiration(contracts map[string]modules.RenterContract) types.BlockHeight

Expiration updates CachedExpiration with the lowest height at which any of the file's contracts will expire and returns the new value.

func (*SiaFile) GoodPieces 

func (sf *SiaFile) GoodPieces(chunkIndex int, offlineMap map[string]bool, goodForRenewMap map[string]bool) (uint64, uint64)

GoodPieces loops over the pieces of a chunk and tracks the number of unique pieces that are good for upload, meaning the host is online, and the number of unique pieces that are good for renew, meaning the contract is set to renew.

func (*SiaFile) GrowNumChunks 

func (sf *SiaFile) GrowNumChunks(numChunks uint64) (err error)

GrowNumChunks increases the number of chunks in the SiaFile to numChunks. If the file already contains >= numChunks chunks then GrowNumChunks is a no-op.

func (*SiaFile) HasPartialChunk 

func (sf *SiaFile) HasPartialChunk() bool

HasPartialChunk returns whether this file is supposed to have a partial chunk or not.

func (*SiaFile) Health 

func (sf *SiaFile) Health(offline map[string]bool, goodForRenew map[string]bool) (h float64, sh float64, uh float64, ush float64, nsc uint64)

Health calculates the health of the file to be used in determining repair priority. Health of the file is the lowest health of any of the chunks and is defined as the percent of parity pieces remaining. The NumStuckChunks will be calculated for the SiaFile and returned.

NOTE: The cached values of the health and stuck health will be set but not saved to disk as Health() does not write to disk. If the cached values need to be updated on disk then a metadata save method should be called in conjunction with Health()

health = 0 is full redundancy, health <= 1 is recoverable, health > 1 needs to be repaired from disk

func (*SiaFile) HostPublicKeys 

func (sf *SiaFile) HostPublicKeys() (spks []types.SiaPublicKey)

HostPublicKeys returns all the public keys of hosts the file has ever been uploaded to. That means some of those hosts might no longer be in use.

func (*SiaFile) IsIncludedPartialChunk 

func (sf *SiaFile) IsIncludedPartialChunk(chunkIndex uint64) bool

IsIncludedPartialChunk returns 'true' if the provided index points to a partial chunk which has been added to the partials sia file already.

func (*SiaFile) IsIncompletePartialChunk 

func (sf *SiaFile) IsIncompletePartialChunk(chunkIndex uint64) bool

IsIncompletePartialChunk returns 'true' if the provided index points to a partial chunk which hasn't been added to a partials siafile yet.

func (*SiaFile) LastHealthCheckTime 

func (sf *SiaFile) LastHealthCheckTime() time.Time

LastHealthCheckTime returns the LastHealthCheckTime timestamp of the file

func (*SiaFile) LocalPath 

func (sf *SiaFile) LocalPath() string

LocalPath returns the path of the local data of the file.

func (*SiaFile) Lock 

func (sf *SiaFile) Lock()

Lock acquires the SiaFile's mutex for calling Unmanaged exported methods.

func (*SiaFile) MasterKey 

func (sf *SiaFile) MasterKey() crypto.CipherKey

MasterKey returns the masterkey used to encrypt the file.

func (*SiaFile) Merge 

func (sf *SiaFile) Merge(newFile *SiaFile) (map[uint64]uint64, error)

Merge merges two PartialsSiafiles into one, returning a map which translates chunk indices in newFile to indices in sf.

func (*SiaFile) Metadata 

func (sf *SiaFile) Metadata() Metadata

Metadata returns the SiaFile's metadata, resolving any fields related to partial chunks.

func (*SiaFile) ModTime 

func (sf *SiaFile) ModTime() time.Time

ModTime returns the ModTime timestamp of the file.

func (*SiaFile) Mode 

func (sf *SiaFile) Mode() os.FileMode

Mode returns the FileMode of the SiaFile.

func (*SiaFile) NumChunks 

func (sf *SiaFile) NumChunks() uint64

NumChunks returns the number of chunks the file consists of. This will return the number of chunks the file consists of even if the file is not fully uploaded yet.

func (*SiaFile) NumStuckChunks 

func (sf *SiaFile) NumStuckChunks() uint64

NumStuckChunks returns the Number of Stuck Chunks recorded in the file's metadata

func (*SiaFile) PartialChunks 

func (sf *SiaFile) PartialChunks() []PartialChunkInfo

PartialChunks returns the partial chunk infos of the siafile.

func (*SiaFile) PieceSize 

func (sf *SiaFile) PieceSize() uint64

PieceSize returns the size of a single piece of the file.

func (*SiaFile) Pieces 

func (sf *SiaFile) Pieces(chunkIndex uint64) ([][]Piece, error)

Pieces returns all the pieces for a chunk in a slice of slices that contains all the pieces for a certain index.

func (*SiaFile) Redundancy 

func (sf *SiaFile) Redundancy(offlineMap map[string]bool, goodForRenewMap map[string]bool) (r, ur float64, err error)

Redundancy returns the redundancy of the least redundant chunk. A file becomes available when this redundancy is >= 1. Assumes that every piece is unique within a file contract. -1 is returned if the file has size 0. It takes two arguments, a map of offline contracts for this file and a map that indicates if a contract is goodForRenew. The first redundancy returned is the one that should be used by the repair code and is more accurate. The other one is the redundancy presented to users.

func (*SiaFile) RemoveLastChunk 

func (sf *SiaFile) RemoveLastChunk() error

RemoveLastChunk removes the last chunk of the SiaFile and truncates the file accordingly.

func (*SiaFile) Rename 

func (sf *SiaFile) Rename(newSiaFilePath string) error

Rename changes the name of the file to a new one. To guarantee that renaming the file is atomic across all operating systems, we create a wal transaction that moves over all the chunks one-by-one and deletes the src file.

func (*SiaFile) SaveHeader 

func (sf *SiaFile) SaveHeader() (err error)

SaveHeader saves the file's header to disk.

func (*SiaFile) SaveMetadata 

func (sf *SiaFile) SaveMetadata() (err error)

SaveMetadata saves the file's metadata to disk.

func (*SiaFile) SaveWithChunks 

func (sf *SiaFile) SaveWithChunks(chunks Chunks) (err error)

SaveWithChunks saves the file's header to disk and appends the raw chunks provided at the end of the file.

func (*SiaFile) SetAllStuck 

func (sf *SiaFile) SetAllStuck(stuck bool) (err error)

SetAllStuck sets the Stuck field of all chunks to stuck.

func (*SiaFile) SetChunkStatusCompleted 

func (sf *SiaFile) SetChunkStatusCompleted(pci uint64) (err error)

SetChunkStatusCompleted sets the CombinedChunkStatus field of the metadata to completed.

func (*SiaFile) SetFileSize 

func (sf *SiaFile) SetFileSize(fileSize uint64) (err error)

SetFileSize changes the fileSize of the SiaFile.

func (*SiaFile) SetLastHealthCheckTime 

func (sf *SiaFile) SetLastHealthCheckTime()

SetLastHealthCheckTime sets the LastHealthCheckTime in memory to the current time but does not update and write to disk.

NOTE: This call should be used in conjunction with a method that saves the SiaFile metadata

func (*SiaFile) SetLocalPath 

func (sf *SiaFile) SetLocalPath(path string) (err error)

SetLocalPath changes the local path of the file which is used to repair the file from disk.

func (*SiaFile) SetMode 

func (sf *SiaFile) SetMode(mode os.FileMode) (err error)

SetMode sets the filemode of the sia file.

func (*SiaFile) SetPartialChunks 

func (sf *SiaFile) SetPartialChunks(combinedChunks []modules.PartialChunk, updates []writeaheadlog.Update) (err error)

SetPartialChunks informs the SiaFile about a partial chunk that has been saved by the partial chunk set. As such it should be exclusively called by the partial chunk set. It updates the metadata of the SiaFile and also adds a new chunk to the partial SiaFile if necessary. At the end it applies the updates of the partial chunk set, the SiaFile and the partial SiaFile atomically.

func (*SiaFile) SetPartialsSiaFile 

func (sf *SiaFile) SetPartialsSiaFile(partialsSiaFile *SiaFile)

SetPartialsSiaFile sets the partialsSiaFile field of the SiaFile. This is usually done for non-partials SiaFiles after loading them from disk.

func (*SiaFile) SetSiaFilePath 

func (sf *SiaFile) SetSiaFilePath(path string)

SetSiaFilePath sets the path of the siafile on disk.

func (*SiaFile) SetStuck 

func (sf *SiaFile) SetStuck(index uint64, stuck bool) (err error)

SetStuck sets the Stuck field of the chunk at the given index

func (*SiaFile) SiaFilePath 

func (sf *SiaFile) SiaFilePath() string

SiaFilePath returns the siaFilePath field of the SiaFile.

func (*SiaFile) Size 

func (sf *SiaFile) Size() uint64

Size returns the file's size.

func (*SiaFile) Snapshot 

func (sf *SiaFile) Snapshot(sp modules.SiaPath) (*Snapshot, error)

Snapshot creates a snapshot of the SiaFile.

func (*SiaFile) SnapshotReader 

func (sf *SiaFile) SnapshotReader() (*SnapshotReader, error)

SnapshotReader creates a io.ReadCloser that can be used to read the raw Siafile from disk. Note that the underlying siafile holds a readlock until the SnapshotReader is closed, which means that no operations can be called to the underlying siafile which may cause it to grab a lock, because that will cause a deadlock.

Operations which require grabbing a readlock on the underlying siafile are also not okay, because if some other thread has attempted to grab a writelock on the siafile, the readlock will block and then the Close() statement may never be reached for the SnapshotReader.

TODO: Things upstream would be a lot easier if we could drop the requirement to hold a lock for the duration of the life of the snapshot reader.

func (*SiaFile) StuckChunkByIndex 

func (sf *SiaFile) StuckChunkByIndex(index uint64) (bool, error)

StuckChunkByIndex returns if the chunk at the index is marked as Stuck or not

func (*SiaFile) UID 

func (sf *SiaFile) UID() SiafileUID

UID returns a unique identifier for this file.

func (*SiaFile) Unlock 

func (sf *SiaFile) Unlock()

Unlock releases the SiaFile's mutex.

func (*SiaFile) UnmanagedSetDeleted 

func (sf *SiaFile) UnmanagedSetDeleted(deleted bool)

UnmanagedSetDeleted sets the deleted field of the SiaFile without holding the lock.

func (*SiaFile) UnmanagedSetSiaFilePath 

func (sf *SiaFile) UnmanagedSetSiaFilePath(newSiaFilePath string)

UnmanagedSetSiaFilePath sets the siaFilePath field of the SiaFile without holding the lock.

func (*SiaFile) UpdateAccessTime 

func (sf *SiaFile) UpdateAccessTime() (err error)

UpdateAccessTime updates the AccessTime timestamp to the current time.

func (*SiaFile) UpdateUniqueID 

func (sf *SiaFile) UpdateUniqueID()

UpdateUniqueID creates a new random uid for the SiaFile.

func (*SiaFile) UpdateUsedHosts 

func (sf *SiaFile) UpdateUsedHosts(used []types.SiaPublicKey) (err error)

UpdateUsedHosts updates the 'Used' flag for the entries in the pubKeyTable of the SiaFile. The keys of all used hosts should be passed to the method and the SiaFile will update the flag for hosts it knows of to 'true' and set hosts which were not passed in to 'false'.

func (*SiaFile) UploadProgressAndBytes 

func (sf *SiaFile) UploadProgressAndBytes() (float64, uint64, error)

UploadProgressAndBytes is the exported wrapped for uploadProgressAndBytes.

type SiafileUID 

type SiafileUID string

SiafileUID is a unique identifier for siafile which is used to track siafiles even after renaming them.

type Snapshot 

type Snapshot struct {
	// contains filtered or unexported fields
}

Snapshot is a snapshot of a SiaFile. A snapshot is a deep-copy and can be accessed without locking at the cost of being a frozen readonly representation of a siafile which only exists in memory.

func SnapshotFromReader 

func SnapshotFromReader(sp modules.SiaPath, r io.Reader) (*Snapshot, error)

SnapshotFromReader reads a siafile from the specified reader and creates a snapshot from it.

func (*Snapshot) ChunkIndexByOffset 

func (s *Snapshot) ChunkIndexByOffset(offset uint64) (chunkIndex uint64, off uint64)

ChunkIndexByOffset will return the chunkIndex that contains the provided offset of a file and also the relative offset within the chunk. If the offset is out of bounds, chunkIndex will be equal to NumChunk().

func (*Snapshot) ChunkSize 

func (s *Snapshot) ChunkSize() uint64

ChunkSize returns the size of a single chunk of the file.

func (*Snapshot) ErasureCode 

func (s *Snapshot) ErasureCode() modules.ErasureCoder

ErasureCode returns the erasure coder used by the file.

func (*Snapshot) IsIncludedPartialChunk 

func (s *Snapshot) IsIncludedPartialChunk(chunkIndex uint64) (PartialChunkInfo, bool)

IsIncludedPartialChunk returns 'true' if the provided index points to a partial chunk which has been added to the partials sia file already.

func (*Snapshot) IsIncompletePartialChunk 

func (s *Snapshot) IsIncompletePartialChunk(chunkIndex uint64) bool

IsIncompletePartialChunk returns 'true' if the provided index points to a partial chunk which hasn't been added to a partials siafile yet.

func (*Snapshot) LocalPath 

func (s *Snapshot) LocalPath() string

LocalPath returns the localPath used to repair the file.

func (*Snapshot) MasterKey 

func (s *Snapshot) MasterKey() crypto.CipherKey

MasterKey returns the masterkey used to encrypt the file.

func (*Snapshot) Mode 

func (s *Snapshot) Mode() os.FileMode

Mode returns the FileMode of the file.

func (*Snapshot) NumChunks 

func (s *Snapshot) NumChunks() uint64

NumChunks returns the number of chunks the file consists of. This will return the number of chunks the file consists of even if the file is not fully uploaded yet.

func (*Snapshot) PartialChunks 

func (s *Snapshot) PartialChunks() []PartialChunkInfo

PartialChunks returns the snapshot's PartialChunks.

func (*Snapshot) PieceSize 

func (s *Snapshot) PieceSize() uint64

PieceSize returns the size of a single piece of the file.

func (*Snapshot) Pieces 

func (s *Snapshot) Pieces(chunkIndex uint64) [][]Piece

Pieces returns all the pieces for a chunk in a slice of slices that contains all the pieces for a certain index.

func (*Snapshot) SiaPath 

func (s *Snapshot) SiaPath() modules.SiaPath

SiaPath returns the SiaPath of the file.

func (*Snapshot) Size 

func (s *Snapshot) Size() uint64

Size returns the size of the file.

func (*Snapshot) UID 

func (s *Snapshot) UID() SiafileUID

UID returns the UID of the file.

type SnapshotReader 

type SnapshotReader struct {
	// contains filtered or unexported fields
}

SnapshotReader is a helper type that allows reading a raw SiaFile from disk while keeping the file in memory locked.

func (*SnapshotReader) Close 

func (sfr *SnapshotReader) Close() error

Close closes the underlying file.

func (*SnapshotReader) Read 

func (sfr *SnapshotReader) Read(b []byte) (int, error)

Read calls Read on the underlying file.

func (*SnapshotReader) Stat 

func (sfr *SnapshotReader) Stat() (os.FileInfo, error)

Stat returns the FileInfo of the underlying file.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.