db

package
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 26, 2018 License: MPL-2.0 Imports: 24 Imported by: 0

Documentation

Overview

Package db provides a set type to track local/remote files with newness checks. We must do a certain amount of normalization in here. We will get fed paths with either native or wire-format separators and encodings depending on who calls us. We transform paths to wire-format (NFC and slashes) on the way to the database, and transform to native format (varying separator and encoding) on the way back out. 

Package db is a generated protocol buffer package.

It is generated from these files:
	structs.proto

It has these top-level messages:
	FileVersion
	VersionList
	FileInfoTruncated
	Counts
	CountsSet

Index

Constants

View Source 
const (
	// KeyTypeDevice <int32 folder ID> <int32 device ID> <file name> = FileInfo
	KeyTypeDevice = 0

	// KeyTypeGlobal <int32 folder ID> <file name> = VersionList
	KeyTypeGlobal = 1

	// KeyTypeBlock <int32 folder ID> <32 bytes hash> <§file name> = int32 (block index)
	KeyTypeBlock = 2

	// KeyTypeDeviceStatistic <device ID as string> <some string> = some value
	KeyTypeDeviceStatistic = 3

	// KeyTypeFolderStatistic <folder ID as string> <some string> = some value
	KeyTypeFolderStatistic = 4

	// KeyTypeVirtualMtime <int32 folder ID> <file name> = dbMtime
	KeyTypeVirtualMtime = 5

	// KeyTypeFolderIdx <int32 id> = string value
	KeyTypeFolderIdx = 6

	// KeyTypeDeviceIdx <int32 id> = string value
	KeyTypeDeviceIdx = 7

	// KeyTypeIndexID <int32 device ID> <int32 folder ID> = protocol.IndexID
	KeyTypeIndexID = 8

	// KeyTypeFolderMeta <int32 folder ID> = CountsSet
	KeyTypeFolderMeta = 9

	// KeyTypeMiscData <some string> = some value
	KeyTypeMiscData = 10

	// KeyTypeSequence <int32 folder ID> <int64 sequence number> = KeyTypeDevice key
	KeyTypeSequence = 11

	// KeyTypeNeed <int32 folder ID> <file name> = <nothing>
	KeyTypeNeed = 12
)

Variables

View Source 
var (
	ErrInvalidLengthStructs = fmt.Errorf("proto: negative length found during unmarshaling")
	ErrIntOverflowStructs   = fmt.Errorf("proto: integer overflow")
)

Functions

func DropDeltaIndexIDs 

func DropDeltaIndexIDs(db *Lowlevel)

DropDeltaIndexIDs removes all delta index IDs from the database. This will cause a full index transmission on the next connection.

func DropFolder 

func DropFolder(ll *Lowlevel, folder string)

DropFolder clears out all information related to the given folder from the database.

func UpdateSchema 

func UpdateSchema(ll *Lowlevel) error

Types

type BlockFinder 

type BlockFinder struct {
	// contains filtered or unexported fields
}

func NewBlockFinder 

func NewBlockFinder(db *Lowlevel) *BlockFinder

func (*BlockFinder) Iterate 

func (f *BlockFinder) Iterate(folders []string, hash []byte, iterFn func(string, string, int32) bool) bool

Iterate takes an iterator function which iterates over all matching blocks for the given hash. The iterator function has to return either true (if they are happy with the block) or false to continue iterating for whatever reason. The iterator finally returns the result, whether or not a satisfying block was eventually found.

func (*BlockFinder) String 

func (f *BlockFinder) String() string

type BlockMap 

type BlockMap struct {
	// contains filtered or unexported fields
}

func NewBlockMap 

func NewBlockMap(db *Lowlevel, folder string) *BlockMap

func (*BlockMap) Add 

func (m *BlockMap) Add(files []protocol.FileInfo) error

Add files to the block map, ignoring any deleted or invalid files.

func (*BlockMap) Discard 

func (m *BlockMap) Discard(files []protocol.FileInfo) error

Discard block map state, removing the given files

func (*BlockMap) Drop 

func (m *BlockMap) Drop() error

Drop block map, removing all entries related to this block map from the db.

func (*BlockMap) Update 

func (m *BlockMap) Update(files []protocol.FileInfo) error

Update block map state, removing any deleted or invalid files.

type Counts added in v0.14.12 

type Counts struct {
	Files       int32  `protobuf:"varint,1,opt,name=files,proto3" json:"files,omitempty"`
	Directories int32  `protobuf:"varint,2,opt,name=directories,proto3" json:"directories,omitempty"`
	Symlinks    int32  `protobuf:"varint,3,opt,name=symlinks,proto3" json:"symlinks,omitempty"`
	Deleted     int32  `protobuf:"varint,4,opt,name=deleted,proto3" json:"deleted,omitempty"`
	Bytes       int64  `protobuf:"varint,5,opt,name=bytes,proto3" json:"bytes,omitempty"`
	Sequence    int64  `protobuf:"varint,6,opt,name=sequence,proto3" json:"sequence,omitempty"`
	DeviceID    []byte `protobuf:"bytes,17,opt,name=deviceID,proto3" json:"deviceID,omitempty"`
	LocalFlags  uint32 `protobuf:"varint,18,opt,name=localFlags,proto3" json:"localFlags,omitempty"`
}

For each folder and device we keep one of these to track the current counts and sequence. We also keep one for the global state of the folder.

func (Counts) Add 

func (c Counts) Add(other Counts) Counts

func (*Counts) Descriptor 

func (*Counts) Descriptor() ([]byte, []int)

func (*Counts) Marshal 

func (m *Counts) Marshal() (dAtA []byte, err error)

func (*Counts) MarshalTo 

func (m *Counts) MarshalTo(dAtA []byte) (int, error)

func (*Counts) ProtoMessage 

func (*Counts) ProtoMessage()

func (*Counts) ProtoSize 

func (m *Counts) ProtoSize() (n int)

func (*Counts) Reset 

func (m *Counts) Reset()

func (*Counts) String 

func (m *Counts) String() string

func (Counts) TotalItems 

func (c Counts) TotalItems() int32

func (*Counts) Unmarshal 

func (m *Counts) Unmarshal(dAtA []byte) error

type CountsSet 

type CountsSet struct {
	Counts  []Counts `protobuf:"bytes,1,rep,name=counts" json:"counts"`
	Created int64    `protobuf:"varint,2,opt,name=created,proto3" json:"created,omitempty"`
}

func (*CountsSet) Descriptor 

func (*CountsSet) Descriptor() ([]byte, []int)

func (*CountsSet) Marshal 

func (m *CountsSet) Marshal() (dAtA []byte, err error)

func (*CountsSet) MarshalTo 

func (m *CountsSet) MarshalTo(dAtA []byte) (int, error)

func (*CountsSet) ProtoMessage 

func (*CountsSet) ProtoMessage()

func (*CountsSet) ProtoSize 

func (m *CountsSet) ProtoSize() (n int)

func (*CountsSet) Reset 

func (m *CountsSet) Reset()

func (*CountsSet) String 

func (m *CountsSet) String() string

func (*CountsSet) Unmarshal 

func (m *CountsSet) Unmarshal(dAtA []byte) error

type FileInfoTruncated 

type FileInfoTruncated struct {
	Name          string                                              `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"`
	Type          protocol.FileInfoType                               `protobuf:"varint,2,opt,name=type,proto3,enum=protocol.FileInfoType" json:"type,omitempty"`
	Size          int64                                               `protobuf:"varint,3,opt,name=size,proto3" json:"size,omitempty"`
	Permissions   uint32                                              `protobuf:"varint,4,opt,name=permissions,proto3" json:"permissions,omitempty"`
	ModifiedS     int64                                               `protobuf:"varint,5,opt,name=modified_s,json=modifiedS,proto3" json:"modified_s,omitempty"`
	ModifiedNs    int32                                               `protobuf:"varint,11,opt,name=modified_ns,json=modifiedNs,proto3" json:"modified_ns,omitempty"`
	ModifiedBy    github_com_syncthing_syncthing_lib_protocol.ShortID `` /* 146-byte string literal not displayed */
	Deleted       bool                                                `protobuf:"varint,6,opt,name=deleted,proto3" json:"deleted,omitempty"`
	RawInvalid    bool                                                `protobuf:"varint,7,opt,name=invalid,proto3" json:"invalid,omitempty"`
	NoPermissions bool                                                `protobuf:"varint,8,opt,name=no_permissions,json=noPermissions,proto3" json:"no_permissions,omitempty"`
	Version       protocol.Vector                                     `protobuf:"bytes,9,opt,name=version" json:"version"`
	Sequence      int64                                               `protobuf:"varint,10,opt,name=sequence,proto3" json:"sequence,omitempty"`
	RawBlockSize  int32                                               `protobuf:"varint,13,opt,name=block_size,json=blockSize,proto3" json:"block_size,omitempty"`
	// repeated BlockInfo Blocks         = 16
	SymlinkTarget string `protobuf:"bytes,17,opt,name=symlink_target,json=symlinkTarget,proto3" json:"symlink_target,omitempty"`
	// see bep.proto
	LocalFlags uint32 `protobuf:"varint,1000,opt,name=local_flags,json=localFlags,proto3" json:"local_flags,omitempty"`
}

Must be the same as FileInfo but without the blocks field

func (FileInfoTruncated) BlockSize 

func (f FileInfoTruncated) BlockSize() int

func (FileInfoTruncated) ConvertToIgnoredFileInfo 

func (f FileInfoTruncated) ConvertToIgnoredFileInfo(by protocol.ShortID) protocol.FileInfo

func (*FileInfoTruncated) Descriptor added in v0.14.12 

func (*FileInfoTruncated) Descriptor() ([]byte, []int)

func (FileInfoTruncated) FileLocalFlags 

func (f FileInfoTruncated) FileLocalFlags() uint32

func (FileInfoTruncated) FileName added in v0.14.12 

func (f FileInfoTruncated) FileName() string

func (FileInfoTruncated) FileSize added in v0.14.12 

func (f FileInfoTruncated) FileSize() int64

func (FileInfoTruncated) FileVersion 

func (f FileInfoTruncated) FileVersion() protocol.Vector

func (FileInfoTruncated) HasPermissionBits added in v0.14.12 

func (f FileInfoTruncated) HasPermissionBits() bool

func (FileInfoTruncated) IsDeleted added in v0.14.12 

func (f FileInfoTruncated) IsDeleted() bool

func (FileInfoTruncated) IsDirectory added in v0.14.12 

func (f FileInfoTruncated) IsDirectory() bool

func (FileInfoTruncated) IsIgnored 

func (f FileInfoTruncated) IsIgnored() bool

func (FileInfoTruncated) IsInvalid added in v0.14.12 

func (f FileInfoTruncated) IsInvalid() bool

func (FileInfoTruncated) IsReceiveOnlyChanged 

func (f FileInfoTruncated) IsReceiveOnlyChanged() bool
func (f FileInfoTruncated) IsSymlink() bool

func (FileInfoTruncated) IsUnsupported 

func (f FileInfoTruncated) IsUnsupported() bool

func (*FileInfoTruncated) Marshal added in v0.14.12 

func (m *FileInfoTruncated) Marshal() (dAtA []byte, err error)

func (*FileInfoTruncated) MarshalTo added in v0.14.12 

func (m *FileInfoTruncated) MarshalTo(dAtA []byte) (int, error)

func (FileInfoTruncated) ModTime added in v0.14.12 

func (f FileInfoTruncated) ModTime() time.Time

func (FileInfoTruncated) MustRescan 

func (f FileInfoTruncated) MustRescan() bool

func (*FileInfoTruncated) ProtoMessage added in v0.14.12 

func (*FileInfoTruncated) ProtoMessage()

func (*FileInfoTruncated) ProtoSize added in v0.14.12 

func (m *FileInfoTruncated) ProtoSize() (n int)

func (*FileInfoTruncated) Reset added in v0.14.12 

func (m *FileInfoTruncated) Reset()

func (FileInfoTruncated) SequenceNo 

func (f FileInfoTruncated) SequenceNo() int64

func (FileInfoTruncated) ShouldConflict 

func (f FileInfoTruncated) ShouldConflict() bool

func (FileInfoTruncated) String added in v0.14.12 

func (f FileInfoTruncated) String() string

func (*FileInfoTruncated) Unmarshal added in v0.14.12 

func (m *FileInfoTruncated) Unmarshal(dAtA []byte) error

type FileIntf 

type FileIntf interface {
	FileSize() int64
	FileName() string
	FileLocalFlags() uint32
	IsDeleted() bool
	IsInvalid() bool
	IsIgnored() bool
	IsUnsupported() bool
	MustRescan() bool
	IsReceiveOnlyChanged() bool
	IsDirectory() bool
	IsSymlink() bool
	ShouldConflict() bool
	HasPermissionBits() bool
	SequenceNo() int64
	BlockSize() int
	FileVersion() protocol.Vector
}

FileIntf is the set of methods implemented by both protocol.FileInfo and FileInfoTruncated.

type FileSet 

type FileSet struct {
	// contains filtered or unexported fields
}

func NewFileSet 

func NewFileSet(folder string, fs fs.Filesystem, ll *Lowlevel) *FileSet

func (*FileSet) Availability 

func (s *FileSet) Availability(file string) []protocol.DeviceID

func (*FileSet) Drop 

func (s *FileSet) Drop(device protocol.DeviceID)

func (*FileSet) Get 

func (s *FileSet) Get(device protocol.DeviceID, file string) (protocol.FileInfo, bool)

func (*FileSet) GetGlobal 

func (s *FileSet) GetGlobal(file string) (protocol.FileInfo, bool)

func (*FileSet) GetGlobalTruncated 

func (s *FileSet) GetGlobalTruncated(file string) (FileInfoTruncated, bool)

func (*FileSet) GlobalSize 

func (s *FileSet) GlobalSize() Counts

func (*FileSet) IndexID added in v0.14.12 

func (s *FileSet) IndexID(device protocol.DeviceID) protocol.IndexID

func (*FileSet) ListDevices added in v0.14.12 

func (s *FileSet) ListDevices() []protocol.DeviceID

func (*FileSet) LocalSize 

func (s *FileSet) LocalSize() Counts

func (*FileSet) MtimeFS added in v0.14.12 

func (s *FileSet) MtimeFS() *fs.MtimeFS

func (*FileSet) ReceiveOnlyChangedSize 

func (s *FileSet) ReceiveOnlyChangedSize() Counts

func (*FileSet) Sequence added in v0.14.12 

func (s *FileSet) Sequence(device protocol.DeviceID) int64

func (*FileSet) SetIndexID added in v0.14.12 

func (s *FileSet) SetIndexID(device protocol.DeviceID, id protocol.IndexID)

func (*FileSet) Update 

func (s *FileSet) Update(device protocol.DeviceID, fs []protocol.FileInfo)

func (*FileSet) WithGlobal 

func (s *FileSet) WithGlobal(fn Iterator)

func (*FileSet) WithGlobalTruncated 

func (s *FileSet) WithGlobalTruncated(fn Iterator)

func (*FileSet) WithHave 

func (s *FileSet) WithHave(device protocol.DeviceID, fn Iterator)

func (*FileSet) WithHaveSequence 

func (s *FileSet) WithHaveSequence(startSeq int64, fn Iterator)

func (*FileSet) WithHaveTruncated 

func (s *FileSet) WithHaveTruncated(device protocol.DeviceID, fn Iterator)

func (*FileSet) WithNeed 

func (s *FileSet) WithNeed(device protocol.DeviceID, fn Iterator)

func (*FileSet) WithNeedTruncated 

func (s *FileSet) WithNeedTruncated(device protocol.DeviceID, fn Iterator)

func (*FileSet) WithPrefixedGlobalTruncated 

func (s *FileSet) WithPrefixedGlobalTruncated(prefix string, fn Iterator)

Except for an item with a path equal to prefix, only children of prefix are iterated. E.g. for prefix "dir", "dir/file" is iterated, but "dir.file" is not.

func (*FileSet) WithPrefixedHaveTruncated added in v0.12.22 

func (s *FileSet) WithPrefixedHaveTruncated(device protocol.DeviceID, prefix string, fn Iterator)

Except for an item with a path equal to prefix, only children of prefix are iterated. E.g. for prefix "dir", "dir/file" is iterated, but "dir.file" is not.

type FileVersion added in v0.14.12 

type FileVersion struct {
	Version protocol.Vector `protobuf:"bytes,1,opt,name=version" json:"version"`
	Device  []byte          `protobuf:"bytes,2,opt,name=device,proto3" json:"device,omitempty"`
	Invalid bool            `protobuf:"varint,3,opt,name=invalid,proto3" json:"invalid,omitempty"`
}

func (*FileVersion) Descriptor added in v0.14.12 

func (*FileVersion) Descriptor() ([]byte, []int)

func (*FileVersion) Marshal added in v0.14.12 

func (m *FileVersion) Marshal() (dAtA []byte, err error)

func (*FileVersion) MarshalTo added in v0.14.12 

func (m *FileVersion) MarshalTo(dAtA []byte) (int, error)

func (*FileVersion) ProtoMessage added in v0.14.12 

func (*FileVersion) ProtoMessage()

func (*FileVersion) ProtoSize added in v0.14.12 

func (m *FileVersion) ProtoSize() (n int)

func (*FileVersion) Reset added in v0.14.12 

func (m *FileVersion) Reset()

func (*FileVersion) String added in v0.14.12 

func (m *FileVersion) String() string

func (*FileVersion) Unmarshal added in v0.14.12 

func (m *FileVersion) Unmarshal(dAtA []byte) error

type Iterator 

type Iterator func(f FileIntf) bool

The Iterator is called with either a protocol.FileInfo or a FileInfoTruncated (depending on the method) and returns true to continue iteration, false to stop.

type Lowlevel 

type Lowlevel struct {
	*leveldb.DB
	// contains filtered or unexported fields
}

Lowlevel is the lowest level database interface. It has a very simple purpose: hold the actual *leveldb.DB database, and the in-memory state that belong to that database. In the same way that a single on disk database can only be opened once, there should be only one Lowlevel for any given *leveldb.DB.

func NewLowlevel 

func NewLowlevel(db *leveldb.DB, location string) *Lowlevel

NewLowlevel wraps the given *leveldb.DB into a *lowlevel

func Open 

func Open(location string) (*Lowlevel, error)

Open attempts to open the database at the given location, and runs recovery on it if opening fails. Worst case, if recovery is not possible, the database is erased and created from scratch.

func OpenMemory 

func OpenMemory() *Lowlevel

OpenMemory returns a new Lowlevel referencing an in-memory database.

func OpenRO 

func OpenRO(location string) (*Lowlevel, error)

OpenRO attempts to open the database at the given location, read only.

func (*Lowlevel) Committed 

func (db *Lowlevel) Committed() int64

Committed returns the number of items committed to the database since startup

func (*Lowlevel) Delete 

func (db *Lowlevel) Delete(key []byte, wo *opt.WriteOptions) error

func (*Lowlevel) ListFolders 

func (db *Lowlevel) ListFolders() []string

ListFolders returns the list of folders currently in the database

func (*Lowlevel) Location 

func (db *Lowlevel) Location() string

Location returns the filesystem path where the database is stored

func (*Lowlevel) Put 

func (db *Lowlevel) Put(key, val []byte, wo *opt.WriteOptions) error

type NamespacedKV 

type NamespacedKV struct {
	// contains filtered or unexported fields
}

NamespacedKV is a simple key-value store using a specific namespace within a leveldb.

func NewDeviceStatisticsNamespace 

func NewDeviceStatisticsNamespace(db *Lowlevel, device string) *NamespacedKV

NewDeviceStatisticsNamespace creates a KV namespace for device statistics for the given device.

func NewFolderStatisticsNamespace 

func NewFolderStatisticsNamespace(db *Lowlevel, folder string) *NamespacedKV

NewFolderStatisticsNamespace creates a KV namespace for folder statistics for the given folder.

func NewMiscDataNamespace 

func NewMiscDataNamespace(db *Lowlevel) *NamespacedKV

NewMiscDateNamespace creates a KV namespace for miscellaneous metadata.

func NewNamespacedKV 

func NewNamespacedKV(db *Lowlevel, prefix string) *NamespacedKV

NewNamespacedKV returns a new NamespacedKV that lives in the namespace specified by the prefix.

func (NamespacedKV) Bool 

func (n NamespacedKV) Bool(key string) (bool, bool)

Bool returns the stored value as a boolean and a boolean that is false if no value was stored at the key.

func (NamespacedKV) Bytes 

func (n NamespacedKV) Bytes(key string) ([]byte, bool)

Bytes returns the stored value as a raw byte slice and a boolean that is false if no value was stored at the key.

func (NamespacedKV) Delete 

func (n NamespacedKV) Delete(key string)

Delete deletes the specified key. It is allowed to delete a nonexistent key.

func (*NamespacedKV) Int64 

func (n *NamespacedKV) Int64(key string) (int64, bool)

Int64 returns the stored value interpreted as an int64 and a boolean that is false if no value was stored at the key.

func (*NamespacedKV) PutBool 

func (n *NamespacedKV) PutBool(key string, val bool)

PutBool stores a new boolean. Any existing value (even if of another type) is overwritten.

func (*NamespacedKV) PutBytes 

func (n *NamespacedKV) PutBytes(key string, val []byte)

PutBytes stores a new byte slice. Any existing value (even if of another type) is overwritten.

func (*NamespacedKV) PutInt64 

func (n *NamespacedKV) PutInt64(key string, val int64)

PutInt64 stores a new int64. Any existing value (even if of another type) is overwritten.

func (*NamespacedKV) PutString 

func (n *NamespacedKV) PutString(key, val string)

PutString stores a new string. Any existing value (even if of another type) is overwritten.

func (*NamespacedKV) PutTime 

func (n *NamespacedKV) PutTime(key string, val time.Time)

PutTime stores a new time.Time. Any existing value (even if of another type) is overwritten.

func (*NamespacedKV) Reset 

func (n *NamespacedKV) Reset()

Reset removes all entries in this namespace.

func (NamespacedKV) String 

func (n NamespacedKV) String(key string) (string, bool)

String returns the stored value interpreted as a string and a boolean that is false if no value was stored at the key.

func (NamespacedKV) Time 

func (n NamespacedKV) Time(key string) (time.Time, bool)

Time returns the stored value interpreted as a time.Time and a boolean that is false if no value was stored at the key.

type VersionList added in v0.14.12 

type VersionList struct {
	Versions []FileVersion `protobuf:"bytes,1,rep,name=versions" json:"versions"`
}

func (*VersionList) Descriptor added in v0.14.12 

func (*VersionList) Descriptor() ([]byte, []int)

func (VersionList) Get 

func (vl VersionList) Get(device []byte) (FileVersion, bool)

func (*VersionList) Marshal added in v0.14.12 

func (m *VersionList) Marshal() (dAtA []byte, err error)

func (*VersionList) MarshalTo added in v0.14.12 

func (m *VersionList) MarshalTo(dAtA []byte) (int, error)

func (*VersionList) ProtoMessage added in v0.14.12 

func (*VersionList) ProtoMessage()

func (*VersionList) ProtoSize added in v0.14.12 

func (m *VersionList) ProtoSize() (n int)

func (*VersionList) Reset added in v0.14.12 

func (m *VersionList) Reset()

func (VersionList) String added in v0.14.12 

func (vl VersionList) String() string

func (*VersionList) Unmarshal added in v0.14.12 

func (m *VersionList) Unmarshal(dAtA []byte) error

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.