metadb

Documentation

Overview

Package metadb package

Index

• Constants
• Variables
• func CheckDifferent(arrA, arrB []string) (int, []string)
• func CheckDirIfNotExist(path string) error
• func CheckRepeated(arr []string) bool
• func GenerateUniqueID() string
• func GetFileSize(filePath string) (int64, error)
• func NewCrcFile(c CRC, fileName string) error
• func NewMetaDataDB(config conf.StorageConfig, chainID string, dbHandle protocol.DBHandle, ...) (meta.MetaData, error)
• type CRC
  • func NewCRC(b []byte) CRC
  • func (c CRC) Update(b []byte) CRC
  • func (c CRC) Value() uint32
• type MetaDataDB
  • func (m *MetaDataDB) AbsolutePath(fileName string, fileSystem string) string
  • func (m *MetaDataDB) CreateNewFile(fileName string, fileSystem string, startIndex uint64) error
  • func (m *MetaDataDB) DoArchive(startIndex uint64, endIndex uint64) (string, error)
  • func (m *MetaDataDB) DoHotColdDataSeparation(startHeight uint64, endHeight uint64) (string, error)
  • func (m *MetaDataDB) FSBalance(fileSystemGroup []string) (string, uint64, error)
  • func (m *MetaDataDB) Get(index uint64) (storePb.MetaFileInfo, error)
  • func (m *MetaDataDB) GetAbsolutePathByFileName(fileName string) (string, error)
  • func (m *MetaDataDB) GetArchiveJobByID(jobID string) (storePb.ArchiveJob, error)
  • func (m *MetaDataDB) GetChainID() string
  • func (m *MetaDataDB) GetFileInfoByFileName(fileName string) (storePb.MetaFileInfo, error)
  • func (m *MetaDataDB) GetFileInfoByHeight(index uint64) (string, storePb.MetaFileInfo, bool, error)
  • func (m *MetaDataDB) GetHotColdDataSeparationMaxHeight() (uint64, error)
  • func (m *MetaDataDB) GetLastFileInfo() (string, storePb.MetaFileInfo, bool, error)
  • func (m *MetaDataDB) GetMetaFS() (storePb.MetaFileSystem, error)
  • func (m *MetaDataDB) GetVersionFromDB() (*storePb.MetaMultVersions, error)
  • func (m *MetaDataDB) IsArchive(fileName string) (bool, error)
  • func (m *MetaDataDB) ReadArchiveFileDataByOffset(fileName string, b []byte, offset int64) (int, error)
  • func (m *MetaDataDB) SetFileEndHeight(fileName string, endHeight uint64) error

Constants

const (
	BUFFERSIZE       = 1000
	ReserveSpaceSize = 1024 * 1024 * 256
)

归档影响的区块:

1.归档是 左闭右开。归档要指定startHeight,endHeight。实际归档时，endHeight 所在的文件，如果正是当前online文件系统 正在写的文件.
这时，不能 对endHeight 对应的文件进行归档，也就说最新的，最近写的在线文件不能归档。
2.归档是 连续区间。如果目前区块是，0-100，如果首次归档 是 50-80，是不允许的，因为<50的还未归档，必须从0号区块开始归档。
3.配置区块归档。配置区块会单独再存到db中。同时配置区块作为普通区块也会和非配置区块一起写到对应文件和索引中。

nolint

Variables

var (
	ErrOnlineFSSpaceNotEnough = errors.New("online FS disk full error")
)

nolint

Functions

func CheckDifferent

func CheckDifferent(arrA, arrB []string) (int, []string)

CheckDifferent A和B进行比较，A的元素个数小于或等于B 返回 (A+B)-(A与B交集) 的元素个数 返回 B-A 的差集，B中有，A中没有的元素 A=["a","b","c"],B=["b",d"] ,返回 (3,["d"]) CheckDifferent find the difference between arrA and arrB Returns the number of elements in the union of A and B minus the number of elements in the intersection of A and B

@Description:
@param arrA []string
@param arrB []string
@return the count(A union B) - count(A intersect B)
@return the elements of arrB minus arrA, that the elements is exist in arrB but not exist in arrA
example: A=["a","b","c"],B=["b",d"] , return (3,["d"])

func CheckDirIfNotExist

func CheckDirIfNotExist(path string) error

CheckDirIfNotExist 检查目录是否存在 CheckDirIfNotExist check the directory is exist or not

@Description:
@param path
@return error

func CheckRepeated

func CheckRepeated(arr []string) bool

CheckRepeated 验证arr中是否有重复元素 CheckRepeated check input elements of array is repeated or not

@Description:
@param arr []string
@return bool

func GenerateUniqueID

func GenerateUniqueID() string

GenerateUniqueID get unique id

@Description:
@return string

func GetFileSize

func GetFileSize(filePath string) (int64, error)

GetFileSize 获得文件大小 GetFileSize get file size

@Description:
@param filePath
@return file size
@return error

func NewCrcFile

func NewCrcFile(c CRC, fileName string) error

NewCrcFile 创建一个保存crc的文件 /todo:处理 文件已存在的情况下，create file error NewCrcFile create a crc file

@Description:
@param crc
@param file name
@return error

func NewMetaDataDB

func NewMetaDataDB(config conf.StorageConfig, chainID string, dbHandle protocol.DBHandle,
	logger protocol.Logger) (meta.MetaData, error)

NewMetaDataDB create MetaData

@Description:
@param storeConfig
@param chainID
@param dbHandle
@param logger
@return MetaData
@return error

Types

type CRC

type CRC uint32

CRC is a CRC-32 checksum computed using Castagnoli's polynomial.

func NewCRC

func NewCRC(b []byte) CRC

NewCRC creates a new crc based on the given bytes.

func (CRC) Update

func (c CRC) Update(b []byte) CRC

Update updates the crc with the given bytes.

func (CRC) Value

func (c CRC) Value() uint32

Value returns a masked crc.

type MetaDataDB

type MetaDataDB struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

MetaDataDB manage meta data of block file , include create meta info of block file , and separate from hot and cold data , and read data from block file which has been separated. Notice 1,if the block file is not be separated ,than read data from block file directly, rather than read from meta data db. Notice 2, the startHeight and endHeight of MetaFileInfo are not the block height, they are the entry index of block file. The entry index of block file is 1 less than the block height. Example :

	MetaFileSystem {
       startHeight:3,
       endHeight:8,
	}

it means this segment or file contains [3,8] entries .A entry is a block. And it means this segment contains the block height range is [2,7] rather than [3,8].

@Description:

func (*MetaDataDB) AbsolutePath

func (m *MetaDataDB) AbsolutePath(fileName string, fileSystem string) string

AbsolutePath 获得绝对路径，根据文件系统和文件名 绝对路径: $filesystem/$chainID/bfdb/$filename.fdb AbsolutePath get absolute path,include file system, separate char, chainID, const var ,and file name

@Description:
@receiver m
@param fileName
@param fileSystem
@return string

func (*MetaDataDB) CreateNewFile

func (m *MetaDataDB) CreateNewFile(fileName string, fileSystem string, startIndex uint64) error

CreateNewFile 创建一个文件，在元数据系统中 CreateNewFile create a new file internal meta info, it is just a meta data not an os file

@Description: MetaFileInfo.startHeight is the index of wal segment,  block number +1 is equal index.
for example MetaFileInfo.startHeight is 6, the block height is 7.
so, MetaFileInfo{StartHeight:6,EndHeight:10} that means the file includes [7,11] blocks

@receiver m
@param fileName
@param fileSystem
@param startHeight
@return error

func (*MetaDataDB) DoArchive

func (m *MetaDataDB) DoArchive(startIndex uint64, endIndex uint64) (string, error)

DoArchive 发起一次归档操作,异步执行,回调结果 /todo:是否要强制要求必须保证 [0,start-1]区间的已经归档了，才能 让 [start,end] 进行归档 目前未限制 /todo:归档完成后，被归档的文件是否要通知 blockfiledb 和 resultfiledb 去重新打开 目前看不需要 /todo: 当前最新的wal文件，后缀名带了 .END ,不能归档 如果归档时，要对最新的wal文件进行归档，不允许 最近10个块，不允许归档 DoArchive do a job for separate hot data and cold data by start height and end height of block

@Description: startIndex -1 is startHeight,  endIndex -1 is endHeight
@receiver m
@param startHeight
@param endHeight
@return string   return a job id ,that client can get job status by job id.
@return error

func (*MetaDataDB) DoHotColdDataSeparation

func (m *MetaDataDB) DoHotColdDataSeparation(startHeight uint64, endHeight uint64) (string, error)

DoHotColdDataSeparation 发起一次冷热数据分离 startHeight,endHeight 需要冷热分离的区块区间块高 DoHotColdDataSeparation do a job for separate hot data and cold data by start height and end height of block

@Description:
@receiver m
@param startHeight
@param endHeight
@return string   return a job id ,that client can get job status by job id.
@return error

func (*MetaDataDB) FSBalance

func (m *MetaDataDB) FSBalance(fileSystemGroup []string) (string, uint64, error)

FSBalance 按照磁盘空间均衡策略，返回最优文件系统 返回值: 剩余空间最大的文件系统，可用空间，错误信息 FSBalance get a file system with the largest amount of free space

@Description:
@receiver m
@param fileSystemGroup
@return string   file system which free space capacity is largest
@return uint64   free space capacity
@return error

func (*MetaDataDB) Get

func (m *MetaDataDB) Get(index uint64) (storePb.MetaFileInfo, error)

Get 按照指定块高，返回对应块高的归属的文件 Get get meta info by height

@Description:
@receiver m
@param height
@return MetaFileInfo
@return error

func (*MetaDataDB) GetAbsolutePathByFileName

func (m *MetaDataDB) GetAbsolutePathByFileName(fileName string) (string, error)

GetAbsolutePathByFileName return file's path as absolute path AbsPath: $filesystem/$chainID/bfdb/$filename.fdb AbsolutePath get absolute path,include file system, separate char, chainID, const var ,and file name

@Description:
@receiver m
@param fileName
@return string
@return error

func (*MetaDataDB) GetArchiveJobByID

func (m *MetaDataDB) GetArchiveJobByID(jobID string) (storePb.ArchiveJob, error)

GetArchiveJobByID 从db读一个归档任务 GetArchiveJobByID get archive job info by job ID

@Description:
@receiver m
@param jobID
@return archiveJob
@return error

func (*MetaDataDB) GetChainID

func (m *MetaDataDB) GetChainID() string

GetChainID 获得metadb中的chainID GetChainID get the chainID from metadb

@Description:
@receiver m
@return chainID

func (*MetaDataDB) GetFileInfoByFileName

func (m *MetaDataDB) GetFileInfoByFileName(fileName string) (storePb.MetaFileInfo, error)

GetFileInfoByFileName 获得文件信息 GetFileInfoByFileName get file info by file name

@Description:
@receiver m
@param fileName
@return MetaFileInfo
@return error

func (*MetaDataDB) GetFileInfoByHeight

func (m *MetaDataDB) GetFileInfoByHeight(index uint64) (string, storePb.MetaFileInfo, bool, error)

GetFileInfoByHeight return the filename,file info,file exist or not, error

func (*MetaDataDB) GetHotColdDataSeparationMaxHeight

func (m *MetaDataDB) GetHotColdDataSeparationMaxHeight() (uint64, error)

GetHotColdDataSeparationMaxHeight get the max height which can be used for do hot-cold data separations

@Description:
@receiver m
@return uint64   return a max height
@return error

func (*MetaDataDB) GetLastFileInfo

func (m *MetaDataDB) GetLastFileInfo() (string, storePb.MetaFileInfo, bool, error)

GetLastFileInfo 获得最后生成的文件 信息,如果没有 返回false GetLastFileInfo get last file info

@Description:
@receiver m
@return MetaFileInfo
@return bool
@return error

func (*MetaDataDB) GetMetaFS

func (m *MetaDataDB) GetMetaFS() (storePb.MetaFileSystem, error)

GetMetaFS 获得metadb中的元数据,返回最新版本的FS GetMetaFS get the meta file system

@Description:
@receiver m
@return MetaFileSystem
@return error

func (*MetaDataDB) GetVersionFromDB

func (m *MetaDataDB) GetVersionFromDB() (*storePb.MetaMultVersions, error)

GetVersionFromDB 从db中获得所有版本 GetVersionFromDB get all versions of meta from db, the meta support multiple version

@Description:
@receiver m
@param storeConfig
@return error

func (*MetaDataDB) IsArchive

func (m *MetaDataDB) IsArchive(fileName string) (bool, error)

IsArchive return true or false if file is archived or not

@Description:
@receiver m
@param fileName
@return bool
@return error

func (*MetaDataDB) ReadArchiveFileDataByOffset

func (m *MetaDataDB) ReadArchiveFileDataByOffset(fileName string, b []byte, offset int64) (int, error)

ReadArchiveFileDataByOffset 读取已归档的文件数据 ReadArchiveFileDataByOffset read data by offset from archive file which file is cold

@Description:
@receiver m
@param fileName
@param b    input []byte, the data is saved to b
@param offset      read position of file
@return int        data length that read
@return error

func (*MetaDataDB) SetFileEndHeight

func (m *MetaDataDB) SetFileEndHeight(fileName string, endHeight uint64) error

SetFileEndHeight 设置file的结束区块的块高 SetFileEndHeight set the end height of file

@Description:
@receiver m
@param fileName
@param endHeight
@return error