Documentation
¶
Overview ¶
Package integrity detects a form of storage corruption introduced by earlier DoltgreSQL releases: prolly tree node messages whose address offset fields (value_address_offsets for value tuples, key_address_offsets for key tuples, at every tree level) omit entries for adaptive-encoded values stored out of band. Nodes written this way are missing chunk references, which causes push and clone to omit the out-of-band chunks, and garbage collection to delete them, resulting in data loss. See cmd/admin for the offline tool that reports and repairs this corruption.
Index ¶
- Constants
- func CheckDatabase(ctx *sql.Context, dbName string, ddb *doltdb.DoltDB) error
- func ChildAddresses(pm *serial.ProllyTreeNode) []hash.Hash
- func DescFingerprint(kd, vd *val.TupleDesc) uint64
- func GetTreeNodeMessage(ctx context.Context, cs chunks.ChunkStore, addr hash.Hash) (serial.Message, error)
- func HasValidSentinel(fs filesys.Filesys, doltDir string) bool
- func WriteSentinel(fs filesys.Filesys, doltDir string) error
- type CacheKey
- type CorruptionError
- type InternalKeyAnalysis
- type LeafAnalysis
- type Scanner
- type Stats
- type TableInfo
Constants ¶
const SentinelFileName = ".integrity_check_passed"
SentinelFileName is the name of the file dropped in a database's .dolt directory once the database has passed the startup integrity check, so that subsequent startups can skip re-checking it
Variables ¶
This section is empty.
Functions ¶
func CheckDatabase ¶
CheckDatabase scans every table in every commit reachable from every branch of the database, as well as each branch's working set, for integrity. It returns a *CorruptionError describing the first corrupted table found, or nil if the database is healthy.
func ChildAddresses ¶
func ChildAddresses(pm *serial.ProllyTreeNode) []hash.Hash
ChildAddresses returns the child chunk addresses of an internal tree node.
func DescFingerprint ¶
DescFingerprint returns a cheap fingerprint of the key and value tuple descriptors' encodings.
func GetTreeNodeMessage ¶
func GetTreeNodeMessage(ctx context.Context, cs chunks.ChunkStore, addr hash.Hash) (serial.Message, error)
GetTreeNodeMessage reads the chunk at |addr| and validates that it is a ProllyTreeNode message.
func HasValidSentinel ¶
HasValidSentinel reports whether the database whose .dolt directory is |doltDir| has already passed the current version of the integrity check.
Types ¶
type CacheKey ¶
CacheKey identifies a scanned subtree: the node chunk hash plus a fingerprint of the tuple descriptors it was interpreted with. Identical chunks shared between branches, commits, and structurally identical tables are only ever processed once.
type CorruptionError ¶
type CorruptionError struct {
Database string
Branch string
// Commit is the hash of the corrupt commit, or empty if the corruption was found in the branch's
// uncommitted working set.
Commit string
Table string
Stats *Stats
}
CorruptionError describes the first corrupted table found by CheckDatabase, with instructions for repairing the database.
func (*CorruptionError) Error ¶
func (e *CorruptionError) Error() string
type InternalKeyAnalysis ¶
type InternalKeyAnalysis struct {
// OutOfBandValues is the number of adaptive-encoded values stored out-of-band in the node's keys.
OutOfBandValues uint64
// CorruptValues is the number of expected chunk address references missing from the node's
// key_address_offsets field.
CorruptValues uint64
// UnexpectedOffsets is the number of recorded offsets that match no address in the node's keys.
UnexpectedOffsets uint64
// Corrupt is true when at least one expected offset is missing; such nodes must be rewritten to
// be repaired.
Corrupt bool
}
InternalKeyAnalysis is the result of examining an internal node's boundary key tuples against its recorded key_address_offsets field.
func AnalyzeInternalKeys ¶
func AnalyzeInternalKeys(pm *serial.ProllyTreeNode, kd *val.TupleDesc) (*InternalKeyAnalysis, error)
AnalyzeInternalKeys recomputes the expected key_address_offsets for an internal (non-leaf) node from its boundary key tuples, mirroring the serializer logic in dolt's go/store/prolly/message package, and compares the result against the offsets actually recorded in the message. Boundary keys are copies of leaf keys, so any out-of-band addresses they embed must be recorded just as they are in the leaves.
type LeafAnalysis ¶
type LeafAnalysis struct {
Stats Stats
// Corrupt is true when at least one expected offset is missing from value_address_offsets or
// key_address_offsets; such nodes must be rewritten to be repaired.
Corrupt bool
// ValueOOBAddrs are the chunk addresses referenced from value tuples (out-of-band adaptive values
// and address-encoded fields).
ValueOOBAddrs []hash.Hash
// KeyOOBAddrs are the chunk addresses referenced by out-of-band adaptive values in key tuples.
KeyOOBAddrs []hash.Hash
}
LeafAnalysis is the result of examining a single leaf node's tuples against its recorded value_address_offsets field.
func AnalyzeLeaf ¶
func AnalyzeLeaf(pm *serial.ProllyTreeNode, kd, vd *val.TupleDesc) (*LeafAnalysis, error)
AnalyzeLeaf recomputes the expected value_address_offsets and key_address_offsets for a leaf node from its tuples
type Scanner ¶
type Scanner struct {
Cs chunks.ChunkStore
// CacheHits counts subtree scans satisfied from the cache.
CacheHits uint64
// contains filtered or unexported fields
}
Scanner walks prolly trees at the chunk level and detects nodes whose address offset fields (value_address_offsets and key_address_offsets) omit references to out-of-band values. Results are cached per chunk, so re-scanning the same table on another branch or commit is nearly free.
func NewScanner ¶
func NewScanner(cs chunks.ChunkStore) *Scanner
func (*Scanner) ScanRootNode ¶
func (s *Scanner) ScanRootNode(ctx context.Context, root *tree.Node, kd, vd *val.TupleDesc) (*Stats, error)
ScanRootNode scans a tree from its in-memory root node. A table's row map root is embedded in the durable table message rather than referenced by address, so it need not exist as an addressable chunk at all (in a cloned database, or after garbage collection). Descendant nodes are always addressable and are fetched from the chunk store.
type Stats ¶
type Stats struct {
// Chunks is the total number of tree node chunks in the subtree.
Chunks uint64
// LeafChunks is the number of leaf node chunks in the subtree.
LeafChunks uint64
// CorruptChunks is the number of chunks, leaf or internal, with at least one missing key or value
// address offset.
CorruptChunks uint64
// Rows is the total number of rows in the subtree.
Rows uint64
// CorruptRows is the number of rows with at least one out-of-band value missing from the node's
// recorded key or value address offsets.
CorruptRows uint64
// AdaptiveValues is the number of non-NULL adaptive-encoded values in value tuples.
AdaptiveValues uint64
// OutOfBandValues is the number of adaptive-encoded values stored out-of-band in value tuples.
OutOfBandValues uint64
// CorruptValues is the number of expected chunk address references (out-of-band adaptive values and
// non-empty address-encoded fields) missing from the node's value_address_offsets field.
CorruptValues uint64
// UnexpectedOffsets is the number of recorded value_address_offsets entries that do not correspond to
// any address in the node's value tuples. Always zero for both healthy and known-corrupt databases.
UnexpectedOffsets uint64
// KeyAdaptiveValues is the number of non-NULL adaptive-encoded values in key tuples.
KeyAdaptiveValues uint64
// KeyOutOfBandValues is the number of adaptive-encoded values stored out-of-band in key tuples.
KeyOutOfBandValues uint64
// KeyCorruptValues is the number of expected chunk address references missing from the node's
// key_address_offsets field. Nodes written before that field existed record no key addresses at
// all, so any out-of-band key value they hold counts here.
KeyCorruptValues uint64
// InternalKeyOutOfBandValues is the number of adaptive-encoded values stored out-of-band in the
// key tuples of internal (non-leaf) nodes. Internal boundary keys are copies of leaf keys, so
// these are duplicates of values counted in KeyOutOfBandValues.
InternalKeyOutOfBandValues uint64
// InternalKeyCorruptValues is the number of expected chunk address references missing from the
// key_address_offsets field of internal (non-leaf) nodes.
InternalKeyCorruptValues uint64
// MissingChunks is the number of out-of-band values (key or value side) whose chunks are absent from
// the chunk store. These values are already lost (e.g. to a previous GC) and cannot be repaired by
// rewriting nodes.
MissingChunks uint64
}
Stats records corruption statistics aggregated over a prolly tree node and all its children.
type TableInfo ¶
type TableInfo struct {
Name doltdb.TableName
Tbl *doltdb.Table
Sch schema.Schema
KeyDesc *val.TupleDesc
ValDesc *val.TupleDesc
// AdaptiveValueCols are the names of value (non-PK) columns with an adaptive encoding.
AdaptiveValueCols []string
// AdaptiveKeyCols are the names of key (PK) columns with an adaptive encoding.
AdaptiveKeyCols []string
}
TableInfo describes one table in one root value, along with the tuple descriptors needed to scan it.
func TablesForRoot ¶
func TablesForRoot(sctx *sql.Context, root doltdb.RootValue, ns tree.NodeStore) ([]*TableInfo, error)
TablesForRoot enumerates all tables in all database schemas of the given root value. |sctx| must be a *sql.Context: doltgres extended type deserialization requires one.
func (*TableInfo) KeyColsImpacted ¶
KeyColsImpacted returns whether this table's key tuples contain adaptive-encoded fields. Out-of-band values in key tuples cannot be recorded in the ProllyTreeNode message format at all, so they are scanned and reported, but cannot be repaired by rewriting nodes.
func (*TableInfo) ValColsImpacted ¶
ValColsImpacted returns whether this table's schema can be affected by the value_address_offsets corruption: its value tuples contain one or more adaptive-encoded fields.
Source Files