Documentation
¶
Index ¶
- Constants
- Variables
- func DefaultPruneDirs() func(path string, name string) bool
- func EncodeCert(cert, signingCert *x509.Certificate, pubKey *ecdsa.PublicKey, ...) ([]byte, error)
- func EncodeKey(key *ecdsa.PrivateKey) ([]byte, error)
- func GenerateLeafCertificate(rootCert *x509.Certificate, rootKey *ecdsa.PrivateKey) (*x509.Certificate, *ecdsa.PrivateKey, error)
- func GenerateRootCert() (cert *x509.Certificate, key *ecdsa.PrivateKey, err error)
- func Log(format string, v ...interface{})
- func PruneDirNames() []string
- func WalkFiles(ctx context.Context, paths []string, opts WalkOptions) (<-chan RepositoryIndexedFile, <-chan WalkStats)
- func WalkRoots(ctx context.Context, roots <-chan IndexedRoot, opts WalkOptions) (<-chan RepositoryIndexedFile, <-chan WalkStats)
- type IndexedRoot
- type LineIndex
- func (li *LineIndex) AppendEOLs(eols []int)
- func (li *LineIndex) GetPositionFromCharacterIndex(pos int64) code.Position
- func (li *LineIndex) IndexBytes(baseOffset int64, data []byte)
- func (li *LineIndex) IndexString(baseOffset int64, data string)
- func (li *LineIndex) Len() int
- func (li *LineIndex) Reset()
- type PathConsumer
- type PathMultiplexer
- type PositionProvider
- type RepositoryIndexedFile
- type ResourceConsumer
- type ResourceMultiplexer
- type UUID
- type WalkOptions
- type WalkStats
Constants ¶
const MaxIndexableFileSize = int64(1<<31 - 1)
MaxIndexableFileSize is the largest file a LineIndex can describe, imposed by the int32 offsets. Two gigabytes is far above the engine's scanning thresholds; offsets beyond it are clamped rather than silently wrapping into negative positions.
Variables ¶
var ( // MaxInMemoryFileSize is the largest source read whole into memory. // // It is set to the engine's own scanning cut-off: files above it are // already skipped unless they are of a recognised parsable type, so in // practice the streaming path is reserved for large recognised files. MaxInMemoryFileSize = int64(1024 * 1000 * 10) // 10Mb )
Functions ¶
func DefaultPruneDirs ¶ added in v1.4.0
DefaultPruneDirs returns the opt-in pruning predicate over defaultPruneDirNames, honouring CHECKMATE_PRUNE_DIRS, or nil when pruning has been disabled by setting that variable empty.
Read defaultPruneDirNames before using this: it removes files from the scan, and some of what it removes contains real secrets.
Returning nil rather than a predicate that always says false keeps the disabled case free: WalkFiles skips the call entirely.
func EncodeCert ¶
func EncodeCert(cert, signingCert *x509.Certificate, pubKey *ecdsa.PublicKey, signingKey *ecdsa.PrivateKey) ([]byte, error)
DER encode certificate
func GenerateLeafCertificate ¶
func GenerateLeafCertificate(rootCert *x509.Certificate, rootKey *ecdsa.PrivateKey) (*x509.Certificate, *ecdsa.PrivateKey, error)
func GenerateRootCert ¶
func GenerateRootCert() (cert *x509.Certificate, key *ecdsa.PrivateKey, err error)
func Log ¶
func Log(format string, v ...interface{})
Log and flush stdout. To be used by plugins, so we can stream plugin output in CheckMate service
func PruneDirNames ¶ added in v1.4.0
func PruneDirNames() []string
PruneDirNames returns the effective prune set, honouring CHECKMATE_PRUNE_DIRS. Exposed for diagnostics and tests.
func WalkFiles ¶ added in v1.4.0
func WalkFiles(ctx context.Context, paths []string, opts WalkOptions) (<-chan RepositoryIndexedFile, <-chan WalkStats)
WalkFiles streams the files under each of paths, tagging every file with the index of the root it was found under.
It replaces the removed FindFiles' materialise-everything-then-return model. On a large estate that slice was hundreds of megabytes that had to be complete before the first byte was scanned; here the first file is available immediately and memory is bounded by the channel.
The returned channels are both closed when the walk finishes. Callers must drain the files channel — abandoning it leaks the walker goroutines until ctx is cancelled. The stats channel is single-slot and latest-wins, so it is safe to ignore, and safe to read only at the end: the final update is retained in the buffer after close.
Per-root file ordering is lexical and depth first, matching filepath.WalkDir. Order *between* roots is not defined, because roots are walked concurrently.
func WalkRoots ¶ added in v1.4.0
func WalkRoots(ctx context.Context, roots <-chan IndexedRoot, opts WalkOptions) (<-chan RepositoryIndexedFile, <-chan WalkStats)
WalkRoots is WalkFiles over a stream of roots, for callers that can begin scanning one root before the next is available.
The walk finishes when roots is closed and every root has been walked, so a caller that never closes the channel will hang until ctx is cancelled. Concurrency defaults to GOMAXPROCS; unlike WalkFiles it cannot be clamped to the number of roots, since that number is not known in advance.
Types ¶
type IndexedRoot ¶ added in v1.4.0
IndexedRoot is one scan root together with the RepositoryIndex every file discovered beneath it will carry.
The index is supplied by the caller rather than derived from arrival order because the callers that need a streaming walk are precisely the ones whose roots arrive out of order: repositories are pipelined into the scan as each clone completes, and a fast local checkout must not take the index of a slow remote one. Findings are mapped back to their repository through this index, so an index that depended on clone timing would attribute findings to the wrong repository from one run to the next.
type LineIndex ¶ added in v1.4.0
type LineIndex struct {
// contains filtered or unexported fields
}
LineIndex maps a character offset in a file onto a code.Position.
It replaces LineKeeper, which had three costs that matter at scale:
- GetPositionFromCharacterIndex scanned the end-of-line slice linearly for every lookup. Every finding costs a scan proportional to the file's line count, so a file with many findings is quadratic in its own size.
- Every lookup and every append took a mutex, even though a file is indexed by exactly one goroutine and then queried by that same goroutine. The lock only existed because the old multiplexer fanned chunks out across goroutines; that fan-out is gone.
- Offsets were stored as int, i.e. 8 bytes each on 64-bit. Line offsets are bounded by MaxIndexableFileSize, so int32 halves the index's memory for no loss.
A LineIndex is reusable: Reset returns it to the empty state while keeping its backing array, so a per-worker index costs one allocation for the lifetime of the scan rather than one per file.
A LineIndex is not safe for concurrent use. Parallelism belongs at the file level, with one index per worker.
func NewLineIndex ¶ added in v1.4.0
NewLineIndex returns an index with room for capacity lines.
func (*LineIndex) AppendEOLs ¶ added in v1.4.0
AppendEOLs records end-of-line offsets expressed relative to the start of the current chunk.
This reproduces LineKeeper.appendEOLs, including its assumption that chunks are split on end-of-line boundaries so the next chunk starts one byte past the previous chunk's final newline. It exists so callers that already have match locations can be ported without changing behaviour; new code should prefer IndexBytes, which carries an explicit base offset and therefore does not depend on how chunks were cut.
func (*LineIndex) GetPositionFromCharacterIndex ¶ added in v1.4.0
GetPositionFromCharacterIndex returns the zero-based line and character position of the given absolute character offset.
Behaviour is identical to LineKeeper.GetPositionFromCharacterIndex, but by binary search rather than a linear scan. TestLineIndexDifferential asserts the two agree over random inputs and offsets.
func (*LineIndex) IndexBytes ¶ added in v1.4.0
IndexBytes appends the offsets of every '\n' in data, where data begins at absolute offset baseOffset in the file.
This is the preferred way to build the index: it is a single pass over the bytes with no intermediate slice of match locations, and it does not care where chunk boundaries fall.
func (*LineIndex) IndexString ¶ added in v1.4.0
IndexString is IndexBytes for a string, avoiding the []byte conversion.
type PathConsumer ¶
type PathConsumer interface {
ConsumePath(path RepositoryIndexedFile)
diagnostics.ExclusionProvider
}
PathConsumer is a sink for paths and URIs
type PathMultiplexer ¶
type PathMultiplexer interface {
SetPathConsumers(consumers ...PathConsumer)
ConsumePath(path RepositoryIndexedFile)
}
PathMultiplexer interface defines an aggregator of analysers that can consume filesystem paths and URIs and process them
func NewPathMultiplexer ¶
func NewPathMultiplexer(consumers ...PathConsumer) PathMultiplexer
NewPathMultiplexer creates a choreographer that orchestrates the consumption of paths by consumers
type PositionProvider ¶
PositionProvider provides a "global" view of code location, given an arbitrary character index.
type RepositoryIndexedFile ¶
type RepositoryIndexedFile struct {
RepositoryIndex int //repository index of the under which the file is found
File string
}
Provide repository index context for every file that is scanned. The index of the scan root is mapped to each file found beneath it during the walk. See WalkFiles in walk.go.
func CollectFiles ¶ added in v1.4.0
func CollectFiles(ctx context.Context, paths []string, opts WalkOptions) []RepositoryIndexedFile
CollectFiles drains WalkFiles into a slice, ordered by root and then lexically within each root.
This is the shape the removed FindFiles had, and it exists only for the callers that genuinely need the whole list — the SDK returns one. Prefer WalkFiles: materialising the list reintroduces exactly the memory cost the streaming walker was built to remove.
type ResourceConsumer ¶
type ResourceConsumer interface {
//Consume allows a source processor receive `source` data streamed in "chunks", with `startIndex` indicating the
//character location of the first character in the stream
Consume(startIndex int64, source string)
//ConsumePath allows resource consumers that process filepaths directly to analyse files on disk
ConsumePath(filePath RepositoryIndexedFile)
SetLineIndex(*LineIndex)
//SetRepositoryFile supplies the file about to be consumed, along with the
//index of the scan root (repository) it was found under.
//
//This is per-file state and is therefore set through the multiplexer for
//every file, exactly like SetLineKeeper. It used to be captured when the
//consumer was constructed, which forced a fresh consumer to be built for
//every file and — where consumers were cached and shared — caused findings
//to report the repository index of whichever file was scanned first.
//Supplying it per file lets a consumer be built once and reused.
SetRepositoryFile(RepositoryIndexedFile)
//ShouldProvideSourceInDiagnostics toggles whether source evidence should be provided with diagnostics, defaults to false
ShouldProvideSourceInDiagnostics(bool)
//used to signal to the consumer that the source stream has ended
End()
}
ResourceConsumer is a sink for streaming source
type ResourceMultiplexer ¶
type ResourceMultiplexer interface {
//SetSource is the source reader to multiplex to multiple consumers, which will be provided with a copy of the source data as it is being streamed in from the source
SetResourceAndConsumers(filePath RepositoryIndexedFile, source *io.Reader, provideSourceInDiagnostics bool, consumers ...ResourceConsumer)
}
ResourceMultiplexer interface defines a path or source reader that can be multiplexed to multiple consumers. It provides additional utility such as mapping a source index to the line and character, i.e. the `code.Position` in the source
func NewResourceMultiplexer ¶
func NewResourceMultiplexer(filePath RepositoryIndexedFile, source *io.Reader, provideSource bool, consumers ...ResourceConsumer) ResourceMultiplexer
NewResourceMultiplexer creates a source multiplexer over an input reader
type WalkOptions ¶ added in v1.4.0
type WalkOptions struct {
//PruneDirs is consulted BEFORE descending into a directory. Returning true
//skips the entire subtree, so the cost of an excluded tree is one call and
//not one call per contained file.
//
//It receives the full path and the base name; name alone covers the common
//"never enter node_modules" case, while path allows root-relative rules.
//
//Pruning removes files from the scan outright. It is not a performance-only
//knob: a pruned subtree is one that will not be searched for secrets.
PruneDirs func(path string, name string) bool
//FollowLinks makes the walker descend into symlinked directories. When
//false (the default, and the historic behaviour) a symlink to a directory
//is emitted as an ordinary file entry, exactly as filepath.WalkDir does.
FollowLinks bool
//Concurrency bounds how many roots are walked at once. Defaults to
//GOMAXPROCS. Per-root parallelism matters for multi-repository scans: a
//slow network mount should not hold up a local checkout.
Concurrency int
}
WalkOptions configures WalkFiles.
The zero value discovers exactly the set of files the engine has always scanned: no pruning, no symlink following. Pruning is opt-in because it trades coverage for speed — see defaultPruneDirNames.
type WalkStats ¶ added in v1.4.0
type WalkStats struct {
//DiscoveredSoFar is the number of files emitted so far across all roots.
DiscoveredSoFar int64
//WalkComplete is true only on the final update, at which point
//DiscoveredSoFar is the exact total.
WalkComplete bool
}
WalkStats reports walk progress.
The walk streams, so the total file count is not known until it finishes. DiscoveredSoFar is therefore a running count that becomes exact exactly when WalkComplete is true. Callers driving a progress bar should treat it as a moving denominator rather than a fixed one.
Source Files