Documentation
¶
Overview ¶
Package content opens a matched file once and shares it with every collector that asked for its bytes.
Three things drive the design.
First, collectors must never receive a raw descriptor: a file offset is shared state, so the second collector to call Read would see an empty file, and a third-party collector that closes the descriptor causes the number to be recycled -- after which a retained reference silently reads a *different* file into the evidence output. What collectors get instead is a revocable, read-only view with per-caller offsets.
Second, measurement: for a typical forensic artifact, reading the whole file into a reusable buffer costs the same as a single streaming pass, but also gives every collector independent random access. So small files are buffered and large ones stay on the descriptor.
Third, the content phase is where a real acquisition spends most of its time -- measured at 44% of a warm scan, and the overwhelming majority of a cold one, because a single thread waits for each open/read/write to complete before starting the next. So the bulk work runs on a bounded pool of workers. The walker itself stays single-threaded, which is what preserves the one-entry stat cache and deterministic rule evaluation.
Parallelism must not change what a collection contains or what order it is written in: two runs of the same image have to be diffable, and an examiner comparing against a colleague's run should not see reordered evidence. So each consumer splits in two. The bulk -- reading, hashing, copying bytes -- happens in the worker. The tail that appends a line to an output file is handed to Emit, and a single sequencer goroutine runs those tails strictly in walk order. Output is therefore byte-identical to a serial run, which TestParallelOutputMatchesSerial asserts directly.
Index ¶
Constants ¶
const DefaultBufferLimit = 4 << 20
DefaultBufferLimit is the size below which a file is read into memory. Buffers are pooled, so the peak is roughly this times the worker count.
Variables ¶
var ErrExpired = errors.New("content: view used after the file was released")
ErrExpired is returned when a collector uses a view after the worker that owned it has finished. Failing loudly beats reading whatever file inherited the descriptor, or whatever the buffer pool handed out next.
Functions ¶
Types ¶
type Broker ¶
type Broker struct {
// BufferLimit is the size below which a file is read whole.
BufferLimit int64
// Workers is the number of files whose content is processed concurrently.
// Zero or one keeps everything on the calling goroutine, which is what the
// unit tests use and what makes a failure easy to read in a stack trace.
Workers int
// OnError receives per-file failures. A collector that cannot read a file
// must not abort the walk -- unreadable files are routine on real images --
// so these are recorded, not returned. It is called only from the sequencer
// goroutine, so implementations need no locking and see files in walk
// order.
OnError func(path, consumer string, err error)
// contains filtered or unexported fields
}
Broker collects requests for one file and then serves them from a single open. Requests are registered while rules are being evaluated; Run performs the open, and nothing is opened at all if nobody asked.
func (*Broker) Reset ¶
func (b *Broker) Reset()
Reset drops any pending requests without serving them.
func (*Broker) Run ¶
Run serves the registered consumers for one file.
With a single worker everything happens inline. With more, the file is queued and Run returns as soon as the queue has room, so the walker carries on stat'ing while workers read. The error it returns is therefore not necessarily about *this* file: it is the first fatal output failure seen so far, which is what stops the walk.
The open is deliberately the last thing to happen, and happens in the worker: it only occurs after a rule matched, so device nodes found in a mounted image are never opened. On an image mounted without nodev they would resolve to the examiner's own hardware.
func (*Broker) Wait ¶
Wait blocks until all queued content work has completed and its output has been written. It must be called before the results are read.
It is safe to call more than once, and to enqueue further work afterwards: the two-phase artifacts collect their files after the walk proper has finished.
type Consumer ¶
Consumer is a collector's request for file bytes, registered during the metadata phase and run once the file is open.
type Content ¶
type Content interface {
// Path is the image-relative path, for error messages and records.
Path() string
Size() int64
// ReadAt has no shared offset: two collectors can read concurrently and
// independently.
ReadAt(p []byte, off int64) (int, error)
// Reader returns a fresh independent stream over the whole file.
Reader() io.Reader
// Bytes returns the buffered contents when the file was small enough to
// hold in memory, avoiding a copy for the common case.
//
// The slice is only valid until the consumer returns: it belongs to a pool
// and will be handed to another file afterwards. Retaining it is the one
// mistake the revocation flag cannot catch, because the slice header has
// already been copied out.
Bytes() ([]byte, bool)
// Emit defers a small piece of work -- in practice appending one line to an
// output file -- until every earlier file's deferred work has run.
//
// This is what keeps parallel output byte-identical to serial output. Do
// the bulk in the consumer, where it runs concurrently with other files,
// and register only the ordered tail here. The closure runs on a single
// sequencer goroutine, so it also needs no locking of its own.
Emit(fn func() error)
}
Content is the read-only, offset-free handle collectors receive.
type FatalError ¶
type FatalError struct{ Err error }
FatalError marks a failure that must stop the scan.
The two kinds of failure are not alike. Failing to *read* a source file is routine on a real image -- a bad sector, a permission denial -- and must not abort an acquisition. Failing to *write* output is not routine: the disk is full, the destination is unwritable, the spool cannot be appended to. Left as a recorded error, that produces a partial acquisition that exits zero and looks complete, which is the worst outcome this tool can have.
func (*FatalError) Error ¶
func (e *FatalError) Error() string
func (*FatalError) Unwrap ¶
func (e *FatalError) Unwrap() error
Source Files