extract

Documentation

Overview

Package extract is yarad's format-aware front-end. It pulls plaintext that is hidden inside structured container formats — primarily the run-length (MS-OVBA) compressed VBA macro source inside OLE2 and OOXML Office documents — so the YARA scanner's keyword rules can actually match it.

Why it exists: ScanMem over a raw .docm sees the zip plus the MS-OVBA compressed macro stream, so VBA-keyword rules never fire. A scanner that only ever sees raw bytes is the weaker design; every real maldoc engine (ClamAV's unpackers, YARA's own pe/dotnet/macho modules) preprocesses structured formats before matching. This package is that preprocessing step, kept in its own package so the scanner core stays format-blind: it calls Extract and scans whatever extra blobs come back, knowing nothing about documents.

Contract: extraction is best-effort enrichment, not a gate. Extract never returns an error and never panics out — for a non-container input, or on any parse failure (truncated, obfuscated, hostile), it reports no streams and the caller scans the raw bytes regardless (fail-open, matching yarad's gozer contract). The Result flags exist only for observability/metrics.

Index

• Constants
• type Options
  • func FullOptions(deadline time.Time) *Options
• type Result
  • func Extract(buf []byte, deadline time.Time) Result
  • func ExtractWithOptions(buf []byte, opts *Options) (res Result)

Constants

const Version = "" /* 892-byte string literal not displayed */

Version identifies the extraction logic. It is folded into the scanner's verdict-cache fingerprint, so a bump here (new extractor behaviour, an oleparse upgrade that changes output) invalidates cached verdicts the same way a rule-set change does — important for the shared Redis L2 that survives an image rebuild. Bump it whenever the bytes Extract emits could change.

Types

type Options

type Options struct {
	Deadline time.Time // zero == no wall-clock limit (mirrors the old deadline arg)

	// DecodeDepth caps the MSD multi-layer static-decode recursion (decodeSourceTree).
	// DecodeIterations caps worklist dequeues per source stream. Both are floored
	// to 1 at the read-site. Effort scales these: a low level unwraps fewer layers.
	DecodeDepth      int
	DecodeIterations int

	// PDFDeepen enables the pdfid-style structural-indicator pass over a PDF
	// (fromPDFIndicators). Disabled at low effort: only the inflated object streams
	// are scanned, not the action/JS/launch name markers.
	PDFDeepen bool

	// XLMFoldSheets caps the number of macrosheets scanned per document by the
	// XLM constant-fold pass. 0 means use the package default (maxXLMFoldSheets).
	XLMFoldSheets int
	// XLMFoldFormulas caps the number of formulas processed per macrosheet by the
	// XLM constant-fold pass. 0 means use the package default (maxXLMFoldFormulas).
	XLMFoldFormulas int

	// ArchivePWEnabled turns on bounded decrypt attempts against password-protected
	// archive members (zip/7z/rar). Default false: an encrypted member emits the
	// ARCHIVE-ENCRYPTED marker and is never decrypted (historical behaviour). When
	// true, the unpacker tries PWCandidates against each encrypted member under a
	// hard attempt budget + per-attempt deadline; on a hit the plaintext is fed to
	// the normal child-scan path. Gated by MAILSTRIX_ARCHIVE_PW at the server.
	ArchivePWEnabled bool
	// PWCandidates is the effective, already-capped+deduped candidate-password list
	// for this request (built-in defaults ∪ wordlist file ∪ filename tokens ∪
	// mail-body candidates). Consulted only when ArchivePWEnabled is true. Nil/empty
	// means "no candidates" → encrypted members stay ARCHIVE-ENCRYPTED.
	PWCandidates []string
}

Options carries the per-request extraction caps (EFFORT-4) plus the time budget. It is resolved once per scan from the effort level and threaded to the two cap-sink chains that honour a per-request bound: the MSD multi-layer decode (DecodeDepth/DecodeIterations) and the PDF structural-indicator pass (PDFDeepen). Every other extractor still consults Deadline directly; only these two read effort-scaled caps today. A higher effort level widens the caps; the scanner builds Options from EffortProfileFor (see yarad.EffortProfile).

Deadline mirrors the old bare deadline argument (zero == no time limit). The cap fields are floored to sane minimums by their read-sites so a zero-value Options (e.g. a test passing &Options{}) degrades to the shallowest safe behaviour rather than an unbounded or zero-depth walk.

func FullOptions

func FullOptions(deadline time.Time) *Options

FullOptions returns an Options at maximum depth for the given deadline — the historical always-on behaviour. Used by tests and any caller that wants no effort scaling. Values mirror the extract package's own ceilings.

type Result

type Result struct {
	// Streams holds the decompressed VBA macro source, one cleartext blob per
	// module. The caller scans each in addition to the raw bytes.
	Streams [][]byte
	// Markers holds yarad's synthetic PURE marker entries (no attacker-controlled
	// data) split out of Streams at the end of extraction — the out-of-band
	// "marker channel" (PLAN-marker-channel Phase 1). The scanner scans these
	// against the full ruleset exactly like Streams, so Phase 1 changes nothing
	// observable beyond the separation; the split is the prerequisite for the
	// Phase 2 collision filter and Phase 3 compiled markers.yac partition.
	// COMBINED markers (marker tag + a real attacker IOC in one string) stay in
	// Streams. Populated by splitPureMarkers (see markers.go).
	Markers [][]byte
	// IsDoc is true when buf was a recognised OLE2/OOXML container (magic hit),
	// whether or not any macro was found.
	IsDoc bool
	// Encrypted is true for an ECMA-376 encrypted OOXML (an OLE2 wrapper holding
	// EncryptionInfo/EncryptedPackage). The real document is AES-wrapped, so no
	// macros are extractable here — we flag it but do not decrypt (that needs a
	// full ECMA-376 implementation; see the package notes / TODO).
	Encrypted bool
	// Failed is true when extraction was attempted on a container (IsDoc) but the
	// parse errored. Distinct from "not a document" (IsDoc=false).
	Failed bool
	// Panicked is true when oleparse panicked on hostile input and was recovered.
	// Worth a separate counter: a spike points at a parser bug or a new evasion.
	Panicked bool
	// IsMSI is true when buf was recognised as a Windows Installer database (an
	// OLE2 with the MSI root CLSID) and its streams were dumped for scanning.
	IsMSI bool
	// IsMSG is true when buf was recognised as an Outlook .msg (an OLE2 MAPI
	// message store) and its nested attachment data streams were pulled out for
	// scanning.
	IsMSG bool
	// IsPDF is true when buf was a PDF whose FlateDecode object streams were
	// inflated and surfaced for scanning.
	IsPDF bool
	// IsLNK is true when buf was a Windows shell link (.lnk) whose StringData
	// fields (command-line arguments, paths) were surfaced for scanning.
	IsLNK bool
	// IsOLEPackage is true when an OLE2 document carried an embedded OLE Package
	// object (Ole10Native stream) whose native file data was carved out.
	IsOLEPackage bool
	// IsArchive is true when buf (or a nested member) was a recognised archive
	// (zip/gz/7z/rar/tar) whose members were unpacked and surfaced for scanning.
	IsArchive bool
	// EncryptedArchive is true when a password-protected member was seen in the
	// archive (any layer). yarad holds no password so it cannot unpack the member;
	// the ARCHIVE-ENCRYPTED marker is emitted instead — a hidden-payload mail tell.
	EncryptedArchive bool
	// DecryptedArchive is true when a password-protected member was successfully
	// decrypted using a candidate password (ArchivePWEnabled path). The decrypted
	// plaintext is surfaced as a normal member stream and the ARCHIVE-DECRYPTED
	// marker is emitted. Independent of EncryptedArchive: an input may carry both
	// (one member decrypted, another not), and a decrypted member is no longer the
	// hidden-payload tell that ARCHIVE-ENCRYPTED is.
	DecryptedArchive bool
	// Polyglot is true when buf is simultaneously a valid PE image and a valid ZIP
	// (file-type confusion): the email gateway parses the ZIP while the endpoint
	// runs the PE. The POLYGLOT-PE-ZIP marker is emitted; extraction is not
	// re-routed.
	Polyglot bool
	// IsXLL is true when buf is a PE that exports the Excel XLL add-in callback
	// contract (xlAutoOpen): an Excel add-in DLL, which runs code on load without
	// a macro prompt. The XLL-ADDIN marker is emitted.
	IsXLL bool
	// IsRTF is true when buf was recognised as an RTF document whose \objdata
	// embedded-object groups were hex-decoded and carved for scanning.
	IsRTF bool
	// IsOneNote is true when buf was recognised as a OneNote section/TOC
	// (.one/.onetoc2) and its embedded FileDataStoreObject payloads were carved
	// out for scanning.
	IsOneNote bool
	// IsTNEF is true when buf was recognised as a TNEF (winmail.dat) blob and its
	// attachment/body parts were unwrapped for scanning.
	IsTNEF bool
	// EncodedScript is true when >=1 MS Script Encoder block (#@~^...^#~@,
	// i.e. an encoded VBScript/JScript, as in .vbe/.jse or embedded in a
	// .wsf/.hta/.html/.sct) was found and decoded to cleartext for scanning.
	EncodedScript bool
	// HasDocProps is true when at least one string was extracted from document
	// properties (OOXML docProps/*, customXml/, word/settings.xml docVars, or
	// OLE2 \x05SummaryInformation / \x05DocumentSummaryInformation streams) and
	// emitted for YARA scanning. Drives the extract_docprops_total metric.
	HasDocProps bool
	// HasXLMFold is true when at least one XLM formula was constant-folded
	// and the folded cleartext was emitted for YARA scanning.
	HasXLMFold bool
	// IsSLK is true when buf was recognised as a SYLK (.slk) spreadsheet, whose
	// C-record E-field formulas were scanned for XLM/DDE droppers.
	IsSLK bool
	// DecodedStreams is how many blobs the single-layer static decode pass
	// (base64/hex/whole-buffer reverse; see decode.go) appended to Streams. These
	// are the trailing len-N entries of Streams; the caller subtracts them so the
	// macro/extracted-stream metrics aren't inflated by decode output. >0 means
	// the pass fired.
	DecodedStreams int

	// VBAStreams holds the decompressed VBA macro-source streams (the codes()
	// output) — a SUBSET of Streams, by content identity. The scanner sets the VBA
	// external variable ONLY when scanning a stream in this set, so VBA-gated rules
	// (`VBA and any of(...)`) fire on real macro source and not on every other
	// extracted stream (PDF/archive/script/marker/decoded), which previously caused
	// false positives. Entries share their backing array with the matching Streams
	// entry, so a content-hash membership test identifies them.
	VBAStreams [][]byte
	// contains filtered or unexported fields
}

Result reports what Extract found in one buffer. Streams is the only field the scanner needs to match rules; the booleans are for /metrics so the new code path is observable (how often docs arrive, yield macros, fail, or are encrypted) rather than invisible.

func Extract

func Extract(buf []byte, deadline time.Time) Result

Extract is the back-compat full-depth entry point: it runs every extractor at maximum depth, bounded only by deadline (zero == no limit). Used by tests and the -extract CLI tool. The server scan path calls ExtractWithOptions to apply per-request effort caps (EFFORT-4).

func ExtractWithOptions

func ExtractWithOptions(buf []byte, opts *Options) (res Result)

ExtractWithOptions reports the plaintext hidden inside an OLE2/OOXML container — the decompressed VBA macro source — plus flags describing what the buffer was. For anything that is not a recognised container it returns the zero Result (IsDoc=false, no streams). It never returns an error and never panics out: a poison attachment degrades to a raw-only scan, never crashes the scan path.

opts carries the time budget (opts.Deadline; zero == no limit) plus the per-request effort caps (EFFORT-4). The deadline bounds the OOXML extraction loop (decompression + oleparse runs are done before any libyara scan, so a small compressed bomb could otherwise burn CPU before the scan budget is ever consulted); the effort caps scale the MSD decode depth and the PDF indicator pass. A nil opts degrades to FullOptions (no time limit, full depth). The cumulative byte/count caps still apply regardless.