olefixture

Documentation

Overview

Package olefixture builds OLE Compound File Binary containers for tests.

Legacy Office documents (.doc/.xls/.ppt) are the one format this repository cannot test from a committed fixture without checking in an opaque binary, and the only real ones available live on a developer's disk — a test that depends on such a file is a test that does not run in CI. So the container is synthesized, exactly as the golden corpus synthesizes .docx/.xlsx zips rather than committing them.

Three packages need this: the redactor, the metadata extractor, and the golden corpus. It is a leaf package (no ferret-scan imports) so all three can use one implementation. A test-only helper file in each would mean three copies of the format's sharp edges, and the format has several.

Container layout

The 512-byte header is NOT part of the sector numbering: sector n begins at offset (n+1)*512. Sectors are then:

0   : the single FAT sector
1   : the single directory sector
2   : the single mini FAT sector
3.. : the mini stream chain, then one chain per regular stream

One FAT sector holds 128 entries, addressing 64KB of file — far more than any fixture needs, so there is no DIFAT or multi-FAT handling. Build returns an error rather than a corrupt file if a caller exceeds that.

Why the mini stream is mandatory

Readers route every stream SMALLER THAN 4096 bytes through the mini FAT. A real document's property streams are a few hundred bytes, so a builder without mini stream support cannot produce a fixture with readable properties: the reader fails with "minisector number is outside minisector range" and returns zero bytes. Properties are exactly what the metadata half of legacy Office support reads, so that would make every metadata assertion vacuous — which is not hypothetical: the first version of this builder had that bug, and its self-check only walked the directory without reading a stream, so it passed.

Stream names

Office writes property streams with a leading 0x05 byte ("\x05SummaryInformation"). Readers strip a non-printable initial character when exposing the name, so consumers match "SummaryInformation". Fixtures use the on-disk name WITH the prefix so they agree with real documents.

Index

• Constants
• Variables
• func Build(streams []Stream) ([]byte, error)
• func BuildFragmented(streams []FragmentedStream) ([]byte, error)
• func DocSummaryInformation(props map[uint32]string) []byte
• func DocSummaryInformationWithVectors(props map[uint32]string, vectors map[uint32][]string) []byte
• func FileTime(year int, month time.Month, day int) uint64
• func LegacyDoc(body string, props map[uint32]string) []byte
• func MustBuild(streams []Stream) []byte
• func MustBuildFragmented(streams []FragmentedStream) []byte
• func SummaryInformation(props map[uint32]string) []byte
• func SummaryInformationWide(props map[uint32]string) []byte
• func SummaryInformationWithTimes(props map[uint32]string, times map[uint32]uint64) []byte
• func SummaryInformationWithVectors(props map[uint32]string, vectors map[uint32][]string) []byte
• func UTF16LE(s string) []byte
• func UserDefinedProperties(props map[string]string) []byte
• type FragmentedStream
• type Stream

Constants

const (
	SectorSize = 512
	// MiniCutoff is the size at or above which a stream lives in regular sectors.
	// Below it, the stream is stored in the mini stream and addressed by the mini
	// FAT.
	MiniCutoff = 4096
)

Container structure constants, per MS-CFB.

const (
	PropTitle        = 0x00000002
	PropSubject      = 0x00000003
	PropAuthor       = 0x00000004
	PropKeywords     = 0x00000005
	PropComments     = 0x00000006
	PropTemplate     = 0x00000007
	PropLastAuthor   = 0x00000008
	PropCreateTime   = 0x0000000C
	PropLastSaveTime = 0x0000000D
	PropAppName      = 0x00000012
)

SummaryInformation property IDs (MS-OLEPS 2.18).

const (
	PropCategory      = 0x00000002
	PropManager       = 0x0000000E
	PropCompany       = 0x0000000F
	PropContentStatus = 0x0000001B
	PropLanguage      = 0x0000001C
)

DocumentSummaryInformation property IDs. Company and Manager live HERE, in a second property set with a different FMTID — not in SummaryInformation.

const (
	StreamSummaryInformation    = "\x05SummaryInformation"
	StreamDocSummaryInformation = "\x05DocumentSummaryInformation"
	StreamWordDocument          = "WordDocument"
	StreamWorkbook              = "Workbook"
	StreamPowerPoint            = "PowerPoint Document"
)

Canonical stream names, including the leading 0x05 byte Office writes.

const PropDocumentParts = 0x0000000D

PropDocumentParts is DocumentSummaryInformation 0x0D: the vector of part names — sheet names in a workbook, slide titles in a presentation. msoleps labels it "Document parts".

const PropHyperlinks = 0x00000015

PropHyperlinks is DocumentSummaryInformation 0x15, also vector-valued.

Variables

var Signature = []byte{0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1}

Signature is the 8-byte magic every compound file starts with.

Functions

func Build

func Build(streams []Stream) ([]byte, error)

Build assembles a valid compound file containing the given streams. Streams under MiniCutoff bytes go through the mini stream, as Office does.

It returns an error rather than a malformed container when a fixture would exceed what a single FAT or directory sector can address: a silently corrupt fixture makes every test built on it pass for the wrong reason.

func BuildFragmented

func BuildFragmented(streams []FragmentedStream) ([]byte, error)

BuildFragmented writes a valid compound file whose stream sectors sit where the caller says, so a value can be contiguous in a stream's logical bytes while its halves live far apart on disk.

That layout is what separates a correct redactor from one that merely looks correct: searching the raw file cannot find such a value, so it is reported by the extractor (which reads reassembled logical bytes) and silently left in cleartext by the writer. Only regular sectors are used — the mini stream has its own indirection and is covered by Build.

func DocSummaryInformation

func DocSummaryInformation(props map[uint32]string) []byte

DocSummaryInformation encodes a DocumentSummaryInformation property stream.

func DocSummaryInformationWithVectors

func DocSummaryInformationWithVectors(props map[uint32]string, vectors map[uint32][]string) []byte

DocSummaryInformationWithVectors additionally encodes VECTOR-valued string properties, which is how a real .xls stores its sheet-name list and a real .ppt its slide titles (DocumentParts, property 0x0D).

Worth encoding exactly as [MS-OLEPS] specifies, because the whole reason vector properties were unreadable is a type-word disagreement: the property type is ONE 32-bit value with VT_VECTOR (0x1000) OR'd into the same 16-bit type field, so a vector of VT_LPSTR is 0x0000101E — NOT 0x001E with a separate flag word. A fixture that encoded it the other way would make a broken reader look correct.

func FileTime

func FileTime(year int, month time.Month, day int) uint64

FileTime converts a date to a Windows FILETIME: 100-nanosecond ticks since 1601-01-01 UTC.

func LegacyDoc

func LegacyDoc(body string, props map[uint32]string) []byte

LegacyDoc builds a .doc-shaped container: body text in WordDocument plus a SummaryInformation property stream. This is the shape most tests want, and having one definition of it keeps the corpus, the extractor tests and the redactor tests describing the same document.

func MustBuild

func MustBuild(streams []Stream) []byte

MustBuild is Build for callers that cannot handle an error, such as a package-level fixture in a corpus definition. It panics on a malformed request, which surfaces at test setup rather than as a mysterious empty scan result.

func MustBuildFragmented

func MustBuildFragmented(streams []FragmentedStream) []byte

MustBuildFragmented is BuildFragmented for callers that cannot handle an error.

func SummaryInformation

func SummaryInformation(props map[uint32]string) []byte

SummaryInformation encodes a SummaryInformation property stream carrying the given string properties.

func SummaryInformationWide

func SummaryInformationWide(props map[uint32]string) []byte

SummaryInformationWide encodes the properties as VT_LPWSTR (UTF-16LE) rather than VT_LPSTR, which is what Office writes for any value that is not representable in the document's code page — in practice, most non-English names.

The distinction is load-bearing for a redactor: a UTF-16LE value shares no bytes with its UTF-8 form, so a redactor that searches only the narrow encoding finds nothing and reports success while the value stays in the file.

func SummaryInformationWithTimes

func SummaryInformationWithTimes(props map[uint32]string, times map[uint32]uint64) []byte

SummaryInformationWithTimes additionally encodes FILETIME properties, so a test can exercise the timestamp path.

func SummaryInformationWithVectors

func SummaryInformationWithVectors(props map[uint32]string, vectors map[uint32][]string) []byte

SummaryInformationWithVectors is the same for the SummaryInformation set, where a Keywords list may be stored as a vector rather than one delimited string.

func UTF16LE

func UTF16LE(s string) []byte

UTF16LE encodes a string the way legacy Office stores wide text: UTF-16 little-endian, with surrogate pairs for anything outside the BMP.

Non-ASCII must be encoded, not refused. A test that needs the on-disk form of "José Ramírez" in order to assert it was redacted cannot get it from a function that gives up on the first accented character — the assertion would silently become vacuous, which is how a real leak in exactly this area went unnoticed.

func UserDefinedProperties

func UserDefinedProperties(props map[string]string) []byte

UserDefinedProperties encodes a user-defined (custom) property stream: the legacy counterpart of docProps/custom.xml.

Unlike the two well-known sets, custom property NAMES are not in any reader's built-in table — they live in the set's own dictionary at property ID 0, which a reader consults to label IDs 2 and up. A stream without that dictionary yields unnamed properties, so the dictionary is what makes these visible at all.

This matters because custom properties are a documented leak channel: a property named "ClientSSN" holding a real SSN is exactly the shape that reaches a scanner only if the dictionary is parsed.

Types

type FragmentedStream

type FragmentedStream struct {
	Name string
	Data []byte
	// Sectors are disk sector numbers in LOGICAL order, one per sector of Data.
	// Listing them out of ascending order is the point: it reproduces what a real
	// allocator leaves behind after a document has been edited.
	Sectors []uint32
}

FragmentedStream is a stream whose sectors are placed at caller-chosen disk locations, in logical order.

type Stream

type Stream struct {
	// Name is the name as Office writes it, including any leading 0x05 byte for a
	// property stream.
	Name string
	Data []byte
}

Stream is one named stream to place in a container.