warc

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 1, 2026 License: BSD-3-Clause Imports: 12 Imported by: 0

README

go-filesystems/warc

warc

A pure-Go reader and writer for WARC (ISO 28500), the format a web crawl is archived in — plus a read-only filesystem.Filesystem view with one entry per record.

No cgo. One dependency: github.com/go-filesystems/interface.

import "github.com/go-filesystems/warc"

// Stream records.
r, err := warc.NewReader(f)
for {
    rec, err := r.Next()
    if errors.Is(err, io.EOF) {
        break
    }
    if err != nil {
        return err
    }
    fmt.Println(rec.Type(), rec.TargetURI(), len(rec.Block))
}

// Write records.
w := warc.NewWriter(out)
err = w.WriteRecord(&warc.Record{
    Header: warc.Header{
        {warc.HeaderType, warc.TypeResource},
        {warc.HeaderRecordID, "<urn:uuid:5a3b...>"},
        {warc.HeaderDate, "2026-09-26T06:00:00Z"},
        {warc.HeaderTargetURI, "http://example.org/a.txt"},
    },
    Block: []byte("hello"),
}) // Content-Length is computed and appended

// See the file as a filesystem: one file per record, contents = the block.
fsys, err := warc.OpenReader(ra, size)
ents, err := fsys.ListDir("/")
if o, ok := fsys.(filesystem.Opener); ok {
    file, err := o.OpenFile("00001-response-one.html") // random access, no full read
}

The format, and the two traps

WARC/1.1 CRLF
Header-Name: value CRLF      (repeated)
CRLF
<exactly Content-Length octets>
CRLF CRLF

The two trailing CRLFs are part of the record, not a separator to trim. A reader that skips whitespace until the next WARC/ accepts a truncated record as a whole one, and mis-frames the file whenever a block ends in a newline — which for a response record carrying an HTTP message is the ordinary case. This package reads four octets and compares them.

The block is opaque. It may contain CRLF, NUL, a lone CR, anything. Never scan for a delimiter; read exactly Content-Length octets. This package also does not allocate Content-Length up front — the length comes from the file, so a seventy-octet header claiming an eight-exabyte block would otherwise be an out-of-memory panic instead of an error.

Plain .warc only

A .warc.gz is not a gzipped .warc: it is one gzip member per record, concatenated, so an index can seek to a record and inflate it alone. This package handles plain .warc; wrap the stream yourself for a single-member file. In this family that is not a gap in practice — github.com/go-filesystems/unarchive peels compression before dispatching on format, so a .warc.gz arrives here already inflated. It is a gap for a caller wanting per-record random access into a .warc.gz, which needs the gzip member offsets this package does not compute.

Entry names in the filesystem view

<5-digit ordinal>-<WARC-Type>[-<sanitised tail of WARC-Target-URI>]

00000-warcinfo
00001-response-one.html
00002-request-one.html
00003-resource-blob.bin

WARC-Record-ID is the obvious unique key and the wrong filename: its value is a URI in angle brackets (<urn:uuid:5a3b…>) whose brackets and colons are illegal on Windows and whose colon is the HFS path separator, so it must be mangled — and what survives is a UUID nobody can predict from the archive in front of them. The target URI alone is worse, because it is not unique: a crawl of one page writes a request and a response under the same WARC-Target-URI, and a revisit record later under the same one again, so three entries would silently become one.

The ordinal is the record's position in the file: unique by construction, sorts into crawl order under a plain lexical sort, and countable by a person. It is not stable across files — a re-fetched crawl renumbers, and concatenating two WARCs renumbers the second. A caller needing identity across files wants WARC-Record-ID, which is what it is for, and (*FS).Record(name).RecordID() hands it over.

Mutating methods return ErrReadOnly: Content-Length fixes each record's extent, so changing one octet of a block either moves every later record or is not the record that was written.

Errors
errors.Is(err, warc.ErrSyntax) this is not a WARC, or not a valid one
errors.Is(err, io.ErrUnexpectedEOF) the file is truncated
neither an I/O failure, returned unwrapped
errors.Is(err, fs.ErrNotExist) the filesystem view's missing-path contract
errors.Is(err, warc.ErrReadOnly) every mutating method
errors.Is(err, warc.ErrTooLarge) a header block over MaxHeaderBytes, or a block over the ceiling

Untrusted input

A WARC file is read as hostile. What a file can make the reader hold, and what bounds it:

What the file controls Bound
one record's version line and named fields MaxHeaderBytes = 1 MiB together (net/http's DefaultMaxHeaderBytes; ISO 28500 and jwarc state none), read in bounded pieces
a folded field value linear: the folds are appended to one buffer
a block held by Next, ReadFile, Record Reader.MaxBlockSize, default DefaultMaxBlockSize = 1 GiB, refused with ErrTooLarge before a byte is read
a block seen by Open not read at all: stepped over with a seek, after checking by subtraction that it lies inside the file; OpenFile has no ceiling
the entries Open keeps one header per record: a small multiple of the file
error messages file text is cut to 200 bytes

Measured before the fix (TotalAlloc around the call):

Input Allocated
50 000 one-byte header folds (200 KB) 2 655 393 224 bytes (13 000x)
one 256 MiB header line with no newline 2 424 052 328 bytes
Open of a file holding one 64 MiB block 268 442 712 bytes (the comment said blocks are not read)

and after: 833 KB, 5.2 MB and 7.6 KB. FuzzReader and FuzzOpen (the second is differential: Open against Next, record for record) found nothing in the first 90 seconds each.

An unregistered WARC-Type is accepted, not refused: §5.5 tells a reader to pass an unknown record type through rather than treat the file as broken.

⚠ Where the fixtures came from, and what that costs in confidence

Neither wget nor warcio was installed on the machine this was written on, so there was no crawl to point at. The corpus is two files, and they are not equal in standing:

  • testdata/handbuilt.warc is our own construction. Its octets were laid out from §5's grammar by testdata/gen_handbuilt.py, which writes CRLFs with string concatenation — deliberately not with our Writer. It cannot fail the way a real archive can: no crawl's idiosyncrasies, no truncated fetch, no header a real crawler emits that we did not think to write.
  • testdata/warcio.warc was written by warcio (Apache-2.0), the Python implementation behind Webrecorder and the Common Crawl tooling, installed into a throwaway virtualenv by testdata/gen_warcio.py. Its framing is not ours. Its payloads are, so committing it redistributes nothing third-party.

No specimen from a public archive is committed: freely downloadable is not the same as redistributable, and a licence we could not establish is one we do not act on.

What compensates:

  • A round trip in both directions — and this is the weaker half, because two halves of one package agree with each other about a misunderstanding every time.
  • Bytes laid out by hand from the specification, read back. This is what stops the two halves agreeing privately.
  • Byte equality with warcio. TestWriterReproducesWarcio re-emits testdata/warcio.warc from the records our Reader parsed; the result is identical, octet for octet.
  • Measured off-repository (warcio cannot be a CI dependency): warcio reads both fixtures and both of our re-emissions — 7 and 4 records, every block the declared length — and agrees on types, lengths, the three repeated WARC-Concurrent-To values and on splitting X-Note: a:b:c: d at the first colon.

One disagreement, stated rather than hidden: for a folded value X-Folded: first CRLF HT second, this package yields first second and warcio yields first\tsecond. RFC 2616 §2.2 — which ISO 28500 §4 takes LWS from — says a recipient may replace linear white space with a single SP, so both conform. We normalise, because a value's meaning should not depend on where a writer broke the line.

What the corpus is asserted to reach

TestCorpusReaches fails if any of these leaves the corpus, so a regenerated or truncated fixture cannot quietly make the suite prove less: a block containing CRLF, a block ending in CRLF, a block containing NUL, a zero-length block, a field value containing a colon, an empty field value, a repeated field name, a WARC/1.0 record, an unregistered WARC-Type, and a folded field value.

Ablations

Each row is a plausible wrong implementation, applied to a scratch copy and run against the suite.

Ablation Result Caught by
Skip whitespace at the trailer instead of consuming exactly CRLF CRLF caught 3 negative cases only — every positive test still passed
Scan for the CRLF CRLF delimiter instead of using Content-Length caught 12 tests, incl. both fixtures and the whole filesystem suite
Keep one value per field name (last wins, as a map does) caught 5 tests, incl. the repeated-header and two-Content-Length cases
Split a header line at the last colon caught 13 tests
Accept a bare LF as a line terminator caught 1 case (LF-terminated version line)
No LWS unfolding: a continuation line becomes its own field caught 10 tests
strings.TrimSpace instead of strings.Trim(value, " \t") PASSED — see below now TestHeaderValueTrimsOnlySPandHT
Drop the ordinal from the entry name (type + URI tail only) caught naming tests + 3 filesystem tests, on duplicate names
Refuse an unregistered WARC-Type (contra §5.5) caught 9 tests
Compare field names case-sensitively caught 2 header tests
Writer trims a value instead of refusing it caught 2 cases in TestWriteRefuses

The one that passed. strings.TrimSpace strips VT, FF and every Unicode space, U+00A0 among them; LWS is SP and HT and nothing else, so those octets are ordinary field-content and a reader that eats them hands back a value missing bytes the file carried, with nothing to say so. The whole suite passed under that change. TestHeaderValueTrimsOnlySPandHT now pins it, with a value whose ends are U+00A0, VT and FF inside real LWS.

The first row is worth reading twice: the trailer ablation was invisible to every positive test, including the fixture whose block ends in CRLF, because reading by count first and then skipping whitespace still lands in the right place. Only the negative cases — a one-CRLF trailer, an LF LF trailer, a trailer eaten by a block longer than its declared length — could see it.

Testing

export GOWORK=off GOFLAGS=-mod=mod
go test -race ./...

CI runs 8 lanes — 4 native (linux/amd64, linux/arm64, darwin/arm64, windows/amd64) and 4 emulated under QEMU (riscv64, loong64, ppc64le, s390x) — behind a 100 % statement coverage gate. Fixtures are //go:embeded because the emulated lanes run a go test -c binary in a container with no testdata/.

To regenerate the fixtures:

python3 testdata/gen_handbuilt.py testdata/handbuilt.warc
python3 -m venv /tmp/warcvenv && /tmp/warcvenv/bin/pip install warcio
/tmp/warcvenv/bin/python testdata/gen_warcio.py testdata/warcio.warc

Both are byte-for-byte reproducible: every record ID and date is fixed in the scripts, so a regeneration that differs means the producer changed its framing, which is worth knowing.

Licence

BSD-3-Clause.

Documentation

Overview

Package warc reads and writes WARC files (ISO 28500), the format a web crawl is archived in, in pure Go with no cgo and no dependency outside github.com/go-filesystems/interface.

Three entry points, one per way a caller wants the file: 

func NewReader(r io.Reader) (*Reader, error)                              // stream records
func NewWriter(w io.Writer) *Writer                                       // write records
func OpenReader(r io.ReaderAt, size int64) (filesystem.Filesystem, error) // one entry per record

Open is OpenReader with the concrete *FS, for the three calls that are not on the interface: Names, Header and Record.

The format, and the two places a reader gets it wrong

A record is 

WARC/1.1 CRLF
Header-Name: value CRLF      (repeated)
CRLF
<exactly Content-Length octets>
CRLF CRLF

The version line, the field names and every CRLF are ASCII; Content-Length is mandatory and counts octets. WARC-Type, WARC-Date (ISO 8601, UTC) and WARC-Record-ID (a URI in angle brackets) are mandatory too.

Two properties of that framing are where implementations fail, and this package is built around both:

THE TWO TRAILING CRLFS ARE PART OF THE RECORD, not a separator to be skipped. A reader that consumes whitespace until the next "WARC/" instead of reading exactly four octets accepts a truncated record as a whole one, and mis-frames the file whenever a block ends in a newline -- which, for a response record holding an HTTP message, is the usual case rather than the odd one. This package reads four octets and compares them.

THE BLOCK IS OPAQUE. It may contain CRLF, NUL, a lone CR, bytes that are not text; nothing may scan it for a delimiter. This package reads exactly Content-Length octets and never looks inside. It also does not ALLOCATE Content-Length octets up front: the length comes from the file, so a seventy-octet header claiming an eight-exabyte block would otherwise be an out-of-memory panic rather than an error.

Plain .warc only

A .warc.gz is not a gzipped .warc: it is a concatenation of one gzip MEMBER per record, so that an index can seek to a record and inflate that record alone. This package handles plain .warc. A caller with a single-member stream wraps it: 

zr, err := gzip.NewReader(f)
r, err := warc.NewReader(zr)

In this family that is not a gap in practice: github.com/go-filesystems/unarchive peels compression before it dispatches on format, so a .warc.gz arrives here already inflated. It IS a gap for a caller that wants per-record random access into a .warc.gz, because that needs the gzip member offsets, which this package does not compute.

The filesystem view, and how entries are named

OpenReader presents the file as a flat, read-only directory with one regular file per record, whose contents are that record's content block. Opening reads only the header blocks; a block is read when a caller asks for it, which is why FS implements the optional filesystem.Opener and can answer a byte range out of a hundred-megabyte block without materialising it. Every mutating method returns ErrReadOnly.

An entry is named 

<5-digit ordinal>-<WARC-Type>[-<sanitised tail of WARC-Target-URI>]

00000-warcinfo
00001-response-one.html
00002-request-one.html
00003-resource-blob.bin

WARC-Record-ID is the obvious unique key and it is the wrong filename. Its value is a URI in angle brackets -- "<urn:uuid:5a3b...>" in practice -- whose brackets and colons are illegal in a Windows filename and whose colon is the HFS path separator, so it has to be mangled; what survives the mangling is a UUID, which tells a person nothing and which they cannot predict from the archive in front of them. The target URI alone is worse, because it is not unique: a crawl of one page writes a request AND a response record under the same WARC-Target-URI, and a revisit record later under the same one again, so three entries would silently become one.

The ordinal is the record's position in the file. It is unique by construction, so no two entries can collide whatever the headers say; it sorts the listing into crawl order under a plain lexical sort; and a person can count to it. It is NOT stable across files -- record 5 of a re-fetched crawl is a different record, and concatenating two WARCs renumbers the second -- and that is the price. A caller that needs an identity across files wants WARC-Record-ID, which is what it is for, and Record(name).RecordID() hands it over.

What it refuses

The reader refuses a version line other than WARC/1.0 or WARC/1.1, a line not terminated by CRLF, a header line with no colon or a field name that is not a token, a Content-Length that is absent, repeated or not a decimal octet count, a missing WARC-Type, WARC-Date or WARC-Record-ID, a WARC-Date that is not a UTC timestamp, a record ID without angle brackets, and a trailer that is not two CRLFs. All of those satisfy errors.Is(err, ErrSyntax); a truncated file satisfies errors.Is(err, io.ErrUnexpectedEOF); an I/O failure is neither, so a caller can tell "this is not a WARC" from "the disk went away".

It does NOT refuse an unregistered WARC-Type. §5.5 tells a reader to pass an unknown record type through rather than treat the file as broken, so refusing one would reject files a conforming writer may produce.

The writer refuses rather than repairs, because each case it refuses would read back as something other than what the caller passed: a missing mandatory field, a field name that is not a token, a value containing CR, LF or NUL -- that one is header injection, and the writer is the only place it can be stopped -- a value with a leading or trailing SP or HT, which a reader is entitled to strip, and a Content-Length disagreeing with len(Block).

⚠ WHERE THE FIXTURES CAME FROM, AND WHAT THAT COSTS IN CONFIDENCE

Neither wget nor warcio was installed on the machine this package was written on, so there was no crawl to point at and no archive to hand. The corpus is therefore two files, and the reader should know which is which:

  • testdata/handbuilt.warc is OUR OWN CONSTRUCTION. Its octets were laid out from §5's grammar by testdata/gen_handbuilt.py, a script that writes CRLFs with string concatenation. It cannot fail the way a real archive can: it contains no crawl's idiosyncrasies, no truncated fetch, no header a real crawler emits that we did not think to write.
  • testdata/warcio.warc was written by warcio (Apache-2.0), the Python implementation behind Webrecorder and the Common Crawl tooling, installed into a throwaway virtualenv for the purpose. Its framing is not ours. Its payloads are, so committing it redistributes nothing third-party.

No specimen from a public archive is committed: what is freely downloadable is not automatically redistributable, and a licence we could not establish is a licence we do not act on.

What compensates for that, and what does not:

  • A round trip in BOTH directions. Reader∘Writer and Writer∘Reader are both tested, and that is the weaker half: two halves of one package agree with each other about a misunderstanding every time.
  • Bytes laid out by hand from the specification and read back. This is what stops the two halves agreeing privately, and it is why testdata/handbuilt.warc is generated by a script rather than by the Writer.
  • Byte equality with warcio. TestWriterReproducesWarcio reads testdata/warcio.warc and re-emits it; the result must be identical, octet for octet. It is. So our Writer puts every CRLF where an implementation we did not write put it.
  • Measured off-repository, not in CI, because warcio cannot be a CI dependency: warcio reads both fixtures and both of our re-emissions, 7 and 4 records, every block the declared length. It agrees with us on the record count, the types, the block lengths, the three repeated WARC-Concurrent-To values and on splitting "X-Note: a:b:c: d" at the FIRST colon.

The one place warcio and this package disagree is worth stating rather than hiding. For a folded field value 

X-Folded: first CRLF HT second CRLF SP SP SP SP third

this package yields "first second third" and warcio yields "first\tsecond third". RFC 2616 §2.2, which ISO 28500 §4 takes LWS from, says a recipient MAY replace linear white space with a single SP before interpreting a value, so both readings conform. This package normalises, because a value's meaning should not depend on where a writer chose to break the line; a caller that needs the octets as they lay on disk should not be reading them through a field value at all.

Fixtures are EMBEDDED with //go:embed rather than read from testdata/ at run time, because this repository's cross-architecture lanes build the test binary with `go test -c` and run it in a container holding the binary and nothing else. A test that read testdata/ would pass on the four native lanes and fail on the four emulated ones.

Index

Constants

View Source 
const (
	// MaxHeaderBytes bounds the version line and every named field of one
	// record together, lines terminators included.
	MaxHeaderBytes = 1 << 20
	// DefaultMaxBlockSize bounds a block held in memory.
	DefaultMaxBlockSize = 1 << 30
)

The ceilings. A WARC file is untrusted input, and every one of these is a number the file alone could otherwise choose.

ISO 28500 (the 1.1 text, checked) states no limit on a header block, and jwarc's parser keeps none either; MaxHeaderBytes is net/http's DefaultMaxHeaderBytes (1 MiB, net/http/server.go), the same kind of thing: a block of "Name: value" lines ended by an empty line. DefaultMaxBlockSize is a policy, not a format fact, chosen so that a block always fits an int on a 32-bit target; a block is held whole by Next, ReadFile and Record. OpenFile has no such limit, since it reads nothing until asked.

View Source 
const (
	Version10 = "WARC/1.0"
	Version11 = "WARC/1.1"
)

The two WARC versions this package reads and writes. ISO 28500:2009 defined 1.0; ISO 28500:2017 defined 1.1, whose only framing-relevant change is that WARC-Date may carry fractional seconds. Anything else in the version line is refused rather than guessed at, because a reader that accepts a version it does not know cannot promise it framed the file correctly.

View Source 
const (
	TypeWarcinfo     = "warcinfo"
	TypeResponse     = "response"
	TypeRequest      = "request"
	TypeMetadata     = "metadata"
	TypeResource     = "resource"
	TypeRevisit      = "revisit"
	TypeConversion   = "conversion"
	TypeContinuation = "continuation"
)

The WARC-Type values ISO 28500 defines. A reader must NOT refuse a type it does not know -- §5.5 says an unknown record type is to be skipped, not treated as a broken file -- so these are here for writers and for callers that switch on the type, and the reader never consults them.

View Source 
const (
	HeaderType          = "WARC-Type"
	HeaderDate          = "WARC-Date"
	HeaderRecordID      = "WARC-Record-ID"
	HeaderContentLength = "Content-Length"

	HeaderConcurrentTo   = "WARC-Concurrent-To"
	HeaderContentType    = "Content-Type"
	HeaderBlockDigest    = "WARC-Block-Digest"
	HeaderPayloadDigest  = "WARC-Payload-Digest"
	HeaderIPAddress      = "WARC-IP-Address"
	HeaderRefersTo       = "WARC-Refers-To"
	HeaderTargetURI      = "WARC-Target-URI"
	HeaderTruncated      = "WARC-Truncated"
	HeaderWarcinfoID     = "WARC-Warcinfo-ID"
	HeaderFilename       = "WARC-Filename"
	HeaderProfile        = "WARC-Profile"
	HeaderIdentifiedPLD  = "WARC-Identified-Payload-Type"
	HeaderSegmentNumber  = "WARC-Segment-Number"
	HeaderSegmentOriginI = "WARC-Segment-Origin-ID"
	HeaderSegmentTotal   = "WARC-Segment-Total-Length"
)

Named fields this package refers to by name. The four mandatory ones are listed first; the rest are here so callers do not have to spell them.

Variables

View Source 
var ErrReadOnly = errors.New("warc: filesystem is read-only")

ErrReadOnly is returned by every mutating method of the filesystem view. A WARC file is an append-only log of what a crawl saw; rewriting a record in place would change a record of fact, so the Filesystem surface refuses it rather than implementing half of it.

View Source 
var ErrSyntax = errors.New("warc: syntax")

ErrSyntax is the class of every failure to frame or parse a WARC file: a version line that is not one, a header line without a colon, a Content-Length that is not a decimal count of octets, a record whose two trailing CRLFs are not there. Callers distinguish "this is not a WARC" from an I/O failure with errors.Is(err, warc.ErrSyntax); the wrapped message says which record and what was wrong.

View Source 
var ErrTooLarge = errors.New("warc: over the size ceiling")

ErrTooLarge is returned when a record asks for more memory than the ceilings above allow. It is not a syntax error: the file may be a well-formed WARC that this reader declines to hold.

Functions

func EntryName 

func EntryName(ordinal int, warcType, targetURI string) string

EntryName is the name record ordinal (0-based) gets in the filesystem view: 

00000-warcinfo
00001-request-index.html
00002-response-index.html
00003-resource-logo.png

Why not WARC-Record-ID

The obvious unique key is WARC-Record-ID, and it is the wrong filename. Its value is a URI in angle brackets, in practice "<urn:uuid:5a3b...>": the angle brackets and the colons are not allowed in a Windows filename and the colon is the HFS path separator, so the name has to be mangled anyway; what is left is a UUID, which tells a person nothing and which they cannot predict from the archive they are looking at. A tool that wants the record ID can ask for it -- Record(name).RecordID() -- and a person reading a directory listing gains nothing from seeing thirty UUIDs.

The target URI alone is worse: it is not unique. A crawl of one page writes a request AND a response record with the same WARC-Target-URI, and a revisit record later with the same one again, so the names would collide -- silently, three entries becoming one.

Why ordinal first

The ordinal is the record's position in the file. It is unique by construction, so no two entries can collide whatever the headers say; it sorts the listing into archive order, which is crawl order, under a plain lexical sort; and it is stable for a given file, which a hash of the record would also be but an ordinal is one a person can count to.

It is NOT stable across files: record 5 of a WARC is not record 5 of the same crawl re-fetched, and concatenating two WARCs renumbers the second. That is the cost, and it is the right one to pay here -- a filesystem view of one file needs names unique within that file, and a caller that needs an identity across files has WARC-Record-ID, which is exactly what it is for.

The type and the URI tail follow so the name says what the entry is. Both are sanitised to ASCII letters, digits, '.', '_' and '-' and truncated, so the name is safe on every filesystem this family writes to; two records whose tails sanitise to the same string still differ by ordinal.

func OpenReader 

func OpenReader(r io.ReaderAt, size int64) (filesystem.Filesystem, error)

OpenReader opens the WARC file in the first size bytes of r and returns it as a filesystem.Filesystem.

It exists so every driver in go-filesystems answers to ONE name for the same thing: github.com/go-filesystems/detect registers a func(io.ReaderAt, int64) (filesystem.Filesystem, error), and Open here has the right shape but the concrete return type, which does not fit. The concrete type is still available from Open for a caller that wants Record, Header or Names.

Types

type FS 

type FS struct {
	// contains filtered or unexported fields
}

FS is a WARC file seen as a read-only filesystem: a flat directory with one regular file per record, whose contents are that record's content block.

Contents are the BLOCK, and the header is beside it

Reading an entry gives the record's content block, not the record's bytes. That is the choice a caller of a filesystem wants -- the archived HTTP message, the fetched resource -- and it is what makes Stat's size the number ReadFile returns. The named fields are not lost: Record and Header on this type hand them back for any entry, and Reader gives both together for a caller that wants to stream.

Every mutating method returns ErrReadOnly

A WARC is a record of what a crawl observed, and the format has no way to rewrite a record in place: Content-Length fixes each record's extent, so changing one byte of a block either changes the length -- moving every later record -- or is not the record that was written. So the write half of filesystem.Filesystem returns ErrReadOnly rather than a partial implementation.

func Open 

func Open(r io.ReaderAt, size int64) (*FS, error)

Open reads the record headers of the WARC file in the first size bytes of r and returns it as a filesystem. Every record is framed on the way through, so Open fails on a file that is not a WARC rather than on the read that reaches the bad record.

Blocks are not read. Open costs one pass over the header blocks plus a seek per record; ReadFile and OpenFile read a block when asked.

func (*FS) Close 

func (f *FS) Close() error

Close releases nothing: FS does not own the io.ReaderAt it was handed, and closing what a caller passed in would be closing something it may still be using. It is here because filesystem.Filesystem has it.

func (*FS) DeleteDir 

func (f *FS) DeleteDir(p string) error

DeleteDir always returns ErrReadOnly.

func (*FS) DeleteFile 

func (f *FS) DeleteFile(p string) error

DeleteFile always returns ErrReadOnly.

func (*FS) Header 

func (f *FS) Header(name string) (Header, error)

Header returns the named fields of the record behind an entry, in file order. The returned Header is a copy.

func (*FS) ListDir 

func (f *FS) ListDir(p string) ([]filesystem.DirEntry, error)

ListDir lists the root, the only directory this filesystem has.

func (*FS) MkDir 

func (f *FS) MkDir(p string, perm os.FileMode) error

MkDir always returns ErrReadOnly.

func (*FS) Names 

func (f *FS) Names() []string

Names returns every entry name, in archive order.

func (*FS) OpenFile 

func (f *FS) OpenFile(p string) (filesystem.File, error)

OpenFile opens a record's content block for random access, which is the reason this driver implements filesystem.Opener: a response record's block can be a hundred megabytes, and a server answering a range request out of a WARC must not have to materialise all of it. Opening reads nothing.

func (*FS) ReadFile 

func (f *FS) ReadFile(p string) ([]byte, error)

ReadFile returns the content block of the record at path. 

func (f *FS) ReadLink(p string) (string, error)

ReadLink always fails: a WARC has no symbolic links. A path that is not there is still reported as not there, so a caller can tell the two apart.

func (*FS) Record 

func (f *FS) Record(name string) (*Record, error)

Record returns the whole record behind an entry: version, headers and block. The block is read now.

func (*FS) Rename 

func (f *FS) Rename(oldPath, newPath string) error

Rename always returns ErrReadOnly.

func (*FS) Stat 

func (f *FS) Stat(p string) (filesystem.Stat, error)

Stat describes the root or one entry.

func (*FS) WriteFile 

func (f *FS) WriteFile(p string, data []byte, perm os.FileMode) error

WriteFile always returns ErrReadOnly.

type Field 

type Field struct {
	Name  string
	Value string
}

Field is one WARC named field: a name and its value, in the order the record carries them.

Fields are a SLICE and not a map because two properties of the format make a map wrong. A WARC record may repeat a field name -- WARC-Concurrent-To is the ordinary case, a record concurrent with several others -- and a map either loses the repeats or has to model them anyway. And a WARC file is evidence: a reader that reorders the header block cannot hand a caller the bytes it read, so a digest computed over a re-serialised header would not match the one the crawler wrote.

type File 

type File struct {
	// contains filtered or unexported fields
}

File is an open content block: random access over the record's extent in the underlying file, and its size.

ReadAt is io.SectionReader's, so it keeps io.ReaderAt's contract to the letter -- a short read always carries an error, an offset at or past the end gives (0, io.EOF), and concurrent calls are safe -- rather than this package reimplementing it and getting one of those wrong.

func (*File) Close 

func (f *File) Close() error

Close releases nothing; a File holds an offset and a length.

func (*File) ReadAt 

func (f *File) ReadAt(p []byte, off int64) (int, error)

ReadAt implements io.ReaderAt over the block.

func (*File) Size 

func (f *File) Size() int64

Size is the block's length in octets, which is the record's Content-Length. 

type Header []Field

Header is a record's named fields, in file order.

func (*Header) Add 

func (h *Header) Add(name, value string)

Add appends a field, keeping any field of the same name that is already there. This is the call for WARC-Concurrent-To and the other repeatable fields.

func (Header) Clone 

func (h Header) Clone() Header

Clone returns a copy that shares no storage with h, so a caller can keep a record's header after handing the record on.

func (*Header) Del 

func (h *Header) Del(name string)

Del removes every field with this name.

func (Header) Get 

func (h Header) Get(name string) string

Get returns the value of the first field with this name, comparing names case-insensitively as the grammar's token production requires, or "" if there is none. Use Values when a name may repeat: Get cannot tell one absent field from one whose value is empty, and Has can.

func (Header) Has 

func (h Header) Has(name string) bool

Has reports whether the name is present at all, which is not the same question as Get returning a non-empty string.

func (*Header) Set 

func (h *Header) Set(name, value string)

Set replaces the value of the first field with this name, in place so the field keeps its position, and deletes any further field of the same name. If the name is absent it is appended.

func (Header) Values 

func (h Header) Values(name string) []string

Values returns every value carried under this name, in file order, or nil if there is none.

type Reader 

type Reader struct {

	// MaxBlockSize is the largest Content-Length Next will read into memory;
	// zero means DefaultMaxBlockSize. A larger declared length is refused with
	// ErrTooLarge BEFORE a byte of the block is read. Set it before the first
	// Next.
	MaxBlockSize int64
	// contains filtered or unexported fields
}

Reader reads WARC records from a stream, one at a time, in file order.

What it does not do: gzip

A .warc.gz is not a gzipped .warc. It is a concatenation of one gzip MEMBER per record, so that an index can seek to a record and inflate it alone; the deflate streams are per-record and the file is only valid gzip by the multi-member rule. This package handles PLAIN .warc and nothing else. Wrap the stream yourself for the single-member case: 

zr, err := gzip.NewReader(f)   // and zr.Multistream(false) per member
r, err := warc.NewReader(zr)

In this family it is not a gap in practice: github.com/go-filesystems/unarchive peels compression before it dispatches on format, so a .warc.gz reaches this package already inflated. It IS a gap for a caller that wants per-record random access into a .warc.gz, which needs the member offsets this package does not compute.

func NewReader 

func NewReader(r io.Reader) (*Reader, error)

NewReader returns a Reader that reads records from r.

It returns an error only for a nil reader. The signature keeps the error because a caller writing `r, err := warc.NewReader(f)` should not have to change shape if a future version validates something up front, and because a nil io.Reader here would otherwise panic on the first Next instead of at the call that was wrong.

func (*Reader) Next 

func (r *Reader) Next() (*Record, error)

Next returns the next record, or io.EOF at a clean end of file -- that is, immediately after a record's second CRLF with nothing following.

The returned Record owns its Block; the Reader keeps no reference to it, so a caller may retain records across calls.

Errors are sticky. A malformed record leaves the stream at an offset this package cannot justify resuming from -- the whole point of Content-Length framing is that there is no resynchronisation point to scan for -- so every later Next returns the same error rather than pretending to recover.

func (*Reader) Records 

func (r *Reader) Records() int

Records returned so far.

type Record 

type Record struct {
	// Version is the version line, e.g. Version11. NewWriter's WriteRecord
	// treats "" as Version11.
	Version string
	// Header is the named fields, in order.
	Header Header
	// Block is the content block, Content-Length octets exactly.
	Block []byte
}

Record is one WARC record: its version line, its named fields in file order, and its content block.

Block is the block EXACTLY as the file carries it, Content-Length octets of it, with nothing trimmed. The block is opaque: a response record's block is an HTTP message, a warcinfo block is an application/warc-fields document, a resource block is whatever was fetched, and any of them may contain CRLF, NUL or bytes that are not text at all. Nothing in this package looks inside it.

func (*Record) ContentType 

func (r *Record) ContentType() string

ContentType returns the block's Content-Type, or "" if unstated.

func (*Record) Date 

func (r *Record) Date() (time.Time, error)

Date parses WARC-Date. The reader has already checked it parses, so the error here is for a Record a caller built.

func (*Record) RecordID 

func (r *Record) RecordID() string

RecordID returns WARC-Record-ID as the file carries it, angle brackets and all: the field's value is a URI in angle brackets, and stripping them here would make the value we hand back differ from the value we read.

func (*Record) TargetURI 

func (r *Record) TargetURI() string

TargetURI returns WARC-Target-URI, or "" for the record types that have no target (warcinfo, and metadata about the archive itself).

func (*Record) Type 

func (r *Record) Type() string

Type returns WARC-Type, or "" if the field is absent. The reader guarantees it is present, so "" means the Record was built by a caller.

type Writer 

type Writer struct {
	// contains filtered or unexported fields
}

Writer writes WARC records to a stream.

It writes PLAIN .warc. A .warc.gz is one gzip member per record, which a caller composes by wrapping each WriteRecord in its own gzip.Writer and closing it before the next; this package does not do it, because doing it halfway -- one gzip stream around the whole file -- produces a file that inflates to a valid WARC and is NOT a valid .warc.gz, since no index can seek into it.

func NewWriter 

func NewWriter(w io.Writer) *Writer

NewWriter returns a Writer that writes records to w. It does not buffer: each WriteRecord issues three Writes (header block, content block, trailer), so a caller writing many small records should hand it a bufio.Writer.

func (*Writer) Records 

func (w *Writer) Records() int

Records written so far.

func (*Writer) WriteRecord 

func (w *Writer) WriteRecord(rec *Record) error

WriteRecord writes one record: version line, named fields, CRLF, the block's octets, CRLF, CRLF.

It REFUSES rather than repairs, in four cases, because each one is a record that would read back as something other than what the caller passed:

  • a missing WARC-Type, WARC-Date or WARC-Record-ID: §5 makes them mandatory, and this package's own Reader would refuse the result.
  • a field name that is not a token, or a value containing CR, LF or NUL: a value with a CRLF in it does not produce a long header, it produces two, and the second is whatever the caller's data happened to say. That is the header-injection shape, and a writer is the only place it can be stopped.
  • a value with leading or trailing space or tab: the grammar lets a reader strip it, so writing it would silently change the value. Trim it first and the round trip is exact.
  • a Content-Length that disagrees with len(Block): one of the two is wrong and the writer cannot know which. A record with no Content-Length gets one, computed from the block, appended after the caller's fields.

Errors are sticky: after a failed write the stream's contents are no longer a WARC, so further records are not appended to it.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.