warc

package module
v0.0.0-...-74239d2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 28, 2026 License: CC0-1.0 Imports: 43 Imported by: 0

README

warc

GoDoc Go Report Card

A Go library for reading and writing WARC files, with advanced features for web archiving.

Features

  • Read and write WARC files with support for multiple compression formats (GZIP, ZSTD)
  • HTTP client with built-in WARC recording capabilities
  • Content deduplication (local URL-agnostic and CDX-based)
  • Configurable file rotation and size limits
  • DNS caching and custom DNS resolution (with DNS archiving)
  • Custom TLS configurations
  • Random local IP assignment for distributed crawling (including Linux kernel AnyIP feature)
  • Smart memory management with disk spooling options
  • IPv4/IPv6 support with configurable preferences

Installation

go get github.com/internetarchive/gowarc

Usage

This library's biggest feature is to provide a standard HTTP client through which you can execute requests that will be recorded automatically to WARC files. It's the basis of Zeno.

HTTP Client with WARC Recording
package main

import (
	"context"
	"fmt"
	"github.com/internetarchive/gowarc"
	"io"
	"net/http"
	"time"
)

func main() {
	// Configure WARC settings
	rotatorSettings := &warc.RotatorSettings{
		WarcinfoContent: warc.Header{
			"software": "My WARC writing client v1.0",
		},
		Prefix:             "WEB",
		Compression:        "gzip",
		WARCWriterPoolSize: 4, // Records will be written to 4 WARC files in parallel, it helps maximize the disk IO on some hardware. To be noted, even if we have multiple WARC writers, WARCs are ALWAYS written by pair in the same file. (req/resp pair)
	}

	// Configure HTTP client settings
	clientSettings := warc.HTTPClientSettings{
		RotatorSettings: rotatorSettings,
		TempDir:         "./temp",
		DNSServers:      []string{"8.8.8.8", "8.8.4.4"},
		DedupeOptions: warc.DedupeOptions{
			LocalDedupe:   true,
			CDXDedupe:     false,
			SizeThreshold: 2048, // Only payloads above that threshold will be deduped
		},
		DialTimeout:           10 * time.Second,
		ResponseHeaderTimeout: 30 * time.Second,
		DNSResolutionTimeout:  5 * time.Second,
		DNSRecordsTTL:         5 * time.Minute,
		DNSCacheSize:          10000,
		MaxReadBeforeTruncate: 1000000000,
		DecompressBody:        true,
		FollowRedirects:       true,
		VerifyCerts:           true,
		RandomLocalIP:         true,
	}

	// Create HTTP client
	client, err := warc.NewWARCWritingHTTPClient(clientSettings)
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// The error channel NEED to be consumed, else it will block the
	// execution of the WARC module
	go func() {
		for err := range client.ErrChan {
			fmt.Errorf("WARC writer error: %s", err.Err.Error())
		}
	}()

	// This is optional but the module give a feedback on a channel passed as context value to the
	// request, this helps knowing when the record has been written to disk. If this is not used, the WARC
	// writing is asynchronous
	req, err := http.NewRequest("GET", "https://archive.org", nil)
	if err != nil {
		panic(err)
	}

	feedbackChan := make(chan warc.FeedbackEvent, 1)
	req = req.WithContext(warc.WithFeedbackChannel(req.Context(), feedbackChan))

	resp, err := client.Do(req)
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()

	// Process response
	// Note: the body NEED to be consumed to be written to the WARC file.
	io.Copy(io.Discard, resp.Body)

	// Will block until records are actually written to the WARC file
	<-feedbackChan
}

CLI Tools

In addition to the Go library, gowarc provides several command-line utilities for working with WARC files:

Installation

Pre-built releases are available on the GitHub releases page.

# Install from source
go install github.com/internetarchive/gowarc/cmd/warc@latest

# Or build locally
cd cmd/warc/
go build -o warc
Available Commands
warc extract

Extract files and content from WARC archives with filtering options.

# Extract all files from WARC archives
warc extract file1.warc.gz file2.warc.gz

# Extract only specific content types
warc extract --content-type "text/html" --content-type "image/jpeg" archive.warc.gz

# Extract to specific directory with multiple threads  
warc extract --output ./extracted --threads 4 *.warc.gz

# Sort extracted files by host
warc extract --host-sort archive.warc.gz
warc mend

Repair and close incomplete gzip-compressed WARC files that were left with .open suffix during crawling.

# Dry run to see what would be fixed
warc mend --dry-run *.warc.gz.open

# Fix files with confirmation prompts  
warc mend corrupted.warc.gz.open

# Auto-fix without prompts
warc mend --yes *.warc.gz.open

# Force verification of any gzip WARC files (not just .open)
warc mend --force --dry-run archive.warc.gz

Features:

  • By default, only processes .open files; use --force to verify any gzip WARC files
  • Verifies gzip format using magic bytes, not just file extension
  • Detects and removes trailing garbage bytes
  • Truncates at corruption points while preserving maximum valid data
  • Removes .open suffix to "close" files when present
  • Provides comprehensive statistics on repairs performed
  • Memory-efficient streaming for large files

See cmd/warc/mend/README.md for detailed documentation.

warc verify

Validate the integrity and structure of WARC files.

# Verify single file
warc verify archive.warc.gz

# Verify multiple files with progress
warc verify -v *.warc.gz

# JSON output for automation
warc verify --json archive.warc.gz
warc completion

Generate shell completion scripts for bash, zsh, fish, or PowerShell.

# Bash completion
warc completion bash > /etc/bash_completion.d/warc

# Zsh completion
warc completion zsh > ~/.zsh/completions/_warc

# Fish completion
warc completion fish > ~/.config/fish/completions/warc.fish

# PowerShell completion
warc completion powershell > warc.ps1
Global Flags

All commands support these global options:

  • -v, --verbose - Enable verbose/debug logging
  • --json - Output logs in JSON format for structured processing
  • -h, --help - Show help for any command

Build tags

  • standard_gzip: Use the standard library gzip implementation instead of the faster one from klauspost
  • klauspost_gzip: Use the faster gzip implementation from klauspost (default, you don't need to specify it)

License

This module is released under CC0 license. You can find a copy of the CC0 License in the LICENSE file.

Documentation

Index

Constants

View Source 
const (
	ContextKeyFeedback    contextKey = "feedback"
	ContextKeyWrappedConn contextKey = "wrappedConn"
	ContextKeySave        contextKey = "save"
)
View Source 
const (
	CompressionNone = compressionType("none")
	CompressionGzip = compressionType("gzip")
	CompressionZstd = compressionType("zstd")
)

Variables

View Source 
var (
	IPv6 *availableIPs
	IPv4 *availableIPs
)
View Source 
var (
	// Create a couple of counters for tracking various stats
	DataTotal atomic.Int64

	CDXDedupeTotalBytes          atomic.Int64
	DoppelgangerDedupeTotalBytes atomic.Int64
	LocalDedupeTotalBytes        atomic.Int64

	CDXDedupeTotal          atomic.Int64
	DoppelgangerDedupeTotal atomic.Int64
	LocalDedupeTotal        atomic.Int64
)
View Source 
var CompressionTypes = []compressionType{CompressionNone, CompressionGzip, CompressionZstd}
View Source 
var DedupeHTTPClient = http.Client{
	Timeout: 10 * time.Second,
	Transport: &http.Transport{
		Dial: (&net.Dialer{
			Timeout: 5 * time.Second,
		}).Dial,
		TLSHandshakeTimeout: 5 * time.Second,
	},
}
View Source 
var ErrUnknownDigestAlgorithm = errors.New("unknown digest algorithm")

Functions

func GetDigest 

func GetDigest(r io.Reader, digestAlgorithm DigestAlgorithm) (string, error)

func GetNextIP 

func GetNextIP(availableIPs *availableIPs) net.IP

func IsDigestSupported 

func IsDigestSupported(algorithm string) bool

func MustCompressionTypeFromString 

func MustCompressionTypeFromString(s string) compressionType

func WithFeedbackChannel 

func WithFeedbackChannel(ctx context.Context, feedbackChan chan FeedbackEvent) context.Context

func WithSaveChannel 

func WithSaveChannel(ctx context.Context, ch chan bool) context.Context

func WithWrappedConnection 

func WithWrappedConnection(ctx context.Context, wrappedConnChan chan *CustomConnection) context.Context

Types

type Compressor 

type Compressor interface {
	io.Writer // Embedded so the compressor can be used as an io.Writer and its underlying writer can be replaced via Reset.
	io.Closer
	Reset(io.Writer)
}

type ConnPoolStats 

type ConnPoolStats struct {
	TotalHosts  int
	IdleConns   int
	MaxIdle     int
	IdleTimeout time.Duration
	TotalDials  int64
	ActiveConns int64
}

type CustomConnection 

type CustomConnection struct {
	net.Conn
	io.Reader
	io.Writer

	sync.WaitGroup
	// contains filtered or unexported fields
}

CustomConnection is kept for backward compatibility with WithWrappedConnection.

func (*CustomConnection) Close 

func (cc *CustomConnection) Close() error

func (*CustomConnection) CloseWithError 

func (cc *CustomConnection) CloseWithError(err error) error

func (*CustomConnection) Read 

func (cc *CustomConnection) Read(b []byte) (int, error)

func (*CustomConnection) Write 

func (cc *CustomConnection) Write(b []byte) (int, error)

type CustomHTTPClient 

type CustomHTTPClient struct {
	WaitGroup *WaitGroupWithCount

	ErrChan    chan *Error
	WARCWriter chan *RecordBatch

	TempDir string

	TLSHandshakeTimeout   time.Duration
	ConnReadDeadline      time.Duration
	MaxReadBeforeTruncate int // todo

	DigestAlgorithm DigestAlgorithm

	DataTotal *atomic.Int64

	DecompressBody bool

	CDXDedupeTotalBytes          *atomic.Int64
	DoppelgangerDedupeTotalBytes *atomic.Int64
	LocalDedupeTotalBytes        *atomic.Int64

	CDXDedupeTotal          *atomic.Int64
	DoppelgangerDedupeTotal *atomic.Int64
	LocalDedupeTotal        *atomic.Int64
	// contains filtered or unexported fields
}

func NewWARCWritingHTTPClient 

func NewWARCWritingHTTPClient(HTTPClientSettings HTTPClientSettings) (httpClient *CustomHTTPClient, err error)

func (*CustomHTTPClient) Close 

func (c *CustomHTTPClient) Close() error

func (*CustomHTTPClient) CloseIdleConnections 

func (c *CustomHTTPClient) CloseIdleConnections()

func (*CustomHTTPClient) Do 

func (c *CustomHTTPClient) Do(req *http.Request) (*http.Response, error)

func (*CustomHTTPClient) Get 

func (c *CustomHTTPClient) Get(url string) (*http.Response, error)

func (*CustomHTTPClient) GetConnPoolStats 

func (c *CustomHTTPClient) GetConnPoolStats() *ConnPoolStats

func (*CustomHTTPClient) Head 

func (c *CustomHTTPClient) Head(url string) (*http.Response, error)

func (*CustomHTTPClient) Post 

func (c *CustomHTTPClient) Post(url, contentType string, body interface{}) (*http.Response, error)

func (*CustomHTTPClient) WriteRecord 

func (c *CustomHTTPClient) WriteRecord(WARCTargetURI, WARCType, contentType, payloadString string, payloadReader io.Reader)

type DedupeOptions 

type DedupeOptions struct {
	CDXURL             string
	DoppelgangerHost   string
	CDXCookie          string
	SizeThreshold      int
	DedupeCacheSize    int
	LocalDedupe        bool
	CDXDedupe          bool
	DoppelgangerDedupe bool
}

type DigestAlgorithm 

type DigestAlgorithm int
const (
	SHA1 DigestAlgorithm = iota
	// According to IIPC, lowercase base 16 is the "popular" encoding for SHA256
	SHA256Base16
	SHA256Base32
	BLAKE3
)

func GetDigestFromPrefix 

func GetDigestFromPrefix(prefix string) DigestAlgorithm

type Error 

type Error struct {
	Err  error
	Func string
}

type FeedbackEvent 

type FeedbackEvent []RecordEvent // WARC-Record-ID header values of the records that have been written

type GzipReaderInterface 

type GzipReaderInterface interface {
	io.ReadCloser
	Multistream(enable bool)
	Reset(r io.Reader) error
}

GzipReaderInterface defines the interface for gzip readers This allows us to switch between standard gzip and klauspost gzip based on build tags

type GzipWriterInterface 

type GzipWriterInterface interface {
	io.WriteCloser
	Flush() error
	Reset(w io.Writer)
}

GzipWriterInterface defines the interface for gzip writers This allows us to switch between standard gzip and klauspost gzip based on build tags

type HTTPClientSettings 

type HTTPClientSettings struct {
	RotatorSettings         *RotatorSettings
	TempDir                 string
	DNSServers              []string
	DNSFallback             *dns.ClientConfig
	DedupeOptions           DedupeOptions
	DialTimeout             time.Duration
	ResponseHeaderTimeout   time.Duration
	DNSResolutionTimeout    time.Duration
	DNSRecordsTTL           time.Duration
	DNSCacheSize            int
	DNSConcurrency          int
	TLSHandshakeTimeout     time.Duration
	ConnReadDeadline        time.Duration
	MaxReadBeforeTruncate   int // todo
	DecompressBody          bool
	FollowRedirects         bool
	InsecureSkipVerifyCerts bool
	RandomLocalIP           bool
	DisableIPv4             bool
	DisableIPv6             bool
	IPv6AnyIP               bool
	DigestAlgorithm         DigestAlgorithm
	EnableKeepAlive         bool
	DefaultUserAgent        string
	ClientProfile           profiles.ClientProfile
	RandomTLSExtensionOrder bool
	MaxIdleConns            int
	MaxIdleConnsPerHost     int
	IdleConnTimeout         time.Duration
	EnableHTTP2             bool
	EnableHTTP3             bool
	ForceProtocol           string // "http/1.1", "h2", "h3"
}
type Header map[string][]string

func NewHeader 

func NewHeader() Header

func (Header) Add 

func (h Header) Add(key, value string)

func (Header) Del 

func (h Header) Del(key string)

func (Header) Get 

func (h Header) Get(key string) string

func (Header) Set 

func (h Header) Set(key, value string)

func (Header) Values 

func (h Header) Values(key string) []string

type ReadOpts 

type ReadOpts int

ReadOpts are options for ReadRecord 

const (
	// ReadOptsNoContentOutput means that the content of the record should not be returned.
	// This is useful for reading only the headers or metadata of the record.
	ReadOptsNoContentOutput ReadOpts = iota
)

type Reader 

type Reader struct {
	// contains filtered or unexported fields
}

Reader stores the bufio.Reader and gzip.Reader for a WARC file

func NewReader 

func NewReader(reader io.Reader) (*Reader, error)

NewReader returns a new WARC reader

func (*Reader) Close 

func (r *Reader) Close() error

Close closes the WARC reader src and dec readers if they are open.

func (*Reader) ReadRecord 

func (r *Reader) ReadRecord(opts ...ReadOpts) (*Record, error)

ReadRecord reads the next record from the opened WARC file.

Returns:

  • *Record: Record guaranteed to be non-nil if no errors occurred.
  • error: any parsing/IO error encountered (io.EOF for clean EOF).

type Record 

type Record struct {
	RecordInfo
	Content spooledtempfile.ReadWriteSeekCloser
}

Record represents a WARC record.

func NewRecord 

func NewRecord(tempDir string) *Record

NewRecord creates a new WARC record.

type RecordBatch 

type RecordBatch struct {
	FeedbackChan chan FeedbackEvent
	CaptureTime  string
	Records      []*Record
}

RecordBatch is a structure that contains a bunch of records to be written at the same time, and a common capture timestamp. FeedbackChan is used to signal when the records have been written.

func NewRecordBatch 

func NewRecordBatch(feedbackChan chan FeedbackEvent) *RecordBatch

NewRecordBatch creates a record batch, it also initialize the capture time.

type RecordEvent 

type RecordEvent struct {
	RecordInfo
	WARCFilename string // WARC filename, no .open extension
}

a record that has been written to a WARC file.

type RecordInfo 

type RecordInfo struct {
	Header  Header
	Offset  int64  // Offset of the record start (-1 if WARC file type is not supported yet)
	Size    int64  // COMPRESSED size of the record (gzip member): header + deflate data + trailer. (-1 if WARC file type is not supported yet)
	Version string // WARC/1.0, WARC/1.1 ...
}

type RotatorSettings 

type RotatorSettings struct {
	// Content of the warcinfo record that will be written
	// to all WARC files
	WarcinfoContent Header
	// Prefix used for WARC filenames, WARC 1.1 specifications
	// recommend to name files this way:
	// Prefix-Timestamp-Serial-Crawlhost.warc.gz
	Prefix string
	// Compression algorithm to use
	Compression compressionType

	// Path to a ZSTD compression dictionary to embed (and use) in .warc.zst files
	CompressionDictionary string
	// Directory where the created WARC files will be stored,
	// default will be the current directory
	OutputDirectory string
	// WARCSize is in Megabytes
	WARCSize float64
	// WARCWriterPoolSize defines the number of parallel WARC writers
	WARCWriterPoolSize int
	// You can get warc filename from this channel after each WARC file is written and closed and renamed to non-temp name.
	// - `nil` to disable
	// - it's your responsibility to close this channel
	WARCFilenameFeedbackChan chan string
	// contains filtered or unexported fields
}

RotatorSettings is used to store the settings needed by recordWriter to write WARC files

func NewRotatorSettings 

func NewRotatorSettings() *RotatorSettings

NewRotatorSettings creates a RotatorSettings structure and initialize it with default values

func (*RotatorSettings) NewWARCRotator 

func (s *RotatorSettings) NewWARCRotator() (recordWriterChan chan *RecordBatch, doneChannels []chan bool, err error)

NewWARCRotator creates and return a channel that can be used to communicate records to be written to WARC files to the recordWriter function running in a goroutine

type TLSProfile 

type TLSProfile struct {
	// contains filtered or unexported fields
}

func DefaultTLSProfile 

func DefaultTLSProfile() *TLSProfile

func NewTLSProfile 

func NewTLSProfile(profile profiles.ClientProfile, randomExtOrder bool) *TLSProfile

type WaitGroupWithCount 

type WaitGroupWithCount struct {
	sync.WaitGroup
	// contains filtered or unexported fields
}

func (*WaitGroupWithCount) Add 

func (wg *WaitGroupWithCount) Add(delta int)

func (*WaitGroupWithCount) Done 

func (wg *WaitGroupWithCount) Done()

func (*WaitGroupWithCount) Size 

func (wg *WaitGroupWithCount) Size() int

type Writer 

type Writer struct {
	Compressor      Compressor
	BufWriter       *bufio.Writer
	FileName        string
	DigestAlgorithm DigestAlgorithm
	ParallelGZIP    bool
}

Writer writes WARC records to WARC files.

func NewWriter 

func NewWriter(writer io.Writer, fileName string, digestAlgorithm DigestAlgorithm, compression compressionType, newFileCreation bool, dictionary []byte) (*Writer, error)

NewWriter creates a new WARC writer.

func (*Writer) FlushAndCloseCompressor 

func (w *Writer) FlushAndCloseCompressor() (err error)

Close will flush the final output and close the stream. The function will block until everything has been written. The Compressor can still be re-used after calling this. If the Compressor is nil, this will just flush the Writer.BufWriter.

func (*Writer) Reset 

func (w *Writer) Reset(output io.Writer)

reset resets the compressed writer to write to a new output. This reuses the encoder's internal buffers.

func (*Writer) WriteInfoRecord 

func (w *Writer) WriteInfoRecord(payload Header) (recordID string, err error)

WriteInfoRecord method can be used to write an information record to the WARC file and flush the data

func (*Writer) WriteRecord 

func (w *Writer) WriteRecord(r *Record) (recordID string, err error)

WriteRecord writes a record to the underlying WARC file and flushes the data. A record consists of a version string, the record header followed by a record content block and two newlines: 

Version CLRF
Header-Key: Header-Value CLRF
CLRF
Content
CLRF
CLRF

Source Files

View all Source files

Directories

Path Synopsis
cmd
testfetch command
warc command
warc/extract
warc/mend
warc/utils
warc/verify
pkg
spooledtempfile

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.