warc

Documentation

Index

• type Header
  • func NewHeader() Header
  • func (h Header) Encode(w io.Writer) error
  • func (h Header) Get(key string) string
  • func (h Header) Set(key, value string)
• type Writer
  • func NewMultiWriter(pattern string, maxSize uint64) (*Writer, error)
  • func NewWriter(w io.WriteCloser) *Writer
  • func (w *Writer) Close() error
  • func (w *Writer) NewRecord(hdr Header) (io.WriteCloser, error)

Types

type Header

type Header map[string]string

Header for a WARC record. Header field names are case-sensitive.

func NewHeader

func NewHeader() Header

NewHeader returns a Header with its own unique ID and the current timestamp.

func (Header) Encode

func (h Header) Encode(w io.Writer) error

Encode the header to a Writer.

func (Header) Get

func (h Header) Get(key string) string

Get the value of a header. If not found, returns an empty string.

func (Header) Set

func (h Header) Set(key, value string)

Set a header to the specified value. Multiple values are not supported.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer can write records to a file in WARC format. It is safe for concurrent access, since writes are serialized internally.

func NewMultiWriter

func NewMultiWriter(pattern string, maxSize uint64) (*Writer, error)

NewMultiWriter initializes a new Writer that writes its output to multiple files of limited size approximately equal to maxSize, rotating them when necessary. The input path should contain a literal '%s' token, which will be replaced with a (lexically sortable) unique token.

func NewWriter

func NewWriter(w io.WriteCloser) *Writer

NewWriter initializes a new Writer and returns it.

func (*Writer) Close

func (w *Writer) Close() error

Close the WARC writer and flush all buffers. This will also call Close on the wrapped io.WriteCloser object.

func (*Writer) NewRecord

func (w *Writer) NewRecord(hdr Header) (io.WriteCloser, error)

NewRecord starts a new WARC record with the provided header. The caller must call Close on the returned writer before creating the next record. Note that this function may block until that condition is satisfied. If this function returns an error, the state of the Writer is invalid and it should no longer be used.