bytebatch

Documentation

Overview

Copyright 2015 by Leipzig University Library, http://ub.uni-leipzig.de
                  The Finc Authors, http://finc.info
                  Martin Czygan, <martin.czygan@uni-leipzig.de>

This file is part of some open source application.

Some open source application is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Some open source application is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Foobar. If not, see <http://www.gnu.org/licenses/>.

@license GPL-3.0+ <http://spdx.org/licenses/GPL-3.0+>

Package bytebatch implements a []byte batch and parallel processing facilities for line oriented input.

Basic usage:

p := bytebatch.NewLineProcessor(os.Stdin, os.Stdout, func(b []byte) ([]byte, error) {
    // implement your business logic here
    return b, nil
})
if err := p.Run(); err != nil {
    // handle error
}

More self contained examples (parallel wc -l, grep -F):

* https://gist.github.com/miku/0687690c68e6104893a7c328d2900ca8.

Index

• type ByteBatch
  • func NewByteBatch(cap int) *ByteBatch
  • func (bb *ByteBatch) Add(b []byte)
  • func (bb *ByteBatch) Reset()
  • func (bb *ByteBatch) Size() int
  • func (bb *ByteBatch) Slice() [][]byte
• type ByteFunc
• type LineProcessor
  • func NewLineProcessor(r io.Reader, w io.Writer, f ByteFunc) *LineProcessor
  • func (p LineProcessor) Run() error

Types

type ByteBatch

type ByteBatch struct {
	// contains filtered or unexported fields
}

ByteBatch is a batch of byte slices.

func NewByteBatch

func NewByteBatch(cap int) *ByteBatch

NewByteBatch creates a new ByteBatch with a given capacity.

func (*ByteBatch) Add

func (bb *ByteBatch) Add(b []byte)

Add adds an element to the batch.

func (*ByteBatch) Reset

func (bb *ByteBatch) Reset()

Reset empties this batch.

func (*ByteBatch) Size

func (bb *ByteBatch) Size() int

Size returns the number of elements in the batch.

func (*ByteBatch) Slice

func (bb *ByteBatch) Slice() [][]byte

Slice returns a slice of byte slices.

type ByteFunc

type ByteFunc func([]byte) ([]byte, error)

ByteFunc is a function, that transforms a byte slice into another byte slice (plus error).

type LineProcessor

type LineProcessor struct {
	BatchSize     int
	LineSeparator byte
	NumWorkers    int
	// contains filtered or unexported fields
}

LineProcessor reads bytes from a reader, feeds them into a ByteFunc and writes the result to a writer. The default unit of work is one line.

func NewLineProcessor

func NewLineProcessor(r io.Reader, w io.Writer, f ByteFunc) *LineProcessor

NewLineProcessor creates a new LineProcessor. Default batch size is 10000, default line separator is `\n`, default number of workers equals the cpu count.

func (LineProcessor) Run

func (p LineProcessor) Run() error

Run starts executing workers, crunching trough the input.