tools

package
v0.1.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2017 License: Apache-2.0 Imports: 18 Imported by: 0

README

Testing Tools

WAL-G offers three prototyping programs to assist with testing and development:

compress

compress takes in a directory and minimum part size and creates a compressed tarball.

Example use cases:

Compress a local directory to disk.

./compress -out=$HOME/directory/to/be/compressed 1000000 directory/to/compress

Connect to Postgres and upload the compressed tarball to S3.

./compress -s 1000000 /dat/9.6/data
extract

extract takes in a target out directory and however many files/urls. It is often used in conjunction with generate to test the accuracy and speed of decompression and extraction.

Example use cases:

Extract local files.

./extract /target/out/directory file1 file2

Extract local files and randomly generated data.

./extract /out/directory file1                     \
https://localhost:8080/stride-10.bytes-100.tar.lzo \
https://localhost:8080/stride-100.bytes-1000.tar.lzo
generate

generate outputs randomly compressed tarballs hosted on localhost:8080. To configure the stride length N and size of the data M, use: https://localhost:8080/stride-N.bytes-M.tar.lzo

Randomly generated data can be downloaded using:

curl -k https://localhost:8080/stride-1.bytes-1.tar.lzo \
-o /path/to/new/file

The URLs can also be fed directly to extract. Currently, generate only supports LZOP compression.

NOTE: compress and extract support profiling options using the flags -p and -m. The first generates a CPU profile to cpu.prof while the latter generates a memory profile to mem.prof.

To access the profiles, use:

go tool pprof wal-g FILENAME

Documentation

Index

Constants

View Source 
const LzopBlockSize = 256 * 1024

LzopBlockSize for modern architectures.

View Source 
const LzopPrefix = "\x89\x4c\x5a\x4f\x00\x0d\x0a\x1a\x0a\x10\x30\x20\xa0\x09\x40" +
	"\x01\x05\x03\x00\x00\x01\x00\x00\x81\xa4\x59\x43\x06\xd0\x00" +
	"\x00\x00\x00\x06\x70\x32\x2e\x74\x61\x72\x51\xf8\x06\x08"

LzopPrefix consists of arbitrary bytes at a specified length to satisfy lzop header format.

Variables

This section is empty.

Functions

func CreateTar 

func CreateTar(w io.Writer, r *io.LimitedReader)

CreateTar creates a new tarball from the passed in reader and writes to a destination writer.

func Handler 

func Handler(w http.ResponseWriter, r *http.Request)

Handler allows for generation of random bytes by configuring the URL 'https://localhost:8080/stride-N.bytes-N.tar.lzo' where byte size and stride length are customizable.

Compressed tar files are automatically generated. Grab using curl ie. 'curl -sk ...'

func MakeDir 

func MakeDir(name string)

MakeDir creates a new directory with mode 0755.

func TimeTrack 

func TimeTrack(start time.Time, name string)

TimeTrack is used to time how long functions take.

Usage Example: defer timeTrack(time.Now(), "EXTRACT ALL")

Types

type BufferTarInterpreter 

type BufferTarInterpreter struct {
	Out []byte
}

BufferTarInterpreter extracts data to a byte slice. Used for testing purposes.

func (*BufferTarInterpreter) Interpret 

func (ti *BufferTarInterpreter) Interpret(tr io.Reader, cur *tar.Header) error

Interpret handles in memory tar formats. Used for testing purposes.

type FileLzWriter 

type FileLzWriter struct {
	Input io.Reader
	Name  string
}

FileLzWriter takes in input and compresses it to local disk.

func (*FileLzWriter) Compress 

func (f *FileLzWriter) Compress()

Compress uses lz4 to compress bytes to disk.

func (*FileLzWriter) Writer 

func (f *FileLzWriter) Writer() io.WriteCloser

Writer creates a new FileLzWriter.

type FileReaderMaker 

type FileReaderMaker struct {
	Key        string
	FileFormat string
}

FileReaderMaker decompresses lzop tarballs from the passed in file.

func (*FileReaderMaker) Format 

func (f *FileReaderMaker) Format() string

func (*FileReaderMaker) Path 

func (f *FileReaderMaker) Path() string

func (*FileReaderMaker) Reader 

func (f *FileReaderMaker) Reader() (io.ReadCloser, error)

Reader creates a new reader from the passed in file.

type FileTarBall 

type FileTarBall struct {
	// contains filtered or unexported fields
}

FileTarBall represents a tarball that is written to disk.

func (*FileTarBall) BaseDir 

func (fb *FileTarBall) BaseDir() string

func (*FileTarBall) CloseTar 

func (fb *FileTarBall) CloseTar() error

CloseTar closes the tar writer and file, flushing any unwritten data to the file before closing.

func (*FileTarBall) Finish 

func (fb *FileTarBall) Finish() error

Finish alerts that compression is complete.

func (*FileTarBall) Nop 

func (fb *FileTarBall) Nop() bool

func (*FileTarBall) Number 

func (fb *FileTarBall) Number() int

func (*FileTarBall) SetSize 

func (fb *FileTarBall) SetSize(i int64)

func (*FileTarBall) SetUp 

func (fb *FileTarBall) SetUp(names ...string)

SetUp creates a new LZ4 writer, tar writer and file for writing bundled compressed bytes to.

func (*FileTarBall) Size 

func (fb *FileTarBall) Size() int64

func (*FileTarBall) Trim 

func (fb *FileTarBall) Trim() string

func (*FileTarBall) Tw 

func (fb *FileTarBall) Tw() *tar.Writer

type FileTarBallMaker 

type FileTarBallMaker struct {
	BaseDir string
	Trim    string
	Out     string
	// contains filtered or unexported fields
}

FileTarBallMaker creates a new FileTarBall with the directory that files should be extracted to.

func (*FileTarBallMaker) Make 

func (f *FileTarBallMaker) Make() walg.TarBall

Make creates a new FileTarBall.

type HTTPReaderMaker 

type HTTPReaderMaker struct {
	Client     *http.Client
	Key        string
	FileFormat string
}

HTTPReaderMaker decompresses lzop tarballs from the passed in url.

func (*HTTPReaderMaker) Format 

func (h *HTTPReaderMaker) Format() string

func (*HTTPReaderMaker) Path 

func (h *HTTPReaderMaker) Path() string

func (*HTTPReaderMaker) Reader 

func (h *HTTPReaderMaker) Reader() (io.ReadCloser, error)

Reader creates a new request to grab the data generated by the random bytes generator.

type LzopReader 

type LzopReader struct {
	Uncompressed io.Reader
	// contains filtered or unexported fields
}

LzopReader takes in uncompressed bytes and compresses using lzo.

func (*LzopReader) Read 

func (lz *LzopReader) Read(p []byte) (n int, err error)

Read compresses byte slice `p` matching lzop format. Does not compress with a checksum.

type NOPTarBall 

type NOPTarBall struct {
	// contains filtered or unexported fields
}

NOPTarBall mocks a tarball. Used for testing purposes.

func (*NOPTarBall) BaseDir 

func (n *NOPTarBall) BaseDir() string

func (*NOPTarBall) CloseTar 

func (n *NOPTarBall) CloseTar() error

func (*NOPTarBall) Finish 

func (n *NOPTarBall) Finish() error

func (*NOPTarBall) Nop 

func (n *NOPTarBall) Nop() bool

func (*NOPTarBall) Number 

func (n *NOPTarBall) Number() int

func (*NOPTarBall) SetSize 

func (n *NOPTarBall) SetSize(i int64)

func (*NOPTarBall) SetUp 

func (n *NOPTarBall) SetUp(params ...string)

func (*NOPTarBall) Size 

func (n *NOPTarBall) Size() int64

func (*NOPTarBall) Trim 

func (n *NOPTarBall) Trim() string

func (*NOPTarBall) Tw 

func (n *NOPTarBall) Tw() *tar.Writer

type NOPTarBallMaker 

type NOPTarBallMaker struct {
	BaseDir string
	Trim    string
	Nop     bool
	// contains filtered or unexported fields
}

NOPTarBallMaker creates a new NOPTarBall. Used for testing purposes.

func (*NOPTarBallMaker) Make 

func (n *NOPTarBallMaker) Make() walg.TarBall

Make creates a new NOPTarBall.

type NOPTarInterpreter 

type NOPTarInterpreter struct{}

NOPTarInterpreter mocks a tar extractor.

func (*NOPTarInterpreter) Interpret 

func (ti *NOPTarInterpreter) Interpret(tr io.Reader, cur *tar.Header) error

Interpret does not do anything except print the 'tar member' name.

type StrideByteReader 

type StrideByteReader struct {
	// contains filtered or unexported fields
}

StrideByteReader allows for customizable "strides" of random bytes. Creates an infinite stream.

func NewStrideByteReader 

func NewStrideByteReader(s int) *StrideByteReader

NewStrideByteReader creates a new random byte stride generator with a seed of 0.

func (*StrideByteReader) Read 

func (sb *StrideByteReader) Read(p []byte) (int, error)

Read creates randomly generated bytes of 'stride' length.

Source Files

View all Source files

Directories

Path Synopsis
cmd
compress
extract
generate

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.