xxhash

package module
Version: v1.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 30, 2018 License: Apache-2.0 Imports: 2 Imported by: 373

README

xxhash GoDoc Build Status Coverage

This is a native Go implementation of the excellent xxhash* algorithm, an extremely fast non-cryptographic Hash algorithm, working at speeds close to RAM limits.

  • The C implementation is (Copyright (c) 2012-2014, Yann Collet)

Install

go get github.com/OneOfOne/xxhash

Features

  • On Go 1.7+ the pure go version is faster than CGO for all inputs.
  • Supports ChecksumString{32,64} xxhash{32,64}.WriteString, which uses no copies when it can, falls back to copy on appengine.
  • The native version falls back to a less optimized version on appengine due to the lack of unsafe.
  • Almost as fast as the mostly pure assembly version written by the brilliant cespare, while also supporting seeds.
  • To manually toggle the appengine version build with -tags safe.

Benchmark

Core i7-4790 @ 3.60GHz, Linux 4.12.6-1-ARCH (64bit), Go tip (+ff90f4af66 2017-08-19)
➤ go test -bench '64' -count 5 -tags cespare | benchstat /dev/stdin
name                          time/op

# https://github.com/cespare/xxhash
XXSum64Cespare/Func-8          160ns ± 2%
XXSum64Cespare/Struct-8        173ns ± 1%
XXSum64ShortCespare/Func-8    6.78ns ± 1%
XXSum64ShortCespare/Struct-8  19.6ns ± 2%

# this package (default mode, using unsafe)
XXSum64/Func-8                 170ns ± 1%
XXSum64/Struct-8               182ns ± 1%
XXSum64Short/Func-8           13.5ns ± 3%
XXSum64Short/Struct-8         20.4ns ± 0%

# this package (appengine, *not* using unsafe)
XXSum64/Func-8                 241ns ± 5%
XXSum64/Struct-8               243ns ± 6%
XXSum64Short/Func-8           15.2ns ± 2%
XXSum64Short/Struct-8         23.7ns ± 5%

CRC64ISO-8                    1.23µs ± 1%
CRC64ISOString-8              2.71µs ± 4%
CRC64ISOShort-8               22.2ns ± 3%

Fnv64-8                       2.34µs ± 1%
Fnv64Short-8                  74.7ns ± 8%
#

Usage

	h := xxhash.New64()
	// r, err := os.Open("......")
	// defer f.Close()
	r := strings.NewReader(F)
	io.Copy(h, r)
	fmt.Println("xxhash.Backend:", xxhash.Backend)
	fmt.Println("File checksum:", h.Sum64())

playground

TODO

  • Rewrite the 32bit version to be more optimized.
  • General cleanup as the Go inliner gets smarter.

License

This project is released under the Apache v2. licence. See LICENCE for more details.

Documentation

Index

Constants

View Source
const Backend = "GoUnsafe"

Backend returns the current version of xxhash being used.

Variables

This section is empty.

Functions

func Checksum32

func Checksum32(in []byte) uint32

Checksum32 returns the checksum of the input data with the seed set to 0.

func Checksum32S

func Checksum32S(in []byte, seed uint32) (h uint32)

Checksum32S returns the checksum of the input bytes with the specific seed.

func Checksum64

func Checksum64(in []byte) uint64

Checksum64 an alias for Checksum64S(in, 0)

func Checksum64S

func Checksum64S(in []byte, seed uint64) uint64

Checksum64S returns the 64bit xxhash checksum for a single input

func ChecksumString32

func ChecksumString32(s string) uint32

ChecksumString32 returns the checksum of the input data, without creating a copy, with the seed set to 0.

func ChecksumString32S

func ChecksumString32S(s string, seed uint32) uint32

ChecksumString32S returns the checksum of the input data, without creating a copy, with the specific seed.

func ChecksumString64

func ChecksumString64(s string) uint64

ChecksumString64 returns the checksum of the input data, without creating a copy, with the seed set to 0.

func ChecksumString64S

func ChecksumString64S(s string, seed uint64) uint64

ChecksumString64S returns the checksum of the input data, without creating a copy, with the specific seed.

Types

type XXHash32

type XXHash32 struct {
	// contains filtered or unexported fields
}

func New32

func New32() *XXHash32

New32 creates a new hash.Hash32 computing the 32bit xxHash checksum starting with the seed set to 0.

func NewS32

func NewS32(seed uint32) (xx *XXHash32)

NewS32 creates a new hash.Hash32 computing the 32bit xxHash checksum starting with the specific seed.

func (*XXHash32) BlockSize

func (xx *XXHash32) BlockSize() int

BlockSize returns the hash's underlying block size. The Write method must be able to accept any amount of data, but it may operate more efficiently if all writes are a multiple of the block size.

func (*XXHash32) Reset

func (xx *XXHash32) Reset()

func (*XXHash32) Size

func (xx *XXHash32) Size() int

Size returns the number of bytes Sum will return.

func (*XXHash32) Sum

func (xx *XXHash32) Sum(in []byte) []byte

Sum appends the current hash to b and returns the resulting slice. It does not change the underlying hash state.

func (*XXHash32) Sum32

func (xx *XXHash32) Sum32() (h uint32)

func (*XXHash32) Write

func (xx *XXHash32) Write(in []byte) (n int, err error)

func (*XXHash32) WriteString

func (xx *XXHash32) WriteString(s string) (int, error)

type XXHash64

type XXHash64 struct {
	// contains filtered or unexported fields
}

func New64

func New64() *XXHash64

New64 creates a new hash.Hash64 computing the 64bit xxHash checksum starting with the seed set to 0x0.

func NewS64

func NewS64(seed uint64) (xx *XXHash64)

NewS64 creates a new hash.Hash64 computing the 64bit xxHash checksum starting with the specific seed.

func (*XXHash64) BlockSize

func (xx *XXHash64) BlockSize() int

BlockSize returns the hash's underlying block size. The Write method must be able to accept any amount of data, but it may operate more efficiently if all writes are a multiple of the block size.

func (*XXHash64) Reset

func (xx *XXHash64) Reset()

func (*XXHash64) Size

func (xx *XXHash64) Size() int

Size returns the number of bytes Sum will return.

func (*XXHash64) Sum

func (xx *XXHash64) Sum(in []byte) []byte

Sum appends the current hash to b and returns the resulting slice. It does not change the underlying hash state.

func (*XXHash64) Sum64

func (xx *XXHash64) Sum64() (h uint64)

func (*XXHash64) Write

func (xx *XXHash64) Write(in []byte) (n int, err error)

func (*XXHash64) WriteString

func (xx *XXHash64) WriteString(s string) (int, error)

Directories

Path Synopsis
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL