fvecs

README

fvecs - vector file formats

The library implement codec for .bvecs, .fvecs and .ivecs vector file formats developed by INRIA LEAR and TEXMEX groups.

Inspiration

Vector file formats (.bvecs, .fvecs and .ivecs) is a serialized sequence of vectors []byte, []float32 or []int32.

Each vector is

      32bit          ⟨𝒅⟩ * n-bits 
|--------------|-------/ ... /-------|
 ⟨𝒅⟩ dimension

• .fvecs stores 32-bit float (n = 32bit)
• .ivecs stores 32-bit signed or unsigned integers (n = 32bits)
• .bvecs stores bytes (n = 8bit)

Getting started

The latest version of the library is available at main branch of this repository. All development, including new features and bug fixes, take place on the main branch using forking and pull requests as described in contribution guidelines. The stable version is available via Golang modules.

import "github.com/kshard/fvecs"

//
// Reading vectors
r, err := os.Open("siftsmall_base.fvecs")
d := fvecs.NewDecoder[float32](r)

for {
  v, err := d.Read()
  if err != nil {
    break
  }
}

//
// Writing vectors
w, err := os.Create("siftsmall_base.fvecs")
e := fvecs.NewEncoder[float32](w)

for _, v := range /* source of []float32 vectors */ {
  err := e.Write(v)
  if err != nil {
    break
  }
}

How To Contribute

The library is MIT licensed and accepts contributions via GitHub pull requests:

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

The build and testing process requires Go version 1.13 or later.

build and test library.

git clone https://github.com/kshard/fvecs
cd fvecs
go test

commit message

The commit message helps us to write a good release note, speed-up review process. The message should address two question what changed and why. The project follows the template defined by chapter Contributing to a Project of Git book.

bugs

If you experience any issues with the library, please let us know via GitHub issues. We appreciate detailed and accurate reports that help us to identity and replicate the issue.

License

Documentation

Index

• type Base
• type Decoder
  • func NewDecoder[T Base](r io.Reader) *Decoder[T]
  • func (d *Decoder[T]) Read() ([]T, error)
• type Encoder
  • func NewEncoder[T Base](w io.Writer) *Encoder[T]
  • func (e *Encoder[T]) Write(v []T) error

Types

type Base

type Base interface {
	float32 | uint32 | byte
}

Vector base types

type Decoder

type Decoder[T Base] struct {
	// contains filtered or unexported fields
}

Vector decoder

func NewDecoder

func NewDecoder[T Base](r io.Reader) *Decoder[T]

New instance of decoder

func (*Decoder[T]) Read

func (d *Decoder[T]) Read() ([]T, error)

Read vector from stream

type Encoder

type Encoder[T Base] struct {
	// contains filtered or unexported fields
}

Vector decoder

func NewEncoder

func NewEncoder[T Base](w io.Writer) *Encoder[T]

New instance of decoder

func (*Encoder[T]) Write

func (e *Encoder[T]) Write(v []T) error

Read vector from stream