embeddings

README

Embeddings

adapter over various popular vector embeddings interfaces: AWS BedRock, OpenAI, word2vec

sub-module	doc	about
		AWS Bedrock embeddings models
		Caching vector embeddings
		OpenAI embeddings models
		Semantic chunking utility
		Word2Vec embeddings model

---

Inspiration

The library implements generic trait to transform text into vector embeddings.

type Embeddings interface {
	Embedding(ctx context.Context, text string) ([]float32, error)
}

The library is structured from submodules, each implements the defined interface towards vendor.

• AWS BedRock embeddings
• OpenAI Embeddings
• word2vec model

Getting started

The latest version of the library is available at main branch of this repository. All development, including new features and bug fixes, take place on the main branch using forking and pull requests as described in contribution guidelines. The stable version is available via Golang modules.

import embeddings "github.com/kshard/embeddings/{your-model}"

text, err := embeddings.New(/* config options */)

// Calculate embeddings
vector, err := text.Embedding(context.Background(), "text embeddings")

// Checks number of tokens consumed by active sessions
text.ConsumedTokens()

How To Contribute

The library is MIT licensed and accepts contributions via GitHub pull requests:

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Added some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

The build and testing process requires Go version 1.13 or later.

build and test library.

git clone https://github.com/kshard/embeddings
cd embeddings
go test ./...

commit message

The commit message helps us to write a good release note, speed-up review process. The message should address two question what changed and why. The project follows the template defined by chapter Contributing to a Project of Git book.

bugs

If you experience any issues with the library, please let us know via GitHub issues. We appreciate detailed and accurate reports that help us to identity and replicate the issue.

License

Documentation

Index

• Constants
• type Embedder
• type Embedding

Constants

const Version = "v0.2.0"

Types

type Embedder

type Embedder interface {
	UsedTokens() int
	Embedding(ctx context.Context, text string) (Embedding, error)
}

type Embedding

type Embedding struct {
	Text       string
	Vector     []float32
	UsedTokens int
}

Embeddings