s3vectors

Documentation

Overview

Package s3vectors contains helper code for maintaining a DynamoDB table that mirrors the records stored in an S3 Vectors index. The table is used by the main database implementation to enable efficient listing by provider or model.

Index

• Constants
• func DynamoDBTables(table_name string, metadata_table_name string) map[string]*dynamodb.CreateTableInput
• type DynamoDBClient
  • func NewDynamoDBClient(ctx context.Context, cfg_uri string) (*DynamoDBClient, error)
  • func (cl *DynamoDBClient) AddModelProviderMetadata(ctx context.Context, model string, provider string) error
  • func (cl *DynamoDBClient) AddRecord(ctx context.Context, rec *embeddingsdb.Record) error
  • func (cl *DynamoDBClient) GetModelsForProvider(ctx context.Context, provider string) ([]string, error)
  • func (cl *DynamoDBClient) GetProvidersForModel(ctx context.Context, model string) ([]string, error)
  • func (cl *DynamoDBClient) GetUniqueMetadataProperty(ctx context.Context, prop string) ([]string, error)
  • func (cl *DynamoDBClient) ListRecordsByModel(ctx context.Context, pg_opts pagination.Options, model string, ...) ([]*DynamoDBRecord, pagination.Results, error)
  • func (cl *DynamoDBClient) ListRecordsByProvider(ctx context.Context, pg_opts pagination.Options, provider string, ...) ([]*DynamoDBRecord, pagination.Results, error)
  • func (cl *DynamoDBClient) RemoveRecord(ctx context.Context, rec *embeddingsdb.Record) error
  • func (cl *DynamoDBClient) SetupTables(ctx context.Context) error
• type DynamoDBRecord

Constants

const DynamoDBTableName string = "s3vectors"

DynamoDBTableName is the default name of the table used to store the record key, provider, model, depiction id and dimension count.

const DynamoDBTableNameMetadata string = "s3vectors_metadata"

Functions

func DynamoDBTables

func DynamoDBTables(table_name string, metadata_table_name string) map[string]*dynamodb.CreateTableInput

Types

type DynamoDBClient

type DynamoDBClient struct {
	// contains filtered or unexported fields
}

DynamoDBClient wraps basic add and remove operations the DynamoDB table used to mirror the S3 Vectors index.

func NewDynamoDBClient

func NewDynamoDBClient(ctx context.Context, cfg_uri string) (*DynamoDBClient, error)

NewDynamoDBClient creates a DynamoDB client using the AWS configuration supplied by cfg_uri.

func (*DynamoDBClient) AddModelProviderMetadata

func (cl *DynamoDBClient) AddModelProviderMetadata(ctx context.Context, model string, provider string) error

func (*DynamoDBClient) AddRecord

func (cl *DynamoDBClient) AddRecord(ctx context.Context, rec *embeddingsdb.Record) error

AddRecord inserts the supplied Record into the DynamoDB table. The function serialises the Record into a DynamoDBRecord and marshals it into a DynamoDB item.

func (*DynamoDBClient) GetModelsForProvider

func (cl *DynamoDBClient) GetModelsForProvider(ctx context.Context, provider string) ([]string, error)

func (*DynamoDBClient) GetProvidersForModel

func (cl *DynamoDBClient) GetProvidersForModel(ctx context.Context, model string) ([]string, error)

func (*DynamoDBClient) GetUniqueMetadataProperty

func (cl *DynamoDBClient) GetUniqueMetadataProperty(ctx context.Context, prop string) ([]string, error)

func (*DynamoDBClient) ListRecordsByModel

func (cl *DynamoDBClient) ListRecordsByModel(ctx context.Context, pg_opts pagination.Options, model string, custom_opts ...options.Option) ([]*DynamoDBRecord, pagination.Results, error)

ListRecordsByModel returns all records that match the given model. The results are paginated according to pg_opts. An optional provider filter may be supplied via options.

func (*DynamoDBClient) ListRecordsByProvider

func (cl *DynamoDBClient) ListRecordsByProvider(ctx context.Context, pg_opts pagination.Options, provider string, custom_opts ...options.Option) ([]*DynamoDBRecord, pagination.Results, error)

ListRecordsByProvider returns all records that match the given provider. The results are paginated according to pg_opts. An optional model filter may be supplied via options.

func (*DynamoDBClient) RemoveRecord

func (cl *DynamoDBClient) RemoveRecord(ctx context.Context, rec *embeddingsdb.Record) error

RemoveRecord deletes the record identified by rec from DynamoDB.

func (*DynamoDBClient) SetupTables

func (cl *DynamoDBClient) SetupTables(ctx context.Context) error

SetupTables creates the necessary DynamoDB tables if they don't not already exist.

type DynamoDBRecord

type DynamoDBRecord struct {
	Key         string
	Provider    string
	Model       string
	DepictionId string
	Dimensions  int
}

DynamoDBRecord represents a minimal record stored in DynamoDB. The primary key is the concatenated provider|model|depiction_id.