vector

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 11, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

README

pg/vector

The pgvector extension for sqlb, the whole query surface of the pgvector README and the types of pgvector-go. Needs CREATE EXTENSION vector. Index DDL (hnsw, ivfflat, operator classes, partial and expression indexes) is versioned SQL the application applies; every query shape those indexes serve is here, as a checked example in example_test.go.

Types

pgvector Go Wire form
vector(n) Vector, a []float32 [1,2,3]
halfvec(n) HalfVector, a []float32; the server rounds to half precision [1,2,3]
sparsevec(n) SparseVector{Dimensions, Indices, Values}, NewSparse(dense) {2:1.5,4:2}/5, 1-based on the wire, 0-based in Go like pgvector-go
bit(n) BitVector, a string of 0 and 1 0101

All four implement Scan and Value, so database/sql and pgx carry them as text without a codec. Dense is the type set with distance operators (Vector | HalfVector | SparseVector); Elementwise is the set with arithmetic and subvector (Vector | HalfVector).

Operators and functions

SQL Helper On
a <-> b L2 L2 Dense
a <=> b cosine Cosine Dense
a <#> b negative inner product InnerProduct Dense
a <+> b L1 L1 Dense
a <~> b Hamming, a <%> b Jaccard Hamming, Jaccard BitVector
l2_distance, cosine_distance, inner_product (positive), l1_distance L2Distance, CosineDistance, InnerProductOf, L1Distance Dense
hamming_distance, jaccard_distance HammingDistance, JaccardDistance BitVector
vector_dims, vector_norm Dims, Norm Vector
l2_norm L2Norm HalfVector, SparseVector
l2_normalize L2Normalize Dense
subvector(v, start, count) Subvector Elementwise
binary_quantize(v) BinaryQuantize, returns BitVector; cast with CastBit Elementwise
sum(v), avg(v) element-wise Sum, Avg Dense; Avg on Elementwise
a + b, a - b, a * b, a || b Add, Sub, Mul, Concat Dense; Concat on Elementwise
e::vector(n), ::halfvec(n), ::sparsevec(n), ::bit(n) CastVector, CastHalf, CastSparse, CastBit; dims 0 omits (n) any expression
$n::vector(n) bound with its dimension ParamVector, ParamHalf, ParamSparse, ParamBit values

Every distance is an Expr[float64]: one expression is the SELECT item, the WHERE operand and the ORDER BY term. Under a dialect other than Postgres, Build returns sqlb.ErrUnsupportedDialect naming the construct.

A bare $n has no type until Postgres sees the other operand. Against a column that is fine, so sqlb.Param(v) works in L2(col, sqlb.Param(v)). As a function argument it is not, so ParamVector adds the cast with the dimension taken from the value.

Query-time settings

SET cannot take a bind parameter; set_config(name, value, true) can, and lasts for the transaction like SET LOCAL. Run the returned expression in a SELECT inside the transaction that runs the search.

Setting Helper
hnsw.ef_search SetEfSearch(n)
ivfflat.probes SetProbes(n)
hnsw.iterative_scan, ivfflat.iterative_scan SetHNSWIterativeScan(mode), SetIVFFlatIterativeScan(mode) with ScanOff, ScanStrictOrder, ScanRelaxedOrder
hnsw.max_scan_tuples, ivfflat.max_probes SetMaxScanTuples(n), SetMaxProbes(n)
enable_seqscan = off, enable_indexscan = off ForceIndex(), ForceExact()

The general form is pg.SetLocal(name, value) and pg.CurrentSetting(name).

The README's queries, as examples

Each is an Example_* in example_test.go and go test checks the SQL.

pgvector README section Example
Insert vectors Example_insert
Nearest neighbours by L2, ORDER BY LIMIT Example_nearest
Nearest neighbours to a row Example_nearestToRow
Cosine similarity, inner product, distance in WHERE Example_distances
Filtering with an approximate index, iterative scans, ef_search Example_filteredWithIterativeScan
Relaxed order with strict results, MATERIALIZED CTE Example_relaxedOrderStrictResult
Half-precision vectors and half-precision indexing by cast Example_halfPrecision
Binary quantization with re-ranking Example_binaryQuantizeRerank
Binary vectors, Hamming and Jaccard Example_binaryVectors
Subvector indexing with re-ranking Example_subvectorRerank
Sparse vectors Example_sparse
Aggregates and arithmetic Example_aggregatesAndArithmetic
Cast from a float array Example_castFromArray
Hybrid search with Reciprocal Rank Fusion Example_hybridSearchRRF

Not here, and why: CREATE INDEX and its parameters are DDL; COPY bulk load is a driver feature (pgx CopyFrom); SET maintenance_work_mem and the parallel worker settings are for index builds, not queries, and pg.SetLocal renders them if a migration runner wants them; Citus distribution is DDL. pgvector-go's pgx binary codec is a driver registration for the integration module, not a query construct.

Documentation

Overview

Package vector is the pgvector extension: the four vector types, every distance operator and function, the utility functions, aggregates, arithmetic, casts, and the query-time settings. Needs CREATE EXTENSION vector on the server. Wire forms are pgvector's text I/O, so any database/sql driver carries them without a codec.

Example (AggregatesAndArithmetic)

Aggregates and arithmetic: the centroid of a category, and a query vector shifted by another. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")

	cat = sqlb.Col[int32](items, "category_id")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	show(sel().Exprs(cat, sqlb.As(vector.Avg(emb), "centroid"), sqlb.As(vector.Sum(emb), "total")).From(items).GroupBy(cat).Build())
	q := vector.ParamVector(vector.Vector{3, 1, 2})
	shift := vector.ParamVector(vector.Vector{0, 0, 1})
	show(sel().Exprs(id).From(items).
		Order(sqlb.OrderBy(vector.L2(emb, vector.L2Normalize(vector.Add(q, shift))), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "items"."category_id", avg("items"."embedding") AS "centroid", sum("items"."embedding") AS "total" FROM "items" AS "items" GROUP BY "items"."category_id"
args: []
SELECT "items"."id" FROM "items" AS "items" ORDER BY ("items"."embedding" <-> l2_normalize(($1::vector(3) + $2::vector(3)))) ASC LIMIT $3
args: [[3 1 2] [0 0 1] 5]
Example (BinaryQuantizeRerank)

Binary quantization with re-ranking: the bit index finds candidates by Hamming distance, the full vector re-ranks them. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamVector(vector.Vector{3, 1, 2})
	candidates := sqlb.Derived(sel().Exprs(id, emb).From(items).
		Order(sqlb.OrderBy(vector.Hamming(vector.CastBit(vector.BinaryQuantize(emb), 3), vector.BinaryQuantize(q)), sqlb.SortAsc)).
		Limit(20), "c")
	c := sqlb.Ref(candidates)
	cID, cEmb := sqlb.Col[int64](c, "id"), sqlb.Col[vector.Vector](c, "embedding")
	show(sel().Exprs(cID).From(candidates).Order(sqlb.OrderBy(vector.L2(cEmb, q), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "c"."id" FROM (SELECT "items"."id", "items"."embedding" FROM "items" AS "items" ORDER BY (binary_quantize("items"."embedding")::bit(3) <~> binary_quantize($1::vector(3))) ASC LIMIT $2) AS "c" ORDER BY ("c"."embedding" <-> $3::vector(3)) ASC LIMIT $4
args: [[3 1 2] 20 [3 1 2] 5]
Example (BinaryVectors)

A native bit column: Hamming and Jaccard directly. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")

	bits = sqlb.Col[vector.BitVector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamBit("101")
	show(sel().Exprs(id, vector.HammingDistance(bits, q)).From(items).
		Order(sqlb.OrderBy(vector.Jaccard(bits, q), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "items"."id", hamming_distance("items"."embedding", $1::bit(3)) FROM "items" AS "items" ORDER BY ("items"."embedding" <%> $2::bit(3)) ASC LIMIT $3
args: [101 101 5]
Example (CastFromArray)

A float array column or literal becomes a vector by cast, the README's way in from an existing schema. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	arr := sqlb.Col[[]float32](items, "arr")
	show(sel().Exprs(id, vector.Dims(vector.CastVector(arr, 0))).From(items).
		Where(sqlb.Less(vector.L2(vector.CastVector(arr, 3), vector.ParamVector(vector.Vector{1, 2, 3})), 1.0)).Build())
}

Output:
SELECT "items"."id", vector_dims("items"."arr"::vector) FROM "items" AS "items" WHERE (("items"."arr"::vector(3) <-> $1::vector(3)) < $2)
args: [[1 2 3] 1]
Example (Distances)

Cosine similarity as 1 minus cosine distance, inner product negated back to positive, and a distance filter in WHERE. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamVector(vector.Vector{3, 1, 2})
	sim := sqlb.Binary[float64](sqlb.Param(1.0), sqlb.OpSub, vector.Cosine(emb, q))
	ip := sqlb.Binary[float64](vector.InnerProduct(emb, q), sqlb.OpMul, sqlb.Param(-1.0))
	show(sel().Exprs(id, sqlb.As(sim, "cosine_similarity"), sqlb.As(ip, "inner_product")).From(items).
		Where(sqlb.Less(vector.L2(emb, q), 5.0)).Build())
}

Output:
SELECT "items"."id", ($1 - ("items"."embedding" <=> $2::vector(3))) AS "cosine_similarity", (("items"."embedding" <#> $3::vector(3)) * $4) AS "inner_product" FROM "items" AS "items" WHERE (("items"."embedding" <-> $5::vector(3)) < $6)
args: [1 [3 1 2] [3 1 2] -1 [3 1 2] 5]
Example (FilteredWithIterativeScan)

Filtering with an approximate index: a WHERE on another column, and the iterative scan setting bound for the transaction so the index keeps scanning until LIMIT rows pass the filter. Two statements, one transaction. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")

	cat = sqlb.Col[int32](items, "category_id")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	show(sel().Exprs(vector.SetHNSWIterativeScan(vector.ScanRelaxedOrder), vector.SetEfSearch(100)).Build())
	q := vector.ParamVector(vector.Vector{3, 1, 2})
	show(sel().Exprs(id).From(items).Where(sqlb.Eq(cat, int32(123))).
		Order(sqlb.OrderBy(vector.L2(emb, q), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT set_config('hnsw.iterative_scan', $1, true), set_config('hnsw.ef_search', $2, true)
args: [relaxed_order 100]
SELECT "items"."id" FROM "items" AS "items" WHERE ("items"."category_id" = $1) ORDER BY ("items"."embedding" <-> $2::vector(3)) ASC LIMIT $3
args: [123 [3 1 2] 5]
Example (HalfPrecision)

Half-precision: the column is halfvec, or a vector column is cast to halfvec so an expression index on the cast serves the search. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
	half  = sqlb.Col[vector.HalfVector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamHalf(vector.HalfVector{3, 1, 2})
	show(sel().Exprs(id).From(items).
		Order(sqlb.OrderBy(vector.L2(vector.CastHalf(emb, 3), q), sqlb.SortAsc)).Limit(5).Build())
	show(sel().Exprs(id, vector.L2Norm(half)).From(items).
		Order(sqlb.OrderBy(vector.Cosine(half, q), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "items"."id" FROM "items" AS "items" ORDER BY ("items"."embedding"::halfvec(3) <-> $1::halfvec(3)) ASC LIMIT $2
args: [[3 1 2] 5]
SELECT "items"."id", l2_norm("items"."embedding") FROM "items" AS "items" ORDER BY ("items"."embedding" <=> $1::halfvec(3)) ASC LIMIT $2
args: [[3 1 2] 5]
Example (HybridSearchRRF)

Hybrid search with Reciprocal Rank Fusion, the README's example: a semantic CTE ranked by cosine distance, a keyword CTE ranked by ts_rank_cd, a FULL JOIN on id, and 1/(k+rank) summed. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	docs := sqlb.Table{Name: "documents", Alias: "documents"}
	dID, content, dEmb := sqlb.Col[int64](docs, "id"), sqlb.Col[string](docs, "content"), sqlb.Col[vector.Vector](docs, "embedding")

	q := vector.ParamVector(vector.Vector{3, 1, 2})
	dist := vector.Cosine(dEmb, q)
	semantic := sel().Exprs(dID, sqlb.As(sqlb.Rank().Over(sqlb.NewWindow().OrderBy(sqlb.OrderBy(dist, sqlb.SortAsc))), "rank")).
		From(docs).Order(sqlb.OrderBy(dist, sqlb.SortAsc)).Limit(20)

	tsq := pg.PlainToTSQuery("english", "postgres vector search")
	tsv := pg.ToTSVector("english", content)
	score := pg.TSRankCD(tsv, tsq)
	keyword := sel().Exprs(dID, sqlb.As(sqlb.Rank().Over(sqlb.NewWindow().OrderBy(sqlb.OrderBy(score, sqlb.SortDesc))), "rank")).
		From(docs).Where(pg.Matches(tsv, tsq)).Order(sqlb.OrderBy(score, sqlb.SortDesc)).Limit(20)

	s := sqlb.Table{Name: "semantic_search", Alias: "s"}
	k := sqlb.Table{Name: "keyword_search", Alias: "k"}
	sID, sRank := sqlb.Col[int64](s, "id"), sqlb.Nullable(sqlb.Col[int64](s, "rank"))
	kID, kRank := sqlb.Col[int64](k, "id"), sqlb.Nullable(sqlb.Col[int64](k, "rank"))
	// 1/(k+rank) is NULL for a side the FULL JOIN did not match; COALESCE makes it 0.
	rrf := func(rank sqlb.Expr[*int64]) sqlb.Expr[float64] {
		score := sqlb.Binary[*float64](sqlb.Param(1.0), sqlb.OpDiv, sqlb.Binary[*float64](sqlb.Param(60), sqlb.OpAdd, rank))
		return sqlb.AssertNotNull(sqlb.Coalesce(score, sqlb.Nullable(sqlb.Param(0.0))))
	}
	total := sqlb.Binary[float64](rrf(sRank), sqlb.OpAdd, rrf(kRank))
	show(sel().With("semantic_search", semantic).With("keyword_search", keyword).
		Exprs(sqlb.As(sqlb.Coalesce(sqlb.Nullable(sID), sqlb.Nullable(kID)), "id"), sqlb.As(total, "score")).
		From(s).Join(sqlb.Join{Type: pg.JoinFull, Table: k, On: []sqlb.Predicate{sqlb.EqExpr(sID, kID)}}).
		Order(sqlb.OrderBy(sqlb.OutputCol[float64]("score"), sqlb.SortDesc)).Limit(5).Build())
}

Output:
WITH "semantic_search" AS (SELECT "documents"."id", rank() OVER (ORDER BY ("documents"."embedding" <=> $1::vector(3)) ASC) AS "rank" FROM "documents" AS "documents" ORDER BY ("documents"."embedding" <=> $2::vector(3)) ASC LIMIT $3),
"keyword_search" AS (SELECT "documents"."id", rank() OVER (ORDER BY ts_rank_cd(to_tsvector('english', "documents"."content"), plainto_tsquery('english', $4)) DESC) AS "rank" FROM "documents" AS "documents" WHERE ((to_tsvector('english', "documents"."content") @@ plainto_tsquery('english', $5))) ORDER BY ts_rank_cd(to_tsvector('english', "documents"."content"), plainto_tsquery('english', $6)) DESC LIMIT $7)
SELECT COALESCE("s"."id", "k"."id") AS "id", (COALESCE(($8 / ($9 + "s"."rank")), $10) + COALESCE(($11 / ($12 + "k"."rank")), $13)) AS "score" FROM "semantic_search" AS "s" FULL JOIN "keyword_search" AS "k" ON ("s"."id" = "k"."id") ORDER BY "score" DESC LIMIT $14
args: [[3 1 2] [3 1 2] 20 postgres vector search postgres vector search postgres vector search 20 1 60 0 1 60 0 5]
Example (Insert)

Insert vectors: pgvector-go's first example. The Vector value renders in text form and the driver sends it as a string. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}

	emb = sqlb.Col[vector.Vector](items, "embedding")
)

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	show(sqlb.NewInsert(d).Into(items).
		Row(sqlb.Set(emb, vector.Vector{1, 2, 3})).
		Row(sqlb.Set(emb, vector.Vector{4, 5, 6})).Build())
}

Output:
INSERT INTO "items" AS "items" ("embedding") VALUES ($1), ($2)
args: [[1 2 3] [4 5 6]]
Example (Nearest)

Nearest neighbours by L2: ORDER BY distance LIMIT n is the shape an HNSW or IVFFlat index serves. The same expression is the SELECT item. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamVector(vector.Vector{3, 1, 2})
	dist := vector.L2(emb, q)
	show(sel().Exprs(id, sqlb.As(dist, "distance")).From(items).
		Order(sqlb.OrderBy(dist, sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "items"."id", ("items"."embedding" <-> $1::vector(3)) AS "distance" FROM "items" AS "items" ORDER BY ("items"."embedding" <-> $2::vector(3)) ASC LIMIT $3
args: [[3 1 2] [3 1 2] 5]
Example (NearestToRow)

Nearest neighbours to a row: the query vector is a subquery. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	target := sqlb.AssertNotNull(sqlb.Subquery[vector.Vector](sel().Exprs(emb).From(items).Where(sqlb.Eq(id, int64(1)))))
	dist := vector.L2(emb, target)
	show(sel().Exprs(id).From(items).Where(sqlb.NotEq(id, int64(1))).
		Order(sqlb.OrderBy(dist, sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "items"."id" FROM "items" AS "items" WHERE ("items"."id" <> $1) ORDER BY ("items"."embedding" <-> (SELECT "items"."embedding" FROM "items" AS "items" WHERE ("items"."id" = $2))) ASC LIMIT $3
args: [1 1 5]
Example (RelaxedOrderStrictResult)

A relaxed-order scan can return rows slightly out of order. The README's fix: a MATERIALIZED CTE for the scan and an outer ORDER BY. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamVector(vector.Vector{3, 1, 2})
	dist := vector.L2(emb, q)
	inner := sel().Exprs(id, sqlb.As(dist, "distance")).From(items).
		Order(sqlb.OrderBy(dist, sqlb.SortAsc)).Limit(5)
	relaxed := sqlb.Table{Name: "relaxed_results", Alias: "relaxed_results"}
	rDist := sqlb.Col[float64](relaxed, "distance")
	show(sel().WithCTE(sqlb.CTE{Name: "relaxed_results", Body: inner, Prefix: pg.Materialized(true)}).
		Exprs(sqlb.Col[int64](relaxed, "id"), rDist).From(relaxed).
		Order(sqlb.OrderBy(rDist, sqlb.SortAsc)).Build())
}

Output:
WITH "relaxed_results" AS MATERIALIZED (SELECT "items"."id", ("items"."embedding" <-> $1::vector(3)) AS "distance" FROM "items" AS "items" ORDER BY ("items"."embedding" <-> $2::vector(3)) ASC LIMIT $3)
SELECT "relaxed_results"."id", "relaxed_results"."distance" FROM "relaxed_results" AS "relaxed_results" ORDER BY "relaxed_results"."distance" ASC
args: [[3 1 2] [3 1 2] 5]
Example (Sparse)

Sparse vectors: the same operators over sparsevec, with the text form {index:value,...}/dims bound as the parameter. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")

	spv = sqlb.Col[vector.SparseVector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamSparse(vector.NewSparse([]float32{0, 1, 0, 2, 0}))
	show(sel().Exprs(id).From(items).Order(sqlb.OrderBy(vector.L2(spv, q), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "items"."id" FROM "items" AS "items" ORDER BY ("items"."embedding" <-> $1::sparsevec(5)) ASC LIMIT $2
args: [{2:1,4:2}/5 5]
Example (SubvectorRerank)

Subvector indexing: an index on the first dimensions of a Matryoshka embedding serves the candidate scan, the full vector re-ranks. 

package main

import (
	"fmt"
	"strings"

	"github.com/ubgo/sqlb"
	"github.com/ubgo/sqlb/pg/vector"
)

var (
	d     = sqlb.PostgresDialect{}
	items = sqlb.Table{Name: "items", Alias: "items"}
	id    = sqlb.Col[int64](items, "id")
	emb   = sqlb.Col[vector.Vector](items, "embedding")
)

func sel() *sqlb.Select[sqlb.PostgresDialect] { return sqlb.NewSelect(d) }

func show(q sqlb.Query, err error) {
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	sql := strings.ReplaceAll(q.SQL, ") SELECT ", ")\nSELECT ")
	sql = strings.ReplaceAll(sql, `), "`, "),\n\"")
	fmt.Println(sql)
	fmt.Println("args:", q.Args)
}

func main() {
	q := vector.ParamVector(vector.Vector{3, 1, 2, 4})
	head := vector.CastVector(vector.Subvector(emb, 1, 2), 2)
	qHead := vector.CastVector(vector.Subvector(q, 1, 2), 2)
	candidates := sqlb.Derived(sel().Exprs(id, emb).From(items).
		Order(sqlb.OrderBy(vector.Cosine(head, qHead), sqlb.SortAsc)).Limit(20), "c")
	c := sqlb.Ref(candidates)
	cID, cEmb := sqlb.Col[int64](c, "id"), sqlb.Col[vector.Vector](c, "embedding")
	show(sel().Exprs(cID).From(candidates).Order(sqlb.OrderBy(vector.Cosine(cEmb, q), sqlb.SortAsc)).Limit(5).Build())
}

Output:
SELECT "c"."id" FROM (SELECT "items"."id", "items"."embedding" FROM "items" AS "items" ORDER BY (subvector("items"."embedding", $1, $2)::vector(2) <=> subvector($3::vector(4), $4, $5)::vector(2)) ASC LIMIT $6) AS "c" ORDER BY ("c"."embedding" <=> $7::vector(4)) ASC LIMIT $8
args: [1 2 [3 1 2 4] 1 2 20 [3 1 2 4] 5]

Index

Examples

Constants

View Source 
const (
	HNSW    pg.IndexMethod = "hnsw"
	IVFFlat pg.IndexMethod = "ivfflat"
)

HNSW and IVFFlat are pgvector's index methods. HNSW builds slower and answers better; IVFFlat needs rows in the table before it is built, because it clusters what it finds.

View Source 
const (

	// Settings pgvector reads at query time. SetEfSearch and friends bind
	// them for the transaction through set_config(..., true).
	SettingHNSWEfSearch          = "hnsw.ef_search"
	SettingHNSWIterativeScan     = "hnsw.iterative_scan"
	SettingHNSWMaxScanTuples     = "hnsw.max_scan_tuples"
	SettingHNSWScanMemMultiplier = "hnsw.scan_mem_multiplier"
	SettingIVFFlatProbes         = "ivfflat.probes"
	SettingIVFFlatIterativeScan  = "ivfflat.iterative_scan"
	SettingIVFFlatMaxProbes      = "ivfflat.max_probes"
	SettingEnableSeqscan         = "enable_seqscan"
	SettingEnableIndexscan       = "enable_indexscan"
)

Construct names, as they appear in ErrUnsupportedDialect.

Variables

View Source 
var (
	CosineOps     = pg.NewOpClass[Vector]("vector_cosine_ops")
	L2Ops         = pg.NewOpClass[Vector]("vector_l2_ops")
	IPOps         = pg.NewOpClass[Vector]("vector_ip_ops")
	L1Ops         = pg.NewOpClass[Vector]("vector_l1_ops")
	HalfCosineOps = pg.NewOpClass[HalfVector]("halfvec_cosine_ops")
	HalfL2Ops     = pg.NewOpClass[HalfVector]("halfvec_l2_ops")
	HalfIPOps     = pg.NewOpClass[HalfVector]("halfvec_ip_ops")
)

The operator classes, one per distance, typed to the vector kind they index: an index is used only by a query ordering by the same distance.

View Source 
var ErrFormat = errors.New("vector: value is not in pgvector text form")

ErrFormat is returned by Scan for a value not in the type's text form.

View Source 
var Extension = pg.Extension{Name: "vector"}

Extension is the pgvector extension this package's constructs need, for pg.CreateExtension(Extension), so a library installs what it uses without spelling its name.

Functions

func Add 

func Add[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[V]

Add renders a + b, element-wise. The two operands must have equal dimensions or the server raises an error.

func Avg 

func Avg[V Elementwise](v sqlb.Expr[V], opts ...sqlb.AggOption) sqlb.Expr[V]

Avg renders avg(v), the element-wise mean: the centroid of a group. There is no avg(sparsevec) in pgvector, so this takes Elementwise only.

func BinaryQuantize 

func BinaryQuantize[V Elementwise](v sqlb.Expr[V]) sqlb.Expr[BitVector]

BinaryQuantize renders binary_quantize(v): one bit per element, positive is 1. Cast it with CastBit to compare with a bit(n) column.

func CastBit 

func CastBit[T any](e sqlb.Expr[T], dims int) sqlb.Expr[BitVector]

CastBit renders e::bit(dims); dims is required, a bare bit is bit(1).

func CastHalf 

func CastHalf[T any](e sqlb.Expr[T], dims int) sqlb.Expr[HalfVector]

CastHalf renders e::halfvec(dims). An expression index on this cast is how a vector column gets a half-precision index.

func CastSparse 

func CastSparse[T any](e sqlb.Expr[T], dims int) sqlb.Expr[SparseVector]

CastSparse renders e::sparsevec(dims).

func CastVector 

func CastVector[T any](e sqlb.Expr[T], dims int) sqlb.Expr[Vector]

CastVector renders e::vector(dims). e may be a float array, text, or another vector type.

func Concat 

func Concat[V Elementwise](a, b sqlb.Expr[V]) sqlb.Expr[V]

Concat renders a || b: the two vectors end to end, so the result has the sum of their dimensions.

func Cosine 

func Cosine[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

Cosine renders a <=> b, the cosine distance.

func CosineDistance 

func CosineDistance[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

CosineDistance renders cosine_distance(a, b), the function form of <=>.

func Dims 

func Dims(v sqlb.Expr[Vector]) sqlb.Expr[int32]

Dims renders vector_dims(v), the element count. It is for vector only.

func EfConstruction 

func EfConstruction(n int64) pg.IndexParam

EfConstruction is HNSW's candidate list while building, 64 by default.

func ForceExact 

func ForceExact() sqlb.Expr[string]

ForceExact binds enable_indexscan = off, so the search is exact and slow. It is how a recall measurement gets its ground truth.

func ForceIndex 

func ForceIndex() sqlb.Expr[string]

ForceIndex binds enable_seqscan = off, so the planner takes the vector index. It is for proving an index is used, never for production traffic.

func Hamming 

func Hamming(a, b sqlb.Expr[BitVector]) sqlb.Expr[float64]

Hamming renders a <~> b on binary vectors.

func HammingDistance 

func HammingDistance(a, b sqlb.Expr[BitVector]) sqlb.Expr[float64]

HammingDistance renders hamming_distance(a, b), the function form of <~>.

func InnerProduct 

func InnerProduct[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

InnerProduct renders a <#> b, the NEGATIVE inner product, because Postgres indexes support ascending order only. InnerProductOf is the positive function form.

func InnerProductOf 

func InnerProductOf[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

InnerProductOf renders inner_product(a, b), the positive value.

func Jaccard 

func Jaccard(a, b sqlb.Expr[BitVector]) sqlb.Expr[float64]

Jaccard renders a <%> b on binary vectors.

func JaccardDistance 

func JaccardDistance(a, b sqlb.Expr[BitVector]) sqlb.Expr[float64]

JaccardDistance renders jaccard_distance(a, b), the function form of <%>.

func L1 

func L1[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

L1 renders a <+> b, the taxicab distance.

func L1Distance 

func L1Distance[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

L1Distance renders l1_distance(a, b), the function form of <+>.

func L2 

func L2[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

L2 renders a <-> b, the Euclidean distance.

func L2Distance 

func L2Distance[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[float64]

L2Distance renders l2_distance(a, b), the same number as the <-> operator. The function form goes where an operator cannot: a function argument, a CASE branch, a check constraint. Only the operator uses an index.

func L2Norm 

func L2Norm[V HalfVector | SparseVector](v sqlb.Expr[V]) sqlb.Expr[float64]

L2Norm renders l2_norm(v), the Euclidean length of a halfvec or sparsevec. Postgres names this function differently per type, hence the two helpers.

func L2Normalize 

func L2Normalize[V Dense](v sqlb.Expr[V]) sqlb.Expr[V]

L2Normalize renders l2_normalize(v): the unit-length vector.

func Lists 

func Lists(n int64) pg.IndexParam

Lists is IVFFlat's cluster count; rows/1000 is pgvector's starting point.

func M 

func M(n int64) pg.IndexParam

M is HNSW's connections per layer, 16 by default: higher answers better and builds slower.

func Mul 

func Mul[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[V]

Mul renders a * b, element-wise, not a dot product. InnerProductOf is the dot product.

func Norm 

func Norm(v sqlb.Expr[Vector]) sqlb.Expr[float64]

Norm renders vector_norm(v), the Euclidean length. It is for vector only; L2Norm is the halfvec and sparsevec spelling of the same thing.

func ParamBit 

func ParamBit(v BitVector) sqlb.Expr[BitVector]

ParamBit binds v as $n::bit(len), the length taken from the string.

func ParamHalf 

func ParamHalf(v HalfVector) sqlb.Expr[HalfVector]

ParamHalf binds v as $n::halfvec(len).

func ParamSparse 

func ParamSparse(v SparseVector) sqlb.Expr[SparseVector]

ParamSparse binds v as $n::sparsevec(dims), the dimension count taken from the value, not from the number of non-zero elements.

func ParamVector 

func ParamVector(v Vector) sqlb.Expr[Vector]

ParamVector binds v and casts it: $n::vector(len). The dimension comes from the value, so it cannot disagree with it.

func SetEfSearch 

func SetEfSearch(n int) sqlb.Expr[string]

SetEfSearch binds hnsw.ef_search: candidates per HNSW scan, default 40.

func SetHNSWIterativeScan 

func SetHNSWIterativeScan(mode IterativeScan) sqlb.Expr[string]

SetHNSWIterativeScan binds hnsw.iterative_scan, so a filtered search keeps scanning until LIMIT rows pass the WHERE clause instead of returning fewer. ScanRelaxedOrder is faster and may return rows slightly out of order; wrap the scan in a MATERIALIZED CTE and sort again to fix that.

func SetIVFFlatIterativeScan 

func SetIVFFlatIterativeScan(mode IterativeScan) sqlb.Expr[string]

SetIVFFlatIterativeScan binds ivfflat.iterative_scan. IVFFlat supports ScanRelaxedOrder only, never ScanStrictOrder.

func SetMaxProbes 

func SetMaxProbes(n int) sqlb.Expr[string]

SetMaxProbes binds ivfflat.max_probes, the IVFFlat ceiling matching SetMaxScanTuples.

func SetMaxScanTuples 

func SetMaxScanTuples(n int) sqlb.Expr[string]

SetMaxScanTuples binds hnsw.max_scan_tuples, the ceiling on how far an iterative scan goes before it gives up and returns what it has.

func SetProbes 

func SetProbes(n int) sqlb.Expr[string]

SetProbes binds ivfflat.probes: lists per IVFFlat scan, default 1.

func Sub 

func Sub[V Dense](a, b sqlb.Expr[V]) sqlb.Expr[V]

Sub renders a - b, element-wise.

func Subvector 

func Subvector[V Elementwise](v sqlb.Expr[V], start, count int) sqlb.Expr[V]

Subvector renders subvector(v, start, count), 1-based, the expression an index on the first dimensions of a Matryoshka embedding is built on.

func Sum 

func Sum[V Dense](v sqlb.Expr[V], opts ...sqlb.AggOption) sqlb.Expr[V]

Sum renders sum(v), the element-wise total. The core Sum is for numbers; this one stays typed as the vector, so the result feeds another operator.

Types

type BitVector 

type BitVector string

BitVector is a bit(n) used as a binary vector, text form "0101".

func (*BitVector) Scan 

func (v *BitVector) Scan(src any) error

Scan accepts "0101". Value renders it unchanged.

func (BitVector) Value 

func (v BitVector) Value() (driver.Value, error)

Value sends the bit string unchanged.

type Dense 

type Dense interface {
	Vector | HalfVector | SparseVector
}

Dense is a vector type with element-wise distance operators.

type Elementwise 

type Elementwise interface{ Vector | HalfVector }

Elementwise is a vector type with arithmetic and concatenation.

type HalfVector 

type HalfVector []float32

HalfVector is a halfvec(n): up to 4000 half-precision elements. Go has no float16, so it carries float32 and the server rounds. Text form "[1,2,3]".

func (*HalfVector) Scan 

func (v *HalfVector) Scan(src any) error

Scan parses "[1,2,3]" into float32. The server has already rounded to half precision, so the value read back is not the value written.

func (HalfVector) Value 

func (v HalfVector) Value() (driver.Value, error)

Value renders "[1,2,3]". The server rounds each element to half precision.

type IterativeScan 

type IterativeScan string

IterativeScan is a value for hnsw.iterative_scan or ivfflat.iterative_scan. 

const (
	ScanOff          IterativeScan = "off"
	ScanStrictOrder  IterativeScan = "strict_order"  // hnsw only
	ScanRelaxedOrder IterativeScan = "relaxed_order" // hnsw and ivfflat
)

The iterative-scan modes. Off is the default and can return fewer rows than LIMIT once a filter rejects candidates; the other two keep scanning, and relaxed order trades exact ordering for speed.

type SparseVector 

type SparseVector struct {
	Dimensions int32
	Indices    []int32 // 0-based, as pgvector-go keeps them
	Values     []float32
}

SparseVector is a sparsevec(n): only the non-zero elements, text form "{1:1.5,3:2}/5" with 1-based indices. Indices must be strictly increasing.

func NewSparse 

func NewSparse(dense []float32) SparseVector

NewSparse builds a SparseVector from a dense slice, keeping the non-zero elements, the same as pgvector-go's NewSparseVector.

func (*SparseVector) Scan 

func (v *SparseVector) Scan(src any) error

Scan parses "{1:1.5,3:2}/5".

func (SparseVector) String 

func (v SparseVector) String() string

String is the text form, so a logged parameter reads as pgvector sees it.

func (SparseVector) Value 

func (v SparseVector) Value() (driver.Value, error)

Value renders "{1:1.5,3:2}/5".

type Vector 

type Vector []float32

Vector is a vector(n): up to 2000 float32 elements, text form "[1,2,3]".

func (*Vector) Scan 

func (v *Vector) Scan(src any) error

Scan parses "[1,2,3]".

func (Vector) Value 

func (v Vector) Value() (driver.Value, error)

Value renders "[1,2,3]".

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.