dalgo2typesense

README

dalgo2typesense

A bounded, read-only native Typesense adapter for DALgo.

It reads documents already stored in an existing Typesense 30.2 collection. It does not write documents, create collections, or require a DALgo envelope. The Typesense document id (a string, including numeric-looking values such as "001") is the DALgo record key.

Status

Adapter checkpoint. The Go CLI and any TypeScript/web UI are separate follow-on deliverables, not part of this module. The module does not link, embed, copy or redistribute the Typesense server or SDK.

Usage

db, err := dalgo2typesense.NewDB(ctx, dalgo2typesense.Config{
    Endpoint:   "https://typesense.example.com:8108",
    Collection: "books",
    Headers: func(ctx context.Context) (map[string]string, error) {
        return map[string]string{"X-TYPESENSE-API-KEY": apiKey}, nil
    },
})

NewDB performs no network I/O. Use dal.As[io.Closer] for Close. The header callback runs once per request, so a rotated scoped key is picked up without constructing a second database. The adapter never logs or stores key values.

Native search

searcher, ok := dal.As[dalgo2typesense.NativeSearcher](db)
result, err := searcher.Search(ctx, dalgo2typesense.SearchRequest{
    Collection: "books",
    Q:          "dune",
    QueryBy:    []string{"title", "author"},
    FilterBy:   "year:>=1965 && genre:=`science fiction`",
    SortBy:     []dalgo2typesense.SortField{{Field: "rank", Descending: true}},
    FacetBy:    []string{"genre"},
    Limit:      20,
})

SearchRequest is typed and bounded. It has no arbitrary option map, raw URL, management endpoint, multi-search, join or RPC surface. FilterBy is validated against a documented subset: joins ($), object filters ({}), control characters, unbalanced backticks and unbalanced parentheses are refused before any request. Facet results are native Typesense facet statistics, not SQL aggregates.

Structured DALgo queries

A single existing root collection, q=*, a positive bounded limit, a bounded offset, top-level scalar AND/OR comparisons, projection and explicit sort are translated. Joins, grouping, HAVING, subqueries, computed columns, nested fields, wildcard projection, unsupported operators and NULL tests are refused before I/O. IS NULL / IS NOT NULL are refused because Typesense does not provide one uniform meaning for absent and explicit-null values.

Structured queries are not text searches. They explicitly send num_typos=0, prefix=false, drop_tokens_threshold=0 and typo_tokens_threshold=0 in addition to q=*. Native Q keeps the server's honest typo, prefix and token-dropping semantics.

Exact structured predicates and honest refusal

A native Typesense filter_by is only a bounded candidate producer. Typesense filtering folds case, accents, punctuation and whitespace, matches array elements for scalar filters, and can return rows that the admitted DALgo predicate does not actually satisfy. The adapter therefore compiles the same condition AST into a small exact verifier and checks every native candidate before exposing it.

• The whole AND/OR tree is evaluated exactly. Scalar Equal and numeric ranges are scalar-only: an array value is not equal to a scalar and is not a numeric range. In intentionally matches an exact array element.
• A missing predicate field never authorizes a row by itself. A true sibling OR branch can still make the whole predicate true.
• If any real candidate fails the exact predicate, the read is refused with ErrSemanticMismatch (which also unwraps to dal.ErrNotSupported) before the first row is exposed. There is no silent row dropping, no scan-around retry, no automatic policy relaxation and no broad whole-collection scan.
• Filtered reads fetch native candidates from offset zero through offset+limit, verify the complete window, and only then apply the requested offset locally. A later-page mismatch or a cancellation during verification cannot leave an apparently successful prefix. The window shape is checked before I/O: a window that cannot fit the configured MaxPages, MaxRowsPerPage, MaxOffset or MaxColumns bounds is refused before any request. Row and response-byte limits, the operation deadline and cancellation are enforced while the window is buffered, so those refusals happen during the read, before any row is exposed. Unfiltered reads keep their lazy native-offset path.
• Predicate helper fields are fetched with the declared projection so the verifier can see them. They count against MaxColumns and are stripped before a projected row is constructed.
• Structured NOT IN is refused before I/O because native negative string folding is not a safe superset. NativeSearcher keeps its native negative filter grammar.

Numeric candidate widening (documented limitation). Typesense 30.2 stores raw JSON numbers but indexes float values through quantised buckets. An actual probe showed strict n:>0.1 omitting stored 0.1000000001 and 0.10000000149, and n:>100000000 returning nothing even though 100000001 is stored. A strict native filter alone therefore cannot be trusted to contain every eligible row. The adapter lowers structured strict > and < candidates to inclusive >= and <=; existing >= and <= are unchanged. The exact verifier still evaluates the original operator and the same compiler wire literal, so an inclusive boundary candidate is refused rather than silently dropped. This means an exact integer boundary (for example year > 1965 when a row has 1965) can refuse the read, and a fractional RHS on an integer field may be rejected by the server. That is an intentional, fail-closed initial limitation, not a bypass. NativeSearcher remains fully engine-native.

Curation and filter safety

Every read path sets:

enable_curations=false
filter_curated_hits=true

Typesense 30.2 defaults are enable_curations=true and filter_curated_hits=false, so a pinned or promoted document can otherwise bypass filter_by. The adapter applies both flags itself and the request types offer no way for a caller to override them. There is no pinned, promoted or native-override bypass.

Permissions

One scoped read-only credential callback is used for every request. A pure documents:search key scoped to one collection can execute native searches and exact-id document reads. Metadata calls made with that key fail explicitly with ErrMetadataForbidden; they are never converted into an empty collection list and never fall back to an admin credential. A key that additionally has collections:get and collections:list can discover the real collection catalog and schema through dbschema.SchemaReader. The adapter never creates collections, keys or documents.

Identity and value fidelity

• The native Typesense id is always a string. "001" and 1 are different.
• JSON numbers are decoded with UseNumber and resolved exactly before any value is exposed. An integral value stays integral even when it is spelled with an exponent or a decimal point: 9007199254740993e0 and 9007199254740993.0 both become exactly 9007199254740993, not the float64-rounded neighbour. Integers that fit int64 become int64; larger non-negative integers that fit uint64 become uint64. An integral value outside those ranges is rejected, never silently substituted. Only non-integral values become finite float64 values with documented IEEE754 semantics.
• A caller-supplied json.Number must use a valid JSON number spelling. Malformed forms such as 1., .1 and 01 are refused rather than normalised.
• Structured numeric comparison reuses the exact wire literal the native filter was built from. A float32 RHS is compared as its shortest float32 wire decimal (for example 0.1), never as the extra binary64 digits of float64(float32(v)). Integral json.Number spellings incl. exponents stay exact int64/uint64; fractional values use the documented finite IEEE754 value. Comparisons use exact rational arithmetic, so adjacent int64/uint64 values above 2^53 stay distinct and are never routed through float64.
• Facet stats keep the exact native JSON class too: an integral min, max, sum or avg is int64/uint64, a fractional one is float64. They stay native Typesense facet statistics, not SQL aggregates.
• Arrays, objects, booleans, strings and null are not stringified.
• A projected DALgo query returns exactly its declared columns in declaration order. The native identity may be fetched as a helper but never leaks into a projected row. A declared but absent field is represented as nil.
• Recordset columns are Column[any] with an explicit any ValueType; no declared affinity coerces a value.
• Missing, null or wrong hits / documents / fields arrays, bad IDs, duplicate IDs within or across pages, mismatched offsets and integer overflow are malformed responses, never an empty successful result.

Pagination and server caps

Typesense accepts at most 250 hits per page. One DALgo query may span several pages under one operation deadline, one byte budget and one page budget. found and out_of are server metadata, distinct from Returned hits. A short or empty page that the server's own found count says is not the end returns ErrPartialResult, so a scoped limit_hits is never presented as a clean EOF. An offset at or beyond found is a valid empty result and is reported as EOF, not as a malformed page.

The adapter also decodes the server's search_cutoff response indicator as a strict boolean. A cutoff makes a native result explicitly Partial with Complete false, even when the cutoff page returned zero hits and found is zero. DALgo readers, Get, Exists and GetMulti refuse a cutoff with ErrPartialResult; it is never mistaken for EOF, a missing document or a full result. search_cutoff_ms is not a strict bound, facets are not bound by it and the adapter does not send it as a hard provider deadline.

No id sort is invented. An explicit application-field sort is the caller's choice; repeatable pages are claimed only for unchanged data with a unique sort value. The adapter does not claim snapshot isolation under concurrent writes.

Configuration bounds

All bounds have modest defaults and hard maxima. Notable defaults:

Option	Default	Maximum
MaxRows	1000	10000
MaxRowsPerPage	100	250
MaxPages	20	100
MaxOffset	10000	1000000
MaxGetMany	100	250
MaxResponseBytes	4 MiB	64 MiB
MaxTotalBytes	16 MiB	64 MiB
MaxRowBytes	1 MiB	64 MiB
Timeout	30s	2m

MaxRowsPerPage is the physical page size of the structured reader and also bounds one native search or one exact-id GetMulti batch. A larger native request is refused before I/O; the configured bound is never silently ignored. MaxRowBytes is enforced by every read path (native search, structured reader and recordset, Get, Exists, GetMulti) before an over-budget document is exposed.

Redirects are refused before a credential can reach another origin. The adapter copies a caller-supplied http.Client and never mutates it. It only closes a transport it created. Error values retain a numeric HTTP status and safe classification, never a URL, query string, body, key or provider error text.

Licensing boundary

The Typesense server is GPL-3.0. This package is an independent MIT-licensed HTTP client. It does not link, embed, copy or redistribute the server or its SDK, and it does not relicense DALgo core or any CLI.

Verification

Unit tests use httptest and cover outbound curation policy, header refresh, redirect refusal, strict response shape, projection, paging, partial results, search_cutoff refusal, offset-at-or-beyond-found EOF, row and page budgets, exact exponent/decimal integer fidelity across readers, recordsets, filters and native facet stats, metadata refusal, cancellation, close and exact-id reads. They also pin the exact structured predicate verifier, filtered-window buffering, numeric candidate widening, float32 wire canonicalisation and adjacent large integers.

The integration suite starts one owned, pinned typesense/typesense:30.2@sha256:610f2d34b1f93d00762869da2c67736775e5798d19a2c8b91b014b8a0cc1e110 container on a random loopback port, seeds native documents and a curation set that promotes a document excluded by a filter, and proves the unprotected server premise and the adapter's red-to-green filter enforcement. It also exercises multi-page reads plus an offset at and beyond the real server's found count, and proves the configured page and row budgets against that server. The same fixture seeds the observed FLOAT boundary values and proves that every strict-eligible raw value appears in the inclusive candidate set, that a boundary candidate refuses before the first row, and that exact integer, string, array, AND/OR, filtered-offset and SecureReadSession residual cases behave as documented.

Documentation

Overview

Package dalgo2typesense provides a bounded, read-only DALgo adapter for native documents in an existing Typesense 30.2 collection.

Documents are read as they are stored: the Typesense string id is the DALgo record key and no DALgo envelope is required or written. Native searches are exposed through the adapter-local NativeSearcher capability, reachable with dal.As. Structured DALgo queries are translated to a deliberately small and validated subset of the Typesense search API.

Every read path disables curations (enable_curations=false) and asks the server to apply filter_by to curated hits (filter_curated_hits=true). A caller cannot override either policy. The adapter never follows redirects, never creates collections, never writes documents and never exposes raw URLs, management endpoints, multi-search, joins or RPC endpoints.

Server licensing: Typesense server is GPL-3.0. This package is an independent MIT-licensed HTTP client. It does not link, embed, copy or redistribute the Typesense server or its SDK.

Index

• Constants
• Variables
• func NewDB(ctx context.Context, cfg Config) (dal.DB, error)
• type Config
• type Facet
• type FacetStats
• type FacetValue
• type GetMultiError
  • func (e *GetMultiError) Error() string
  • func (e *GetMultiError) Unwrap() error
• type HTTPError
  • func (e *HTTPError) Error() string
  • func (e *HTTPError) Unwrap() error
• type NativeSearcher
• type SearchHit
• type SearchRequest
• type SearchResult
• type SortField

Constants

const ProvenanceSearch = "typesense-native-search"

ProvenanceSearch is the only provenance the adapter reports for a native search. It makes clear that the result came from the Typesense HTTP search API, not from a sampled document or a local recomputation.

Variables

var (
	// ErrRequestTooLarge reports that the encoded request exceeds the
	// configured byte budget.
	ErrRequestTooLarge = errors.New("typesense: request exceeds configured byte budget")

	// ErrResponseTooLarge reports that a response body or the retained row data
	// of one operation exceeds a configured byte budget.
	ErrResponseTooLarge = errors.New("typesense: response exceeds configured byte budget")

	// ErrRowTooLarge reports that one decoded document exceeds MaxRowBytes.
	ErrRowTooLarge = errors.New("typesense: document exceeds configured row byte budget")

	// ErrTooManyRows reports that a response or operation returned more rows
	// than the configured row budget.
	ErrTooManyRows = errors.New("typesense: result exceeds configured row budget")

	// ErrTooManyPages reports that completing an operation would need more
	// search pages than the configured page budget.
	ErrTooManyPages = errors.New("typesense: result exceeds configured page budget")

	// ErrMalformedResponse reports a response that is not the expected
	// Typesense JSON shape. It never includes response text.
	ErrMalformedResponse = errors.New("typesense: malformed response")

	// ErrUnsafeValue reports an identifier, field, header or constant that the
	// adapter will not send. It never includes the rejected value.
	ErrUnsafeValue = errors.New("typesense: unsafe value")

	// ErrPartialResult reports that the server returned fewer hits than the
	// response's own found count says are available. It is used for caps such
	// as a scoped limit_hits and for search cutoffs, so a capped result is
	// never presented as a clean end of data.
	ErrPartialResult = errors.New("typesense: partial result")

	// ErrDuplicateID reports that a paged search repeated a native document ID.
	// It prevents duplicate rows silently entering a result and detects an
	// unstable page boundary.
	ErrDuplicateID = errors.New("typesense: duplicate document ID across pages")

	// ErrMetadataForbidden reports that a metadata request was refused by the
	// credential. It is never converted into an empty collection list.
	ErrMetadataForbidden = errors.New("typesense: metadata is not permitted by this credential")

	// ErrPermissionDenied reports an HTTP 401 or 403 response. The numeric
	// status remains available through *HTTPError.
	ErrPermissionDenied = errors.New("typesense: request is not permitted")

	// ErrRedirectRefused reports that the server attempted a redirect. The
	// adapter never follows a redirect because a credential must not reach a
	// different origin.
	ErrRedirectRefused = errors.New("typesense: redirect refused")

	// ErrHeaderProvider reports that the configured header provider failed.
	// The provider's own error text is not returned because it may contain a
	// credential.
	ErrHeaderProvider = errors.New("typesense: header provider failed")

	// ErrRequestFailed is the safe, detail-free error returned for a transport
	// failure that is neither a context cancellation nor a deadline.
	ErrRequestFailed = errors.New("typesense: request failed")

	// ErrClosed reports that the adapter has been closed.
	ErrClosed = errors.New("typesense: adapter is closed")

	// ErrCollectionNotConfigured reports a structured query or configured
	// metadata request on an adapter without a configured collection.
	ErrCollectionNotConfigured = errors.New("typesense: no collection is configured")

	// ErrSemanticMismatch reports that a native Typesense candidate does not
	// satisfy the exact admitted DALgo predicate. Native string folding and
	// array matching make filter_by a candidate producer, never the final
	// semantic authority; such a candidate is refused instead of silently
	// dropped, exposed or retried around. The same refusal covers a widened
	// strict numeric boundary candidate: a strict > or < is lowered to an
	// inclusive >= or <= so a quantised FLOAT index cannot omit eligible raw
	// values, and the exact verifier then rejects the boundary row. It also
	// unwraps to dal.ErrNotSupported because the adapter honestly cannot
	// produce the requested exact result under the existing bounds.
	ErrSemanticMismatch = errors.New("typesense: exact structured predicate cannot be matched by the native candidate")
)

The adapter returns sentinel errors for conditions callers may need to distinguish without parsing text. Every message is deliberately static: endpoints, credentials, headers, query strings and response bodies are never included in an error.

var (
	ErrCanceled         = fmt.Errorf("typesense: request canceled: %w", context.Canceled)
	ErrDeadlineExceeded = fmt.Errorf("typesense: request deadline exceeded: %w", context.DeadlineExceeded)
)

ErrCanceled and ErrDeadlineExceeded preserve errors.Is semantics for context.Canceled and context.DeadlineExceeded without exposing the wrapped transport error, whose text may contain the endpoint or credentials.

Functions

func NewDB

func NewDB(ctx context.Context, cfg Config) (dal.DB, error)

NewDB validates cfg and returns a sealed DALgo database. It performs no network request. Use dal.As[io.Closer] on the returned DB for Close.

Types

type Config

type Config struct {
	// Endpoint is the HTTPS origin of the Typesense server, for example
	// https://typesense.example.com:8108. It must not contain credentials, a
	// path, a query or a fragment. Plain HTTP is accepted only for a loopback
	// host when AllowLoopbackHTTP is true.
	Endpoint string

	// Collection optionally pins every request to one existing collection.
	// When empty, native searches may name any collection the credential can
	// read, while structured DALgo queries and configured metadata are refused
	// because there is no configured target.
	Collection string

	// Headers is called for every HTTP request and returns that request's
	// credential headers (typically X-TYPESENSE-API-KEY). It is the only
	// credential source. Names and values must be CR/LF/NUL-safe; hop-by-hop,
	// protocol and framing headers are refused. Provider error text is never
	// returned to the caller.
	Headers func(ctx context.Context) (map[string]string, error)

	// HTTPClient optionally supplies the client. The adapter copies it, blocks
	// redirects on the copy, and never mutates the caller's client.
	HTTPClient *http.Client

	// AllowLoopbackHTTP permits plain HTTP only for localhost, 127.0.0.1 and
	// ::1. It is a development option; production endpoints must use HTTPS.
	AllowLoopbackHTTP bool

	// MaxRows is the largest accepted DALgo query or native search limit.
	// Default 1000. Native search is one page, so it is additionally bounded by
	// MaxRowsPerPage and the server's 250-hit page limit.
	MaxRows int
	// MaxRowsPerPage bounds one search request. Default 100, hard maximum 250.
	// The structured reader uses it as its physical page size. Native search
	// and exact-id GetMulti batches refuse a request larger than this bound
	// instead of silently ignoring it.
	MaxRowsPerPage int
	// MaxPages bounds how many search requests one DALgo query may issue.
	// Default 20.
	MaxPages int
	// MaxOffset is the largest accepted offset. Default 10000.
	MaxOffset int
	// MaxGetMany is the largest accepted GetMulti batch. Default 100.
	MaxGetMany int
	// MaxRequestBytes bounds the encoded search query sent to the server.
	// Default 64 KiB.
	MaxRequestBytes int64
	// MaxResponseBytes bounds one HTTP response body. Default 4 MiB.
	MaxResponseBytes int64
	// MaxTotalBytes bounds the cumulative response bytes of one operation,
	// including metadata, paging and batches. Default 16 MiB.
	MaxTotalBytes int64
	// MaxRowBytes bounds one decoded native document's JSON size. Every read
	// path enforces it before exposing the row: the structured reader, native
	// search, Get, Exists and GetMulti. Default 1 MiB.
	MaxRowBytes int64
	// MaxFilterBytes bounds a native FilterBy expression and a compiled DALgo
	// filter. Default 8 KiB.
	MaxFilterBytes int
	// MaxFacetFields bounds the number of facet_by fields. Default 5.
	MaxFacetFields int
	// MaxFacetValues bounds max_facet_values. Default 100.
	MaxFacetValues int
	// MaxColumns bounds a structured projection and a query_by field list.
	// Default 100.
	MaxColumns int
	// MaxSortFields bounds sort fields. Typesense accepts at most 3. Default 3.
	MaxSortFields int
	// Timeout bounds one operation, including every page and request in it.
	// Default 30s.
	Timeout time.Duration
}

Config configures one adapter instance. The adapter addresses an existing Typesense server; it never creates collections or writes documents.

type Facet

type Facet struct {
	Field  string
	Counts []FacetValue
	Stats  *FacetStats
}

Facet is a native facet result. Counts are native facet statistics, not SQL aggregates.

type FacetStats

type FacetStats struct {
	Min         any
	Max         any
	Sum         any
	Avg         any
	TotalValues int
}

FacetStats retains the native stats object when Typesense returns one. The numeric fields keep the exact JSON class: an integral statistic becomes int64 or uint64 and a fractional one becomes float64. Stats are never presented as SQL aggregates.

type FacetValue

type FacetValue struct {
	Value       string
	Count       int
	Highlighted string
}

FacetValue is one native facet value count.

type GetMultiError

type GetMultiError struct {
	Index int
	Err   error
}

GetMultiError reports which item of a GetMulti batch stopped the batch. Records before Index are populated; the record at Index and all later records are not. It unwraps to the underlying cause, so errors.Is and errors.As keep working for sentinels and record.ErrRecordNotFound.

func (*GetMultiError) Error

func (e *GetMultiError) Error() string

func (*GetMultiError) Unwrap

func (e *GetMultiError) Unwrap() error

type HTTPError

type HTTPError struct {
	StatusCode int
}

HTTPError is a safe Typesense HTTP status error. Only the numeric status is retained; response headers and bodies are discarded.

func (*HTTPError) Error

func (e *HTTPError) Error() string

func (*HTTPError) Unwrap

func (e *HTTPError) Unwrap() error

Unwrap classifies authentication and authorization failures without retaining provider text.

type NativeSearcher

type NativeSearcher interface {
	Search(ctx context.Context, request SearchRequest) (SearchResult, error)
}

NativeSearcher is the adapter-local native Typesense search capability. Obtain it with dal.As[NativeSearcher](db). It is deliberately not a DALgo core interface.

type SearchHit

type SearchHit struct {
	ID        string
	Document  map[string]any
	TextMatch int64
	Curated   bool
}

SearchHit retains the native string ID and the losslessly decoded document.

type SearchRequest

type SearchRequest struct {
	// Collection is an existing collection. When Config.Collection is set it
	// must match it exactly; an empty value uses the configured collection.
	Collection string

	// Q is the native query text. "*" is the wildcard query.
	Q string

	// QueryBy lists full-text fields. It is required unless Q is "*".
	QueryBy []string

	// FilterBy is a bounded native Typesense filter expression. It is not a
	// URL and it is validated before any request. Currency, joins, object
	// filters and other unsupported grammar are refused.
	FilterBy string

	// SortBy lists simple application fields and directions. Typesense accepts
	// at most three. No id sort is invented.
	SortBy []SortField

	// FacetBy lists facet-enabled fields.
	FacetBy []string

	// MaxFacetValues bounds the returned facet values. Zero uses the smaller
	// of 10 and Config.MaxFacetValues; it can never exceed
	// Config.MaxFacetValues.
	MaxFacetValues int

	// Limit is the positive number of hits to return. It is bounded by
	// Config.MaxRows, Config.MaxRowsPerPage and the server's 250-hit page
	// limit. Zero is refused.
	Limit int

	// Offset is the non-negative hit offset.
	Offset int
}

SearchRequest is one bounded Typesense search. It has no arbitrary option bag, raw URL, management endpoint, multi-search, join or RPC surface.

type SearchResult

type SearchResult struct {
	Collection   string
	Hits         []SearchHit
	Facets       []Facet
	Found        int
	OutOf        int
	Page         int
	Returned     int
	SearchTimeMS int
	Partial      bool
	Complete     bool
	Truncated    bool
	Provenance   string
}

SearchResult is the bounded native result. Found and OutOf are server metadata, distinct from the hits actually returned. Partial and Complete are explicit: a server cap or cutoff is never presented as a clean end of data. A Typesense search_cutoff response indicator sets Partial and clears Complete even when the cutoff page returned zero hits and found is zero.

type SortField

type SortField struct {
	Field      string
	Descending bool
}

SortField is one explicit native sort term.