dalgo2http

README

dalgo2http

HTTP/JSON adapter for DALgo: expose read-only REST endpoints (public reference data, internal JSON APIs) as DALgo collections so that the same dal.Query model, and the same access policies, apply to them as to SQL, SQLite, Firestore and inGitDB sources.

Status: v0.x implemented 2026-09-09 (founder decision: a generic DALgo adapter for HTTP rather than a consumer-private fetcher). First consumer: DataTug's demo knowledge project. The two example descriptors under examples/ (CountriesNow currency-by-country, Frankfurter exchange rates) replace REST Countries — restcountries.com's public v3.1 API is now fully deprecated; see examples/countries/README.md.

Design constraints (do not relax without a recorded decision)

• Declarative collections. Each collection is a descriptor: URL template, HTTP method (GET only in v0.x of this adapter), which query fields map to path/query parameters, the JSON path to the rows, the key field, timeout. Descriptors carry no secrets; header values come from the environment.
• Fail closed on pushdown. A dal.Query is executed only when every condition can be expressed by the endpoint (equality on declared parameter fields). Anything else returns a typed "not supported" error. The adapter never fetches a superset and filters client-side unless the descriptor explicitly opts in, because an access-policy predicate that cannot be pushed down must refuse, not leak. A SelectColumns() projection is different: even without a schema, this adapter CAN enforce a plain field-name projection itself, after fetching — see "Projection" below — so it is applied, not refused; only a column this adapter cannot evaluate (anything other than a bare field reference) is refused.
• Projection is enforced at the adapter boundary, not pushed down or refused. A SelectColumns() request over plain field references drops every un-requested field from each row after extractRows, before the row is ever converted into the returned record — an un-requested field never reaches the caller. KeyField is always retained even when not itself requested (it identifies the row, not a value under projection). A requested field the live response does not carry is left absent in the output, never synthesized as an explicit null.
• Declared capabilities. Callers can ask the adapter what it supports per collection so a policy layer can decide before executing.
• Snapshots. An optional recorded snapshot store answers when the endpoint is unreachable; every result says whether it came from live or snapshot.
• Read-only. Writes and transactions that mutate return "not supported".
• HTTPS only. URLTemplate must use https://; http:// is refused at config time (ErrInvalidConfig), except for Collection.InsecureAllowLoopback — a TEST-ONLY escape hatch, never for a real descriptor, and only when the host is literally loopback (127.0.0.1, ::1, localhost). It is not loadable from LoadConfigYAML/LoadConfigJSON, only from a Go Collection{} literal.
• Address-guarded dialing. The default client (Config.Client left nil) never connects to a private (RFC1918 + IPv6 ULA), loopback, link-local (including the 169.254.169.254 cloud metadata address), multicast or unspecified address — resolved once and dialed by IP literal, so a later DNS rebind cannot redirect the connection. A caller who supplies their own Config.Client is responsible for equivalent protections on it.
• No redirects. The default client refuses every redirect response (ErrRedirectNotAllowed, wrapping ErrUpstreamClient — never fallback-eligible). There is no config knob to re-enable following redirects in this package; a descriptor must target its final host directly.
• Bounded responses. A live response body over 2 MiB fails explicitly with ErrResponseTooLarge rather than being silently truncated and then failing JSON decoding with a misleading error.
• No secrets in query strings. Neither a declared query-location Param name nor a literal query-string key already in URLTemplate may look like a credential (token, apikey, api_key, secret, password, authorization, case-insensitive substring match) — rejected at config time. Headers (an environment variable, never a literal) is the documented place for a credential.

Usage

db, err := dalgo2http.NewDB(dalgo2http.Config{
	Collections: []dalgo2http.Collection{
		{
			Name:        "countries",
			URLTemplate: "https://countriesnow.space/api/v0.1/countries/currency/q?country={name}",
			KeyField:    "name",
			RowsPath:    "data",
			Params:      map[string]dalgo2http.Param{"name": {Location: dalgo2http.ParamQuery}},
			Timeout:     10 * time.Second,
		},
	},
	// Snapshots: os.DirFS("testdata"), // optional recorded-fixture fallback
	// Mode:      dalgo2http.ModeLiveThenSnapshot, // the default
})

// Get by key (only works when KeyField is itself a declared Param — see
// Capabilities.SupportsGet below):
target := map[string]any{}
rec := record.NewRecordWithData(record.NewKeyWithID("countries", "France"), &target)
err = db.Get(ctx, rec)

// Query: only an equality condition on a declared Param field is pushed
// into the URL; anything else fails closed with dal.ErrNotSupported unless
// the collection sets ClientSideFilter.
q := dal.NewQueryBuilder(dal.From(dal.NewRootCollectionRef("countries", ""))).
	Where(dal.WhereField("name", dal.Equal, "France")).
	SelectIntoRecord(nil)
reader, err := db.ExecuteQueryToRecordsReader(ctx, q)

// Capabilities is an optional adapter capability (see dal.As's doc comment
// on why a plain type assertion on a dal.DB does not see it):
caps, err := dal.As[dalgo2http.CapabilitiesProvider](db)

Building a recorded-snapshot fixture (used both for offline tests and as the live-request fallback the design constraints describe):

path, err := dalgo2http.Record(ctx, http.DefaultClient, coll, map[string]string{"name": "France"}, "testdata")

See examples/countries and examples/frankfurter for complete, runnable descriptors with recorded fixtures and offline tests.

Descriptor reference

Field	Meaning
Name	Collection name a record.Key or dal.Query.From() names.
URLTemplate	Request URL with {name} placeholders for declared Params. A placeholder can sit in the path or be embedded in a literal query string (e.g. ...?symbols={to}); its declared Param.Location decides the escaping used, not its position in the string.
Method	Empty or dalgo2http.MethodGET — this adapter is read-only GET-only in v0.x.
Params	map[string]Param{name: {Location: ParamPath | ParamQuery}}. A ParamQuery entry whose name never appears in URLTemplate is instead appended as an extra ?name=value when a query supplies a value for it.
RowsPath	Dot-separated JSON object-field path to the rows. Empty means the response root: a JSON array of row objects, or a single JSON object treated as one row.
KeyField	The row field that becomes a record.Key's ID. Get/Exists only work when KeyField is ALSO a declared Param — see Capabilities.SupportsGet — otherwise they fail closed with dal.ErrNotSupported.
Headers	map[headerName]envVarName. A header value is never a literal in a descriptor; an unset/empty variable means the header is simply not sent.
Timeout	Bounds one request to this collection's endpoint. Zero means no adapter-imposed timeout beyond the context's own deadline.
ClientSideFilter	Opt-in escape hatch for a dal.Query condition that cannot be fully pushed into the URL: the equality parts that CAN be pushed still narrow the request, and the remainder is evaluated in memory. Set this ONLY on a collection where over-fetching cannot leak anything a caller was not already allowed to see (public reference data) — see the "Fail closed on pushdown" design constraint above.
InsecureAllowLoopback	TEST-ONLY. Lets URLTemplate use http:// instead of https://, and lets the default guarded client dial a loopback address, for THIS collection only — and only when the host is literally loopback. Not loadable from YAML/JSON config; see the "HTTPS only" design constraint above.

Config also loads from YAML or JSON via LoadConfigYAML/LoadConfigJSON (a collections: list of the fields above, plus a repo-level mode:; a collection's timeout is a duration string like "10s"; InsecureAllowLoopback is deliberately excluded from this schema).

Query support

ExecuteQueryToRecordsReader (via dal.StructuredQuery) supports:

• An equality condition (dal.Equal) on a declared Param field, combined with AND (a GroupCondition with any other operator, or a bare OR, is not pushable and fails closed unless ClientSideFilter is set).
• Limit, applied AFTER fetching (never pushed into the request).

It does not support (fails closed with dal.ErrNotSupported): joins, GROUP BY/HAVING, ORDER BY, Offset, or start cursors.

A SelectColumns() projection over plain field references IS supported, applied after fetch (see "Projection is enforced at the adapter boundary" above) — an un-requested field never reaches the returned record. A column whose expression is not a bare field reference (something this adapter cannot evaluate) is refused with dal.ErrNotSupported before any request is made.

ExecuteQueryToRecordsetReader is not implemented (returns dal.ErrNotSupported): this adapter's rows are schemaless HTTP/JSON objects, and recordset.Recordset's typed columnar shape is not something a declarative descriptor can derive without a schema. dalgo2fs, the reference minimal read-only adapter, makes the same choice for the same reason.

Errors

Sentinel errors (errors.Is-checkable), beyond DALgo's own dal.ErrNotSupported:

Error	Meaning	Fallback-eligible under ModeLiveThenSnapshot?
ErrInvalidConfig	A Collection/Config failed validation.	n/a — never reaches a live request.
ErrUnknownCollection	A record.Key/dal.Query named an undeclared collection.	n/a
ErrMissingParam	A required URL template parameter had no value.	n/a — never reaches a live request.
ErrUpstream	A network error, timeout, or 5xx/429 response.	Yes — the only fallback-eligible class.
ErrUpstreamClient	A 4xx response, a refused redirect (wraps ErrRedirectNotAllowed), or a blocked address (wraps ErrAddressBlocked) — every one a caller/config problem, not a transient failure.	No.
ErrRedirectNotAllowed	The live endpoint tried to redirect; redirects are always refused. Also wraps ErrUpstreamClient.	No.
ErrAddressBlocked	The guarded dialer refused a private/loopback/link-local/metadata/multicast/unspecified target address. Also wraps ErrUpstreamClient.	No.
ErrResponseTooLarge	A live response body exceeded 2 MiB.	No.
ErrSnapshotMiss	No recorded snapshot exists for a request.	n/a

Spec

See spec/features/http-json-adapter/README.md.

Documentation

Overview

Package dalgo2http exposes read-only HTTP/JSON endpoints as DALgo collections.

The adapter is declarative and fails closed: a query runs only when every condition can be pushed to the endpoint. See README.md for the constraints.

Index

• Variables
• func ContextWithProvenanceObserver(ctx context.Context, obs Observer) context.Context
• func NewDB(cfg Config) (dal.DB, error)
• func Record(ctx context.Context, client *http.Client, coll Collection, ...) (path string, err error)
• func SnapshotKey(collection string, params map[string]string) string
• type Capabilities
• type CapabilitiesProvider
• type Collection
• type Config
  • func LoadConfigJSON(data []byte) (Config, error)
  • func LoadConfigYAML(data []byte) (Config, error)
• type Method
• type Mode
• type Observer
• type Param
• type ParamLocation
• type Provenance
• type Recorder
  • func NewRecorder() *Recorder
  • func (r *Recorder) Last() (Provenance, bool)
  • func (r *Recorder) WithContext(ctx context.Context) context.Context
• type Source

Variables

var (
	// ErrInvalidConfig indicates a Collection or Config value failed
	// validation (LoadConfigYAML, LoadConfigJSON or NewDB).
	ErrInvalidConfig = errors.New("dalgo2http: invalid config")

	// ErrUnknownCollection indicates a record.Key or query named a collection
	// that is not declared in the Config.
	ErrUnknownCollection = errors.New("dalgo2http: unknown collection")

	// ErrMissingParam indicates a request could not be built because a
	// required URL template parameter had no value.
	ErrMissingParam = errors.New("dalgo2http: missing required parameter")

	// ErrUpstream indicates a live request failed for a reason snapshot
	// fallback treats as transient: a network error, a timeout, or a 5xx /
	// 429 response.
	ErrUpstream = errors.New("dalgo2http: upstream request failed")

	// ErrUpstreamClient indicates a live request received a 4xx response.
	// This is treated as a caller/config error, not a transient failure, so
	// it never triggers snapshot fallback.
	ErrUpstreamClient = errors.New("dalgo2http: upstream rejected the request")

	// ErrSnapshotMiss indicates no recorded snapshot exists for a request.
	ErrSnapshotMiss = errors.New("dalgo2http: no snapshot available")

	// ErrRedirectNotAllowed indicates a live request received a redirect
	// response. Redirects are disabled by default (Phase 1 HTTP bounds:
	// "Redirects are disabled for the demo") — a redirecting endpoint is a
	// caller/config problem (the descriptor should target the final host
	// directly), never a transient failure, so doLiveFetch reports it
	// wrapping ErrUpstreamClient, never ErrUpstream: it is never
	// fallback-eligible.
	ErrRedirectNotAllowed = errors.New("dalgo2http: redirects are not allowed")

	// ErrResponseTooLarge indicates a live response body exceeded
	// maxBodyBytes. Reported explicitly rather than silently truncated
	// (which would otherwise fail JSON decoding downstream with a
	// misleading "unexpected end of JSON input").
	ErrResponseTooLarge = errors.New("dalgo2http: response body exceeds the size limit")

	// ErrAddressBlocked indicates the guarded dialer refused to connect to a
	// resolved address because it is private, loopback, link-local or a
	// metadata-service address (Phase 1 HTTP bounds: "Deny private,
	// loopback, link-local and metadata-service addresses"). A caller/config
	// problem, not a transient failure.
	ErrAddressBlocked = errors.New("dalgo2http: target address is blocked (private/loopback/link-local/metadata)")
)

Sentinel errors this adapter returns. All are typed so callers can use errors.Is; several also wrap dal.ErrNotSupported (see descriptor.go and query.go) so a caller that only checks for dal.ErrNotSupported still sees the fail-closed refusal.

Functions

func ContextWithProvenanceObserver

func ContextWithProvenanceObserver(ctx context.Context, obs Observer) context.Context

ContextWithProvenanceObserver returns a context that, when passed to a dalgo2http database method, calls obs with that call's Provenance before the method returns. See also Recorder for a ready-made Observer that keeps the last value for the caller to read back afterwards.

func NewDB

func NewDB(cfg Config) (dal.DB, error)

NewDB validates cfg and returns a dal.DB backed by it. cfg.Client defaults to a guarded client (see security.go's newDefaultClient — a custom DialContext rejecting private/loopback/link-local/metadata addresses, and redirects disabled) when nil, per the Phase 1 HTTP bounds; cfg.Mode defaults to ModeLiveThenSnapshot when empty.

func Record

func Record(ctx context.Context, client *http.Client, coll Collection, params map[string]string, dir string) (path string, err error)

Record performs one live GET for coll with params — ignoring Config.Mode entirely, it always calls the network — and writes the response as a snapshot fixture file under dir, named by SnapshotKey. It is a development/test tool for building fixtures (see examples/), not part of request-serving at runtime.

func SnapshotKey

func SnapshotKey(collection string, params map[string]string) string

SnapshotKey returns the deterministic file name a recorded snapshot for collection with the given request params is stored under: the collection name followed by its params sorted by key, so the same logical request always resolves to the same file regardless of map iteration order.

Types

type Capabilities

type Capabilities struct {
	Collection string

	// Fields lists the declared parameter fields an equality condition may
	// target and have pushed into the request.
	Fields []string

	// Operators lists the dal.Operator values usable against Fields.
	// Equality is always supported (it is the only operator ever pushed into
	// the URL); the ordering/inequality operators appear only when
	// ClientSideFilter is set, since those are evaluated in memory after a
	// broader fetch, never pushed down.
	Operators []dal.Operator

	// SupportsGet reports whether dal.DB.Get/Exists work for this
	// collection: true only when KeyField is itself a declared Param.
	SupportsGet bool

	// ClientSideFilter mirrors Collection.ClientSideFilter.
	ClientSideFilter bool
}

Capabilities describes what a policy layer may push down to one collection, so it can decide whether a query is answerable before executing it (see query.go for the fail-closed enforcement this describes).

type CapabilitiesProvider

type CapabilitiesProvider interface {
	Capabilities(collection string) (Capabilities, error)
}

CapabilitiesProvider is the optional capability a dalgo2http database implements. dal.NewDB decorates the Backend passed to it, so a plain type assertion on a dal.DB does not see Capabilities; use dal.As[CapabilitiesProvider](db) instead, per the convention dal.As documents (the same one dbschema.SchemaReader and ddl.SchemaModifier use).

type Collection

type Collection struct {
	// Name is the collection name a record.Key or dal.Query.From() names.
	Name string `yaml:"name" json:"name"`

	// URLTemplate is the request URL, with {name} placeholders for every
	// declared Param that is substituted directly into the template (a path
	// segment, or a value embedded in a literal query string such as
	// "...?symbols={to}"). A Param declared with ParamLocation "query" whose
	// name does NOT appear in the template is instead appended as an extra
	// "?name=value" query parameter when a value is supplied.
	URLTemplate string `yaml:"urlTemplate" json:"urlTemplate"`

	// Method is the HTTP method. Empty means MethodGET; any other value
	// fails validation.
	Method Method `yaml:"method,omitempty" json:"method,omitempty"`

	// Params declares every field a dal.Query equality condition, or a
	// record.Key ID via KeyField, may be pushed down as.
	Params map[string]Param `yaml:"params,omitempty" json:"params,omitempty"`

	// RowsPath is the dot-separated JSON path (object-field traversal only)
	// to the row array in the response body. Empty means the root of the
	// response IS the rows value: a JSON array of row objects, or a single
	// JSON object treated as one row.
	RowsPath string `yaml:"rowsPath,omitempty" json:"rowsPath,omitempty"`

	// KeyField is the row field whose value becomes a record.Key's ID.
	// dal.DB.Get/Exists for this collection only work when KeyField is also
	// a declared Param (see query.go); otherwise Get/Exists fail closed with
	// an error wrapping dal.ErrNotSupported, and only ExecuteQueryToRecords*
	// is usable.
	KeyField string `yaml:"keyField" json:"keyField"`

	// Headers maps an HTTP header name to the name of an environment
	// variable holding its value. A variable that is unset or empty at
	// request time means the header is simply not sent.
	Headers map[string]string `yaml:"headers,omitempty" json:"headers,omitempty"`

	// Timeout bounds one request to this collection's endpoint. Zero means
	// no adapter-imposed timeout beyond the context's own deadline.
	//
	// This field is not (de)serialized directly: LoadConfigYAML/LoadConfigJSON
	// read it from a "timeout" string (e.g. "10s", parsed with
	// time.ParseDuration) since YAML/JSON have no native duration type.
	Timeout time.Duration `yaml:"-" json:"-"`

	// ClientSideFilter opts a collection into fetching a broader result than
	// a dal.Query's Where() strictly asks for, and filtering the remainder
	// in memory. It exists ONLY for public, non-confidential reference data:
	// the point of failing closed by default (see query.go) is that an
	// access-policy predicate that cannot be pushed down must refuse, not
	// silently fetch a superset and hide the extra rows after the fact. Set
	// this only on a collection where over-fetching cannot leak anything a
	// caller was not already allowed to see.
	ClientSideFilter bool `yaml:"clientSideFilter,omitempty" json:"clientSideFilter,omitempty"`

	// InsecureAllowLoopback is a TEST-ONLY escape hatch, never intended for a
	// real descriptor. Setting it relaxes two of the Phase 1 HTTP bounds for
	// THIS collection alone, and only when URLTemplate's host is literally a
	// loopback address (127.0.0.1, ::1, or "localhost" — validate() rejects
	// it otherwise even with this set): (1) URLTemplate may use http://
	// instead of https://; (2) the default guarded dialer (see
	// security.go's guardedDialContext) permits dialing that loopback
	// address. It exists only so this package's own test suite — and a
	// consumer's — can exercise real HTTP against an httptest.Server, which
	// always speaks plain HTTP on 127.0.0.1. It is deliberately NOT
	// (de)serializable from LoadConfigYAML/LoadConfigJSON (see collectionFile
	// in config.go): a production descriptor loaded from a file can never
	// enable it, only Go code building a Collection literal directly can.
	// Every other blocked address class (private, link-local, metadata,
	// multicast, unspecified) stays blocked even with this set.
	InsecureAllowLoopback bool `yaml:"-" json:"-"`
}

Collection declaratively describes one read-only HTTP/JSON source: the endpoint to call, which fields of a dal.Query can be pushed into it, where the rows live in the JSON response, and which field identifies a row's dal.DB record key.

A Collection carries no secrets: Headers maps a header name to the name of an environment variable dalgo2http reads at request time, never a literal value (see README.md "Design constraints").

type Config

type Config struct {
	Collections []Collection

	// Client performs live HTTP requests. Nil means http.DefaultClient.
	Client *http.Client

	// Snapshots is an optional read-only store of recorded fixtures, keyed
	// by SnapshotKey(collection, params). Nil means no snapshot fallback is
	// possible regardless of Mode. Use Record to populate a directory that
	// can then be wrapped with os.DirFS (or embed.FS for shipped fixtures).
	Snapshots fs.FS

	// Mode governs live-vs-snapshot behaviour. Empty means
	// ModeLiveThenSnapshot.
	Mode Mode
}

Config configures a dalgo2http database: the collections it serves, the HTTP client and optional recorded-snapshot store it reads through, and the Mode governing live-vs-snapshot behaviour.

func LoadConfigJSON

func LoadConfigJSON(data []byte) (Config, error)

LoadConfigJSON parses a Config from JSON. See LoadConfigYAML.

func LoadConfigYAML

func LoadConfigYAML(data []byte) (Config, error)

LoadConfigYAML parses a Config from YAML in the shape documented on configFile/collectionFile (see also the example descriptors under examples/), and validates it exactly as NewDB does.

type Method

type Method string

Method is the HTTP method a Collection is fetched with. v0.x supports GET only — see README.md "Design constraints".

const MethodGET Method = "GET"

MethodGET is the only Method a Collection may declare.

type Mode

type Mode string

Mode selects where a collection's rows come from.

const (
	// ModeLive always fetches live and never falls back to a snapshot: a
	// live failure is returned to the caller as-is.
	ModeLive Mode = "live"

	// ModeSnapshot never calls the network: every read is served from
	// Config.Snapshots, failing with ErrSnapshotMiss when none exists.
	ModeSnapshot Mode = "snapshot"

	// ModeLiveThenSnapshot (the default — see NewDB) fetches live and, only
	// when the live request fails for a transient reason (network error,
	// timeout, or a 5xx/429 response — never a 4xx, which is treated as a
	// caller/config error) AND Config.Snapshots has a matching snapshot,
	// falls back to it. This is the behaviour item 4 of the stream brief
	// describes; naming it as its own Mode (rather than making it the only
	// behaviour) lets a caller pin ModeLive for "never serve stale data" or
	// ModeSnapshot for fully offline runs.
	ModeLiveThenSnapshot Mode = "live-then-snapshot"
)

type Observer

type Observer func(Provenance)

Observer is called once per collection read (Get, Exists, or a query) with that read's Provenance.

type Param

type Param struct {
	Location ParamLocation `yaml:"location" json:"location"`
}

Param declares one substitutable field of a Collection's URL template.

type ParamLocation

type ParamLocation string

ParamLocation says where a declared parameter's value is placed in the request: substituted into a {name} placeholder in the path portion of the URL template, or attached as a query-string value (either by substituting a {name} placeholder embedded in the template's query string, or, if the name never appears in the template, appended as an extra query parameter).

const (
	ParamPath  ParamLocation = "path"
	ParamQuery ParamLocation = "query"
)

type Provenance

type Provenance struct {
	Collection string
	Source     Source
	StatusCode int
	FetchedAt  time.Time
}

Provenance describes where one collection read's rows came from.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder captures the Provenance of the most recent call made with its context, so a caller that just wants "was that live or snapshot?" back after a call does not have to write its own Observer.

func NewRecorder

func NewRecorder() *Recorder

NewRecorder creates a Recorder with no observed Provenance yet.

func (*Recorder) Last

func (r *Recorder) Last() (Provenance, bool)

Last returns the most recently observed Provenance, and whether any call has been observed yet.

func (*Recorder) WithContext

func (r *Recorder) WithContext(ctx context.Context) context.Context

WithContext returns ctx wired so calls made with it are observed by r.

type Source

type Source string

Source says whether a result came from a live request or a recorded snapshot.

const (
	SourceLive     Source = "live"
	SourceSnapshot Source = "snapshot"
)