Documentation
¶
Overview ¶
Package truegrain is the Go client for a truegrain semantic engine.
truegrain compiles an Apache Ossie semantic model into governed warehouse SQL. It answers questions phrased as metrics, dimensions and structured filters, and refuses the ones that would return a plausible wrong number. See https://github.com/rk-chavali/truegrain.
This package depends only on the standard library. It is deliberately a separate module from the engine: importing the engine would pull in the BigQuery client, an OIDC verifier and an MCP server, none of which a caller asking for a number needs.
Getting started ¶
client, err := truegrain.FromEnv() // SEMANTIC_URL, SEMANTIC_TOKEN
if err != nil {
return err
}
result, err := client.Query(ctx, truegrain.Request{
Metrics: []string{"sales.order_revenue"},
Dimensions: []string{"sales.customers.region"},
Filters: []truegrain.Filter{truegrain.Eq("sales.orders.status", "shipped")},
})
Every Result carries CompiledSQL and ModelVersion. That is how a disagreement about a number gets settled: two results with the same digest were produced by exactly the same definitions.
There is no way to send SQL ¶
No method on Client accepts a SQL string, because no endpoint on the engine accepts one. The only expressible request is a semantic one, which is what makes an ungoverned query impossible rather than merely discouraged.
Refusals are answers ¶
The engine refuses questions it cannot answer correctly. A refusal is not a failure of the request so much as a statement about it, and it says what to do next. Branch on that rather than on the message:
result, err := client.Run(ctx, req)
var refusal *truegrain.Refused
if errors.As(err, &refusal) {
switch {
case refusal.ShouldModify():
// Answerable, but not as written. refusal.Hint names the metrics
// defined at the grain where the question is well defined.
case refusal.ShouldWait():
// Nothing about the request is wrong. The same call may work shortly.
case refusal.IsFinal():
// A denial. Say so rather than substituting a different metric that
// answers a different question.
}
}
That distinction is the difference between a self-correcting agent loop and an infinite one.
Slow warehouses ¶
A warehouse query can run longer than an HTTP request should be held open. Client.Run submits the query, polls with backoff, collects every page and returns one Result, so no caller writes that loop:
result, err := client.Run(ctx, truegrain.Request{
Metrics: []string{"sales.order_revenue"},
})
Cancelling the context cancels the job on the engine as well, rather than leaving the warehouse spending on an answer nobody is waiting for.
Index ¶
- Constants
- Variables
- type Cell
- type Client
- func (c *Client) CancelJob(ctx context.Context, id string) (*Job, error)
- func (c *Client) Compile(ctx context.Context, req Request) (*Compiled, error)
- func (c *Client) Dimensions(ctx context.Context, metric string) ([]Dimension, error)
- func (c *Client) Health(ctx context.Context) (*Health, error)
- func (c *Client) Job(ctx context.Context, id, cursor string, pageSize int) (*Job, error)
- func (c *Client) Metric(ctx context.Context, name string) (*Metric, error)
- func (c *Client) Metrics(ctx context.Context, search string) ([]Metric, error)
- func (c *Client) ModelVersion(ctx context.Context) (string, error)
- func (c *Client) Namespaces(ctx context.Context) ([]Namespace, error)
- func (c *Client) Query(ctx context.Context, req Request) (*Result, error)
- func (c *Client) Run(ctx context.Context, req Request, opts ...RunOptions) (*Result, error)
- func (c *Client) Submit(ctx context.Context, req Request) (*Job, error)
- func (c *Client) Wait(ctx context.Context, id string, opts ...RunOptions) (*Job, error)
- type Compiled
- type DialectSecurity
- type Dimension
- type Filter
- func Between(dimension string, low, high any) Filter
- func Eq(dimension string, value any) Filter
- func Gt(dimension string, value any) Filter
- func Gte(dimension string, value any) Filter
- func In(dimension string, values ...any) Filter
- func IsNotNull(dimension string) Filter
- func IsNull(dimension string) Filter
- func Lt(dimension string, value any) Filter
- func Lte(dimension string, value any) Filter
- func Ne(dimension string, value any) Filter
- func NotIn(dimension string, values ...any) Filter
- type Governance
- type Health
- type Job
- type JobState
- type Metric
- type Namespace
- type Option
- type Order
- type Refused
- type Request
- type Result
- type Retry
- type RunOptions
- type TransportError
- type UnauthorizedError
Constants ¶
const DefaultTimeout = 60 * time.Second
DefaultTimeout bounds a single HTTP call.
Variables ¶
var Operations = map[string]string{
"getHealth": "Health",
"getModelVersion": "ModelVersion",
"listNamespaces": "Namespaces",
"listMetrics": "Metrics",
"describeMetric": "Metric",
"listDimensions": "Dimensions",
"query": "Query",
"compile": "Compile",
"submitJob": "Submit",
"getJob": "Job",
"cancelJob": "CancelJob",
}
Operations maps each OpenAPI operationId to the Client method implementing it.
TestClientCoversTheSpec asserts this covers every operation in spec/openapi.yaml, so an endpoint the engine gains cannot quietly go unsupported here.
Functions ¶
This section is empty.
Types ¶
type Cell ¶
type Cell any
Cell is one value in a result row. The engine returns JSON, so a cell is whatever JSON gave: a string, a float64, a bool, or nil.
type Client ¶
type Client struct {
// BaseURL is where the engine is served, without a trailing slash.
BaseURL string
// HTTPClient is used for every request. Replace it to add tracing, a
// proxy, or a different timeout.
HTTPClient *http.Client
// UserAgent identifies the caller. The engine's audit log records it,
// which is how one agent is told from another.
UserAgent string
// contains filtered or unexported fields
}
Client is a connection to one truegrain engine.
It is safe for concurrent use. Build one with New or FromEnv.
func FromEnv ¶
FromEnv builds a client from SEMANTIC_URL and SEMANTIC_TOKEN.
Keeping the token in the environment keeps it out of source and out of the process list.
func (*Client) CancelJob ¶
CancelJob stops a running query. It is idempotent: cancelling a finished job returns its existing state rather than an error.
func (*Client) Compile ¶
Compile returns the SQL a request compiles to, without running it.
A dry run is still governed: a request you may not run returns a refusal and no SQL, and the inspection is audited. It is a way to see what a query would do, not a way around the gate.
func (*Client) Dimensions ¶
Dimensions lists what is available for grouping and filtering.
Pass a metric name to get only the dimensions valid for it, which is almost always what you want.
func (*Client) Health ¶
Health reports what this deployment enforces, including what it does not.
Worth reading before trusting the deployment with anything sensitive: Governance.Note and Executor state the gaps in plain language.
func (*Client) Job ¶
Job polls a job and reads one page of its rows.
cursor comes from a previous page's NextCursor; pass an empty string for the first page. pageSize of zero uses the engine's default.
func (*Client) Metric ¶
Metric returns one metric's definition and the dimensions legal for it.
name is qualified as `namespace.metric`, or bare when unambiguous across the workspace.
func (*Client) Metrics ¶
Metrics lists every metric this identity may read.
search is a case-insensitive substring over name and description; pass an empty string for all of them.
func (*Client) ModelVersion ¶
ModelVersion returns the workspace digest currently served.
Every Result carries this too. Two results with the same digest were produced by exactly the same definitions.
func (*Client) Namespaces ¶
Namespaces lists the namespaces in the workspace, their owners and whether each one loaded.
func (*Client) Query ¶
Query answers a question and returns rows.
It holds the connection open for the whole query. Use Client.Run when the warehouse might be slow.
A refusal comes back as a *Refused; check it with errors.As and branch on ShouldModify, ShouldWait and IsFinal rather than on the message.
func (*Client) Run ¶
Run executes a query that may take longer than an HTTP request should.
It submits the query, polls until it finishes, collects every page and returns one Result, so no caller writes that loop. A large warehouse scan routinely outlives the default timeout of an HTTP client, and a synchronous call that times out leaves the query running and billable with nobody reading it.
The governance gate runs during submission, so a request this identity may not make returns a refusal immediately rather than after a wait.
Cancelling ctx cancels the job on the engine too, rather than leaving the warehouse spending on an answer nobody is waiting for.
func (*Client) Submit ¶
Submit starts a query in the background and returns immediately.
The returned Job already carries CompiledSQL, because compilation happened synchronously. A job coming back at all means the query was authorized, not merely that it was accepted.
Prefer Client.Run unless you genuinely need to do something else while the query runs.
func (*Client) Wait ¶
Wait polls until a job reaches a terminal state.
It backs off from a fast first poll to a two second ceiling, so a query that finishes quickly is not delayed and a slow one is not hammered.
If ctx is cancelled the job is cancelled on the engine before returning, so giving up here also stops the warehouse work.
type Compiled ¶
type Compiled struct {
SQL string `json:"compiled_sql"`
Columns []string `json:"columns"`
Dialect string `json:"dialect"`
Namespace string `json:"namespace"`
ModelVersion string `json:"model_version"`
// Parts is how many facts were aggregated separately and joined. More than
// one means the query spanned grains or namespaces.
Parts int `json:"parts"`
}
Compiled is the SQL a request compiles to, without having run it.
type DialectSecurity ¶
type DialectSecurity struct {
// Dialect names the compilation target, for example bigquery or duckdb.
Dialect string `json:"dialect"`
// ColumnLevelSecurity reports whether the warehouse restricts columns on
// its own, such as through BigQuery policy tags.
ColumnLevelSecurity bool `json:"column_level_security"`
// RowLevelSecurity reports whether the warehouse filters rows on its own.
RowLevelSecurity bool `json:"row_level_security"`
// Note states the limits in plain language.
Note string `json:"note,omitempty"`
}
DialectSecurity reports what the target warehouse enforces by itself, independently of this engine.
When both flags are false, the engine's own gate is the only control, and it applies only to queries that go through the engine.
type Dimension ¶
type Dimension struct {
Name string `json:"name"`
Namespace string `json:"namespace,omitempty"`
Datatype string `json:"datatype,omitempty"`
Description string `json:"description,omitempty"`
Synonyms []string `json:"synonyms,omitempty"`
// IsTime reports whether this dimension accepts a Grain.
IsTime bool `json:"is_time,omitempty"`
// Grains are the legal time buckets. Populated only when IsTime.
Grains []string `json:"grains,omitempty"`
}
Dimension is something a metric can be grouped or filtered by.
type Filter ¶
type Filter struct {
// Dimension is `dataset.field`, or `namespace.dataset.field`.
Dimension string `json:"dimension"`
// Op is the comparison. Use the constructors rather than setting it.
Op string `json:"op"`
// Values holds one value for the comparisons, two for between, one or more
// for in and not_in, and none for the null checks.
Values []any `json:"values,omitempty"`
}
Filter is a structured predicate.
It is never a SQL fragment. That is what keeps an ungoverned predicate inexpressible, and it is why these are constructors rather than a string builder: a misspelled operator is caught at the call site rather than as a refusal from the server.
func Between ¶
Between matches rows where the dimension falls within the inclusive range.
The engine refuses a reversed range rather than matching no rows, because an empty result that looks like a real answer is worse than an error.
type Governance ¶
type Governance struct {
// Resolver names the implementation, for example bigquery-policy-tags,
// file, or allow-all.
Resolver string `json:"resolver"`
// ColumnLevel reports whether real column-level decisions are made. False
// means every caller may read every column.
ColumnLevel bool `json:"column_level"`
// Note states the limits in plain language.
Note string `json:"note"`
}
Governance describes what this engine's policy resolver enforces.
It is deliberately honest about a resolver that enforces nothing: a deployment that grants everything says so here, because overstating a guarantee is worse than not offering one.
type Health ¶
type Health struct {
// Workspace names the namespaces served.
Workspace string `json:"workspace"`
// WorkspaceDigest identifies the exact set of definitions in force. Two
// results carrying the same digest came from the same model.
WorkspaceDigest string `json:"workspace_digest"`
// OssieSpecVersion is the Apache Ossie spec version the models declare.
OssieSpecVersion string `json:"ossie_spec_version,omitempty"`
// Namespaces reports each namespace and whether it loaded.
Namespaces []Namespace `json:"namespaces,omitempty"`
// Dialect describes the compilation target and what it secures itself.
Dialect DialectSecurity `json:"dialect"`
// Governance describes the policy resolver this engine applies.
Governance Governance `json:"governance"`
// Executor names the warehouse driver and states its limits, including
// whether queries run as the caller.
Executor string `json:"executor"`
// MetricCount and DimensionCount are what this identity may see.
MetricCount int `json:"metric_count"`
DimensionCount int `json:"dimension_count"`
// SupportedFilterOps are the operators [Filter] may use.
SupportedFilterOps []string `json:"supported_filter_ops,omitempty"`
// SupportedGrains are the legal values for Request.Grain.
SupportedGrains []string `json:"supported_grains,omitempty"`
// EnforcementNotes states the gaps in plain language. Read these.
EnforcementNotes []string `json:"enforcement_notes,omitempty"`
}
Health describes what a deployment enforces, including what it does not.
Worth reading before trusting a deployment with anything sensitive. Overstating a governance guarantee is worse than not offering one, so the engine reports its gaps here in plain language.
Two separate things are reported and they are not the same: Governance says what this engine enforces before it emits SQL, and DialectSecurity says what the warehouse enforces on its own. A caller with direct warehouse credentials is subject only to the second.
type Job ¶
type Job struct {
ID string `json:"job_id"`
State JobState `json:"state"`
// CompiledSQL is available from the moment of submission, because
// compilation and the governance gate run before the job exists.
CompiledSQL string `json:"compiled_sql,omitempty"`
Columns []string `json:"columns,omitempty"`
// Rows is one page. Use [Client.Run] to collect every page.
Rows [][]Cell `json:"rows,omitempty"`
// RowCount is the size of the whole result, not of this page.
RowCount int `json:"row_count,omitempty"`
// NextCursor is present when more rows remain. Pass it back to read on.
NextCursor string `json:"next_cursor,omitempty"`
ModelVersion string `json:"model_version,omitempty"`
Namespace string `json:"namespace,omitempty"`
Dialect string `json:"dialect,omitempty"`
// The refusal fields, set only when State is StateFailed. They carry the
// same vocabulary a synchronous refusal does.
Code string `json:"code,omitempty"`
Reason string `json:"reason,omitempty"`
Hint string `json:"hint,omitempty"`
Retry Retry `json:"retry,omitempty"`
}
Job is one asynchronously executing query.
type JobState ¶
type JobState string
JobState is where an asynchronous query has got to.
StateRunning is the only non-terminal one, and StateCancelled is distinct from StateFailed because the caller stopped it rather than the warehouse failing.
type Metric ¶
type Metric struct {
// Name is qualified as `namespace.metric`.
Name string `json:"name"`
Namespace string `json:"namespace"`
// Description is the grounding text an agent reads to decide whether this
// metric answers the question asked. It is not decoration.
Description string `json:"description"`
Datatype string `json:"datatype,omitempty"`
Synonyms []string `json:"synonyms,omitempty"`
// Definition is the metric's expression. Present on [Client.Metric],
// absent from a listing.
Definition string `json:"definition,omitempty"`
// Dimensions lists only the dimensions this identity may group by.
Dimensions []Dimension `json:"-"`
}
Metric is something the engine can measure.
type Namespace ¶
type Namespace struct {
Name string `json:"name"`
Available bool `json:"available"`
MetricCount int `json:"metric_count"`
Digest string `json:"digest,omitempty"`
Owners []string `json:"owners,omitempty"`
// Error is set only when the namespace failed to load. An unavailable
// namespace is reported rather than omitted, so absence is never mistaken
// for a model that simply had no such metrics.
Error string `json:"error,omitempty"`
}
Namespace is one team's independently owned set of definitions.
type Option ¶
type Option func(*Client)
Option configures a Client.
func WithHTTPClient ¶
WithHTTPClient replaces the HTTP client, which is how to add a proxy, a tracing transport, or a different timeout.
func WithToken ¶
WithToken sets the bearer token identifying the workload.
Read it from the environment; never hard-code one. A token passed as a command-line flag ends up in the process list.
func WithUserAgent ¶
WithUserAgent overrides the identifying header, which is useful when you want the engine's audit log to distinguish one agent from another.
type Order ¶
type Order struct {
// Field is a metric or dimension name from the same request.
Field string `json:"field"`
// Desc sorts descending.
Desc bool `json:"desc,omitempty"`
}
Order is one sort key.
type Refused ¶
type Refused struct {
// Code is the machine-readable reason. Branch on this, never on the text.
Code string `json:"code"`
// Reason is one sentence stating what was wrong.
Reason string `json:"reason"`
// Hint is what to do instead. For a fan-out refusal it names the metrics
// defined at the grain where the question is well defined.
Hint string `json:"hint,omitempty"`
// Retry classifies the refusal. See [Retry].
Retry Retry `json:"retry"`
// Status is the HTTP status that carried the refusal, or 0 when the
// refusal came from a failed job rather than from a response.
Status int `json:"-"`
}
Refused is a structured denial from the engine.
The engine understood the request and declined it, and said what to do about that. Use errors.As to recover it:
var refusal *truegrain.Refused
if errors.As(err, &refusal) && refusal.ShouldModify() {
// adjust the request and try again
}
func (*Refused) IsFinal ¶
IsFinal reports that no version of this request from this caller will succeed.
Usually a denial. Say so rather than substituting a different metric that answers a different question.
func (*Refused) ShouldModify ¶
ShouldModify reports that the request is answerable, but not as written.
Change the arguments and try again. Repeating it unchanged will not work. Hint usually names what to change.
func (*Refused) ShouldWait ¶
ShouldWait reports that nothing about the request is wrong.
Something outside it failed, such as the policy source being unreachable. The same request may succeed shortly.
type Request ¶
type Request struct {
// Metrics names what to measure. Metrics at different grains, or in
// different namespaces, are aggregated separately and joined on the shared
// dimensions rather than refused.
Metrics []string `json:"metrics"`
// Dimensions group the result: `dataset.field`, or
// `namespace.dataset.field` when a bare name would be ambiguous.
Dimensions []string `json:"dimensions,omitempty"`
// Filters are structured predicates, never SQL text. Build them with [Eq],
// [In], [Between] and the rest.
Filters []Filter `json:"filters,omitempty"`
// Grain buckets the selected time dimension: day, week, month, quarter,
// year and the sub-day units.
Grain string `json:"grain,omitempty"`
// Limit caps the rows returned. Zero applies the engine's own default.
Limit int `json:"limit,omitempty"`
// OrderBy sorts the result.
OrderBy []Order `json:"order_by,omitempty"`
}
Request is a semantic question.
It is the same shape for Client.Query, Client.Compile, Client.Submit and Client.Run. There is no field for SQL, by design.
type Result ¶
type Result struct {
Columns []string `json:"columns"`
Rows [][]Cell `json:"rows"`
// CompiledSQL is the exact statement executed. Carried on every response
// on purpose: it is how a disagreement about a number gets settled.
CompiledSQL string `json:"compiled_sql"`
ModelVersion string `json:"model_version"`
Namespace string `json:"namespace"`
Dialect string `json:"dialect"`
}
Result is rows, plus the provenance needed to defend the numbers in them.
type Retry ¶
type Retry string
Retry is what a caller should do about a refusal.
It is the field an agent branches on. Without it a caller either gives up on a typo or loops forever on a denial.
const ( // RetryModify means the request was understood and cannot be answered as // written. Changing the arguments may work; repeating it will not. RetryModify Retry = "modify" // RetryLater means the same request may succeed later. Something outside // the request failed, such as the policy source being unreachable, or this // caller is at their concurrency limit. RetryLater Retry = "later" // RetryNever means no version of this request from this caller will // succeed. A denial, or something the model itself has to change. RetryNever Retry = "never" )
type RunOptions ¶
type RunOptions struct {
// PollInterval is the first gap between polls. It backs off to two seconds.
// Zero uses a sensible default.
PollInterval time.Duration
// PageSize is rows per page while collecting. Lower it when rows are wide.
PageSize int
}
RunOptions tunes Client.Run.
type TransportError ¶
type TransportError struct {
// URL is what was being called.
URL string
// Err is the underlying cause.
Err error
}
TransportError means the engine could not be reached, or answered with something unparseable.
It is a network or deployment problem, never a statement about the request, so retrying the same request is reasonable.
func (*TransportError) Error ¶
func (e *TransportError) Error() string
func (*TransportError) Unwrap ¶
func (e *TransportError) Unwrap() error
Unwrap returns the underlying cause.
type UnauthorizedError ¶
type UnauthorizedError struct {
// credential from a malformed one, which would let a caller probe for
// valid tokens.
Message string
}
UnauthorizedError means credentials are missing or were not recognised.
func (*UnauthorizedError) Error ¶
func (e *UnauthorizedError) Error() string