scio

package module
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 23, 2026 License: MIT Imports: 11 Imported by: 0

README

scio

CI codecov Go Report Card CodeQL Go Reference Go Version Release

URI-based data catalog with atomic operations for Go.

scio is the system's authoritative map of data sources — it knows where data lives and provides type-agnostic access via atoms.

The Map

scio registers storage resources and routes operations through URIs:

s := scio.New()

// Register resources (user code provides typed grub wrappers)
s.RegisterDatabase("db://users", usersDB.Atomic())
s.RegisterStore("kv://sessions", sessionsStore.Atomic())
s.RegisterBucket("bcs://documents", docsBucket.Atomic())

// Operations via URI — scio routes to the right provider
atom, _ := s.Get(ctx, "db://users/123")
s.Set(ctx, "kv://sessions/abc", sessionAtom)

// Query databases
results, _ := s.Query(ctx, "db://users", stmt, params)

// Introspect the topology
s.Sources()                    // all registered resources
s.FindBySpec(spec)             // resources sharing a type
s.Related("db://users")        // other resources with same spec

Install

go get github.com/zoobz-io/scio

Requires Go 1.24 or higher.

Quick Start

package main

import (
    "context"
    "fmt"

    "github.com/zoobz-io/grub"
    "github.com/zoobz-io/scio"
)

func main() {
    // Create scio instance
    s := scio.New()

    // Register a database (assumes usersDB is a grub.Database[User])
    err := s.RegisterDatabase("db://users", usersDB.Atomic(),
        scio.WithDescription("User accounts"),
        scio.WithTag("owner", "auth-team"),
    )
    if err != nil {
        panic(err)
    }

    ctx := context.Background()

    // Get a record
    atom, err := s.Get(ctx, "db://users/123")
    if err != nil {
        panic(err)
    }

    // Access fields via typed maps
    fmt.Println(atom.Strings["email"])
    fmt.Println(atom.Ints["age"])

    // Find related resources
    for _, r := range s.Related("db://users") {
        fmt.Printf("Related: %s (%s)\n", r.URI, r.Variant)
    }
}

URI Semantics

Variant Pattern Example
db:// table/key db://users/123
kv:// store/key kv://sessions/abc
bcs:// bucket/path bcs://docs/reports/q4.pdf

Why scio?

  • URI-based routing — logical addressing decoupled from physical storage
  • Type-agnostic — operates on atoms, never needs to know your types
  • Topology awareness — knows what resources exist and their relationships
  • Spec tracking — auto-detects when multiple resources share the same type
  • Metadata support — annotate resources for system use (LLM context, ownership, versioning)

Documentation

Learn
Reference

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT License - see LICENSE for details.

Documentation

Overview

Package scio provides a URI-based data catalog with atomic operations. It serves as the authoritative map of data sources, providing topology intelligence and type-agnostic access via atoms.

Index

Constants

This section is empty.

Variables

View Source 
var (
	// ErrInvalidURI indicates the URI could not be parsed.
	ErrInvalidURI = errors.New("invalid URI")

	// ErrUnknownVariant indicates an unrecognized URI scheme.
	ErrUnknownVariant = errors.New("unknown variant")

	// ErrResourceNotFound indicates the requested resource is not registered.
	ErrResourceNotFound = errors.New("resource not found")

	// ErrResourceExists indicates a resource is already registered at the URI.
	ErrResourceExists = errors.New("resource already exists")

	// ErrVariantMismatch indicates an operation was attempted on the wrong variant.
	ErrVariantMismatch = errors.New("variant mismatch")

	// ErrKeyRequired indicates a key is required but was not provided.
	ErrKeyRequired = errors.New("key required")

	// ErrKeyNotExpected indicates a key was provided but is not used by the operation.
	ErrKeyNotExpected = errors.New("key not expected")

	// ErrInvalidUUID indicates the key could not be parsed as a UUID.
	ErrInvalidUUID = errors.New("invalid UUID")
)

Semantic errors for scio operations.

View Source 
var (
	ErrNotFound        = grub.ErrNotFound
	ErrDuplicate       = grub.ErrDuplicate
	ErrConflict        = grub.ErrConflict
	ErrConstraint      = grub.ErrConstraint
	ErrInvalidKey      = grub.ErrInvalidKey
	ErrReadOnly        = grub.ErrReadOnly
	ErrTTLNotSupported = grub.ErrTTLNotSupported
	ErrInvalidQuery    = grub.ErrInvalidQuery
)

Re-export grub errors for convenience.

Functions

This section is empty.

Types

type Catalog 

type Catalog interface {
	// Sources returns all registered resources.
	Sources() []Resource

	// Databases returns all db:// resources.
	Databases() []Resource

	// Stores returns all kv:// resources.
	Stores() []Resource

	// Buckets returns all bcs:// resources.
	Buckets() []Resource

	// Indexes returns all idx:// resources.
	Indexes() []Resource

	// Searches returns all srch:// resources.
	Searches() []Resource

	// Spec returns the atom spec for a specific resource.
	Spec(uri string) (atom.Spec, error)

	// FindBySpec returns all resources sharing the given spec (by FQDN).
	FindBySpec(spec atom.Spec) []Resource

	// FindByField returns all resources containing the given field.
	FindByField(field string) []Resource

	// Related returns other resources with the same spec as the given URI.
	Related(uri string) []Resource
}

Catalog defines topology introspection operations.

type IndexOperations 

type IndexOperations interface {
	// GetVector retrieves a vector at the given idx:// URI.
	// The URI should include a UUID key component.
	// Returns ErrNotFound if the ID does not exist.
	GetVector(ctx context.Context, uri string) (*grub.AtomicVector, error)

	// UpsertVector stores a vector with metadata at the given idx:// URI.
	// The URI should include a UUID key component.
	UpsertVector(ctx context.Context, uri string, vector []float32, metadata *atom.Atom) error

	// DeleteVector removes the vector at the given idx:// URI.
	// The URI should include a UUID key component.
	// Returns ErrNotFound if the ID does not exist.
	DeleteVector(ctx context.Context, uri string) error

	// VectorExists checks whether a vector exists at the given idx:// URI.
	// The URI should include a UUID key component.
	VectorExists(ctx context.Context, uri string) (bool, error)

	// SearchVectors performs similarity search against an idx:// resource.
	// The URI should reference the index without a key component.
	// Returns the k nearest neighbors, optionally filtered by metadata.
	SearchVectors(ctx context.Context, uri string, vector []float32, k int, filter *atom.Atom) ([]grub.AtomicVector, error)

	// QueryVectors performs similarity search with vecna filter support.
	// The URI should reference the index without a key component.
	QueryVectors(ctx context.Context, uri string, vector []float32, k int, filter *vecna.Filter) ([]grub.AtomicVector, error)

	// FilterVectors returns vectors matching the metadata filter without similarity search.
	// The URI should reference the index without a key component.
	FilterVectors(ctx context.Context, uri string, filter *vecna.Filter, limit int) ([]grub.AtomicVector, error)
}

IndexOperations defines vector index access operations.

type Metadata 

type Metadata struct {
	Description string
	Version     string
	Tags        map[string]string
}

Metadata holds resource annotations for system use.

type Operations 

type Operations interface {
	// Get retrieves an atom at the given URI.
	// Returns ErrNotFound if the key does not exist.
	Get(ctx context.Context, uri string) (*atom.Atom, error)

	// Set stores an atom at the given URI.
	Set(ctx context.Context, uri string, data *atom.Atom) error

	// Delete removes the record at the given URI.
	// Returns ErrNotFound if the key does not exist.
	Delete(ctx context.Context, uri string) error

	// Exists checks whether a record exists at the given URI.
	Exists(ctx context.Context, uri string) (bool, error)

	// Query executes a query statement against a database resource.
	// The URI should reference a db:// resource without a key.
	Query(ctx context.Context, uri string, stmt edamame.QueryStatement, params map[string]any) ([]*atom.Atom, error)

	// Select executes a select statement against a database resource.
	// The URI should reference a db:// resource without a key.
	Select(ctx context.Context, uri string, stmt edamame.SelectStatement, params map[string]any) (*atom.Atom, error)

	// SetWithTTL stores an atom at the given kv:// URI with a TTL.
	// TTL of 0 means no expiration.
	SetWithTTL(ctx context.Context, uri string, data *atom.Atom, ttl time.Duration) error

	// Put stores a blob object at the given bcs:// URI.
	Put(ctx context.Context, uri string, obj *grub.AtomicObject) error
}

Operations defines data access operations.

type RegistrationOption 

type RegistrationOption func(*registrationConfig)

RegistrationOption configures resource registration.

func WithDescription 

func WithDescription(desc string) RegistrationOption

WithDescription sets the resource description.

func WithTag 

func WithTag(key, value string) RegistrationOption

WithTag adds a tag to the resource metadata.

func WithVersion 

func WithVersion(ver string) RegistrationOption

WithVersion sets the resource version.

type Registry 

type Registry interface {
	// RegisterDatabase registers an atomic database at the given URI.
	// The URI should be in the form db://table.
	RegisterDatabase(uri string, db grub.AtomicDatabase, opts ...RegistrationOption) error

	// RegisterStore registers an atomic store at the given URI.
	// The URI should be in the form kv://store.
	RegisterStore(uri string, store grub.AtomicStore, opts ...RegistrationOption) error

	// RegisterBucket registers an atomic bucket at the given URI.
	// The URI should be in the form bcs://bucket.
	RegisterBucket(uri string, bucket grub.AtomicBucket, opts ...RegistrationOption) error

	// RegisterIndex registers an atomic index at the given URI.
	// The URI should be in the form idx://index.
	RegisterIndex(uri string, index grub.AtomicIndex, opts ...RegistrationOption) error

	// RegisterSearch registers an atomic search at the given URI.
	// The URI should be in the form srch://index.
	RegisterSearch(uri string, search grub.AtomicSearch, opts ...RegistrationOption) error
}

Registry defines resource registration operations.

type Resource 

type Resource struct {
	URI      string
	Variant  Variant
	Name     string
	Spec     atom.Spec
	Metadata Metadata
}

Resource represents a registered data source.

type Scio 

type Scio struct {
	// contains filtered or unexported fields
}

Scio is the central data catalog and access point.

func New 

func New() *Scio

New creates a new Scio instance.

func (*Scio) Buckets 

func (s *Scio) Buckets() []Resource

Buckets returns all bcs:// resources.

func (*Scio) Databases 

func (s *Scio) Databases() []Resource

Databases returns all db:// resources.

func (*Scio) Delete 

func (s *Scio) Delete(ctx context.Context, uri string) error

Delete removes the record at the given URI. Returns ErrNotFound if the key does not exist. Returns ErrResourceNotFound if the resource is not registered.

func (*Scio) DeleteDocument 

func (s *Scio) DeleteDocument(ctx context.Context, uri string) error

DeleteDocument removes the document at the given srch:// URI. The URI should include a document ID key component. Returns ErrNotFound if the ID does not exist. Returns ErrResourceNotFound if the resource is not registered.

func (*Scio) DeleteVector 

func (s *Scio) DeleteVector(ctx context.Context, uri string) error

DeleteVector removes the vector at the given idx:// URI. The URI should include a UUID key component. Returns ErrNotFound if the ID does not exist. Returns ErrResourceNotFound if the resource is not registered.

func (*Scio) DocumentExists 

func (s *Scio) DocumentExists(ctx context.Context, uri string) (bool, error)

DocumentExists checks whether a document exists at the given srch:// URI. The URI should include a document ID key component. Returns ErrResourceNotFound if the resource is not registered.

func (*Scio) Exists 

func (s *Scio) Exists(ctx context.Context, uri string) (bool, error)

Exists checks whether a record exists at the given URI. Returns ErrResourceNotFound if the resource is not registered.

func (*Scio) FilterVectors 

func (s *Scio) FilterVectors(ctx context.Context, uri string, filter *vecna.Filter, limit int) ([]grub.AtomicVector, error)

FilterVectors returns vectors matching the metadata filter without similarity search. The URI should reference the index without a key component. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyNotExpected if a key component is provided.

func (*Scio) FindByField 

func (s *Scio) FindByField(field string) []Resource

FindByField returns all resources containing the given field.

func (*Scio) FindBySpec 

func (s *Scio) FindBySpec(spec atom.Spec) []Resource

FindBySpec returns all resources sharing the given spec (by FQDN).

func (*Scio) Get 

func (s *Scio) Get(ctx context.Context, uri string) (*atom.Atom, error)

Get retrieves an atom at the given URI. Returns ErrNotFound if the key does not exist. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyRequired if no key is provided.

func (*Scio) GetDocument 

func (s *Scio) GetDocument(ctx context.Context, uri string) (*grub.AtomicDocument, error)

GetDocument retrieves a document at the given srch:// URI. The URI should include a document ID key component. Returns ErrNotFound if the ID does not exist. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyRequired if no key is provided.

func (*Scio) GetVector 

func (s *Scio) GetVector(ctx context.Context, uri string) (*grub.AtomicVector, error)

GetVector retrieves a vector at the given idx:// URI. The URI should include a UUID key component. Returns ErrNotFound if the ID does not exist. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyRequired if no key is provided.

func (*Scio) IndexDocument 

func (s *Scio) IndexDocument(ctx context.Context, uri string, data *atom.Atom) error

IndexDocument stores a document with atomized content at the given srch:// URI. The URI should include a document ID key component. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyRequired if no key is provided.

func (*Scio) Indexes 

func (s *Scio) Indexes() []Resource

Indexes returns all idx:// resources.

func (*Scio) Put 

func (s *Scio) Put(ctx context.Context, uri string, obj *grub.AtomicObject) error

Put stores a blob object at the given bcs:// URI. Returns ErrVariantMismatch if the URI is not a bcs:// resource.

func (*Scio) Query 

func (s *Scio) Query(ctx context.Context, uri string, stmt edamame.QueryStatement, params map[string]any) ([]*atom.Atom, error)

Query executes a query statement against a database resource. The URI should reference a db:// resource without a key component. Returns ErrVariantMismatch if the URI is not a db:// resource. Returns ErrKeyNotExpected if a key component is provided.

func (*Scio) QueryVectors 

func (s *Scio) QueryVectors(ctx context.Context, uri string, vector []float32, k int, filter *vecna.Filter) ([]grub.AtomicVector, error)

QueryVectors performs similarity search with vecna filter support. The URI should reference the index without a key component. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyNotExpected if a key component is provided.

func (*Scio) RegisterBucket 

func (s *Scio) RegisterBucket(uri string, bucket grub.AtomicBucket, opts ...RegistrationOption) error

RegisterBucket registers an atomic bucket at the given URI. The URI should be in the form bcs://bucket.

func (*Scio) RegisterDatabase 

func (s *Scio) RegisterDatabase(uri string, db grub.AtomicDatabase, opts ...RegistrationOption) error

RegisterDatabase registers an atomic database at the given URI. The URI should be in the form db://table.

func (*Scio) RegisterIndex 

func (s *Scio) RegisterIndex(uri string, index grub.AtomicIndex, opts ...RegistrationOption) error

RegisterIndex registers an atomic index at the given URI. The URI should be in the form idx://index.

func (*Scio) RegisterSearch 

func (s *Scio) RegisterSearch(uri string, search grub.AtomicSearch, opts ...RegistrationOption) error

RegisterSearch registers an atomic search at the given URI. The URI should be in the form srch://index.

func (*Scio) RegisterStore 

func (s *Scio) RegisterStore(uri string, store grub.AtomicStore, opts ...RegistrationOption) error

RegisterStore registers an atomic store at the given URI. The URI should be in the form kv://store.

func (*Scio) Related 

func (s *Scio) Related(uri string) []Resource

Related returns other resources with the same spec as the given URI. The resource at the given URI is excluded from the results.

func (*Scio) Resource 

func (s *Scio) Resource(uri string) *Resource

Resource returns the resource metadata for a specific URI. Returns nil if the resource is not registered.

func (*Scio) SearchDocuments 

func (s *Scio) SearchDocuments(ctx context.Context, uri string, search *lucene.Search) ([]grub.AtomicDocument, error)

SearchDocuments performs a full-text search against a srch:// resource. The URI should reference the search index without a key component. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyNotExpected if a key component is provided.

func (*Scio) SearchVectors 

func (s *Scio) SearchVectors(ctx context.Context, uri string, vector []float32, k int, filter *atom.Atom) ([]grub.AtomicVector, error)

SearchVectors performs similarity search against an idx:// resource. The URI should reference the index without a key component. Returns the k nearest neighbors, optionally filtered by metadata. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyNotExpected if a key component is provided.

func (*Scio) Searches 

func (s *Scio) Searches() []Resource

Searches returns all srch:// resources.

func (*Scio) Select 

func (s *Scio) Select(ctx context.Context, uri string, stmt edamame.SelectStatement, params map[string]any) (*atom.Atom, error)

Select executes a select statement against a database resource. The URI should reference a db:// resource without a key component. Returns ErrVariantMismatch if the URI is not a db:// resource. Returns ErrKeyNotExpected if a key component is provided.

func (*Scio) Set 

func (s *Scio) Set(ctx context.Context, uri string, data *atom.Atom) error

Set stores an atom at the given URI. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyRequired if no key is provided. For kv:// resources, use SetWithTTL to specify expiration.

func (*Scio) SetWithTTL 

func (s *Scio) SetWithTTL(ctx context.Context, uri string, data *atom.Atom, ttl time.Duration) error

SetWithTTL stores an atom at the given kv:// URI with a TTL. TTL of 0 means no expiration. Returns ErrVariantMismatch if the URI is not a kv:// resource.

func (*Scio) Sources 

func (s *Scio) Sources() []Resource

Sources returns all registered resources.

func (*Scio) Spec 

func (s *Scio) Spec(uri string) (atom.Spec, error)

Spec returns the atom spec for a specific resource. Returns ErrResourceNotFound if the resource is not registered.

func (*Scio) Stores 

func (s *Scio) Stores() []Resource

Stores returns all kv:// resources.

func (*Scio) UpsertVector 

func (s *Scio) UpsertVector(ctx context.Context, uri string, vector []float32, metadata *atom.Atom) error

UpsertVector stores a vector with metadata at the given idx:// URI. The URI should include a UUID key component. Returns ErrResourceNotFound if the resource is not registered. Returns ErrKeyRequired if no key is provided.

func (*Scio) VectorExists 

func (s *Scio) VectorExists(ctx context.Context, uri string) (bool, error)

VectorExists checks whether a vector exists at the given idx:// URI. The URI should include a UUID key component. Returns ErrResourceNotFound if the resource is not registered.

type SearchOperations 

type SearchOperations interface {
	// GetDocument retrieves a document at the given srch:// URI.
	// The URI should include a document ID key component.
	// Returns ErrNotFound if the ID does not exist.
	GetDocument(ctx context.Context, uri string) (*grub.AtomicDocument, error)

	// IndexDocument stores a document with atomized content at the given srch:// URI.
	// The URI should include a document ID key component.
	IndexDocument(ctx context.Context, uri string, data *atom.Atom) error

	// DeleteDocument removes the document at the given srch:// URI.
	// The URI should include a document ID key component.
	// Returns ErrNotFound if the ID does not exist.
	DeleteDocument(ctx context.Context, uri string) error

	// DocumentExists checks whether a document exists at the given srch:// URI.
	// The URI should include a document ID key component.
	DocumentExists(ctx context.Context, uri string) (bool, error)

	// SearchDocuments performs a full-text search against a srch:// resource.
	// The URI should reference the search index without a key component.
	SearchDocuments(ctx context.Context, uri string, search *lucene.Search) ([]grub.AtomicDocument, error)
}

SearchOperations defines full-text search access operations.

type URI 

type URI struct {
	Variant  Variant
	Resource string // table, store, or bucket name
	Key      string // record key or blob path (may be empty for queries)
}

URI represents a parsed data address.

func ParseURI 

func ParseURI(raw string) (*URI, error)

ParseURI parses a raw URI string into its components. Supported formats:

  • db://table/key -> Variant: db, Resource: table, Key: key
  • kv://store/key -> Variant: kv, Resource: store, Key: key
  • bcs://bucket/path -> Variant: bcs, Resource: bucket, Key: path (full remainder)

func (*URI) ResourceURI 

func (u *URI) ResourceURI() string

ResourceURI returns the URI without the key component.

func (*URI) String 

func (u *URI) String() string

String returns the URI as a string.

type Variant 

type Variant string

Variant represents a storage type. 

const (
	// VariantDatabase represents SQL database storage (db://).
	VariantDatabase Variant = "db"

	// VariantStore represents key-value storage (kv://).
	VariantStore Variant = "kv"

	// VariantBucket represents blob/object storage (bcs://).
	VariantBucket Variant = "bcs"

	// VariantIndex represents vector index storage (idx://).
	VariantIndex Variant = "idx"

	// VariantSearch represents full-text search storage (srch://).
	VariantSearch Variant = "srch"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.