knowledge-engine-toolkit

module

v0.0.5 Latest Latest Go to latest Published: Oct 12, 2026 License: MIT

README ¶

Tech AI Knowledge Engine

A generic, layered service substrate for building multi-language (Go + Python) backend systems — the reusable "how we build services" library. It carries no product-domain logic: just the foundation, clients, repositories, services, pipelines, workflows, transport, and batch-execution machinery that a service stack is assembled from, behind dependency-free core interfaces.

MIT-licensed and semantically versioned (the git tag is the source of truth). The design rules — layers, swappable components, decorators, error codes — are in ARCHITECTURE.md; contribution rules are in CONTRIBUTING.md.

Layout

go.mod                  ONE Go module: github.com/gt-tech-ai/knowledge-engine-toolkit (packages under go/)
go/
  core/                 Layer 0 — pure interfaces + types (no external deps)
  foundation/           Layer 1 — config, logger, metrics, tracer, resilience, cache, stream (bounded backpressured results)
  clients/              Layer 2 — external-system integration (db, cache, messaging, rpc, storage, lock)
  repos/                Layer 3 — data-access base (repository, cursor, decorators)
  services/             Layer 4 — domain-service base
  pipelines/            Layer 5 — stateless transforms
  workflows/            Layer 6 — multi-step orchestration
  transport/            Layer 7 — HTTP/RPC/JSON:API
  execution/            batch fan-out substrate (discover → map → fan-out → aggregate)
  helpers/              small shared utilities
  tests/                black-box tests (unit + integration via testcontainers), fixtures, generated mocks
python/
  techai_webutils/      the Python mirror of the Go substrate (published to PyPI)

Layers import downward only (core → foundation → clients → … → transport); .golangci.yml carries depguard rules for the boundaries. Every swappable component is a Kind + Config + factory (NewFromConfig in Go, new_<type>_from_config in Python), selected by configuration and injected at a composition root. See ARCHITECTURE.md.

Consuming

Go:

go get github.com/gt-tech-ai/knowledge-engine-toolkit@latest

import "github.com/gt-tech-ai/knowledge-engine-toolkit/go/foundation/logger"

Python:

pip install techai-webutils   # or: uv add techai-webutils

Developing

Mocks are generated by mockgen into go/tests/mocks/; regenerate them (as CI does) after changing a mocked interface:

go install go.uber.org/mock/mockgen@v0.6.0
go generate ./go/tests/mocks/...

Then run the checks:

# Go
go vet ./go/...
golangci-lint run ./go/...          # layer-boundary rules + the linter set
go test ./go/...                    # unit
go test -tags=integration ./go/...  # integration (real deps via testcontainers)

# Python
cd python/techai_webutils
uv run ruff check . && uv run basedpyright src && uv run pytest

Integration tests use real dependencies in throwaway Docker containers (testcontainers) — never fakes.

License

MIT — see LICENSE.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
go
clients/analytics Package analytics builds the AnalyticsStore: the zero-infrastructure stub (the default) or the Cassandra-protocol store (Apache Cassandra locally, Amazon Keyspaces in staging, chosen by the nested Cassandra config).	Package analytics builds the AnalyticsStore: the zero-infrastructure stub (the default) or the Cassandra-protocol store (Apache Cassandra locally, Amazon Keyspaces in staging, chosen by the nested Cassandra config).
clients/analytics/cassandra Package cassandra is the Cassandra-protocol AnalyticsStore (Apache Cassandra or Amazon Keyspaces behind clients/cassandra.Session).	Package cassandra is the Cassandra-protocol AnalyticsStore (Apache Cassandra or Amazon Keyspaces behind clients/cassandra.Session).
clients/analytics/decorators Package decorators wraps an AnalyticsStore with resilience and observability, in the order recovery → metrics → tracing → circuit breaker → timeout (outermost first), and wraps a clients/cassandra.Session's writes in the client stack.	Package decorators wraps an AnalyticsStore with resilience and observability, in the order recovery → metrics → tracing → circuit breaker → timeout (outermost first), and wraps a clients/cassandra.Session's writes in the client stack.
clients/analytics/stub Package stub is the default analytics store: it accepts every write and answers every query with an empty result, so the graph boots with no Cassandra.	Package stub is the default analytics store: it accepts every write and answers every query with an empty result, so the graph boots with no Cassandra.
clients/cache Package cache provides a builder for ByteCache implementations with multiple backends.	Package cache provides a builder for ByteCache implementations with multiple backends.
clients/cache/decorators Package decorators provides cache decorators for cross-cutting concerns.	Package decorators provides cache decorators for cross-cutting concerns.
clients/cache/redis Package redis provides a Redis-backed cache implementation with singleflight support.	Package redis provides a Redis-backed cache implementation with singleflight support.
clients/cassandra Package cassandra builds a Cassandra-protocol Session — self-managed Apache Cassandra (KindCassandra, local and dev) or Amazon Keyspaces (KindKeyspaces, staging) — behind the consumer-side Session seam, so the choice is a config change.	Package cassandra builds a Cassandra-protocol Session — self-managed Apache Cassandra (KindCassandra, local and dev) or Amazon Keyspaces (KindKeyspaces, staging) — behind the consumer-side Session seam, so the choice is a config change.
clients/cassandra/cassandra Package cassandra builds the gocql cluster configuration for a self-managed Apache Cassandra (local Compose, dev): a token-aware, DC-aware host policy, optional password auth and TLS.	Package cassandra builds the gocql cluster configuration for a self-managed Apache Cassandra (local Compose, dev): a token-aware, DC-aware host policy, optional password auth and TLS.
clients/cassandra/keyspaces Package keyspaces builds the gocql cluster configuration for Amazon Keyspaces: the regional endpoint on port 9142, mandatory TLS with host verification, SigV4 authentication (the AWS SigV4 gocql plugin, credentials from an AWS credentials provider — the default chain, IRSA in-cluster) and DisableInitialHostLookup, per AWS guidance.	Package keyspaces builds the gocql cluster configuration for Amazon Keyspaces: the regional endpoint on port 9142, mandatory TLS with host verification, SigV4 authentication (the AWS SigV4 gocql plugin, credentials from an AWS credentials provider — the default chain, IRSA in-cluster) and DisableInitialHostLookup, per AWS guidance.
clients/connector Package connector builds ConnectorSources for the connector family: S3 now, a future GDrive/Salesforce kind added as a new case + sibling package (ARCHITECTURE.md#the-one-idea; the foundation/logger NewFromConfig pattern).	Package connector builds ConnectorSources for the connector family: S3 now, a future GDrive/Salesforce kind added as a new case + sibling package (ARCHITECTURE.md#the-one-idea; the foundation/logger NewFromConfig pattern).
clients/connector/decorators Package decorators wraps a connector ConnectorSource with the shared client observability stack (metrics/tracing/logging).	Package decorators wraps a connector ConnectorSource with the shared client observability stack (metrics/tracing/logging).
clients/connector/s3 Package s3 implements the S3 ConnectorSource for the connector family: a "test connection" probe that lists a bounded page under a connector's bucket/prefix using the connector's OWN credentials, over the shared StorageClient (INJECTED — this package never imports go/clients/storage, so there is no lateral same-layer import).	Package s3 implements the S3 ConnectorSource for the connector family: a "test connection" probe that lists a bounded page under a connector's bucket/prefix using the connector's OWN credentials, over the shared StorageClient (INJECTED — this package never imports go/clients/storage, so there is no lateral same-layer import).
clients/database Package database is the database client tier: it selects a SQL connection-pool backend — PostgreSQL today (postgres/) — by Kind and returns the interfaces.DatabasePool contract, so switching database engines is a config change, not a caller edit.	Package database is the database client tier: it selects a SQL connection-pool backend — PostgreSQL today (postgres/) — by Kind and returns the interfaces.DatabasePool contract, so switching database engines is a config change, not a caller edit.
clients/database/postgres Package postgres is the PostgreSQL backend of the database client tier: a lifecycle-managed *sql.DB pool (Client) plus ORM-independent pool utilities — a boot-time reachability probe (Ping) and a Prometheus pool-stats collector.	Package postgres is the PostgreSQL backend of the database client tier: a lifecycle-managed *sql.DB pool (Client) plus ORM-independent pool utilities — a boot-time reachability probe (Ping) and a Prometheus pool-stats collector.
clients/decorators Package decorators provides the shared client-boundary resilience stack: one reusable builder that composes, from outermost to innermost,	Package decorators provides the shared client-boundary resilience stack: one reusable builder that composes, from outermost to innermost,
clients/interceptorcore Package interceptorcore holds the framework-agnostic resilience decisions shared by the gRPC and Connect interceptor stacks.	Package interceptorcore holds the framework-agnostic resilience decisions shared by the gRPC and Connect interceptor stacks.
clients/jobs Package jobs provides a factory for background job queue implementations with multiple backends.	Package jobs provides a factory for background job queue implementations with multiple backends.
clients/jobs/decorators Package decorators composes the Job decorator stack (ARCHITECTURE.md#decorator-order) around a scheduled job function: LeaderElection → RateLimit → Retry → Timeout → Tracing → Metrics → Logging → Job.	Package decorators composes the Job decorator stack (ARCHITECTURE.md#decorator-order) around a scheduled job function: LeaderElection → RateLimit → Retry → Timeout → Tracing → Metrics → Logging → Job.
clients/jobs/river Package river provides River-based job queue integration: Runtime (a pgxpool-backed River client that works jobs, with leader-elected periodic jobs), InsertClient (an enqueue-only producer), and Migrate (River's own schema).	Package river provides River-based job queue integration: Runtime (a pgxpool-backed River client that works jobs, with leader-elected periodic jobs), InsertClient (an enqueue-only producer), and Migrate (River's own schema).
clients/lock Package lock builds a config-selected core/interfaces.Locker — a DistributedLock (keyed, token-fenced, best-effort) plus Hold, the managed acquire-with-watchdog helper a single-active-operation guard depends on.	Package lock builds a config-selected core/interfaces.Locker — a DistributedLock (keyed, token-fenced, best-effort) plus Hold, the managed acquire-with-watchdog helper a single-active-operation guard depends on.
clients/lock/decorators Package decorators composes cross-cutting concerns — resilience (retry, circuit-breaker, timeout) and observability (logging, metrics, tracing) — around any core/interfaces.DistributedLock, without touching the backend.	Package decorators composes cross-cutting concerns — resilience (retry, circuit-breaker, timeout) and observability (logging, metrics, tracing) — around any core/interfaces.DistributedLock, without touching the backend.
clients/lock/local Package local provides an in-process DistributedLock backend for single-pod deployments (dev / replica=1), where cross-process coordination is unnecessary.	Package local provides an in-process DistributedLock backend for single-pod deployments (dev / replica=1), where cross-process coordination is unnecessary.
clients/lock/redis Package redis provides a cross-pod DistributedLock backend over Redis, for multi-replica deployments (staging/prod).	Package redis provides a cross-pod DistributedLock backend over Redis, for multi-replica deployments (staging/prod).
clients/messaging Package messaging provides a factory for message publisher and subscriber implementations with multiple backends.	Package messaging provides a factory for message publisher and subscriber implementations with multiple backends.
clients/messaging/decorators Package decorators wraps the messaging MessagePublisher and MessageHandler with cross-cutting concerns.	Package decorators wraps the messaging MessagePublisher and MessageHandler with cross-cutting concerns.
clients/messaging/memory Package memory provides an in-process messaging backend for no-infra builds and tests.	Package memory provides an in-process messaging backend for no-infra builds and tests.
clients/messaging/redis Package redis provides a Redis Pub/Sub messaging integration (publisher + subscriber) implementing the core MessagePublisher / MessageConsumer contracts over PUBLISH / SUBSCRIBE / PSUBSCRIBE.	Package redis provides a Redis Pub/Sub messaging integration (publisher + subscriber) implementing the core MessagePublisher / MessageConsumer contracts over PUBLISH / SUBSCRIBE / PSUBSCRIBE.
clients/messaging/sqs Package sqs provides an AWS-SDK-backed SQS messaging integration (publisher + subscriber) usable against real AWS SQS and against ElasticMQ for local dev.	Package sqs provides an AWS-SDK-backed SQS messaging integration (publisher + subscriber) usable against real AWS SQS and against ElasticMQ for local dev.
clients/outbox Package outbox builds the OutboxSink a transactional-outbox relay delivers through: the zero-infrastructure stub (the default), an SQS queue, or an S3 bucket.	Package outbox builds the OutboxSink a transactional-outbox relay delivers through: the zero-infrastructure stub (the default), an SQS queue, or an S3 bucket.
clients/outbox/decorators Package decorators wraps the outbox sinks' SDK seams in the client stack (Bulkhead → Retry → CircuitBreaker → Timeout → Tracing → Metrics → Logging) and codes every SDK error so the stack retries only transient failures.	Package decorators wraps the outbox sinks' SDK seams in the client stack (Bulkhead → Retry → CircuitBreaker → Timeout → Tracing → Metrics → Logging) and codes every SDK error so the stack retries only transient failures.
clients/outbox/s3 Package s3 is the S3 OutboxSink: it writes each record as one object, keyed by a template, with a SHA-256 integrity checksum (Object Lock buckets require an integrity checksum and accept it in place of Content-MD5) and SSE-KMS when the record names a key.	Package s3 is the S3 OutboxSink: it writes each record as one object, keyed by a template, with a SHA-256 integrity checksum (Object Lock buckets require an integrity checksum and accept it in place of Content-MD5) and SSE-KMS when the record names a key.
clients/outbox/sqs Package sqs is the SQS OutboxSink: it routes each record to a queue, sends each queue's records with SendMessageBatch (at most 10 per request) and maps each entry's result back to its record.	Package sqs is the SQS OutboxSink: it routes each record to a queue, sends each queue's records with SendMessageBatch (at most 10 per request) and maps each entry's result back to its record.
clients/outbox/stub Package stub is the zero-infrastructure OutboxSink: it accepts every record and delivers nothing, so the graph boots with no queue or bucket.	Package stub is the zero-infrastructure OutboxSink: it accepts every record and delivers nothing, so the graph boots with no queue or bucket.
clients/prometheus Package prometheus is the client for a Prometheus server's HTTP query API: instant queries reduced to one sample, and range queries reduced to one series.	Package prometheus is the client for a Prometheus server's HTTP query API: instant queries reduced to one sample, and range queries reduced to one series.
clients/prometheus/stub Package stub is the zero-infrastructure metrics querier: every instant query is 0 and every range query an empty series, with no network I/O.	Package stub is the zero-infrastructure metrics querier: every instant query is 0 and every range query an empty series, with no network I/O.
clients/replaybuffer Package replaybuffer selects and builds a ReplayBuffer backend from configuration and wraps it in the resilience + observability decorator stack, mirroring clients/lock.	Package replaybuffer selects and builds a ReplayBuffer backend from configuration and wraps it in the resilience + observability decorator stack, mirroring clients/lock.
clients/replaybuffer/decorators Package decorators composes cross-cutting concerns — a per-op timeout and the observability trio (logging, metrics, tracing) — around any core/interfaces.ReplayBuffer, without touching the backend.	Package decorators composes cross-cutting concerns — a per-op timeout and the observability trio (logging, metrics, tracing) — around any core/interfaces.ReplayBuffer, without touching the backend.
clients/replaybuffer/memory Package memory provides an in-process ReplayBuffer backend for single-pod deployments (dev / replica=1), where a reconnect lands on the same pod so cross-pod sharing is unnecessary.	Package memory provides an in-process ReplayBuffer backend for single-pod deployments (dev / replica=1), where a reconnect lands on the same pod so cross-pod sharing is unnecessary.