Documentation
¶
Overview ¶
Package mcp wires ken's internal search package to the Model Context Protocol via the modelcontextprotocol/go-sdk. It exposes the same two tools semble's Python MCP server does — `search` and `find_related` — with matching argument shapes and the same formatted-string output so any MCP-compatible agent (Claude Code, Cursor, Codex, OpenCode, VS Code, GitHub Copilot CLI) can use ken-mcp as a drop-in replacement for semble's MCP server.
The package name is "mcp" to match docs/DESIGN.md §1's layout; the SDK's own "mcp" package is imported as `sdk` everywhere in this package to keep the two unambiguous.
Output format and arg schemas are ports of /tmp/semble/src/semble/mcp.py and utils._format_results, verified against the live source.
Index ¶
- Constants
- Variables
- func CloneShallow(ctx context.Context, urlStr string) (string, func(), error)
- func FormatResults(header string, results []search.Result) string
- func LogLevelNames() []string
- func NewServer(cfg Config) *sdk.Server
- func NormalizeKey(source string) (string, bool, error)
- func Run(ctx context.Context, fsys fs.FS, opts Options) error
- func ValidateEnum(name, raw string, allowed []string, fallback string, lg *Logger) string
- type Builder
- type Cache
- type Config
- type DBIntegration
- type FindRelatedArgs
- type LogLevel
- type Logger
- type Options
- type ReindexDBArgs
- type ReindexResult
- type SearchArgs
Constants ¶
const DefaultCacheSize = 16
DefaultCacheSize is the LRU bound when KEN_MCP_CACHE_SIZE is unset.
const DefaultTopK = 5
DefaultTopK is the default value for top_k across both tools (matches semble's `top_k: int = 5`; the Stage-5 prompt's "default 10" was a reconstruction).
Variables ¶
var ErrPrivateCloneTarget = errors.New("mcp: clone target resolves to a private/loopback/link-local address")
ErrPrivateCloneTarget is returned by CloneShallow when the supplied http(s) URL resolves to a loopback, link-local, RFC1918, or RFC4193 address. M2 SSRF guard: a hostile agent passing `repo="http://169.254.169.254/..."` or `repo="http://127.0.0.1:..."` would otherwise coax ken-mcp into issuing outbound HTTP requests from the host's network position. Operators with a legitimate internal git host can opt out via `KEN_ALLOW_PRIVATE_CLONE_TARGETS=1`.
Functions ¶
func CloneShallow ¶
CloneShallow shallow-clones a public http(s) repository into a deterministic per-URL subdirectory of $TMPDIR/ken-mcp/. Returns the clone directory and a cleanup that rm -rf's it. No authentication is configured — private repos are out of scope for v1 (semble matches).
Determinism is by sha256(url) so concurrent calls don't collide, and stale dirs from previous processes are detectable and reusable should we ever add cross-process caching (we don't yet; Close() always rms).
M2 SSRF guard: before invoking go-git, the URL's host is resolved and rejected if any of its A/AAAA records points at a loopback / link-local / RFC1918 / RFC4193 / unspecified address. This is a pre-flight defense: it doesn't survive a DNS rebinding TOCTOU, but it blocks the dominant attack shape (a hostile agent naming a metadata or internal endpoint by literal IP or by a hostname that resolves to one). Operators with a legitimate internal git host can opt out via the documented env var.
L3 (documented limitation): there is no max-bytes / max-objects cap on the clone. A malicious host can serve a huge or pathological pack file; the only timeout in play is the MCP request ctx. If this becomes a real problem in production, the fix is a custom http.Transport wrapping the go-git client with a bounded-body reader — deferred until a deployment scenario justifies the per-platform complexity. Mitigation in the meantime: enforce reasonable ctx timeouts at the MCP layer, run ken-mcp in a network-bandwidth-limited container if the input source is untrusted.
func FormatResults ¶ added in v0.3.0
FormatResults mirrors semble utils._format_results: a header, then each result as a numbered, fenced code block with score=X.XXX. Returning a preformatted string keeps wire compatibility with semble — agents see the same text they're already trained against in semble-using prompts.
Exported because bench/tokens/ measures token budgets against this exact wire format (not the in-memory chunk text), so it has to call the same formatter ken-mcp emits over the JSON-RPC channel.
func LogLevelNames ¶ added in v0.6.0
func LogLevelNames() []string
LogLevelNames returns the canonical log-level strings in ascending severity order. Callers that build env-var or CLI allowed-value lists should use this rather than hardcoding.
func NewServer ¶
NewServer returns an MCP server with `search` and `find_related` registered. The server speaks JSON-RPC over whatever Transport the caller passes to server.Run — ken-mcp uses sdk.StdioTransport.
func NormalizeKey ¶
NormalizeKey canonicalizes a user-supplied repo string into a cache key. https/http URLs keep their scheme but get lowercased host + a trailing ".git" and "/" stripped; other URL-shaped inputs (anything containing "://" or matching SCP-form `user@host:path`) are rejected with an error, matching semble/mcp.py's http(s)-only guard. Inputs with no `://` are treated as local paths and resolved to an absolute path (existence is not checked here; defer to the Builder).
L1 hardening: previously this was an allow-list-then-default-to- local-path pattern, which meant `file:///etc`, `ftp://host/`, or any other unknown scheme silently degraded to a local-path resolve (producing a junky absolute path that confused the Builder error). The scheme allow-list is now the security boundary: anything URL- shaped that isn't https/http is rejected with a typed error.
func Run ¶ added in v0.6.0
Run starts an MCP server (over the SDK's default StdioTransport) serving search and find_related over the single fixed corpus rooted at fsys. Blocks until ctx is canceled or the client closes the transport; returns nil for either clean-shutdown path.
The tool wire format and argument schemas are identical to the stock cmd/ken-mcp binary's, so any agent already trained against semble's or ken's MCP server works unchanged. The agent-supplied `repo` argument is ignored (logged at debug level when non-empty): the corpus is fixed by fsys for the lifetime of the call.
Run is the embedded-corpus build pattern (ADR-016). Canonical use:
//go:embed docs/*.md
var corpus embed.FS
//go:embed model/*
var modelFS embed.FS
func main() {
if err := mcp.Run(context.Background(), corpus, mcp.Options{
Mode: "hybrid", ChunkerName: "markdown", ModelFS: modelFS,
}); err != nil { log.Fatal(err) }
}
For multi-repo / per-request URL clone / file-watching use cases, use cmd/ken-mcp directly — Run intentionally doesn't carry that machinery.
func ValidateEnum ¶ added in v0.6.0
ValidateEnum returns raw if it appears in allowed; otherwise warns and returns fallback. Empty raw also returns fallback (no warning). Used by both mcp.Run (validating Options fields against allowed enums) and cmd/ken-mcp's env.envEnum (which wraps this with an os.Getenv lookup and the same warn-then-fallback semantics).
Case-sensitive on purpose: "Hybrid" should be a loud "fix your config" rather than a silent acceptance — matches ADR-009.
Types ¶
type Builder ¶
Builder constructs an Index for an already-normalized source identifier (either a canonical http(s) URL or an absolute filesystem path). The returned cleanup is called when the entry is evicted from the cache — used to rm -rf temp clone dirs; pass nil for local-path entries.
As of v0.3, the returned *search.WatchedIndex wraps the index plus a file-watcher goroutine. The cache calls (*WatchedIndex).Close() on eviction (and on Cache.Close()) to stop the watcher before invoking the user-supplied cleanup; without this the goroutine outlives the cache entry and the temp clone dir gets rm-rf'd while the watcher holds inotify fds pointing into it.
type Cache ¶
type Cache struct {
// contains filtered or unexported fields
}
Cache is the per-process repo→Index cache that backs the MCP server. Concurrent uncached requests for the same key dedupe via singleflight, and entries are LRU-evicted at the configured bound.
func (*Cache) Close ¶
func (c *Cache) Close()
Close releases every cached entry. Stops the watcher goroutine for each (wix.Close()) and runs the user-supplied cleanup (rm -rf for temp clones). Safe to call multiple times.
M8: sets c.closed under c.mu before draining the map. Concurrent Get() calls whose singleflight build is in flight observe the flag once they re-acquire the lock and reap the just-built watcher rather than repopulating the now-cleared map. This avoids a use-after-close where a stale entry survives past Close() and outlives the cache's intent.
func (*Cache) Get ¶
Get returns a cached WatchedIndex for source, building it once on first access. Concurrent first-access calls for the same key share a single build via singleflight. The returned *WatchedIndex is shared across all callers and across subsequent Get calls until evicted; callers MUST NOT call wix.Close() themselves — the cache owns the lifecycle.
type Config ¶
type Config struct {
Cache *Cache // repo→Index cache (required)
DefaultRepo string // optional pre-configured source; if set, tools may be called without a `repo` arg
Mode search.Mode // default ModeHybrid
Chunker string // default "regex"
Instructions string // server-instructions string; default mirrors semble's
ServerName string // default "ken"
ServerVersion string // default "0"
// DB, when non-nil, registers the v0.8.0 reindex_db MCP tool
// (ADR-020 Part 2) and wires Tier-2 chunk integration via the
// caller's swap target (ADR-020 Part 3 addendum). nil = tool
// not registered + no chunk integration; the agent's tools/list
// won't show reindex_db at all when no DB is configured, which
// keeps the tool surface honest.
//
// cmd/ken-mcp wires this with a *mcp/db.Refresher whose Start
// callback writes to the WatchedIndex.SetExtraChunks of the
// pre-warmed default-repo Index. SDK authors using mcp.Run wire
// the same *mcp/db.Refresher; mcp.Run's Start callback updates
// an atomic.Pointer[search.Index] via WithExtraChunks.
DB DBIntegration
}
Config is the wiring for a ken-mcp server. Defaults applied by NewServer for any zero value.
type DBIntegration ¶ added in v0.8.0
type DBIntegration interface {
// Start registers onExtras as the swap callback that fires on
// each refresh. Starts whatever interval / LISTEN goroutines the
// implementation needs. Returns a cleanup func the caller MUST
// defer; canceling ctx is also honored.
Start(ctx context.Context, onExtras func([]chunk.Chunk)) (cleanup func(), err error)
// TryRefresh triggers an immediate refresh and returns the
// outcome. Fail-fast on contention: in-flight refreshes (interval
// tick, LISTEN burst, SIGHUP, prior TryRefresh) cause this call
// to return ReindexResult{InProgress: true} without queuing.
TryRefresh(ctx context.Context) ReindexResult
}
DBIntegration is the seam between mcp.Run / mcp.NewServer and a Tier-2 DB provider (typically *mcp/db.Refresher; mock impls in tests). The v0.8.0 Part 3 addendum (ADR-020) bundles both chunk-integration (Start) and tool invocation (TryRefresh) into one interface so the same provider drives both surfaces.
Lifecycle:
Start is called once by mcp.Run (or NewServer's caller) AFTER the index is built. The onExtras callback fires each time the provider has fresh DB chunks ready (initial introspection, interval ticks, LISTEN/NOTIFY, agent-triggered reindex). The receiver should treat each onExtras call as a complete snapshot replacement, not an append — see *Index.WithExtraChunks for the production pattern. The returned cleanup func is deferred by the caller; canceling ctx is the secondary signal.
TryRefresh is invoked by the reindex_db MCP tool handler. It MUST be fail-fast on contention (return InProgress: true rather than block) because the agent is waiting on the MCP response.
Implementations must be safe to call TryRefresh concurrently with in-flight onExtras callbacks AND with other TryRefresh callers. *mcp/db.Refresher satisfies this via the underlying internal/db.Refresher's mutex serialization (the four blocking trigger sources + TryLock for the fifth — ADR-020 Part 2).
The mcp package stays internal/db-free: this interface is what callers (cmd/ken-mcp; SDK authors using mcp.Run + mcp/db) implement in their layer. The v0.6.0 binary-size contract holds — see TestBinary_MCPPackageStaysDBFree.
type FindRelatedArgs ¶
type FindRelatedArgs struct {
FilePath string `json:"file_path" jsonschema:"Path to the file as stored in the index (use file_path from a search result)."`
Line int `json:"line" jsonschema:"Line number (1-indexed)."`
Repo string `` /* 145-byte string literal not displayed */
TopK int `json:"top_k,omitempty" jsonschema:"Number of similar chunks to return."`
}
FindRelatedArgs is the argument schema for `find_related`.
type LogLevel ¶ added in v0.6.0
type LogLevel int
LogLevel selects which Logger.Logf calls actually write.
func ParseLogLevel ¶ added in v0.6.0
ParseLogLevel maps a string ("debug"/"info"/"warn"/"error") to a LogLevel. Case-insensitive. Unknown strings return LogWarn (the default) and the caller can warn on the mismatch separately.
type Logger ¶ added in v0.6.0
type Logger struct {
Level LogLevel
// contains filtered or unexported fields
}
Logger is the level-aware logger used by mcp.Run and shared with cmd/ken-mcp. Writes to its underlying io.Writer (stderr by default); must never be wired to os.Stdout.
type Options ¶ added in v0.6.0
type Options struct {
// Mode is the search mode: "bm25", "semantic", or "hybrid". Default
// "hybrid". If mode != "bm25" but the model can't be loaded (neither
// ModelFS nor ModelDir is usable), Run downgrades to "bm25" with a
// stderr warning — matches cmd/ken-mcp's first-launch behavior.
Mode string
// ChunkerName picks the chunker for the embedded corpus. Default
// "regex". Other values: "treesitter" (requires importing
// internal/chunk/treesitter for side-effect registration), "line"
// (universal fallback), "markdown" (requires importing
// internal/chunk/markdown; recommended for docs corpora).
ChunkerName string
// ModelDir is a directory path to a Model2Vec snapshot, used when
// ModelFS is nil. The standard HF layout applies: tokenizer.json,
// config.json, model.safetensors. Ignored entirely if ModelFS is set.
ModelDir string
// ModelFS is an optional fs.FS whose root contains a Model2Vec
// snapshot. Typical use: bake the model into a single-binary MCP
// server via //go:embed. When set, ModelDir is ignored.
//
// Root your embed.FS at the snapshot directory itself (or use fs.Sub
// to rebase). The model files must live at ./tokenizer.json,
// ./config.json, ./model.safetensors inside ModelFS.
ModelFS fs.FS
// LogLevel is "debug", "info", "warn", or "error" (default "warn").
LogLevel string
// LogWriter is the destination for server logs (default os.Stderr).
// MUST NOT be os.Stdout when using the default StdioTransport — see
// the stdout/stderr contract in cmd/ken-mcp/main.go.
LogWriter io.Writer
// DB, when non-nil, wires Tier-2 DB support: registers the
// reindex_db MCP tool AND integrates DB chunks into the embedded
// search results. SDK authors using mcp.Run wire this via
// mcp/db.Setup → mcp/db.Refresher. mcp.Run itself stays DB-free
// (no pgx / sqlite / mysql in the import graph) so embedded-corpus
// binaries that don't import mcp/db keep the v0.6.0 small-binary
// posture — verified by mcp/binary_contract_test.go.
//
// Lifecycle:
// 1. mcp.Run builds the embedded *search.Index from fsys.
// 2. Calls opts.DB.Start(ctx, onExtras) — onExtras is an
// mcp.Run-internal closure that calls
// baseIx.WithExtraChunks(extras) and atomic-stores the
// result. Cleanup is deferred for process shutdown.
// 3. Search / find_related handlers read the current Index via
// atomic.Pointer.Load() — they see the latest snapshot
// including DB chunks after each refresh.
// 4. The reindex_db tool calls opts.DB.TryRefresh which
// eventually fires onExtras with the latest chunks.
//
// v0.8.0 Part 3 addendum (ADR-020): this completes the chunk-
// integration loop that v0.8.0 Part 3's initial ship deferred
// to v0.9.0. The deferral is no longer applicable — DB chunks
// become searchable in the next search/find_related call after
// a successful refresh.
DB DBIntegration
// PrebuiltIndex is an optional pre-serialized index produced by
// `ken build-index` (or search.BuildAndSerializeIndex programmatically).
// When non-nil, mcp.Run loads it via search.LoadSerializedIndex
// instead of walking + chunking + embedding fsys at startup —
// the v0.8.3 cold-start optimization (ADR-024, closes #10).
//
// SDK authors who follow the convention (write the bytes to
// `<corpus>/.ken/index.bin`) can leave this nil; mcp.Run
// auto-discovers the file in fsys at that path. The explicit
// override is for SDK authors using a non-conventional layout
// (index outside the corpus FS, in a sibling embed.FS, etc.).
//
// On any load failure (corrupt bytes, format-version mismatch,
// mode/chunker mismatch vs Options.Mode/Options.ChunkerName),
// mcp.Run logs a stderr warning naming the reason and falls
// back to building from fsys — the pre-built path is purely an
// optimization, never a requirement.
//
// v0.8.3 (ADR-024).
PrebuiltIndex []byte
}
Options configures Run. The zero value is meaningful: it serves a BM25-only index over the supplied fsys with the regex chunker, logging warnings to os.Stderr at the "warn" level.
Validation mirrors cmd/ken-mcp's env-var contract (ADR-009): a typoed enum logs a stderr warning and falls back to the documented default rather than failing the run, so a misconfigured callsite still produces a usable server.
type ReindexDBArgs ¶ added in v0.8.0
type ReindexDBArgs struct{}
ReindexDBArgs is the argument schema for the v0.8.0 reindex_db tool. Argument-free by design (ADR-020 Part 2) — operates on the process-configured KEN_DB_DSN. Future v0.8.x+ refinements (async return, per-engine selectors) can extend this struct; for v0.8.0 it's deliberately empty so the agent invokes the tool with no parameters.
type ReindexResult ¶ added in v0.8.0
type ReindexResult struct {
// InProgress is true if another refresh is currently holding the
// Refresher's mutex (interval ticker, SIGHUP, LISTEN/NOTIFY
// listener, or a prior reindex_db call). The handler reports
// "already in progress" without waiting. NEVER set true when
// Err is also non-nil — they're mutually exclusive.
InProgress bool
// Elapsed is wall-clock duration of the refresh, measured by the
// callback wrapper. Surfaced to the agent so it can reason about
// whether to retry-on-stale or trust the freshness.
Elapsed time.Duration
// Err is non-nil on real failure (connection error, query failure).
// InProgress is false in this case; the handler reports the error
// text verbatim so the agent can decide whether to surface or retry.
Err error
}
ReindexResult is the outcome of one reindex attempt. Exactly one of InProgress=true, Err!=nil, or "everything else (success)" is the agent-facing case; the handler picks the message based on that.
type SearchArgs ¶
type SearchArgs struct {
Query string `json:"query" jsonschema:"Natural language or code query."`
Repo string `` /* 270-byte string literal not displayed */
Mode string `` /* 283-byte string literal not displayed */
TopK int `json:"top_k,omitempty" jsonschema:"Number of results to return."`
}
SearchArgs is the argument schema for the `search` tool. The Query / Repo / TopK fields and their jsonschema descriptions mirror semble/mcp.py verbatim so the wire schema matches across implementations. Mode is a ken-side extension (semble's MCP search has no equivalent) — see Mode's jsonschema and runSearch for per-call override semantics.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package mcpdb is the opt-in v0.8.0 Part 3 helper for SDK authors using mcp.Run who want Tier 2 DB support — schema introspection, LISTEN/NOTIFY push notifications, interval reindex, and the reindex_db MCP tool from Part 2.
|
Package mcpdb is the opt-in v0.8.0 Part 3 helper for SDK authors using mcp.Run who want Tier 2 DB support — schema introspection, LISTEN/NOTIFY push notifications, interval reindex, and the reindex_db MCP tool from Part 2. |