llm

Documentation

Overview

Package llm is a small, shared client for large language model providers.

It exists so the ProjectDiscovery ecosystem has one LLM client rather than several: nuclei semantic matchers and the DSL llm_prompt helper both build on it. Any OpenAI-compatible endpoint is supported, which covers local runtimes (ollama, llama.cpp, vLLM, LM Studio) as well as hosted providers. There is no implicit hosted default: a caller must set Provider or BaseURL so a scan cannot send data off-box by accident.

Index

• Constants
• type Client
  • func New(config Config) (*Client, error)
  • func (client *Client) Complete(ctx context.Context, req Request) (string, error)
• type Config
• type Format
• type Request

Constants

const APIKeyEnv = "LLM_API_KEY"

APIKeyEnv is the environment variable consulted when Config.APIKey is empty.

const ProviderOpenAICompatible = "openai-compatible"

ProviderOpenAICompatible talks to any /v1/chat/completions endpoint. BaseURL is required; this name never maps to a hosted vendor.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is a provider-agnostic LLM client with caching, a call budget, and a concurrency limit. It is safe for concurrent use.

func New

func New(config Config) (*Client, error)

New builds a client from config. Hosted presets require Config.APIKey or LLM_API_KEY; local presets and a custom BaseURL do not.

func (*Client) Complete

func (client *Client) Complete(ctx context.Context, req Request) (string, error)

Complete returns a completion for the request.

A cached response is returned without consuming the budget or a concurrency slot, so a warm cache both costs nothing and cannot be rate-limited. The budget is checked and consumed only for calls that actually wait to reach the provider; a cancelled wait refunds the reservation.

type Config

type Config struct {
	// Provider selects a preset endpoint. Ignored when BaseURL is set.
	Provider string
	// APIKey overrides LLM_API_KEY. Empty means read the environment. Hosted
	// presets require a key; local presets and a custom BaseURL do not.
	APIKey string
	// BaseURL is any OpenAI-compatible endpoint, including a local one.
	BaseURL string
	// Model is the model identifier passed to the provider.
	Model string
	// Timeout bounds a single completion call. Zero uses a default.
	Timeout time.Duration
	// Cache enables in-memory response caching keyed on the request.
	Cache bool
	// MaxCalls caps completions for the lifetime of the client. Zero is
	// unlimited. It exists so a misbehaving caller cannot run up an unbounded
	// bill against a paid provider.
	MaxCalls int
	// MaxConcurrency caps in-flight completions. Zero or negative means one.
	MaxConcurrency int
}

Config describes how to reach a provider and how to bound its use.

type Format

type Format struct {
	// JSON asks the provider to return a single JSON value. Callers that need a
	// specific shape state it in the prompt and validate the result; this only
	// requests JSON rather than prose.
	JSON bool
}

Format controls the shape of a completion.

type Request

type Request struct {
	// System is an optional system instruction.
	System string
	// Prompt is the user input.
	Prompt string
	// Format optionally constrains the output.
	Format Format
}

Request is a single completion.