jpf

README

jpf - A Lightweight Framework for AI-Powered Applications

jpf is a Go library for building lightweight AI-powered applications. It provides essential building blocks, including model construction, embedding generation, and robust LLM interaction interfaces, enabling you to craft custom solutions without the bloat.

jpf is aimed at using AI as a tool - not as a chatbot (this is not to say you cannot use it to make a chatbot, however there is no framework provided for this yet). It focusses on adding AI features locally, as opposed to relying too heavily on external APIs - this makes the package particularly flexible when switching models or providers.

Features

• Flexible Orchestration: Design iterative workflows for reasoning, task automation, or application backends.
• Retry and Feedback Handling: Resilient mechanisms for retrying tasks and incorporating feedback into interactions.
• Embedding Utilities: Generate and manipulate vector embeddings for tasks like similarity, search, or clustering.
• Customizable Models: Seamlessly integrate LLMs, including reasoning chains and hybrid models.
• Token Usage Tracking: Stay informed of API token consumption for cost-effective development.
• Easy-to-use Caching: Reduce the calls made to models by composing a caching layer onto an existing model.

Installation

Install jpf in your Go project via:

go get github.com/JoshPattman/jpf

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome! Open an issue or submit a pull request on GitHub.

FAQ

• Will streaming (token-by-token) ever be supported?
  • No. This framework is designed to be more of a back-end tool, and character streaming would add extra complexity that most applications of this package would not benefit from (in my opinion).
• Are there any pre-built formatters / parsers?
  • There are a few built in implementations, however the aim of this package is to create the framework, not the functionality.
  • If you have any ideas of useful functions, feel free to put them on an issue, and if enough arise, I can make a new repo for these.
• Where are the agents?
  • I removed the agent interface recently as I think it was far too restrictuve.
  • I would like to instead get the core building blocks ironed out before moving on to coming up with an agent interface.
• Why does this not support MCP tools on the OpenAI API / Tool calling / Other advanced API feature?
  • The aim of this package is to put the advanced stuff, like using tools, to you to figure out. IMO this allows you to do cooler, more flexible things (like a tree of agents).
  • Also, to a degree tool calls / MCP tools lock you in to one API or another, more than just using the chat completions endpoint.
  • I might consider adding them in the future, but for now I think that implementing your own tool calling is best.

Author

Developed by Josh Pattman. Learn more at GitHub.

Documentation

Index

• Variables
• func CosineSimilarity(vec1, vec2 []float64) (float64, error)
• func HashMessages(msgs []Message) string
• type Cache
  • func NewInMemoryCache() Cache
  • func NewSQLCache(db *sql.DB) (Cache, error)
• type ConcurrentLimiter
  • func NewMaxConcurrentLimiter(n int) ConcurrentLimiter
  • func NewOneConcurrentLimiter() ConcurrentLimiter
• type Embedder
  • func NewCachedEmbedder(emb Embedder, cache EmbedderResponseCache) Embedder
  • func NewOpenAIEmbedder(key, model string, opts ...openAIEmbedderOpt) Embedder
• type EmbedderResponseCache
• type FeedbackGenerator
  • func NewRawMessageFeedbackGenerator() FeedbackGenerator
• type ImageAttachment
  • func (i *ImageAttachment) ToBase64Encoded(useCompression bool) (string, error)
• type MapFunc
  • func NewFeedbackMapFunc[T, U any](enc MessageEncoder[T], pars ResponseDecoder[U], fed FeedbackGenerator, ...) MapFunc[T, U]
  • func NewOneShotMapFunc[T, U any](enc MessageEncoder[T], pars ResponseDecoder[U], model Model) MapFunc[T, U]
• type Message
• type MessageEncoder
  • func NewRawStringMessageEncoder(systemPrompt string) MessageEncoder[string]
  • func NewTemplateMessageEncoder[T any](systemTemplate, userTemplate string) MessageEncoder[T]
• type Model
  • func NewCachedModel(model Model, cache ModelResponseCache) Model
  • func NewConcurrentLimitedModel(model Model, limiter ConcurrentLimiter) Model
  • func NewFakeReasoningModel(reasoner Model, answerer Model, opts ...fakeReasoningModelOpt) Model
  • func NewLoggingModel(model Model, logger ModelLogger) Model
  • func NewOpenAIModel(key, modelName string, opts ...openAIModelOpt) Model
  • func NewRetryModel(model Model, opts ...retryModelOpt) Model
  • func NewSystemReasonModel(model Model, opts ...systemReasonOpt) Model
  • func NewUsageCountingModel(model Model, counter *UsageCounter) Model
• type ModelLogger
  • func NewJsonModelLogger(to io.Writer) ModelLogger
• type ModelLoggingInfo
• type ModelResponseCache
• type ReasoningEffort
• type ResponseDecoder
  • func NewJsonResponseDecoder[T any]() ResponseDecoder[T]
  • func NewRawStringResponseDecoder() ResponseDecoder[string]
• type Role
  • func (r Role) String() string
• type Usage
  • func (u Usage) Add(u2 Usage) Usage
• type UsageCounter
  • func NewUsageCounter() *UsageCounter
  • func (u *UsageCounter) Add(usage Usage)
  • func (u *UsageCounter) Get() Usage
• type WithDelay
• type WithHTTPHeader
• type WithReasoningEffort
• type WithReasoningPrefix
• type WithReasoningPrompt
• type WithRetries
• type WithTemperature
• type WithURL

Variables

var (
	ErrInvalidResponse = errors.New("llm produced an invalid response")
)

Functions

func CosineSimilarity

func CosineSimilarity(vec1, vec2 []float64) (float64, error)

CosineSimilarity takes the cosine similarity between two vectors.

func HashMessages

func HashMessages(msgs []Message) string

Types

type Cache

type Cache interface {
	ModelResponseCache
	EmbedderResponseCache
}

func NewInMemoryCache

func NewInMemoryCache() Cache

NewInMemoryCache creates an in-memory implementation of ModelResponseCache. It stores model responses in memory using a hash of the input messages as a key.

func NewSQLCache

func NewSQLCache(db *sql.DB) (Cache, error)

type ConcurrentLimiter

type ConcurrentLimiter chan struct{}

func NewMaxConcurrentLimiter

func NewMaxConcurrentLimiter(n int) ConcurrentLimiter

NewMaxConcurrentLimiter creates a ConcurrentLimiter that allows up to n concurrent operations. The limiter is implemented as a buffered channel with capacity n.

func NewOneConcurrentLimiter

func NewOneConcurrentLimiter() ConcurrentLimiter

NewOneConcurrentLimiter creates a ConcurrentLimiter that allows only one operation at a time. This is a convenience function equivalent to NewMaxConcurrentLimiter(1).

type Embedder

type Embedder interface {
	Embed(text string) ([]float64, error)
}

Embedder defines an object that is capable of embedding a string into a vector.

func NewCachedEmbedder

func NewCachedEmbedder(emb Embedder, cache EmbedderResponseCache) Embedder

func NewOpenAIEmbedder

func NewOpenAIEmbedder(key, model string, opts ...openAIEmbedderOpt) Embedder

type EmbedderResponseCache

type EmbedderResponseCache interface {
	GetCachedEmbedding(string) (bool, []float64, error)
	SetCachedEmbedding(string, []float64) error
}

type FeedbackGenerator

type FeedbackGenerator interface {
	FormatFeedback(Message, error) string
}

FeedbackGenerator takes an error and converts it to a piece of text feedback to send to the LLM.

func NewRawMessageFeedbackGenerator

func NewRawMessageFeedbackGenerator() FeedbackGenerator

NewRawMessageFeedbackGenerator creates a FeedbackGenerator that formats feedback by returning the error message as a string.

type ImageAttachment

type ImageAttachment struct {
	Source image.Image
}

func (*ImageAttachment) ToBase64Encoded

func (i *ImageAttachment) ToBase64Encoded(useCompression bool) (string, error)

type MapFunc

type MapFunc[T, U any] interface {
	Call(T) (U, Usage, error)
}

MapFunc transforms input of type T into output of type U using an LLM. It handles the encoding of input, interaction with the LLM, and decoding of output.

func NewFeedbackMapFunc

func NewFeedbackMapFunc[T, U any](
	enc MessageEncoder[T],
	pars ResponseDecoder[U],
	fed FeedbackGenerator,
	model Model,
	feedbackRole Role,
	maxRetries int,
) MapFunc[T, U]

NewFeedbackMapFunc creates a MapFunc that adds feedback to the conversation when errors are detected. It will only add to the conversation if the error returned from the parser is an ErrInvalidResponse (using errors.Is).

func NewOneShotMapFunc

func NewOneShotMapFunc[T, U any](enc MessageEncoder[T], pars ResponseDecoder[U], model Model) MapFunc[T, U]

type Message

type Message struct {
	Role    Role
	Content string
	Images  []ImageAttachment
}

Message defines a text message to/from an LLM.

type MessageEncoder

type MessageEncoder[T any] interface {
	BuildInputMessages(T) ([]Message, error)
}

MessageEncoder encodes a structured piece of data into a set of messages for an LLM.

func NewRawStringMessageEncoder

func NewRawStringMessageEncoder(systemPrompt string) MessageEncoder[string]

NewRawStringMessageEncoder creates a MessageEncoder that encodes a system prompt and user input as raw string messages.

func NewTemplateMessageEncoder

func NewTemplateMessageEncoder[T any](systemTemplate, userTemplate string) MessageEncoder[T]

NewTemplateMessageEncoder creates a MessageEncoder that uses Go's text/template for formatting messages. It accepts templates for both system and user messages, allowing dynamic content insertion. The data parameter to BuildInputMessages should be a struct or map with fields accessible to the template. If either systemTemplate or userTemplate is an empty string, that message will be skipped.

type Model

type Model interface {
	// Tokens specifies how many tokens are allowed to be sent.
	Tokens() (int, int)
	// Responds to a set of input messages, with a set of auxilliary messages and a final message.
	// There may be no auxilliary messages, or things like tool calls, function calls, and reasoning may go in the auxilliary messages,
	Respond([]Message) ([]Message, Message, Usage, error)
}

Model defines an interface to an LLM.

func NewCachedModel

func NewCachedModel(model Model, cache ModelResponseCache) Model

NewCachedModel wraps a Model with response caching functionality. It stores responses in the provided ModelResponseCache implementation, returning cached results for identical input messages to avoid redundant model calls.

func NewConcurrentLimitedModel

func NewConcurrentLimitedModel(model Model, limiter ConcurrentLimiter) Model

NewConcurrentLimitedModel wraps a Model with concurrency control. It ensures that only a limited number of concurrent calls can be made to the underlying model, using the provided ConcurrentLimiter to manage access.

func NewFakeReasoningModel

func NewFakeReasoningModel(reasoner Model, answerer Model, opts ...fakeReasoningModelOpt) Model

NewFakeReasoningModel creates a model that uses two underlying models to simulate reasoning. It first calls the reasoner model to generate reasoning about the input messages, then passes that reasoning along with the original messages to the answerer model. The reasoning is included as a ReasoningRole message in the auxiliary messages output. Optional parameters allow customization of the reasoning prompt.

func NewLoggingModel

func NewLoggingModel(model Model, logger ModelLogger) Model

NewLoggingModel wraps a Model with logging functionality. It logs all interactions with the model using the provided ModelLogger. Each model call is logged with input messages, output messages, usage statistics, and timing information.

func NewOpenAIModel

func NewOpenAIModel(key, modelName string, opts ...openAIModelOpt) Model

NewOpenAIModel creates a Model that uses the OpenAI API. It requires an API key and model name, with optional configuration via variadic options.

func NewRetryModel

func NewRetryModel(model Model, opts ...retryModelOpt) Model

NewRetryModel wraps a Model with retry functionality. If the underlying model returns an error, this wrapper will retry the operation up to a configurable number of times with an optional delay between retries.

func NewSystemReasonModel

func NewSystemReasonModel(model Model, opts ...systemReasonOpt) Model

NewSystemReasonModel converts ReasoningRole messages to SystemRole messages. This allows using models that don't natively support a reasoning role by converting reasoning messages into system messages with a customizable prefix. Options: - WithReasoningPrefix: customizes the prefix text added before reasoning content (default provided)

func NewUsageCountingModel

func NewUsageCountingModel(model Model, counter *UsageCounter) Model

NewUsageCountingModel wraps a Model with token usage tracking functionality. It aggregates token usage statistics in the provided UsageCounter, which allows monitoring total token consumption across multiple model calls.

type ModelLogger

type ModelLogger interface {
	ModelLog(ModelLoggingInfo) error
}

ModelLogger specifies a method of logging a call to a model.

func NewJsonModelLogger

func NewJsonModelLogger(to io.Writer) ModelLogger

NewJsonModelLogger creates a ModelLogger that outputs logs in JSON format. The logs are written to the provided io.Writer, with each log entry being a JSON object containing the model interaction details.

type ModelLoggingInfo

type ModelLoggingInfo struct {
	Messages             []Message
	ResponseAuxMessages  []Message
	ResponseFinalMessage Message
	Usage                Usage
	Err                  error
	Duration             time.Duration
}

ModelLoggingInfo contains all information about a model interaction to be logged. It includes input messages, output messages, usage statistics, and any error that occurred.

type ModelResponseCache

type ModelResponseCache interface {
	GetCachedResponse([]Message) (bool, []Message, Message, error)
	SetCachedResponse(inputs []Message, aux []Message, out Message) error
}

type ReasoningEffort

type ReasoningEffort uint8

ReasoningEffort defines how hard a reasoning model should think.

const (
	LowReasoning ReasoningEffort = iota
	MediumReasoning
	HighReasoning
)

type ResponseDecoder

type ResponseDecoder[T any] interface {
	ParseResponseText(string) (T, error)
}

ResponseDecoder converts an LLM response into a structured piece of data. When the LLM response is invalid, it should return ErrInvalidResponse (or an error joined on that).

func NewJsonResponseDecoder

func NewJsonResponseDecoder[T any]() ResponseDecoder[T]

NewJsonResponseDecoder creates a ResponseDecoder that tries to parse a json object from the response. It can ONLY parse json objects with an OBJECT as top level (i.e. it cannot parse a list directly).

func NewRawStringResponseDecoder

func NewRawStringResponseDecoder() ResponseDecoder[string]

NewRawStringResponseDecoder creates a ResponseDecoder that returns the response as a raw string without modification.

type Role

type Role uint8

Role is an enum specifying a role for a message. It is not 1:1 with openai roles (i.e. there is a reasoning role here).

const (
	SystemRole Role = iota
	UserRole
	AssistantRole
	ReasoningRole
)

func (Role) String

func (r Role) String() string

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int
}

Usage defines how many tokens were used when making calls to LLMs.

func (Usage) Add

func (u Usage) Add(u2 Usage) Usage

type UsageCounter

type UsageCounter struct {
	// contains filtered or unexported fields
}

Counts up the sum usage. Is completely concurrent-safe.

func NewUsageCounter

func NewUsageCounter() *UsageCounter

NewUsageCounter creates a new UsageCounter with zero initial usage. The counter is safe for concurrent use across multiple goroutines.

func (*UsageCounter) Add

func (u *UsageCounter) Add(usage Usage)

Add the given usage to the counter.

func (*UsageCounter) Get

func (u *UsageCounter) Get() Usage

Get the current usage in the counter.

type WithDelay

type WithDelay struct{ X time.Duration }

type WithHTTPHeader

type WithHTTPHeader struct {
	K string
	V string
}

type WithReasoningEffort

type WithReasoningEffort struct{ X ReasoningEffort }

type WithReasoningPrefix

type WithReasoningPrefix struct{ X string }

type WithReasoningPrompt

type WithReasoningPrompt struct{ X string }

type WithRetries

type WithRetries struct{ X int }

type WithTemperature

type WithTemperature struct{ X float64 }

type WithURL

type WithURL struct{ X string }