jpf

README

jpf: A Lightweight Framework for AI-Powered Applications

jpf is a Go library for building lightweight AI-powered applications. It provides essential building blocks, including agent orchestration, embedding generation, and robust LLM interaction interfaces, enabling you to craft custom solutions without the bloat.

Features

• Flexible Orchestration: Design iterative agents or workflows for reasoning, task automation, or application backends.
• Retry and Feedback Handling: Resilient mechanisms for retrying tasks and incorporating feedback into interactions.
• Embedding Utilities: Generate and manipulate vector embeddings for tasks like similarity, search, or clustering.
• Customizable Models: Seamlessly integrate LLMs, including reasoning chains and hybrid models.
• Token Usage Tracking: Stay informed of API token consumption for cost-effective development.

Installation

Install jpf in your Go project via:

go get github.com/JoshPattman/jpf

Usage

Example code is in the examples subdirectory. However, a brief overview of the components is as follows:

• Model: An interface defining a model that can create a message given a set of other messages. This encompasses both normal and reasoning models. Models can also be wrapped by other models to achieve retry logic, hybrid reasoning, and more.
• Function: An interface that defines a stateless typed function that performs one LLM call, and includes logic for formatting and parsing the text responses.
• RetryFunction: An extension of the above, but including logic to generate feedback for the LLM upon a failed parse, allowing the LLM call to be run in a loop until valid.
• Agent: An interface that combines a RetryFunction (to generate an action from a state), and a function to integrate that action into the next state. Agents can be run as an iterator, allowing fine control at each step (i.e. showing each step to the user as it is generated).
• Embedder: A string to vector embbedding interface.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome! Open an issue or submit a pull request on GitHub.

FAQ

• Will streaming (token-by-token) ever be supported?
  • No. This framework is designed to be more of a back-end tool, and character streaming would add extra complexity that most applications of this package would not benefit from (in my opinion).
• Are there any pre-built functions / agents?
  • The aim of this package is to create the framework, not the functionality. If you have any ideas of useful functions / agents, feel free to put them on an issue, and if enough arise, I can make a new repo for these.

Author

Developed by Josh Pattman. Learn more at GitHub.

Documentation

Index

• Constants
• func CosineSimilarity(vec1, vec2 []float64) (float64, error)
• func RunAgent[T, U any](model Model, agent Agent[T, U], initialState T) iter.Seq[AgentStep[T, U]]
• type Agent
• type AgentStep
• type Embedder
  • func NewOpenAIEmbedder(key string, model string) Embedder
• type Function
• type Message
• type Model
  • func NewFakeReasoningModel(reasoner Model, answerer Model, reasoningPrompt string) Model
  • func NewReasoningOpenAIModel(key, modelName string, maxInput, maxOutput int, ...) Model
  • func NewRetryModel(model Model, tries int, delay time.Duration) Model
  • func NewStandardOpenAIModel(key, modelName string, maxInput, maxOutput int, temperature float64) Model
• type ParseError
  • func (e *ParseError) Error() string
• type ReasoningEffort
• type RetryFunction
• type Role
• type Usage
  • func RunOneShot[T, U any](model Model, f Function[T, U], input T) (U, Usage, error)
  • func RunWithRetries[T, U any](model Model, f RetryFunction[T, U], maxRetries int, feedbackRole Role, input T) (U, Usage, error)
  • func (u Usage) Add(u2 Usage) Usage

Constants

const DefaultFakeReasoningPromptA = `` /* 525-byte string literal not displayed */

Functions

func CosineSimilarity

func CosineSimilarity(vec1, vec2 []float64) (float64, error)

CosineSimilarity takes the cosine similarity between two vectors.

func RunAgent

func RunAgent[T, U any](model Model, agent Agent[T, U], initialState T) iter.Seq[AgentStep[T, U]]

RunAgent[state, action] will run the agent indefinitely with the model, starting with the inital state.

Types

type Agent

type Agent[T, U any] interface {
	// Function is the functin that is called to generate the next action of the agent.
	Function[T, U]
	// Handle integrates the given action into the state.
	// It returns a new state, a boolean that is tru if that action was terminal, and a terminal error (if any).
	Handle(T, U) (T, bool, error)
}

Agent[state, action] defines some agentic behaviour. It does not contain any state or models itself, only configuration. Abstractly, an agent is somthing that can pick a next action from the given state, then apply that action to its state.

type AgentStep

type AgentStep[T, U any] struct {
	// State is the newest state of the agent (after taking the action).
	State T
	// Action is the most recent action to be taken.
	Action U
	// Error is only populated when somthing unrecoverable happened (stopping iteration).
	Error error
}

AgentStep[state, action] defines a step the agent has taken this iteration.

type Embedder

type Embedder interface {
	Embed(text string) ([]float64, error)
}

Embedder defines an object that is capable of embedding a string into a vector.

func NewOpenAIEmbedder

func NewOpenAIEmbedder(key string, model string) Embedder

NewOpenAIEmbedder creates a new embedding model that uses the openai API.

type Function

type Function[T, U any] interface {
	// Create the input messages from the input value
	BuildInputMessages(T) ([]Message, error)
	// Parse the raw LLM response into an output value,
	// returning a [ParseError] if the response cannot be parsed, or another error if somthing else.
	ParseResponseText(string) (U, error)
}

Function[input, output] is a short-lived task-specific LLM configuration. They are intended to be used to perform single tasks, and should not be used for long-running conversations.

type Message

type Message struct {
	Role    Role
	Content string
}

Message defines a text message to/from an LLM.

type Model

type Model interface {
	// Tokens specifies how many tokens are allowed to be sent.
	Tokens() (int, int)
	// Responds to a set of input messages, with a set of auxilliary messages and a final message.
	// There may be no auxilliary messages, or things like tool calls, function calls, and reasoning may go in the auxilliary messages,
	Respond([]Message) ([]Message, Message, Usage, error)
}

Model defines an interface to an LLM.

func NewFakeReasoningModel

func NewFakeReasoningModel(reasoner Model, answerer Model, reasoningPrompt string) Model

Creates a new model that simulates reasoning by asking one model to reason and the other to answer.

func NewReasoningOpenAIModel

func NewReasoningOpenAIModel(key, modelName string, maxInput, maxOutput int, reasoningEffort ReasoningEffort) Model

NewReasoningOpenAIModel creates a new reasoning model (i.e. o1, o3, ...) from openai.

func NewRetryModel

func NewRetryModel(model Model, tries int, delay time.Duration) Model

NewRetryModel wraps a model such that it will retry calling it if an error occurs, with intermediate delays.

func NewStandardOpenAIModel

func NewStandardOpenAIModel(key, modelName string, maxInput, maxOutput int, temperature float64) Model

NewStandardOpenAIModel creates a new standard model (i.e. gpt4o, gpt4.1, ...) from openai.

type ParseError

type ParseError struct {
	// The latest response of the LLM
	Response string
	// The error that occured
	Err error
}

A parse error is a specific type of error that is created when a function fails to parse an LLM response

func (*ParseError) Error

func (e *ParseError) Error() string

type ReasoningEffort

type ReasoningEffort uint8

ReasoningEffort defines how hard a reasoning model should think.

const (
	LowReasoning ReasoningEffort = iota
	MediumReasoning
	HighReasoning
)

type RetryFunction

type RetryFunction[T, U any] interface {
	Function[T, U]
	// Format the feedback from this parse error
	FormatFeedback(*ParseError) string
}

A retry function is a special type of function that can provide feedback to the LLM when the parse failed. It is still not intended for long-running conversations, however under the hood it does create a long conversation until the parse is sucsessful.

type Role

type Role uint8

Role is an enum specifying a role for a message. It is not 1:1 with openai roles (i.e. there is a reasoning role here).

const (
	SystemRole Role = iota
	UserRole
	AssistantRole
	ReasoningRole
)

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int
}

Usage defines how many tokens were used when making calls to LLMs.

func RunOneShot

func RunOneShot[T, U any](model Model, f Function[T, U], input T) (U, Usage, error)

Runs a function with one try, i.e. it asks the LLM once and tries to parse once.

func RunWithRetries

func RunWithRetries[T, U any](model Model, f RetryFunction[T, U], maxRetries int, feedbackRole Role, input T) (U, Usage, error)

Runs a function with a number of retries, providing feedback at each parse fail, i.e. asks the llm the inital messages at the start of the conversation and continues to provide feedback until the answer is parseable.

func (Usage) Add

func (u Usage) Add(u2 Usage) Usage