jpf

README

jpf: A Lightweight Framework for AI-Powered Applications

jpf is a Go library for building lightweight AI-powered applications. It provides essential building blocks, including agent orchestration, embedding generation, and robust LLM interaction interfaces, enabling you to craft custom solutions without the bloat.

Features

• Flexible Orchestration: Design iterative agents or workflows for reasoning, task automation, or application backends.
• Retry and Feedback Handling: Resilient mechanisms for retrying tasks and incorporating feedback into interactions.
• Embedding Utilities: Generate and manipulate vector embeddings for tasks like similarity, search, or clustering.
• Customizable Models: Seamlessly integrate LLMs, including reasoning chains and hybrid models.
• Token Usage Tracking: Stay informed of API token consumption for cost-effective development.

Installation

Install jpf in your Go project via:

go get github.com/JoshPattman/jpf

Usage

Example code is in the examples subdirectory. However, a brief overview of the components is as follows:

• Model: An interface defining a model that can create a message given a set of other messages. This encompasses both normal and reasoning models. Models can also be wrapped by other models to achieve retry logic, hybrid reasoning, and more.
• Function: An interface that defines a stateless typed function that performs one LLM call, and includes logic for formatting and parsing the text responses.
• RetryFunction: An extension of the above, but including logic to generate feedback for the LLM upon a failed parse, allowing the LLM call to be run in a loop until valid.
• Agent: An interface that combines a RetryFunction (to generate an action from a state), and a function to integrate that action into the next state. Agents can be run as an iterator, allowing fine control at each step (i.e. showing each step to the user as it is generated).
• Embedder: A string to vector embbedding interface.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome! Open an issue or submit a pull request on GitHub.

FAQ

• Will streaming (token-by-token) ever be supported?
  • No. This framework is designed to be more of a back-end tool, and character streaming would add extra complexity that most applications of this package would not benefit from (in my opinion).
• Are there any pre-built functions / agents?
  • The aim of this package is to create the framework, not the functionality. If you have any ideas of useful functions / agents, feel free to put them on an issue, and if enough arise, I can make a new repo for these.

Author

Developed by Josh Pattman. Learn more at GitHub.

Documentation

Index

• func CosineSimilarity(vec1, vec2 []float64) (float64, error)
• func RunAgent[T any](model Model, agent Agent) iter.Seq[AgentStep]
• type Action
• type Agent
• type AgentStep
• type Embedder
• type FakeReasoningModelBuilder
  • func BuildFakeReasoningModel(reasoner Model, answerer Model) *FakeReasoningModelBuilder
  • func (b *FakeReasoningModelBuilder) Validate() (Model, error)
  • func (b *FakeReasoningModelBuilder) WithReasoningPrompt(prompt string) *FakeReasoningModelBuilder
• type Function
• type Message
• type Model
• type OpenAIEmbedderBuilder
  • func BuildOpenAIEmbedder(key, model string) *OpenAIEmbedderBuilder
  • func (b *OpenAIEmbedderBuilder) Validate() (Embedder, error)
  • func (b *OpenAIEmbedderBuilder) WithURL(url string) *OpenAIEmbedderBuilder
• type OpenAIModelBuilder
  • func BuildOpenAIModel(key, modelName string, isReasoning bool) *OpenAIModelBuilder
  • func (b *OpenAIModelBuilder) Validate() (Model, error)
  • func (b *OpenAIModelBuilder) WithHeader(key, val string) *OpenAIModelBuilder
  • func (b *OpenAIModelBuilder) WithReasoningEffort(re ReasoningEffort) *OpenAIModelBuilder
  • func (b *OpenAIModelBuilder) WithTemperature(temp float64) *OpenAIModelBuilder
  • func (b *OpenAIModelBuilder) WithTokens(input, output int) *OpenAIModelBuilder
  • func (b *OpenAIModelBuilder) WithURL(url string) *OpenAIModelBuilder
• type ParseError
  • func (e *ParseError) Error() string
• type ReasoningEffort
• type RetryFunction
• type RetryModelBuilder
  • func BuildRetryModel(model Model) *RetryModelBuilder
  • func (b *RetryModelBuilder) Validate() (Model, error)
  • func (b *RetryModelBuilder) WithDelay(delay time.Duration) *RetryModelBuilder
  • func (b *RetryModelBuilder) WithMaxRetries(maxRetries int) *RetryModelBuilder
• type Role
• type SystemReasinModelBuilder
  • func BuildSystemReasonModel(model Model) *SystemReasinModelBuilder
  • func (b *SystemReasinModelBuilder) Validate() (Model, error)
  • func (b *SystemReasinModelBuilder) WithPrefix(prefix string) *SystemReasinModelBuilder
• type Usage
  • func RunOneShot[T, U any](model Model, f Function[T, U], input T) (U, Usage, error)
  • func RunWithRetries[T, U any](model Model, f RetryFunction[T, U], maxRetries int, feedbackRole Role, input T) (U, Usage, error)
  • func (u Usage) Add(u2 Usage) Usage

Functions

func CosineSimilarity

func CosineSimilarity(vec1, vec2 []float64) (float64, error)

CosineSimilarity takes the cosine similarity between two vectors.

func RunAgent

func RunAgent[T any](model Model, agent Agent) iter.Seq[AgentStep]

RunAgent will run the agent indefinitely with the model, starting with the inital state.

Types

type Action

type Action interface {
	// DoAction runs the action, updates the agent state.
	DoAction() error
}

Action defines an action that an agent has produced. An action may do anything, but usually it updates the agent's state.

type Agent

type Agent interface {
	// BuildInputMessages builds the input messages to be sent to the LLM, given the agents current state.
	BuildInputMessages() ([]Message, error)
	// ParseResponseText converts the raw text output of an agent step into the next action.
	ParseResponseText(string) (Action, error)
}

Agent defines a stateful agent, capable of generating next actions with no inputs.

type AgentStep

type AgentStep struct {
	// Action is the most recent action to be taken.
	Action Action
	// Error is only populated when somthing unrecoverable happened (stopping iteration).
	Error error
}

AgentStep defines a step the agent has taken this iteration.

type Embedder

type Embedder interface {
	Embed(text string) ([]float64, error)
}

Embedder defines an object that is capable of embedding a string into a vector.

type FakeReasoningModelBuilder

type FakeReasoningModelBuilder struct {
	// contains filtered or unexported fields
}

func BuildFakeReasoningModel

func BuildFakeReasoningModel(reasoner Model, answerer Model) *FakeReasoningModelBuilder

func (*FakeReasoningModelBuilder) Validate

func (b *FakeReasoningModelBuilder) Validate() (Model, error)

func (*FakeReasoningModelBuilder) WithReasoningPrompt

func (b *FakeReasoningModelBuilder) WithReasoningPrompt(prompt string) *FakeReasoningModelBuilder

type Function

type Function[T, U any] interface {
	// Create the input messages from the input value
	BuildInputMessages(T) ([]Message, error)
	// Parse the raw LLM response into an output value,
	// returning a [ParseError] if the response cannot be parsed, or another error if somthing else.
	ParseResponseText(string) (U, error)
}

Function[input, output] is a short-lived task-specific LLM configuration. They are intended to be used to perform single tasks, and should not be used for long-running conversations.

type Message

type Message struct {
	Role    Role
	Content string
}

Message defines a text message to/from an LLM.

type Model

type Model interface {
	// Tokens specifies how many tokens are allowed to be sent.
	Tokens() (int, int)
	// Responds to a set of input messages, with a set of auxilliary messages and a final message.
	// There may be no auxilliary messages, or things like tool calls, function calls, and reasoning may go in the auxilliary messages,
	Respond([]Message) ([]Message, Message, Usage, error)
}

Model defines an interface to an LLM.

type OpenAIEmbedderBuilder

type OpenAIEmbedderBuilder struct {
	// contains filtered or unexported fields
}

func BuildOpenAIEmbedder

func BuildOpenAIEmbedder(key, model string) *OpenAIEmbedderBuilder

func (*OpenAIEmbedderBuilder) Validate

func (b *OpenAIEmbedderBuilder) Validate() (Embedder, error)

func (*OpenAIEmbedderBuilder) WithURL

func (b *OpenAIEmbedderBuilder) WithURL(url string) *OpenAIEmbedderBuilder

type OpenAIModelBuilder

type OpenAIModelBuilder struct {
	// contains filtered or unexported fields
}

func BuildOpenAIModel

func BuildOpenAIModel(key, modelName string, isReasoning bool) *OpenAIModelBuilder

func (*OpenAIModelBuilder) Validate

func (b *OpenAIModelBuilder) Validate() (Model, error)

func (*OpenAIModelBuilder) WithHeader

func (b *OpenAIModelBuilder) WithHeader(key, val string) *OpenAIModelBuilder

func (*OpenAIModelBuilder) WithReasoningEffort

func (b *OpenAIModelBuilder) WithReasoningEffort(re ReasoningEffort) *OpenAIModelBuilder

func (*OpenAIModelBuilder) WithTemperature

func (b *OpenAIModelBuilder) WithTemperature(temp float64) *OpenAIModelBuilder

func (*OpenAIModelBuilder) WithTokens

func (b *OpenAIModelBuilder) WithTokens(input, output int) *OpenAIModelBuilder

func (*OpenAIModelBuilder) WithURL

func (b *OpenAIModelBuilder) WithURL(url string) *OpenAIModelBuilder

type ParseError

type ParseError struct {
	// The latest response of the LLM
	Response string
	// The error that occured
	Err error
}

A parse error is a specific type of error that is created when a function fails to parse an LLM response

func (*ParseError) Error

func (e *ParseError) Error() string

type ReasoningEffort

type ReasoningEffort uint8

ReasoningEffort defines how hard a reasoning model should think.

const (
	LowReasoning ReasoningEffort = iota
	MediumReasoning
	HighReasoning
)

type RetryFunction

type RetryFunction[T, U any] interface {
	Function[T, U]
	// Format the feedback from this parse error
	FormatFeedback(*ParseError) string
}

A retry function is a special type of function that can provide feedback to the LLM when the parse failed. It is still not intended for long-running conversations, however under the hood it does create a long conversation until the parse is sucsessful.

type RetryModelBuilder

type RetryModelBuilder struct {
	// contains filtered or unexported fields
}

func BuildRetryModel

func BuildRetryModel(model Model) *RetryModelBuilder

func (*RetryModelBuilder) Validate

func (b *RetryModelBuilder) Validate() (Model, error)

func (*RetryModelBuilder) WithDelay

func (b *RetryModelBuilder) WithDelay(delay time.Duration) *RetryModelBuilder

func (*RetryModelBuilder) WithMaxRetries

func (b *RetryModelBuilder) WithMaxRetries(maxRetries int) *RetryModelBuilder

type Role

type Role uint8

Role is an enum specifying a role for a message. It is not 1:1 with openai roles (i.e. there is a reasoning role here).

const (
	SystemRole Role = iota
	UserRole
	AssistantRole
	ReasoningRole
)

type SystemReasinModelBuilder

type SystemReasinModelBuilder struct {
	// contains filtered or unexported fields
}

func BuildSystemReasonModel

func BuildSystemReasonModel(model Model) *SystemReasinModelBuilder

func (*SystemReasinModelBuilder) Validate

func (b *SystemReasinModelBuilder) Validate() (Model, error)

func (*SystemReasinModelBuilder) WithPrefix

func (b *SystemReasinModelBuilder) WithPrefix(prefix string) *SystemReasinModelBuilder

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int
}

Usage defines how many tokens were used when making calls to LLMs.

func RunOneShot

func RunOneShot[T, U any](model Model, f Function[T, U], input T) (U, Usage, error)

Runs a function with one try, i.e. it asks the LLM once and tries to parse once.

func RunWithRetries

func RunWithRetries[T, U any](model Model, f RetryFunction[T, U], maxRetries int, feedbackRole Role, input T) (U, Usage, error)

Runs a function with a number of retries, providing feedback at each parse fail, i.e. asks the llm the inital messages at the start of the conversation and continues to provide feedback until the answer is parseable.

func (Usage) Add

func (u Usage) Add(u2 Usage) Usage