inference

README

Doppel

Simple go bindings for AI inference

The goal of this library is to offer, a dependency free, way of talking to various llm providers.

Contract

For the high level view of this lib, go take a took at the contract this library is exporting.

Impls

The main priority of this library is to be able to talk with a llama.cpp server. It uses an API very similar to openai's API, so with a little bit of work, we should be able to support openai endpoints.

Stability

Please note the API may change drastically whilst the version is less than v1.0.0

TODO

Write tests

Documentation

Index

• func Chat(ctx context.Context, provider TextProvider, reasoningTokens int, in io.Reader, ...) error
• func HandleStreamedResponse(ctx context.Context, tokens <-chan *TextResponse, errs <-chan error, ...) error
• type MockLLM
  • func NewMockLLM() *MockLLM
  • func (m *MockLLM) Ask(ctx context.Context, history []string, reasoningTokens int) (*TextResponse, error)
  • func (m *MockLLM) StreamedAsk(ctx context.Context, history []string, reasoningTokens int) (<-chan *TextResponse, <-chan error)
• type TextProvider
• type TextResponse

Functions

func Chat

func Chat(ctx context.Context, provider TextProvider, reasoningTokens int, in io.Reader, out io.Writer) error

Chat is a basic util that reads a line of text from in, and writes the response to out

func HandleStreamedResponse

func HandleStreamedResponse(ctx context.Context, tokens <-chan *TextResponse, errs <-chan error, handler func(token *TextResponse) error) error

*

• A utility method to assist with reading the streamed response, returns the first encountered error

Types

type MockLLM

type MockLLM struct {
	Answers  []string
	Thoughts []string
}

func NewMockLLM

func NewMockLLM() *MockLLM

Useful for internal testing

func (*MockLLM) Ask

func (m *MockLLM) Ask(ctx context.Context, history []string, reasoningTokens int) (*TextResponse, error)

func (*MockLLM) StreamedAsk

func (m *MockLLM) StreamedAsk(ctx context.Context, history []string, reasoningTokens int) (<-chan *TextResponse, <-chan error)

type TextProvider

type TextProvider interface {
	Ask(ctx context.Context, history []string, reasoningTokens int) (*TextResponse, error)
	StreamedAsk(ctx context.Context, history []string, reasoningTokens int) (<-chan *TextResponse, <-chan error)
}

* An abstracted concept of an LLM provider

type TextResponse

type TextResponse struct {
	Answer    string // If the empty string, then it implies its a reasoning token/response
	Reasoning string // If the empty string, then it implies its an answer with no reasoning
}

* The most generic response from a provider * this may be used for entire enswers, or single tokens * If both fields are empty, then its undefined