talkative

README

Talkative

A Golang Ollama REST API Wrapper

The talkative package simplifies interaction with the Ollama REST API in Golang applications. It provides a user-friendly interface to access Ollama functionalities without dealing with the raw HTTP requests and responses.

Features

• Abstraction: Hides the underlying HTTP details, allowing you to focus on the Ollama API functionality.
• Type Safety: Provides strongly typed methods for interacting with the API.
• Error Handling: Handles errors gracefully, returning informative error messages.

• Flexible Interactions: Offers two interaction modes for both/chat and/generate endpoints:

  • Typed Response: Suitable for scenarios where you are utilising the response directly in your program to display or process further.
  • Untyped Response: Ideal for scenarios where you are not utilising that response by yourself other than forwarding this to your own APIs i.e: built your own APIs utilizing talkative package to interact with Ollama server.

• Typed Response: Suitable for scenarios where you are utilizing the response directly in your program to display or process further.

• Untyped Response: Ideal for forwarding the raw response from Ollama to your own REST API endpoints without additional processing. This is useful when you don't need to modify the Ollama response within your Golang application

Installation

go get github.com/rifaideen/talkative

Usage

    package main

    import (
    "fmt"
  
    "github.com/rifaideen/talkative"
    )

    func main() {
        // Create a new talkative client with Ollama server url
        client, err := talkative.New("http://localhost:11434")
  
        if err != nil {
            panic("Failed to create talkative client")
        }

        // client is ready, start fuelling with your curiosity.
    }

Examples

To explore practical examples of using the talkative package for various tasks, navigate to the _examples directory within the package.

Feedback

• Submit feedback

Contributing

We welcome contributions to the talkative package! While we don't have a formal CONTRIBUTING.md file yet, feel free to submit pull requests with clear descriptions of your changes. We'll be happy to review them.

Disclaimer of Non-Liability

This project is provided "as is" and without any express or implied warranties, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. In no event shall the authors or copyright holders be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the software.

License

The talkative package is licensed under the Apache License, Version 2.0.

Additional Resources

Ollama API documentation

Documentation

Index

• Constants
• Variables
• func StreamPlainResponse(body io.ReadCloser, cb func(string, error))
• func StreamResponse[T any](body io.ReadCloser, cb func(*T, error))
• type ChatCallBack
• type ChatMessage
• type ChatMetrics
• type ChatParams
• type ChatRequest
• type ChatResponse
• type Client
  • func New(url string) (*Client, error)
  • func (c *Client) Chat(model string, cb ChatCallBack, params *ChatParams, msgs ...ChatMessage) (<-chan bool, error)
  • func (c *Client) Completion(model string, cb CompletionCallback, msg *CompletionMessage) (<-chan bool, error)
  • func (c *Client) PlainChat(model string, cb PlainChatCallBack, params *ChatParams, msgs ...ChatMessage) (<-chan bool, error)
  • func (c *Client) PlainCompletion(model string, cb PlainCompletionCallback, msg *CompletionMessage) (<-chan bool, error)
• type CompletionCallback
• type CompletionMessage
• type CompletionMetrics
• type CompletionParams
• type CompletionRequest
• type CompletionResponse
• type PlainChatCallBack
• type PlainCompletionCallback
• type Role

Constants

const (
	// User role for regular users.
	USER Role = "user"

	// Assistant role for AI assistants or chatbots.
	ASSISTANT Role = "assistant"

	// Default model to be used when model is not specified.
	DEFAULT_MODEL string = "llama2"
)

Variables

var (
	ErrUrl        = errors.New("url cannot be empty")         // Error for missing URL
	ErrCallback   = errors.New("callback cannot be empty")    // Error for missing callback function.
	ErrMessage    = errors.New("message cannot be empty")     // Error for empty message list.
	ErrInvoke     = errors.New("unable to invoke ollama api") // Error for failing to call the Ollama API.
	ErrEncoding   = errors.New("unable to encode")            // Error for problems encoding data to JSON.
	ErrDecoding   = errors.New("unable to decode")            // Error for problems encoding data to JSON.
	ErrBadRequest = errors.New("")                            // Error for bad request response from Ollama API. This just acts as a placeholder, the actual response will be wrapped under this error
)

Pre-defined errors used throughout the code for consistency.

Functions

func StreamPlainResponse

func StreamPlainResponse(body io.ReadCloser, cb func(string, error))

Streaming the plain response from the server asynchronously.

This function takes an io.ReadCloser object (`body`) representing the response body and a callback function (`cb`) for handling individual responses and errors. It iterates through the response and invoking the callback with plain string for processing.

In case of errors during decoding or processing, the callback is invoked with the error and processing stops. The function closes the response body before exiting.

func StreamResponse

func StreamResponse[T any](body io.ReadCloser, cb func(*T, error))

Streaming the response from the server asynchronously.

This function takes an io.ReadCloser object (`body`) representing the response body and a callback function (`cb`) for handling individual responses and errors. It iterates through the response, decoding each message and invoking the callback for processing.

In case of errors during decoding or processing, the callback is invoked with the error and processing stops. The function closes the response body before exiting.

Types

type ChatCallBack

type ChatCallBack func(*ChatResponse, error)

Callback function type used for handling individual chat responses and errors. Takes a pointer to a ChatResponse struct and an error as arguments.

type ChatMessage

type ChatMessage struct {
	Role    Role   `json:"role"`    // Role of the sender (user or assistant).
	Content string `json:"content"` // Content of the message.
}

ChatMessage struct represents a single message sent or received in the chat.

type ChatMetrics

type ChatMetrics struct {
	TotalDuration      int `json:"total_duration"`       // Total processing time in milliseconds.
	LoadDuration       int `json:"load_duration"`        // Time spent loading the model (milliseconds).
	PromptEvalCount    int `json:"prompt_eval_count"`    // Number of prompt evaluations performed.
	PromptEvalDuration int `json:"prompt_eval_duration"` // Time spent on prompt evaluation (milliseconds).
	EvalCount          int `json:"eval_count"`           // Number of overall evaluations performed.
	EvalDuration       int `json:"eval_duration"`        // Time spent on overall evaluation (milliseconds).
}

type ChatParams

type ChatParams struct {
	Format    string                 `json:"format,omitempty"`     // The format to be used in the completion response
	Options   map[string]interface{} `json:"options,omitempty"`    // The additional model parameters  listed in the Modelfile documentation
	Template  string                 `json:"template,omitempty"`   // The prompt template to use (overrides what is defined in the Modelfile)
	Stream    *bool                  `json:"stream,omitempty"`     // Whether to get response in single shot rather than streaming
	KeepAlive string                 `json:"keep_alive,omitempty"` // How long to keep the model will stay loaded into the memory. Default to 5m(inutes)
}

CompletionParams represents the advanced parameters (Optional) to be supplied to the completion request.

type ChatRequest

type ChatRequest struct {
	Model    string        `json:"model"`    // The model to be used for processing the chat.
	Messages []ChatMessage `json:"messages"` // List of messages to be processed.

	*ChatParams `json:",omitempty"` // The additional parameters for the chat
}

ChatRequest struct represents the request body sent to the Ollama API for chat processing.

type ChatResponse

type ChatResponse struct {
	Model       string      `json:"model"`      // The model used for processing.
	Message     ChatMessage `json:"message"`    // The response message.
	CreatedAt   time.Time   `json:"created_at"` // Time the response was created on the server.
	Done        bool        `json:"done"`       // Indicates if processing is complete.
	ChatMetrics             // The metrics associated about the chat
}

ChatResponse struct represents the response received from the Ollama API after processing chat messages.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client struct holds information for interacting with the Ollama API.

func New

func New(url string) (*Client, error)

New function creates a new Client instance for interacting with the Ollama API. Takes the base URL of the Ollama API as an argument.

func (*Client) Chat

func (c *Client) Chat(model string, cb ChatCallBack, params *ChatParams, msgs ...ChatMessage) (<-chan bool, error)

Initiates a chat process and asynchronously handles responses through a callback function.

This function takes model name, callback function (`cb`) and a variable number of messages (`msgs`) as arguments. It performs the following steps:

1. Validates the model, callback and message arguments.
2. When model is empty, it uses the model to DEFAULT_MODEL to perform the operation.
3. Prepares a request body with the messages and model information.
4. Sends a POST request to the chat endpoint from this client.
5. Handles the response status code and potential errors.
6. Launches a goroutine to process the chat response asynchronously.
7. Returns a channel (`<-chan bool`) that signals completion of the chat process and any errors encountered.

The callback function (`cb`) is responsible for handling individual chat responses and errors. The completion channel (`<-chan bool`) allows the caller to track the progress of the chat process if needed.

Note that the channel (`chDone`) is not explicitly closed in this example. However, the goroutine running `processChat` terminates naturally after sending the completion signal (`true`), effectively indicating no more data will be received on the channel.

func (*Client) Completion

func (c *Client) Completion(model string, cb CompletionCallback, msg *CompletionMessage) (<-chan bool, error)

Completion initiates a completion request to the server and returns a channel that signals when the operation is done.

This method takes a CompletionCallback function and a CompletionMessage as arguments. The callback function is invoked with the completion response and any error that occurred during the request. The CompletionMessage contains the prompt and any associated images for the completion request.

Parameters: - cb CompletionCallback: The callback function to be called upon completion of the request. It must not be nil. - msg *CompletionMessage: A pointer to the CompletionMessage containing the prompt and images for the completion. It must not be nil.

Returns: - <-chan bool: A channel that signals when the completion operation is done. The channel receives a boolean value of true upon completion. - error: An error if the callback function or the message is nil, if there's an error encoding the request, or if the server responds with an error.

The method constructs a CompletionRequest from the provided message, encodes it into JSON, and sends it to the server. It handles HTTP response status codes, specifically checking for a BadRequest (400) to return any server-side error messages. Upon a successful request, it starts a goroutine to stream the response and invoke the provided callback function, signaling completion through the returned channel.

func (*Client) PlainChat

func (c *Client) PlainChat(model string, cb PlainChatCallBack, params *ChatParams, msgs ...ChatMessage) (<-chan bool, error)

Initiates a plain chat process and asynchronously handles responses through a callback function.

This method is identical to Chat(), except that it invokes the callback with plain json string without further processing.

func (*Client) PlainCompletion

func (c *Client) PlainCompletion(model string, cb PlainCompletionCallback, msg *CompletionMessage) (<-chan bool, error)

Completion initiates a plain completion request to the server and returns a channel that signals when the operation is done.

This method is identical to Completion(), except that it invokes the callback with plain json string without further processing.

type CompletionCallback

type CompletionCallback func(*CompletionResponse, error)

CompletionCallback defines a function type that is used as a callback for handling completion responses. It takes a pointer to a CompletionResponse and an error as arguments.

Parameters: - *CompletionResponse: A pointer to the CompletionResponse received after a completion request. - error: An error that might have occurred during the completion process.

type CompletionMessage

type CompletionMessage struct {
	Prompt string   `json:"prompt"` // The text prompt to be completed.
	Images []string `json:"images"` // A list of image URLs associated with the prompt.

	*CompletionParams `json:",omitempty"` // The additional parameters for the completion
}

CompletionMessage represents the message structure for initiating a completion request.

type CompletionMetrics

type CompletionMetrics struct {
	TotalDuration      int   `json:"total_duration"`       // Total processing time in milliseconds.
	LoadDuration       int   `json:"load_duration"`        // Time spent loading the model (milliseconds).
	PromptEvalCount    int   `json:"prompt_eval_count"`    // Number of prompt evaluations performed.
	PromptEvalDuration int   `json:"prompt_eval_duration"` // Time spent on prompt evaluation (milliseconds).
	EvalCount          int   `json:"eval_count"`           // Number of overall evaluations performed.
	EvalDuration       int   `json:"eval_duration"`        // Time spent on overall evaluation (milliseconds).
	Context            []int `json:"context"`              // Encoding of the conversation used in this response.
}

CompletionMetrics struct encapsulates various metrics related to the completion process. It includes total processing time, model loading time, counts and durations of prompt and overall evaluations, and the context encoding of the conversation used in the response.

type CompletionParams

type CompletionParams struct {
	Format    string                 `json:"format,omitempty"`     // The format to be used in the completion response
	Options   map[string]interface{} `json:"options,omitempty"`    // The additional model parameters  listed in the Modelfile documentation
	System    string                 `json:"system,omitempty"`     // The system message to use (overrides what is defined in the Modelfile)
	Template  string                 `json:"template,omitempty"`   // The template to use (overrides what is defined in the Modelfile)
	Context   interface{}            `json:"context,omitempty"`    // The context parameter returned from previous request to completion, to keep short conversational memory
	Stream    *bool                  `json:"stream,omitempty"`     // Whether to get response in single shot rather than streaming
	Raw       bool                   `json:"raw,omitempty"`        // If true, no formatting will be applied to the prompt
	KeepAlive string                 `json:"keep_alive,omitempty"` // How long to keep the model will stay loaded into the memory. Default to 5m(inutes)
}

CompletionParams represents the advanced parameters (Optional) to be supplied to the completion request.

type CompletionRequest

type CompletionRequest struct {
	Model             string              `json:"model"`  // The model to use for completion.
	Prompt            string              `json:"prompt"` // The prompt for completion.
	Images            []string            `json:"images"` // The images associated with the completion.
	*CompletionParams `json:",omitempty"` // The additional parameters for the completion
}

CompletionRequest represents a request for completion.

type CompletionResponse

type CompletionResponse struct {
	Model     string `json:"model"`      // The model used for the completion.
	Response  string `json:"response"`   // The generated response based on the prompt.
	CreatedAt string `json:"created_at"` // The timestamp when the response was created.
	Done      bool   `json:"done"`       // A boolean indicating if the completion process is finished.

	CompletionMetrics // embeds CompletionMetrics
}

CompletionResponse represents the response received after a completion request.

It also embeds CompletionMetrics which includes upon completion

type PlainChatCallBack

type PlainChatCallBack func(string, error)

PlainChatCallBack function type used for handling individual chat responses and errors. Takes a string and an error as arguments.

type PlainCompletionCallback

type PlainCompletionCallback func(string, error)

PlainCompletionCallback defines a function type that is used as a callback for handling plain completion responses. It takes string and an error as arguments.

Parameters: - string: Json encoded plain string received after a completion request. - error: An error that might have occurred during the completion process.

type Role

type Role string

Define an enum-like type to represent different user roles in the chat system.