ggllm

package module
v0.0.0-...-3f0a32e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 9, 2023 License: MIT Imports: 6 Imported by: 1

README

Go Reference go-ggllm.cpp

ggllm.cpp golang bindings.

The go-ggllm.cpp bindings are high level, as such most of the work is kept into the C/C++ code to avoid any extra computational cost, be more performant and lastly ease out maintenance, while keeping the usage as simple as possible.

Check out this and this write-ups which summarize the impact of a low-level interface which calls C functions from Go.

If you are looking for an high-level OpenAI compatible API, check out here.

Usage

Note: This repository uses git submodules to keep track of ggllm.cpp.

Clone the repository locally:

git clone --recurse-submodules https://github.com/mudler/go-ggllm.cpp

To build the bindings locally, run:

cd go-ggllm.cpp
make libgglm.a

Now you can run the example with:

LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14

Acceleration

OpenBLAS

To build and run with OpenBLAS, for example:

BUILD_TYPE=openblas make libgglm.a
CGO_LDFLAGS="-lopenblas" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14
CuBLAS

To build with CuBLAS:

BUILD_TYPE=cublas make libgglm.a
CGO_LDFLAGS="-lcublas -lcudart -L/usr/local/cuda/lib64/" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14
OpenCL
BUILD_TYPE=clblas CLBLAS_DIR=... make libgglm.a
CGO_LDFLAGS="-lOpenCL -lclblast -L/usr/local/lib64/" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14

You should see something like this from the output when using the GPU:

ggml_opencl: selecting platform: 'Intel(R) OpenCL HD Graphics'                                            
ggml_opencl: selecting device: 'Intel(R) Graphics [0x46a6]'                                               
ggml_opencl: device FP16 support: true

GPU offloading

Metal (Apple Silicon)
BUILD_TYPE=metal make libgglm.a
CGO_LDFLAGS="-framework Foundation -framework Metal -framework MetalKit -framework MetalPerformanceShaders" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go build ./examples/main.go
cp build/bin/ggml-metal.metal .
./main -m "/model/path/here" -t 1 -ngl 1

Enjoy!

The documentation is available here and the full example code is here.

License

MIT

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Falcon 

type Falcon struct {
	// contains filtered or unexported fields
}

func New 

func New(model string, opts ...ModelOption) (*Falcon, error)

func (*Falcon) Free 

func (l *Falcon) Free()

func (*Falcon) LoadState 

func (l *Falcon) LoadState(state string) error

func (*Falcon) Predict 

func (l *Falcon) Predict(text string, opts ...PredictOption) (string, error)

func (*Falcon) SaveState 

func (l *Falcon) SaveState(dst string) error

func (*Falcon) SetTokenCallback 

func (l *Falcon) SetTokenCallback(callback func(token string) bool)

SetTokenCallback registers a callback for the individual tokens created when running Predict. It will be called once for each token. The callback shall return true as long as the model should continue predicting the next token. When the callback returns false the predictor will return. The tokens are just converted into Go strings, they are not trimmed or otherwise changed. Also the tokens may not be valid UTF-8. Pass in nil to remove a callback.

It is save to call this method while a prediction is running.

type ModelOption 

type ModelOption func(p *ModelOptions)
var EnableEmbeddings ModelOption = func(p *ModelOptions) {
	p.Embeddings = true
}
var EnableF16Memory ModelOption = func(p *ModelOptions) {
	p.F16Memory = true
}
var EnableMLock ModelOption = func(p *ModelOptions) {
	p.MLock = true
}
var VocabOnly ModelOption = func(p *ModelOptions) {
	p.VocabOnly = true
}

func SetContext 

func SetContext(c int) ModelOption

SetContext sets the context size.

func SetGPULayers 

func SetGPULayers(n int) ModelOption

SetGPULayers sets the number of GPU layers to use to offload computation

func SetMMap 

func SetMMap(b bool) ModelOption

SetContext sets the context size.

func SetMainGPU 

func SetMainGPU(maingpu string) ModelOption

SetMainGPU sets the main_gpu

func SetModelSeed 

func SetModelSeed(c int) ModelOption

func SetNBatch 

func SetNBatch(n_batch int) ModelOption

SetNBatch sets the n_Batch

func SetTensorSplit 

func SetTensorSplit(maingpu string) ModelOption

Set sets the tensor split for the GPU

type ModelOptions 

type ModelOptions struct {
	ContextSize int
	Seed        int
	NBatch      int
	F16Memory   bool
	MLock       bool
	MMap        bool
	VocabOnly   bool
	Embeddings  bool
	NGPULayers  int
	MainGPU     string
	TensorSplit string
}
var DefaultModelOptions ModelOptions = ModelOptions{
	ContextSize: 512,
	Seed:        0,
	F16Memory:   false,
	MLock:       false,
	Embeddings:  false,
	MMap:        true,
}

func NewModelOptions 

func NewModelOptions(opts ...ModelOption) ModelOptions

Create a new PredictOptions object with the given options.

type PredictOption 

type PredictOption func(p *PredictOptions)
var Debug PredictOption = func(p *PredictOptions) {
	p.DebugMode = true
}
var EnableF16KV PredictOption = func(p *PredictOptions) {
	p.F16KV = true
}
var EnablePromptCacheAll PredictOption = func(p *PredictOptions) {
	p.PromptCacheAll = true
}
var EnablePromptCacheRO PredictOption = func(p *PredictOptions) {
	p.PromptCacheRO = true
}
var IgnoreEOS PredictOption = func(p *PredictOptions) {
	p.IgnoreEOS = true
}

func SetBatch 

func SetBatch(size int) PredictOption

SetBatch sets the batch size.

func SetFrequencyPenalty 

func SetFrequencyPenalty(fp float64) PredictOption

SetFrequencyPenalty sets the frequency penalty parameter, freq_penalty.

func SetLogitBias 

func SetLogitBias(lb string) PredictOption

SetLogitBias sets the logit bias parameter.

func SetMemoryMap 

func SetMemoryMap(b bool) PredictOption

SetMemoryMap sets memory mapping.

func SetMirostat 

func SetMirostat(m int) PredictOption

SetMirostat sets the mirostat parameter.

func SetMirostatETA 

func SetMirostatETA(me float64) PredictOption

SetMirostatETA sets the mirostat ETA parameter.

func SetMirostatTAU 

func SetMirostatTAU(mt float64) PredictOption

SetMirostatTAU sets the mirostat TAU parameter.

func SetMlock 

func SetMlock(b bool) PredictOption

SetMlock sets the memory lock.

func SetNKeep 

func SetNKeep(n int) PredictOption

SetKeep sets the number of tokens from initial prompt to keep.

func SetPathPromptCache 

func SetPathPromptCache(f string) PredictOption

SetPathPromptCache sets the session file to store the prompt cache.

func SetPenalizeNL 

func SetPenalizeNL(pnl bool) PredictOption

SetPenalizeNL sets whether to penalize newlines or not.

func SetPenalty 

func SetPenalty(penalty float64) PredictOption

SetPenalty sets the repetition penalty for text generation.

func SetPredictionMainGPU 

func SetPredictionMainGPU(maingpu string) PredictOption

SetPredictionMainGPU sets the main_gpu

func SetPredictionTensorSplit 

func SetPredictionTensorSplit(maingpu string) PredictOption

SetPredictionTensorSplit sets the tensor split for the GPU

func SetPresencePenalty 

func SetPresencePenalty(pp float64) PredictOption

SetPresencePenalty sets the presence penalty parameter, presence_penalty.

func SetRepeat 

func SetRepeat(repeat int) PredictOption

SetRepeat sets the number of times to repeat text generation.

func SetSeed 

func SetSeed(seed int) PredictOption

SetSeed sets the random seed for sampling text generation.

func SetStopWords 

func SetStopWords(stop ...string) PredictOption

SetStopWords sets the prompts that will stop predictions.

func SetTailFreeSamplingZ 

func SetTailFreeSamplingZ(tfz float64) PredictOption

SetTailFreeSamplingZ sets the tail free sampling, parameter z.

func SetTemperature 

func SetTemperature(temp float64) PredictOption

SetTemperature sets the temperature value for text generation.

func SetThreads 

func SetThreads(threads int) PredictOption

SetThreads sets the number of threads to use for text generation.

func SetTokenCallback 

func SetTokenCallback(fn func(string) bool) PredictOption

SetTokenCallback sets the prompts that will stop predictions.

func SetTokens 

func SetTokens(tokens int) PredictOption

SetTokens sets the number of tokens to generate.

func SetTopK 

func SetTopK(topk int) PredictOption

SetTopK sets the value for top-K sampling.

func SetTopP 

func SetTopP(topp float64) PredictOption

SetTopP sets the value for nucleus sampling.

func SetTypicalP 

func SetTypicalP(tp float64) PredictOption

SetTypicalP sets the typicality parameter, p_typical.

type PredictOptions 

type PredictOptions struct {
	Seed, Threads, Tokens, TopK, Repeat, Batch, NKeep int
	TopP, Temperature, Penalty                        float64
	F16KV                                             bool
	DebugMode                                         bool
	StopPrompts                                       string
	IgnoreEOS                                         bool

	TailFreeSamplingZ float64
	TypicalP          float64
	FrequencyPenalty  float64
	PresencePenalty   float64
	Mirostat          int
	MirostatETA       float64
	MirostatTAU       float64
	PenalizeNL        bool
	LogitBias         string
	TokenCallback     func(string) bool

	PathPromptCache             string
	MLock, MMap, PromptCacheAll bool
	PromptCacheRO               bool
	MainGPU                     string
	TensorSplit                 string
}
var DefaultOptions PredictOptions = PredictOptions{
	Seed:              -1,
	Threads:           4,
	Tokens:            128,
	Penalty:           1.1,
	Repeat:            64,
	Batch:             512,
	NKeep:             0,
	TopK:              40,
	TopP:              0.95,
	TailFreeSamplingZ: 1.0,
	TypicalP:          1.0,
	Temperature:       0.8,
	FrequencyPenalty:  0.0,
	PresencePenalty:   0.0,
	Mirostat:          0,
	MirostatTAU:       5.0,
	MirostatETA:       0.1,
	MMap:              true,
}

func NewPredictOptions 

func NewPredictOptions(opts ...PredictOption) PredictOptions

Create a new PredictOptions object with the given options.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.