pytorch

README

go-pytorch

Go binding for Pytorch C++ API. This is used by the Pytorch agent in MLModelScope to perform model inference in Go.

Installation

The binding requires Pytorch C++ (libtorch) and other Go packages.

Pytorch C++ (libtorch) Library

The Pytorch C++ library is expected to be under /opt/libtorch.

The binding is built using libtorch 1.8.1.

To install Pytorch C++ on your system, you can

1. download pre-built binary from Pytorch website: Choose Pytorch Build = Stable, Your OS = <fill>, Package = LibTorch, Language = C++ and CUDA = <fill>. Then download cxx11 ABI version. Unzip the packaged directory and copy to /opt/libtorch (or modify the corresponding CFLAGS and LDFLAGS paths if using a custom location).

2. build it from source: Refer to our scripts or the LIBRARY INSTALLATION section in the dockefiles.

• The default blas is OpenBLAS. The default OpenBLAS path for macOS is /usr/local/opt/openblas if installed throught homebrew (openblas is keg-only, which means it was not symlinked into /usr/local, because macOS provides BLAS and LAPACK in the Accelerate framework).

• The default pytorch C++ installation path is /opt/libtorch for linux, darwin and ppc64le without powerai

• The default CUDA path is /usr/local/cuda

See lib.go for details.

If you get an error about not being able to write to /opt then perform the following

sudo mkdir -p /opt/libtorch
sudo chown -R `whoami` /opt/libtorch

If you are using Pytorch docker images or other libary paths, change CGO_CFLAGS, CGO_CXXFLAGS and CGO_LDFLAGS enviroment variables. Refer to Using cgo with the go command.

For example,

    export CGO_CFLAGS="${CGO_CFLAGS} -I/tmp/libtorch/include"
    export CGO_CXXFLAGS="${CGO_CXXFLAGS} -I/tmp/libtorch/include"
    export CGO_LDFLAGS="${CGO_LDFLAGS} -L/tmp/libtorch/lib"

There is an issue when using libtorch with version < 1.6.0, the work around here is to set LRU_CACHE_CAPACITY=1 in the environmental variable.

Configure Environmental Variables

Configure the linker environmental variables since the Pytorch C++ library is under a non-system directory. Place the following in either your ~/.bashrc or ~/.zshrc file

Linux

export LIBRARY_PATH=$LIBRARY_PATH:/opt/libtorch/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/opt/libtorch/lib

macOS

export LIBRARY_PATH=$LIBRARY_PATH:/opt/libtorch/lib
export DYLD_LIBRARY_PATH=$DYLD_LIBRARY_PATH:/opt/libtorch/lib

Check the Build

Run go build to check the dependences installation and library paths set-up. On linux, the default is to use GPU, if you don't have a GPU, do go build -tags=nogpu instead of go build.

Note : The CGO interface passes go pointers to the C API. This is an error by the CGO runtime. Disable the error by placing

export GODEBUG=cgocheck=0

in your ~/.bashrc or ~/.zshrc file and then run either source ~/.bashrc or source ~/.zshrc

Credits

Parts of the implementation have been borrowed from orktes/go-torch

Documentation

Index

• Variables
• func GetError() error
• func GetErrorString() string
• func HasError() bool
• func PanicOnError()
• func PrintTensors(inputs ...*Tensor)
• func ResetError()
• type DType
• type DeviceKind
• type Error
  • func (te *Error) Error() string
• type Predictor
  • func New(ctx context.Context, opts ...options.Option) (*Predictor, error)
  • func (p *Predictor) Close()
  • func (p *Predictor) Predict(ctx context.Context, inputs []tensor.Tensor) error
  • func (p *Predictor) ReadPredictionOutput(ctx context.Context) ([]tensor.Tensor, error)
  • func (p *Predictor) ReadProfile() (string, error)
• type Tensor
  • func NewTensor(value interface{}, device DeviceKind, deviceId int) (*Tensor, error)
  • func NewTensorWithShape(value interface{}, shape []int64, dt DType, device DeviceKind, deviceId int) (*Tensor, error)
  • func (t *Tensor) DType() DType
  • func (t *Tensor) Shape() []int64
  • func (t *Tensor) Value() interface{}
• type Trace
  • func NewTrace(data string, start_time int64) (*Trace, error)
  • func (t Trace) Len() int
  • func (t Trace) Less(i, j int) bool
  • func (t *Trace) Publish(ctx context.Context, lvl tracer.Level, opts ...opentracing.StartSpanOption) error
  • func (t Trace) Swap(i, j int)
• type TraceEvent
  • func (t TraceEvent) ID() string
  • func (event *TraceEvent) Publish(ctx context.Context, lvl tracer.Level, opts ...opentracing.StartSpanOption) error
• type TraceEvents
  • func (t TraceEvents) Len() int
  • func (t TraceEvents) Less(i, j int) bool
  • func (t TraceEvents) Swap(i, j int)
• type Tuple
  • func (t Tuple) Get(index int) interface{}

Variables

var (
	Version   = "1.3.1"
	BuildDate = "undefined"
	GitCommit = "undefined"
)

Functions

func GetError

func GetError() error

func GetErrorString

func GetErrorString() string

func HasError

func HasError() bool

func PanicOnError

func PanicOnError()

func PrintTensors

func PrintTensors(inputs ...*Tensor)

PrintTensors prints tensors contents

func ResetError

func ResetError()

Types

type DType

type DType C.Torch_DataType

DType tensor scalar data type

type DeviceKind

type DeviceKind C.Torch_DeviceKind

const (
	UnknownDeviceKind DeviceKind = C.UNKNOWN_DEVICE_KIND
	CPUDeviceKind     DeviceKind = C.CPU_DEVICE_KIND
	CUDADeviceKind    DeviceKind = C.CUDA_DEVICE_KIND
)

type Error

type Error struct {
	// contains filtered or unexported fields
}

Error errors returned by torch functions

func (*Error) Error

func (te *Error) Error() string

type Predictor

type Predictor struct {
	// contains filtered or unexported fields
}

func New

func New(ctx context.Context, opts ...options.Option) (*Predictor, error)

func (*Predictor) Close

func (p *Predictor) Close()

func (*Predictor) Predict

func (p *Predictor) Predict(ctx context.Context, inputs []tensor.Tensor) error

func (*Predictor) ReadPredictionOutput

func (p *Predictor) ReadPredictionOutput(ctx context.Context) ([]tensor.Tensor, error)

func (*Predictor) ReadProfile

func (p *Predictor) ReadProfile() (string, error)

type Tensor

type Tensor struct {
	// contains filtered or unexported fields
}

Tensor holds a multi-dimensional array of elements of a single data type.

func NewTensor

func NewTensor(value interface{}, device DeviceKind, deviceId int) (*Tensor, error)

NewTensor converts from a Go value to a Tensor. Valid values are scalars, slices, and arrays. Every element of a slice must have the same length so that the resulting Tensor has a valid shape.

func NewTensorWithShape

func NewTensorWithShape(value interface{}, shape []int64, dt DType, device DeviceKind, deviceId int) (*Tensor, error)

NewTensorWithShape converts a single dimensional Go array or slice into a Tensor with given shape

func (*Tensor) DType

func (t *Tensor) DType() DType

DType returns tensors datatype

func (*Tensor) Shape

func (t *Tensor) Shape() []int64

Shape returns tensors shape

func (*Tensor) Value

func (t *Tensor) Value() interface{}

Value returns tensors value as a go type

type Trace

type Trace struct {
	StartTime   time.Time
	TraceEvents TraceEvents
}

func NewTrace

func NewTrace(data string, start_time int64) (*Trace, error)

func (Trace) Len

func (t Trace) Len() int

func (Trace) Less

func (t Trace) Less(i, j int) bool

func (*Trace) Publish

func (t *Trace) Publish(ctx context.Context, lvl tracer.Level, opts ...opentracing.StartSpanOption) error

func (Trace) Swap

func (t Trace) Swap(i, j int)

type TraceEvent

type TraceEvent struct {
	Name            string    `json:"name,omitempty"`
	Phase           string    `json:"ph,omitempty"`
	Timestamp       float64   `json:"ts,omitempty"`
	Duration        float64   `json:"dur,omitempty"`
	ProcessID       string    `json:"pid,omitempty"`
	ThreadID        int64     `json:"tid,omitempty"`
	Shape           string    `json:"shape,omitempty"`
	AllocatedMemory int64     `json:"allocated_memory,omitempty"`
	PeakMemory      int64     `json:"peak_memory,omitempty"`
	Index           int64     `json:"layer_sequence_index,omitempty"`
	Start           int64     `json:"-"`
	End             int64     `json:"-"`
	StartTime       time.Time `json:"-"`
	EndTime         time.Time `json:"-"`
	Seq             int64     `json:"-"`
}

func (TraceEvent) ID

func (t TraceEvent) ID() string

func (*TraceEvent) Publish

func (event *TraceEvent) Publish(ctx context.Context, lvl tracer.Level, opts ...opentracing.StartSpanOption) error

type TraceEvents

type TraceEvents []TraceEvent

func (TraceEvents) Len

func (t TraceEvents) Len() int

func (TraceEvents) Less

func (t TraceEvents) Less(i, j int) bool

func (TraceEvents) Swap

func (t TraceEvents) Swap(i, j int)

type Tuple

type Tuple []interface{}

Tuple a tuple type

func (Tuple) Get

func (t Tuple) Get(index int) interface{}

Get returns a type in specific tuple index (otherwise returns nil)