README
¶
go-mobileclip
Go package for deriving vector embeddings for text and images from a MobileCLIP "service".
Motivation
This is a Go package for deriving vector embeddings for text and images from a MobileCLIP "service" where, for the time being, that "service" is expected to be a gRPC service as implemented by the sfomuseum/swift-mobileclip package.
In effect this is client-code, written in Go, for a service written in Swift.
Documentation
godoc documentation is incomplete at this time.
Usage
import (
"context"
"os"
"github.com/sfomuseum/go-mobileclip"
)
func main() {
body, _ := os.ReadAll("/path/to/image.png")
cl, _ := mobileclip.NewEmbeddingsClient(ctx, "grpc://localhost:8080")
req := &mobileclip.EmbeddingsRequest{
Model: "s0",
Body: body,
}
embeddings, _ := cl.ComputeImageEmbeddings(ctx, req)
}
Error handling removed for the sake of brevity.
Where embeddings is a struct of type Embeddings:
type Embeddings struct {
Id string `json:"id,omitempty"`
Embeddings []float32 `json:"embeddings"`
Dimensions int32 `json:"dimensions"`
Model string `json:"model"`
Created int64 `json:"created"`
}
Tools
$> make cli
go build -mod vendor -ldflags="-s -w" -o bin/embeddings cmd/embeddings/main.go
embeddings
Command-line tool for deriving text or image embeddings from a MobileCLIP "service". Results are written as a JSON-encoded string to STDOUT.
$> ./bin/embeddings -h
Command-line tool for deriving text or image embeddings from a MobileCLIP "service". Results are written as a JSON-encoded string to STDOUT.
Usage:
./bin/embeddings [options] text|image arg(N) arg(N)
If the first argument is "image" then the body of the image to derive embeddings for will be read from the second argument.
If the first argument is "text" then the body of the text to derive embeddings for will be determined as follows: If there are only two arguments and the second argument is "-" then the body will be read from STDIN. Otherwise, if there are only two arguments the body of the text will be read from the file specified in the second argument. Finally, if there are more than two arguments then the body of the text will be the concatenation of the second to the last argument.
Valid options are:
-client-uri string
A valid sfomuseum/go-mobileclip.EmbeddingsClient URI. (default "grpc://localhost:8080")
-model string
The name of the MobileCLIP model to use to derive embeddings. Valid options are: s0, s1, s2, blt (default "s0")
embeddings text
$> echo "hello world" | ./bin/embeddings text -
{"embeddings":[-0.3161621,-0.1697998,1.4482422,0.04 ... and so on
embeddings image
$> ./bin/embeddings image ~/Desktop/test14.png
{"embeddings":[-0.025161743,-0.027786255,-0.0014038086,0.0035247803,-0.006515503,-0.012527466,-0.041168213 ... and so on
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func RegisterEmbeddingsClient ¶
func RegisterEmbeddingsClient(ctx context.Context, scheme string, init_func EmbeddingsClientInitializationFunc) error
RegisterEmbeddingsClient registers 'scheme' as a key pointing to 'init_func' in an internal lookup table used to create new `Client` instances by the `NewEmbeddingsClient` method.
Types ¶
type Embeddings ¶
type EmbeddingsClient ¶
type EmbeddingsClient interface {
ComputeTextEmbeddings(context.Context, *EmbeddingsRequest) (*Embeddings, error)
ComputeImageEmbeddings(context.Context, *EmbeddingsRequest) (*Embeddings, error)
}
func NewEmbeddingsClient ¶
func NewEmbeddingsClient(ctx context.Context, uri string) (EmbeddingsClient, error)
NewClient returns a new `Client` instance configured by 'uri'. The value of 'uri' is parsed as a `url.URL` and its scheme is used as the key for a corresponding `ClientInitializationFunc` function used to instantiate the new `Client`. It is assumed that the scheme (and initialization function) have been registered by the `RegisterClient` method.
func NewGrpcEmbeddingsClient ¶
func NewGrpcEmbeddingsClient(ctx context.Context, uri string) (EmbeddingsClient, error)
func NewNullEmbeddingsClient ¶
func NewNullEmbeddingsClient(ctx context.Context, uri string) (EmbeddingsClient, error)
type EmbeddingsClientInitializationFunc ¶
type EmbeddingsClientInitializationFunc func(ctx context.Context, uri string) (EmbeddingsClient, error)
ClientInitializationFunc is a function defined by individual client package and used to create an instance of that client
type EmbeddingsRequest ¶
type GrpcEmbeddingsClient ¶
type GrpcEmbeddingsClient struct {
// contains filtered or unexported fields
}
func (*GrpcEmbeddingsClient) ComputeImageEmbeddings ¶
func (e *GrpcEmbeddingsClient) ComputeImageEmbeddings(ctx context.Context, req *EmbeddingsRequest) (*Embeddings, error)
func (*GrpcEmbeddingsClient) ComputeTextEmbeddings ¶
func (e *GrpcEmbeddingsClient) ComputeTextEmbeddings(ctx context.Context, req *EmbeddingsRequest) (*Embeddings, error)
type NullEmbeddingsClient ¶
type NullEmbeddingsClient struct{}
func (*NullEmbeddingsClient) ComputeImageEmbeddings ¶
func (e *NullEmbeddingsClient) ComputeImageEmbeddings(ctx context.Context, req *EmbeddingsRequest) (*Embeddings, error)
func (*NullEmbeddingsClient) ComputeTextEmbeddings ¶
func (e *NullEmbeddingsClient) ComputeTextEmbeddings(ctx context.Context, req *EmbeddingsRequest) (*Embeddings, error)
Source Files
Directories