README
¶
What is Vald?
Vald is a highly scalable distributed fast approximate nearest neighbor (ANN) dense vector search engine.
Vald is designed and implemented based on Cloud-Native architecture.
Vald has automatic vector indexing and index backup, and horizontal scaling which made for searching from billions of feature vector data.
Vald is easy to use, feature-rich and highly customizable as you needed.
It uses the fastest ANN Algorithm NGT to search neighbors.
(If you are interested in ANN benchmarks, please refer to ann-benchmarks.com.)
For more information, please refer to Official Web Site.
Vald can handle any object data, image, audio processing, video, text, binary, or etc., if converting to the vector, and be used for:
- Recognition
- Recommendation
- Detecting
- Grammar checker
- Real-time translator
- also you want to do!
Requirements
- Kubernetes 1.19~
- AVX2 instructions (required by Vald Agent NGT)
Get Started
Go to Get Started page to try out Vald !
Installation
Using Helm
helm repo add vald https://vald.vdaas.org/charts
helm install vald-cluster vald/vald
If you use the default values.yaml, the nightly images will be installed.
Using Helm-operator
Please refer to vald-helm-operator.
Components
| Component | Docker image | latest image | nightly image |
|---|---|---|---|
| Agent NGT |
|
|
|
| Agent Sidecar |
|
|
|
| Discoverer |
|
|
|
| Gateways |
|
|
|
| Index Manager |
|
|
|
| Helm Operator |
|
|
|
Docker images tagging policy:
nightly... latest build of main branchvX.X.X... released versionslatest... latest build of release versionsstable... latest long-term supported version
Tools
Vald Users
Contribution
Please read the contribution guide.
Before your first commit to this repository, it is strongly recommended to run the commands below.
git clone https://github.com/vdaas/vald && cd vald
make init
Contributors
Thanks goes to these wonderful people (emoji key):
LICENSE
vald released under Apache 2.0 license, refer LICENSE file.
Directories
¶
| Path | Synopsis |
|---|---|
|
apis
module
|
|
|
grpc/v1/vald
Package vald provides vald server interface
|
Package vald provides vald server interface |
|
charts
|
|
|
cmd
|
|
|
agent/core/ngt
Package main provides program main
|
Package main provides program main |
|
agent/sidecar
Package main provides program main
|
Package main provides program main |
|
discoverer/k8s
Package main provides program main
|
Package main provides program main |
|
gateway/filter
Package main provides program main
|
Package main provides program main |
|
gateway/lb
Package main provides program main
|
Package main provides program main |
|
manager/index
Package main provides program main
|
Package main provides program main |
|
example
|
|
|
client
Module
|
|
|
hack
|
|
|
benchmark/core/benchmark
Package benchmark provides benchmark frame
|
Package benchmark provides benchmark frame |
|
benchmark/core/benchmark/strategy
Package strategy provides benchmark strategy
|
Package strategy provides benchmark strategy |
|
benchmark/internal/core/algorithm
Package algorithm provides core interface
|
Package algorithm provides core interface |
|
benchmark/internal/core/algorithm/ngt
Package ngt provides ngt
|
Package ngt provides ngt |
|
benchmark/internal/starter
Package starter provides starter interfaces
|
Package starter provides starter interfaces |
|
benchmark/internal/starter/agent/core/ngt
Package ngt provides ngt agent starter functionality
|
Package ngt provides ngt agent starter functionality |
|
benchmark/internal/starter/gateway/vald
Package vald provides vald starter functionality
|
Package vald provides vald starter functionality |
|
internal
|
|
|
backoff
Package backoff provides backoff function controller
|
Package backoff provides backoff function controller |
|
cache
Package cache provides implementation of cache
|
Package cache provides implementation of cache |
|
cache/cacher
Package cacher provides implementation of cache type definition
|
Package cacher provides implementation of cache type definition |
|
cache/gache
Package gache provides implementation of cache using gache
|
Package gache provides implementation of cache using gache |
|
client/v1/client
Package client provides vald component client interfaces
|
Package client provides vald component client interfaces |
|
client/v1/client/agent/core
Package core provides agent ngt gRPC client functions
|
Package core provides agent ngt gRPC client functions |
|
client/v1/client/discoverer
Package discoverer
|
Package discoverer |
|
client/v1/client/filter/egress
Package egress provides egress filter client logic
|
Package egress provides egress filter client logic |
|
client/v1/client/filter/ingress
Package ingress provides ingress filter client logic
|
Package ingress provides ingress filter client logic |
|
client/v1/client/vald
Package vald provides vald grpc client library
|
Package vald provides vald grpc client library |
|
compress
Package compress provides compressor interface
|
Package compress provides compressor interface |
|
config
Package config providers configuration type and load configuration logic
|
Package config providers configuration type and load configuration logic |
|
core/algorithm
Package algorithm defines vald's core ann algorithm interface & constraint.
|
Package algorithm defines vald's core ann algorithm interface & constraint. |
|
core/algorithm/ngt
Package ngt provides implementation of Go API for https://github.com/yahoojapan/NGT
|
Package ngt provides implementation of Go API for https://github.com/yahoojapan/NGT |
|
db/kvs/redis
Package redis provides implementation of Go API for redis interface
|
Package redis provides implementation of Go API for redis interface |
|
db/rdb/mysql
Package mysql provides the functions for handling vald's data with MySQL.
|
Package mysql provides the functions for handling vald's data with MySQL. |
|
db/rdb/mysql/dbr
Package dbr provides the dbr functions for handling with databases.
|
Package dbr provides the dbr functions for handling with databases. |
|
db/storage/blob/s3/reader
Package reader provides the reader functions for handling with s3.
|
Package reader provides the reader functions for handling with s3. |
|
errgroup
Package errgroup provides server global wait group for graceful kill all goroutine
|
Package errgroup provides server global wait group for graceful kill all goroutine |
|
errors
Package errors provides error types and function
|
Package errors provides error types and function |
|
file
Package file provides file I/O functionality
|
Package file provides file I/O functionality |
|
info
Package info provides build-time info
|
Package info provides build-time info |
|
io
Package io provides io functions
|
Package io provides io functions |
|
k8s
Package k8s provides kubernetes control functionality
|
Package k8s provides kubernetes control functionality |
|
k8s/metrics/node
Package node provides kubernetes node information and preriodically update
|
Package node provides kubernetes node information and preriodically update |
|
k8s/metrics/pod
Package pod provides kubernetes pod information and preriodically update
|
Package pod provides kubernetes pod information and preriodically update |
|
k8s/node
Package node provides kubernetes node information and preriodically update
|
Package node provides kubernetes node information and preriodically update |
|
k8s/pod
Package pod provides kubernetes pod information and preriodically update
|
Package pod provides kubernetes pod information and preriodically update |
|
net
Package net provides net functionality for vald's network connection
|
Package net provides net functionality for vald's network connection |
|
net/control
Package control provides network socket option
|
Package control provides network socket option |
|
net/grpc
Package grpc provides generic functionality for grpc
|
Package grpc provides generic functionality for grpc |
|
net/grpc/codes
Package codes provides status codes of grpc
|
Package codes provides status codes of grpc |
|
net/grpc/credentials
Package credentials provides generic functionality for grpc credentials setting
|
Package credentials provides generic functionality for grpc credentials setting |
|
net/grpc/errdetails
Package errdetails provides error detail for grpc status
|
Package errdetails provides error detail for grpc status |
|
net/grpc/health
Package health provides generic functionality for grpc health checks.
|
Package health provides generic functionality for grpc health checks. |
|
net/grpc/interceptor/server/logging
Package logging provides gRPC interceptors for access logging
|
Package logging provides gRPC interceptors for access logging |
|
net/grpc/interceptor/server/recover
Package recover provides gRPC interceptors for recovery
|
Package recover provides gRPC interceptors for recovery |
|
net/grpc/keepalive
Package keepalive provides grpc keepalive configuration
|
Package keepalive provides grpc keepalive configuration |
|
net/grpc/pool
Package pool provides grpc connection pool client
|
Package pool provides grpc connection pool client |
|
net/grpc/proto
Package proto provides proto file logic
|
Package proto provides proto file logic |
|
net/grpc/reflection
Package reflection provides grpc reflection for providing grpc server proto specs
|
Package reflection provides grpc reflection for providing grpc server proto specs |
|
net/grpc/status
Package status provides statuses and errors returned by grpc handler functions
|
Package status provides statuses and errors returned by grpc handler functions |
|
net/grpc/types
Package types provides alias of protobuf library types
|
Package types provides alias of protobuf library types |
|
net/http/dump
Package dump provides http request/response dump logic
|
Package dump provides http request/response dump logic |
|
net/http/metrics
Package metrics provides pprof profiler handler
|
Package metrics provides pprof profiler handler |
|
net/http/middleware
Package middleware provides rest.Func Middleware
|
Package middleware provides rest.Func Middleware |
|
net/http/rest
Package rest provides REST API common logic & variable
|
Package rest provides REST API common logic & variable |
|
net/http/transport
Package transport provides http transport roundtrip option
|
Package transport provides http transport roundtrip option |
|
observability/exporter/jaeger
Package jaeger provides a jaeger exporter.
|
Package jaeger provides a jaeger exporter. |
|
observability/trace
Package trace provides trace functions.
|
Package trace provides trace functions. |
|
params
Package params provides implementation of Go API for argument parser
|
Package params provides implementation of Go API for argument parser |
|
rand
Package rand provides random number algorithms
|
Package rand provides random number algorithms |
|
runner
Package runner provides implementation of process runner
|
Package runner provides implementation of process runner |
|
safety
Package safety provides safety functionality like revcover
|
Package safety provides safety functionality like revcover |
|
servers
Package servers provides implementation of Go API for managing server flow
|
Package servers provides implementation of Go API for managing server flow |
|
servers/server
Package servers provides implementation of Go API for managing server flow
|
Package servers provides implementation of Go API for managing server flow |
|
servers/starter
Package starter provides server startup and shutdown flow control
|
Package starter provides server startup and shutdown flow control |
|
singleflight
Package singleflight represents zero time caching.
|
Package singleflight represents zero time caching. |
|
test
Package test provides functions for general testing use
|
Package test provides functions for general testing use |
|
test/data/request
Package request provides functions to generate proto request for testing
|
Package request provides functions to generate proto request for testing |
|
test/mock
Package mock provides mock implementation for testing.
|
Package mock provides mock implementation for testing. |
|
tls
Package tls provides implementation of Go API for tls certificate provider
|
Package tls provides implementation of Go API for tls certificate provider |
|
version
Package version provides version comparison functionality
|
Package version provides version comparison functionality |
|
worker
Package worker provides worker processes
|
Package worker provides worker processes |
|
pkg
|
|
|
agent/core/ngt/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
|
agent/core/ngt/handler/grpc
Package grpc provides grpc server logic
|
Package grpc provides grpc server logic |
|
agent/core/ngt/handler/rest
Package rest provides rest api logic
|
Package rest provides rest api logic |
|
agent/core/ngt/model
Package model defines object structure
|
Package model defines object structure |
|
agent/core/ngt/router
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func
|
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func |
|
agent/core/ngt/service
Package service manages the main logic of server.
|
Package service manages the main logic of server. |
|
agent/core/ngt/service/vqueue
Package vqueue manages the vector cache layer for reducing FFI overhead for fast Agent processing.
|
Package vqueue manages the vector cache layer for reducing FFI overhead for fast Agent processing. |
|
agent/internal/metadata
Package metadata provides agent metadata structs and info.
|
Package metadata provides agent metadata structs and info. |
|
agent/sidecar/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
|
agent/sidecar/handler/grpc
Package grpc provides grpc server logic
|
Package grpc provides grpc server logic |
|
agent/sidecar/handler/rest
Package rest provides rest api logic
|
Package rest provides rest api logic |
|
agent/sidecar/router
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func
|
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func |
|
agent/sidecar/service
Package service manages the main logic of server.
|
Package service manages the main logic of server. |
|
agent/sidecar/service/observer
Package observer provides storage observer
|
Package observer provides storage observer |
|
agent/sidecar/service/restorer
Package restorer provides restorer service
|
Package restorer provides restorer service |
|
agent/sidecar/service/storage
Package storage provides blob storage service
|
Package storage provides blob storage service |
|
discoverer/k8s/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
|
discoverer/k8s/handler/grpc
Package grpc provides grpc server logic
|
Package grpc provides grpc server logic |
|
discoverer/k8s/handler/rest
Package rest provides rest api logic
|
Package rest provides rest api logic |
|
discoverer/k8s/router
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func
|
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func |
|
discoverer/k8s/service
Package service manages the main logic of server.
|
Package service manages the main logic of server. |
|
gateway/filter/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
|
gateway/filter/handler/grpc
Package grpc provides grpc server logic
|
Package grpc provides grpc server logic |
|
gateway/filter/handler/rest
Package rest provides rest api logic
|
Package rest provides rest api logic |
|
gateway/filter/router
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func
|
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func |
|
gateway/lb/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
|
gateway/lb/handler/grpc
Package grpc provides grpc server logic
|
Package grpc provides grpc server logic |
|
gateway/lb/handler/rest
Package rest provides rest api logic
|
Package rest provides rest api logic |
|
gateway/lb/router
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func
|
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func |
|
gateway/lb/service
Package service manages the main logic of server.
|
Package service manages the main logic of server. |
|
gateway/lb/usecase
Package usecase represents gateways usecase layer
|
Package usecase represents gateways usecase layer |
|
manager/index/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
|
manager/index/handler/grpc
Package grpc provides grpc server logic
|
Package grpc provides grpc server logic |
|
manager/index/handler/rest
Package rest provides rest api logic
|
Package rest provides rest api logic |
|
manager/index/router
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func
|
Package router provides implementation of Go API for routing http Handler wrapped by rest.Func |
|
manager/index/service
Package service manages the main logic of server.
|
Package service manages the main logic of server. |
|
tools/cli/loadtest/config
Package setting stores all server application settings
|
Package setting stores all server application settings |
Click to show internal directories.
Click to hide internal directories.