basalt

package module
v0.0.0-...-a7e186f Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2020 License: Apache-2.0 Imports: 36 Imported by: 0

README

基于Raft的数据一致性分布式的 bitmap 服务

bitmap(位图)技术是数据库、大数据和互联网业务等场景下经常使用的一种技术。

  • 存在性判断
    • 爬虫url去重
    • 垃圾邮件过滤
    • 用户已阅读
    • 用户已赞
    • ...
  • 去重
  • 数据库
  • 大数据计算

架构图

支持三种协议读写: HTTP、redcis和rpcx。

写操作可以发给任意节点，但最终是由raft leader节点进行操作。 读操作可以选择任意的节点进行读取。

服务

进入cmd/server, 运行go run server.go启动一个bitmap服务。

它同时支持三种服务:

  • rpcx: 你可以使用rpcx服务获取高性能的网络调用, cmd/rpcx_client是一个rpcx客户端的demo
  • redis: 你可以使用redis客户端访问Bitmap服务(如果你的redis client支持自定义命令), 方便兼容redis调用代码， cmd/redis_client是redis demo
  • http: 通过http服务调用，调用简单,支持各种编程语言和脚本，cmd/http_client/curl.sh是通过curl调用服务

集群模式

支持raft集群模式: basalt集群

API接口

basalt位图服务支持三种接口模式：

  • HTTP API: 通过http api的方式进行访问
  • Redis模式: 扩展了redis命令，可以通过redis client进行访问
  • rpcx模式: 可以通过rpcx框架进行访问
Redis命令
  • ping: ping-pong消息
  • quit: 退出连接
  • bmadd name value: 在名为name的bitmap增加一个uint32值value
  • bmaddmany name value1 value2 value3...: 为名为name的bitmap增加一批值
  • bmdel name value: 在名为name的bitmap删除一个uint32值value
  • bmdrop name: 删除名为name的bitmap
  • bmclear name: 清空名为name的bitmap
  • bmcard name: 获取为name的bitmap包含的元素数
  • bmexists name value: 检查uint32值value是否存在于名为name的bitmap中，整数1代表存在，0代表不存在
  • bminter name1 name2 name3...: 求几个bitmap的交集，返回交集的uint32整数列表
  • bminterstore dst name1 name2 name3...: 求几个bitmap(name1、name2、name3...)的交集，并将结果保存到dst中
  • bmunion name1 name2 name3...: 求几个bitmap的并集，返回并集的uint32整数列表
  • bmunionstore dst name1 name2 name3...: 求几个bitmap(name1、name2、name3...)的并集，并将结果保存到dst中
  • bmxor name1 name2.: 求两个bitmap的xor集(双方不共有的集合,相当于并集减交集)，返回xor集的uint32整数列表
  • bmxorstore dst name1 name2: 求两个bitmap的xor集，并将结果保存到dst中
  • bmdiff name1 name2: 求name1中和name2没有交集的数据，返回结果的uint32整数列表
  • bmdiffstore dst name1 name2: 求name1中和name2没有交集的数据，并将结果保存到dst中
  • bmstats name: 返回name的bitmap的统计信息
rpcx 服务

查看 godoc以了解提供的rpcx服务

HTTP 服务

HTTP 服务提供和 redis、rpcx服务相同的功能，通过http调用就可以访问Bitmap服务。

所有的参数都是在路径中提供，路径格式为/action/param1/param2。 复数形式values、names包含多个元素，元素以逗号,分隔。

为了简化操作，所有的http服务都是通过GET方法提供的。

返回的HTTP StatusCode代表的含义如下：

  • 200 代表OK、存在
  • 400 代表参数不对，比如应该是uint32格式，结果却是无法解析的字符串
  • 404 代表不存在
  • 500 代表内部处理错误

HTTP服务路径列表如下：

  • /add/:name/:value
  • /addmany/:name/:values
  • /remove/:name/:value
  • /drop/:name
  • /clear/:name
  • /exists/:name/:value
  • /card/:name
  • /inter/:names
  • /interstore/:dst/:names
  • /union/:names
  • /unionstore/:dst/:names
  • /xor/:name1/:name2
  • /xorstore/:dst/:name1/:name2
  • /diff/:name1/:name2
  • /diffstore/:dst/:name1/:name2
  • /stats/:name

例子

以微博关注关系数据集做例子，我们使用Bitmap服务来存储某人是否关注了某人，以及两人是否互相关注。

例子参看 weibo_follow

Roadmap

  • Multiple-key Bitmap
  • rpcx services for Bitmap
  • HTTP services for Bitmap
  • Redis services for Bitmap
  • Persistence
  • Cluster mode

Credits

Documentation

Index

Constants

View Source 
const (
	BmOpAdd     OP = 1
	BmOpAddMany    = 2
	BmOpRemove     = 3
	BmOpDrop       = 4
	BmOpClear      = 5
)

Variables

View Source 
var (
	ErrPersistFileNotFound = errors.New("persist file not found")
)

Errors for bitmaps

Functions

func NewRaftNode 

func NewRaftNode(id int, peers []string, join bool, getSnapshot func() ([]byte, error), proposeC <-chan string,
	confChangeC <-chan raftpb.ConfChange) (<-chan *string, <-chan error, <-chan *snap.Snapshotter)

newRaftNode initiates a raft instance and returns a committed log entry channel and error channel. Proposals for log updates are sent over the provided the proposal channel. All log entries are replayed over the commit channel, followed by a nil message (to indicate the channel is current), then new log entries. To shutdown, close proposeC and read errorC.

Types

type AddNodeRequest 

type AddNodeRequest struct {
	ID   uint64
	Addr string
}

type Bitmap 

type Bitmap struct {
	// contains filtered or unexported fields
}

Bitmap is the goroutine-safe bitmap.

type BitmapDstAndPairRequest 

type BitmapDstAndPairRequest struct {
	Destination string
	Name1       string
	Name2       string
}

BitmapDstAndPairRequest contains destination and the name of two bitmaps.

type BitmapPairRequest 

type BitmapPairRequest struct {
	Name1 string
	Name2 string
}

BitmapPairRequest contains the name of two bitmaps.

type BitmapStoreRequest 

type BitmapStoreRequest struct {
	Destination string
	Names       []string
}

BitmapStoreRequest contains the name of destination and names of bitmaps.

type BitmapValueRequest 

type BitmapValueRequest struct {
	Name  string
	Value uint32
}

BitmapValueRequest contains the name of bitmap and value.

type BitmapValuesRequest 

type BitmapValuesRequest struct {
	Name   string
	Values []uint32
}

BitmapValuesRequest contains the name of bitmap and values.

type Bitmaps 

type Bitmaps struct {
	// contains filtered or unexported fields
}

Bitmaps contains all bitmaps of namespace.

func NewBitmaps 

func NewBitmaps() *Bitmaps

NewBitmaps creates a Bitmaps.

func (*Bitmaps) Add 

func (bs *Bitmaps) Add(name string, v uint32, callback bool)

Add adds a value.

func (*Bitmaps) AddMany 

func (bs *Bitmaps) AddMany(name string, v []uint32, callback bool)

AddMany adds multiple values.

func (*Bitmaps) Card 

func (bs *Bitmaps) Card(name string) uint64

Card returns the number of integers contained in the bitmap.

func (*Bitmaps) ClearBitmap 

func (bs *Bitmaps) ClearBitmap(name string, callback bool)

ClearBitmap clear a bitmap.

func (*Bitmaps) Diff 

func (bs *Bitmaps) Diff(name1, name2 string) []uint32

Diff computes the difference between two bitmaps and returns the result.

func (*Bitmaps) DiffStore 

func (bs *Bitmaps) DiffStore(destination, name1, name2 string) uint64

DiffStore computes the difference between two bitmaps and save the result to destination.

func (*Bitmaps) Exists 

func (bs *Bitmaps) Exists(name string, v uint32) bool

Exists checks whether a value exists.

func (*Bitmaps) Inter 

func (bs *Bitmaps) Inter(names ...string) []uint32

Inter computes the intersection (AND) of all provided bitmaps.

func (*Bitmaps) InterStore 

func (bs *Bitmaps) InterStore(destination string, names ...string) uint64

InterStore computes the intersection (AND) of all provided bitmaps and save to destination.

func (*Bitmaps) Read 

func (bs *Bitmaps) Read(r io.Reader) error

Read restores bitmaps from a io.Reader.

func (*Bitmaps) Remove 

func (bs *Bitmaps) Remove(name string, v uint32, callback bool)

Remove removes a value.

func (*Bitmaps) RemoveBitmap 

func (bs *Bitmaps) RemoveBitmap(name string, callback bool)

RemoveBitmap removes a bitmap.

func (*Bitmaps) Save 

func (bs *Bitmaps) Save(w io.Writer) error

Save saves bitmaps to the io.Writer.

func (*Bitmaps) Stats 

func (bs *Bitmaps) Stats(name string) Stats

Stats gets the stats of named bitmap.

func (*Bitmaps) Union 

func (bs *Bitmaps) Union(names ...string) []uint32

Union computes the union (OR) of all provided bitmaps.

func (*Bitmaps) UnionStore 

func (bs *Bitmaps) UnionStore(destination string, names ...string) uint64

UnionStore computes the union (OR) of all provided bitmaps and store to destination.

func (*Bitmaps) Xor 

func (bs *Bitmaps) Xor(name1, name2 string) []uint32

Xor computes the symmetric difference between two bitmaps and returns the result

func (*Bitmaps) XorStore 

func (bs *Bitmaps) XorStore(destination, name1, name2 string) uint64

XorStore computes the symmetric difference between two bitmaps and save the result to destination.

type ConfChange 

type ConfChange interface {
	AddNode(id uint64, addr []byte) error
	RemoveNode(id uint64) error
}

type ConfigRpcxOption 

type ConfigRpcxOption func(*Server, *server.Server)

ConfigRpcxOption defines the rpcx config function.

type HTTPService 

type HTTPService struct {
	// contains filtered or unexported fields
}

HTTPService is a http service.

func (*HTTPService) Serve 

func (s *HTTPService) Serve(ln net.Listener) error

Serve serves http service.

type OP 

type OP byte

OP bitmaps operations

type RaftServer 

type RaftServer struct {
	// contains filtered or unexported fields
}

func NewRaftServer 

func NewRaftServer(bmServer *Server, snapshotter *snap.Snapshotter, confChangeC chan raftpb.ConfChange, proposeC chan<- string, commitC <-chan *string, errorC <-chan error) *RaftServer

func (*RaftServer) AddNode 

func (s *RaftServer) AddNode(id uint64, addr []byte) error

func (*RaftServer) GetSnapshot 

func (s *RaftServer) GetSnapshot() ([]byte, error)

func (*RaftServer) Propose 

func (s *RaftServer) Propose(op OP, value string)

func (*RaftServer) RemoveNode 

func (s *RaftServer) RemoveNode(id uint64) error

type RedisService 

type RedisService struct {
	// contains filtered or unexported fields
}

RedisService is a redis service only supports bitmap commands.

type RpcxBitmapService 

type RpcxBitmapService struct {
	// contains filtered or unexported fields
}

RpcxBitmapService provides the rpcx service for Bitmaps.

func (*RpcxBitmapService) Add 

func (s *RpcxBitmapService) Add(ctx context.Context, req *BitmapValueRequest, reply *bool) error

Add adds a value in the bitmap with name.

func (*RpcxBitmapService) AddMany 

func (s *RpcxBitmapService) AddMany(ctx context.Context, req *BitmapValuesRequest, reply *bool) error

AddMany adds multiple values in the bitmap with name.

func (*RpcxBitmapService) AddNode 

func (s *RpcxBitmapService) AddNode(ctx context.Context, req *AddNodeRequest, reply *bool) error

AddNode adds a raft node.

func (*RpcxBitmapService) Card 

func (s *RpcxBitmapService) Card(ctx context.Context, name string, reply *uint64) error

Card gets number of integers in the bitmap.

func (*RpcxBitmapService) ClearBitmap 

func (s *RpcxBitmapService) ClearBitmap(ctx context.Context, name string, reply *bool) error

ClearBitmap clears the bitmap and set it to be empty.

func (*RpcxBitmapService) Diff 

func (s *RpcxBitmapService) Diff(ctx context.Context, names *BitmapPairRequest, reply *[]uint32) error

Diff gets the difference between two bitmaps.

func (*RpcxBitmapService) DiffStore 

func (s *RpcxBitmapService) DiffStore(ctx context.Context, names *BitmapDstAndPairRequest, reply *bool) error

DiffStore gets the difference between two bitmaps and stores into destination.

func (*RpcxBitmapService) Exists 

func (s *RpcxBitmapService) Exists(ctx context.Context, req *BitmapValueRequest, reply *bool) error

Exists checks whether the value exists.

func (*RpcxBitmapService) Inter 

func (s *RpcxBitmapService) Inter(ctx context.Context, names []string, reply *[]uint32) error

Inter gets the intersection of bitmaps.

func (*RpcxBitmapService) InterStore 

func (s *RpcxBitmapService) InterStore(ctx context.Context, req *BitmapStoreRequest, reply *bool) error

InterStore gets the intersection of bitmaps and stores into destination.

func (*RpcxBitmapService) Remove 

func (s *RpcxBitmapService) Remove(ctx context.Context, req *BitmapValueRequest, reply *bool) error

Remove removes a value in the bitmap with name.

func (*RpcxBitmapService) RemoveBitmap 

func (s *RpcxBitmapService) RemoveBitmap(ctx context.Context, name string, reply *bool) error

RemoveBitmap removes the bitmap.

func (*RpcxBitmapService) RemoveNode 

func (s *RpcxBitmapService) RemoveNode(ctx context.Context, req uint64, reply *bool) error

RemoveNode removes a raft node.

func (*RpcxBitmapService) Save 

func (s *RpcxBitmapService) Save(ctx context.Context, dummy string, reply *bool) error

Save persists bitmaps.

func (*RpcxBitmapService) Stats 

func (s *RpcxBitmapService) Stats(ctx context.Context, name string, reply *Stats) error

Stats get the stats of bitmap `name`.

func (*RpcxBitmapService) Union 

func (s *RpcxBitmapService) Union(ctx context.Context, names []string, reply *[]uint32) error

Union gets the union of bitmaps.

func (*RpcxBitmapService) UnionStore 

func (s *RpcxBitmapService) UnionStore(ctx context.Context, req *BitmapStoreRequest, reply *bool) error

UnionStore gets the union of bitmaps and stores into destination.

func (*RpcxBitmapService) Xor 

func (s *RpcxBitmapService) Xor(ctx context.Context, names *BitmapPairRequest, reply *[]uint32) error

Xor gets the symmetric difference between bitmaps.

func (*RpcxBitmapService) XorStore 

func (s *RpcxBitmapService) XorStore(ctx context.Context, names *BitmapDstAndPairRequest, reply *bool) error

XorStore gets the symmetric difference between bitmaps and stores into destination.

type Server 

type Server struct {
	// contains filtered or unexported fields
}

Server is the bitmap server that supports multiple services.

func NewServer 

func NewServer(addr string, bitmaps *Bitmaps, rpcxOptions []ConfigRpcxOption, persistFile string) *Server

NewServer returns a server.

func (*Server) Close 

func (s *Server) Close() error

Close closes this server.

func (*Server) Restore 

func (s *Server) Restore() error

Restore retores the data from file.

func (*Server) Save 

func (s *Server) Save() error

Save saves the data into file.

func (*Server) Serve 

func (s *Server) Serve() error

Serve serves basalt services.

func (*Server) SetConfChangeCallback 

func (s *Server) SetConfChangeCallback(confChangeCallback ConfChange)

SetConfChangeCallback must invoke before Serve.

type Stats 

type Stats struct {
	Cardinality uint64
	Containers  uint64

	ArrayContainers      uint64
	ArrayContainerBytes  uint64
	ArrayContainerValues uint64

	BitmapContainers      uint64
	BitmapContainerBytes  uint64
	BitmapContainerValues uint64

	RunContainers      uint64
	RunContainerBytes  uint64
	RunContainerValues uint64
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
raft_server
redis_client
rpcx_client
server
examples
weibo

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.