twins

README

Twins

Service twins is used for CRUD and update of digital twins. Twin is a semantic representation of a real world data system consisting of data producers and consumers. It stores the sequence of attribute based definitions of a system and refers to a time series of definition based states that store the system historical data.

Configuration

The service is configured using the environment variables presented in the following table. Note that any unset variables will be replaced with their default values.

Variable	Description	Default
MF_TWINS_LOG_LEVEL	Log level for twin service (debug, info, warn, error)	error
MF_TWINS_HTTP_PORT	Twins service HTTP port	9021
MF_TWINS_SERVER_CERT	Path to server certificate in PEM format
MF_TWINS_SERVER_KEY	Path to server key in PEM format
MF_JAEGER_URL	Jaeger server URL
MF_TWINS_DB	Database name	mainflux
MF_TWINS_DB_HOST	Database host address	localhost
MF_TWINS_DB_PORT	Database host port	27017
MF_TWINS_SINGLE_USER_EMAIL	User email for single user mode (no gRPC communication with users)
MF_TWINS_SINGLE_USER_TOKEN	User token for single user mode that should be passed in auth header
MF_TWINS_CLIENT_TLS	Flag that indicates if TLS should be turned on	false
MF_TWINS_CA_CERTS	Path to trusted CAs in PEM format
MF_TWINS_CHANNEL_ID	NATS notifications channel ID
MF_NATS_URL	Mainflux NATS broker URL	nats://localhost:4222
MF_AUTHN_GRPC_URL	AuthN service gRPC URL	localhost:8181
MF_AUTHN_GRPC_TIMEOUT	AuthN service gRPC request timeout in seconds	1s
MF_TWINS_CACHE_URL	Cache database URL	localhost:6379
MF_TWINS_CACHE_PASS	Cache database password
MF_TWINS_CACHE_DB	Cache instance name	0

Deployment

The service itself is distributed as Docker container. The following snippet provides a compose file template that can be used to deploy the service container locally:

version: "3"
services:
  twins:
    image: mainflux/twins:[version]
    container_name: [instance name]
    ports:
      - [host machine port]:[configured HTTP port]
    environment:
      MF_TWINS_LOG_LEVEL: [Twins log level]
      MF_TWINS_HTTP_PORT: [Service HTTP port]
      MF_TWINS_SERVER_CERT: [String path to server cert in pem format]
      MF_TWINS_SERVER_KEY: [String path to server key in pem format]
      MF_JAEGER_URL: [Jaeger server URL]
      MF_TWINS_DB: [Database name]
      MF_TWINS_DB_HOST: [Database host address]
      MF_TWINS_DB_PORT: [Database host port]
      MF_TWINS_SINGLE_USER_EMAIL: [User email for single user mode]
      MF_TWINS_SINGLE_USER_TOKEN: [User token for single user mode]
      MF_TWINS_CLIENT_TLS: [Flag that indicates if TLS should be turned on]
      MF_TWINS_CA_CERTS: [Path to trusted CAs in PEM format]
      MF_TWINS_CHANNEL_ID: [NATS notifications channel ID]
      MF_NATS_URL: [Mainflux NATS broker URL]
      MF_AUTHN_GRPC_URL: [AuthN service gRPC URL]
      MF_AUTHN_GRPC_TIMEOUT: [AuthN service gRPC request timeout in seconds]
      MF_TWINS_ES_URL: [Event store URL]
      MF_TWINS_ES_PASS: [Event store password]
      MF_TWINS_ES_DB: [Event store instance name]

To start the service outside of the container, execute the following shell script:

# download the latest version of the service
go get gitee.com/shtemmi/iotflux

cd $GOPATH/src/gitee.com/shtemmi/iotflux

# compile the twins
make twins

# copy binary to bin
make install

# set the environment variables and run the service
MF_TWINS_LOG_LEVEL: [Twins log level] \
MF_TWINS_HTTP_PORT: [Service HTTP port] \
MF_TWINS_SERVER_CERT: [String path to server cert in pem format] \
MF_TWINS_SERVER_KEY: [String path to server key in pem format] \
MF_JAEGER_URL: [Jaeger server URL] MF_TWINS_DB: [Database name] \
MF_TWINS_DB_HOST: [Database host address] \
MF_TWINS_DB_PORT: [Database host port] \
MF_TWINS_SINGLE_USER_EMAIL: [User email for single user mode] \
MF_TWINS_SINGLE_USER_TOKEN: [User token for single user mode] \
MF_TWINS_CLIENT_TLS: [Flag that indicates if TLS should be turned on] \
MF_TWINS_CA_CERTS: [Path to trusted CAs in PEM format] \
MF_TWINS_CHANNEL_ID: [NATS notifications channel ID] \
MF_NATS_URL: [Mainflux NATS broker URL] \
MF_AUTHN_GRPC_URL: [AuthN service gRPC URL] \
MF_AUTHN_GRPC_TIMEOUT: [AuthN service gRPC request timeout in seconds] \
$GOBIN/mainflux-twins

Usage

Starting twins service

The twins service publishes notifications on a NATS subject of the format channels.<MF_TWINS_CHANNEL_ID>.messages.<twinID>.<crudOp>, where crudOp stands for the crud operation done on twin - create, update, delete or retrieve - or state - save state. In order to use twin service notifications, one must inform it - via environment variables - about the Mainflux channel used for notification publishing. You must use an already existing channel, since you cannot know in advance or set the channel ID (Mainflux does it automatically).

To set the environment variable, please go to .env file and set the following variable:

MF_TWINS_CHANNEL_ID=

with the corresponding values of the desired channel. If you are running mainflux natively, than do the same thing in the corresponding console environment.

For more information about service capabilities and its usage, please check out the API documentation.

Documentation

Overview

Package twins contains the domain concept definitions needed to support Mainflux twins service functionality. Twin is a digital representation of a real world data system consisting of data producers and consumers. It stores the sequence of attribute based definitions of a data system and refers to a time series of definition based states that store the system historical data.

Index

• Constants
• Variables
• type Attribute
• type Definition
• type Metadata
• type Page
• type PageMetadata
• type Service
  • func New(publisher messaging.Publisher, auth mainflux.AuthNServiceClient, ...) Service
• type State
• type StateRepository
• type StatesPage
• type Twin
• type TwinCache
• type TwinRepository

Constants

const (
	SubtopicWildcard = ">"
)

Variables

var (
	// ErrMalformedEntity indicates malformed entity specification (e.g.
	// invalid username or password).
	ErrMalformedEntity = errors.New("malformed entity specification")

	// ErrUnauthorizedAccess indicates missing or invalid credentials provided
	// when accessing a protected resource.
	ErrUnauthorizedAccess = errors.New("missing or invalid credentials provided")

	// ErrNotFound indicates a non-existent entity request.
	ErrNotFound = errors.New("non-existent entity")

	// ErrConflict indicates that entity already exists.
	ErrConflict = errors.New("entity already exists")
)

Types

type Attribute

type Attribute struct {
	Name         string `json:"name"`
	Channel      string `json:"channel"`
	Subtopic     string `json:"subtopic"`
	PersistState bool   `json:"persist_state"`
}

Attribute stores individual attribute data

type Definition

type Definition struct {
	ID         int         `json:"id"`
	Created    time.Time   `json:"created"`
	Attributes []Attribute `json:"attributes"`
	Delta      int64       `json:"delta"`
}

Definition stores entity's attributes

type Metadata

type Metadata map[string]interface{}

Metadata stores arbitrary twin data

type Page

type Page struct {
	PageMetadata
	Twins []Twin
}

Page contains page related metadata as well as a list of twins that belong to this page.

type PageMetadata

type PageMetadata struct {
	Total  uint64
	Offset uint64
	Limit  uint64
}

PageMetadata contains page metadata that helps navigation.

type Service

type Service interface {
	// AddTwin adds new twin related to user identified by the provided key.
	AddTwin(ctx context.Context, token string, twin Twin, def Definition) (tw Twin, err error)

	// UpdateTwin updates twin identified by the provided Twin that
	// belongs to the user identified by the provided key.
	UpdateTwin(ctx context.Context, token string, twin Twin, def Definition) (err error)

	// ViewTwin retrieves data about twin with the provided
	// ID belonging to the user identified by the provided key.
	ViewTwin(ctx context.Context, token, twinID string) (tw Twin, err error)

	// RemoveTwin removes the twin identified with the provided ID, that
	// belongs to the user identified by the provided key.
	RemoveTwin(ctx context.Context, token, twinID string) (err error)

	// ListTwins retrieves data about subset of twins that belongs to the
	// user identified by the provided key.
	ListTwins(ctx context.Context, token string, offset uint64, limit uint64, name string, metadata Metadata) (Page, error)

	// ListStates retrieves data about subset of states that belongs to the
	// twin identified by the id.
	ListStates(ctx context.Context, token string, offset uint64, limit uint64, twinID string) (StatesPage, error)

	// SaveStates persists states into database
	SaveStates(msg *messaging.Message) error
}

Service specifies an API that must be fullfiled by the domain service implementation, and all of its decorators (e.g. logging & metrics).

func New

func New(publisher messaging.Publisher, auth mainflux.AuthNServiceClient, twins TwinRepository, tcache TwinCache, sr StateRepository, idp mainflux.UUIDProvider, chann string, logger logger.Logger) Service

New instantiates the twins service implementation.

type State

type State struct {
	TwinID     string
	ID         int64
	Definition int
	Created    time.Time
	Payload    map[string]interface{}
}

State stores actual snapshot of entity's values

type StateRepository

type StateRepository interface {
	// Save persists the state
	Save(ctx context.Context, state State) error

	// Update updates the state
	Update(ctx context.Context, state State) error

	// Count returns the number of states related to state
	Count(ctx context.Context, twin Twin) (int64, error)

	// RetrieveAll retrieves the subset of states related to twin specified by id
	RetrieveAll(ctx context.Context, offset uint64, limit uint64, twinID string) (StatesPage, error)

	// RetrieveLast retrieves the last saved state
	RetrieveLast(ctx context.Context, twinID string) (State, error)
}

StateRepository specifies a state persistence API.

type StatesPage

type StatesPage struct {
	PageMetadata
	States []State
}

StatesPage contains page related metadata as well as a list of twins that belong to this page.

type Twin

type Twin struct {
	Owner       string
	ID          string
	Name        string
	Created     time.Time
	Updated     time.Time
	Revision    int
	Definitions []Definition
	Metadata    Metadata
}

Twin is a Mainflux data system representation. Each twin is owned by a single user, and is assigned with the unique identifier.

type TwinCache

type TwinCache interface {
	// Save stores twin ID as element of channel-subtopic keyed set and vice versa.
	Save(ctx context.Context, twin Twin) error

	// SaveIDs stores twin IDs as elements of channel-subtopic keyed set and vice versa.
	SaveIDs(ctx context.Context, channel, subtopic string, twinIDs []string) error

	// Update updates update twin id and channel-subtopic attributes mapping
	Update(ctx context.Context, twin Twin) error

	// ID returns twin IDs for given attribute.
	IDs(ctx context.Context, channel, subtopic string) ([]string, error)

	// Removes twin from cache based on twin id.
	Remove(ctx context.Context, twinID string) error
}

TwinCache contains twin caching interface.

type TwinRepository

type TwinRepository interface {
	// Save persists the twin
	Save(ctx context.Context, twin Twin) (string, error)

	// Update performs an update to the existing twin. A non-nil error is
	// returned to indicate operation failure.
	Update(ctx context.Context, twin Twin) error

	// RetrieveByID retrieves the twin having the provided identifier.
	RetrieveByID(ctx context.Context, twinID string) (Twin, error)

	// RetrieveByAttribute retrieves twin ids whose definition contains
	// the attribute with given channel and subtopic
	RetrieveByAttribute(ctx context.Context, channel, subtopic string) ([]string, error)

	// RetrieveAll retrieves the subset of twins owned by the specified user.
	RetrieveAll(ctx context.Context, owner string, offset, limit uint64, name string, metadata Metadata) (Page, error)

	// Remove removes the twin having the provided identifier.
	Remove(ctx context.Context, twinID string) error
}

TwinRepository specifies a twin persistence API.