cninfra

package module
v1.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 17, 2017 License: Apache-2.0 Imports: 0 Imported by: 0

README

CN-Infra

Build Status Coverage Status Go Report Card GoDoc GitHub license

CN-Infra (cloud-native infrastructure) is a Golang platform for building cloud-native microservices. Although it was originally intended for development/implementation of custom management/control plane agents for cloud-native Virtual Network Functions (VNFs), it can be used to develop any microservice.

Architecture

Each management/control plane app built on top of the CN-Infra platform is basically a set of modules called "plugins" in CN-Infra lingo, where each plugin provides a very specific/focused functionality. Some plugins are provided by the CN-Infra platform itself, some are written by the app's implementors. In other words, the CN-Infra platform itself is implemented as a set of plugins that together provide the platform's functionality, such as logging, health checks, messaging (e.g. Kafka), a common front-end API and back-end connectivity to various KV data stores (Etcd, Cassandra, Redis, ...), and REST and gRPC APIs.

The architecture of the CN-Infra platform is shown in the following figure.

arch

The CN-Infra platform consists of a Core that provides plugin lifecycle management (initialization and graceful shutdown of plugins) and a set of platform plugins. Note that the figure shows not only CN-Infra plugins that are a part of the CN-Infra platform, but also app plugins that use the platform. CN-Infra platform plugins provide APIs that are consumed by app plugins. App plugins themselves may provide their own APIs consumed by external clients.

The platform is modular and extensible. Plugins supporting new functionality (e.g. another KV store or another message bus) can be easily added to the existing set of CN-Infra platform plugins. Moreover, CN-Infra based apps can be built in layers: a set of app plugins together with CN-Infra plugins can form a new platform providing APIs/services to higher layer apps. This approach was used in the VPP Agent - a management/control agent for VPP based software data planes.,

Extending the code base does not mean that all plugins end up in all apps - app writers can pick and choose only those platform plugins that are required by their app; for example, if an app does not need a KV store, the CN-Infra platform KV data store plugins would not be included in the app. All plugins used in an app are statically linked into the app.

CN-Infra Plugins

A CN-Infra plugin is typically implemented as a library providing the plugin's functionality/APIs wrapped in a plugin wrapper. A CN-Infra library can also be used standalone in 3rd party apps that do not use the CN-Infra platform. The plugin wrapper provides lifecycle management for the plugin component.

Platform plugins in the current CN-Infra release provide functionality in one of the following functional areas:

  • RPC - allows to expose application's API via REST or gRPC:

    • REST - HTTP requests and allows app plugins to define their own REST APIs.
  • Data Stores - provides a common data store API for app plugins (the Data Broker) and back-end clients for Etcd, Redis and Cassandra. The data store related plugin are as follows:

    • Etcd - implements keyval skeleton provides access to etcd
    • Redis - implements keyval skeleton provides access to redis
    • Casssandra - implements sql skeleton provides access to cassandra
  • Messaging - provides a common API and connectivity to message buses:

    • Kafka - provides access to a Kafka broker (Sarama)
  • Logging:

    • Logrus wrapper - implements logging skeleton using the Logrus library. An app writer can create multiple loggers - for example, each app plugin can have its own logger. Log level for each logger can be controlled individually at run time through the Log Manager REST API.
    • Log Manager - allows the operator to set log level for each logger using a REST API.
  • Health - Self health check mechanism between plugins plus RPCs:

    • StatusCheck - allows to monitor the status of plugins and exposes it via HTTP
    • Probes (callable remotely from K8s)
  • Miscellaneous - value-add plugins supporting the operation of a CN-Infra based application:

    • Config - helpers for loading plugin configuration.
    • Datasync - provides data resynchronization after HA events (restart or connectivity restoration after an outage) for data stores, gRPC and REST.
    • Flavors - predefined reusable collection of plugins.
    • IDX Map - reusable thread-safe map with advanced features:
      • multiple subscribers for watching changes in the map
      • secondary indexes
    • ServiceLabel - provides setting and retrieval of a unique identifier for a CN-Infra based app. A cloud app typically needs a unique identifier so that it can differentiated from other instances of the same app or from other apps (e.g. to have its own space in a kv data store).

Quickstart

The following code shows the initialization/start of a simple agent application built on the CN-Infra platform. The code for this example can be found here.

func main() {
	flavor := rpc.FlavorRPC{}
	agent := core.NewAgent(logroot.Logger(), 15*time.Second, flavor.Plugins()...)

	err := core.EventLoopWithInterrupt(agent, nil)
	if err != nil {
		os.Exit(1)
	}
}

You can run this example code by using pre-build Docker images:

For quick start with the VPP Agent, you can use pre-build Docker images with the Agent and VPP on Dockerhub.

  1. Run ETCD and Kafka on your host (e.g. in Docker using this procedure).

  2. Run cn-infra example simple-agent.

docker pull ligato/dev-cn-infra
docker run -it --name dev-cn-infra --rm ligato/dev-cn-infra

Documentation

GoDoc can be browsed online.

Contributing

If you are interested in contributing, please see the contribution guidelines.

Documentation

Overview

Package cninfra is the parent package for all packages that are parts of the CN-Infra platform - a Golang platform for building cloud-native microservices.

The CN-Infra platform is a modular platform comprising a core and a set of plugins. The core provides lifecycle management for plugins, the plugins provide the functionality of the platform. A plugin can consist of one or more Golang packages. Out of the box, the CN-Infra platform provides reusable plugins for logging, health checks, messaging (e.g. Kafka), a common front-end API and back-end connectivity to various data stores (Etcd, Cassandra, Redis, ...), and REST and gRPC APIs.

The CN-Infra platform can be extended by adding a new platform plugin, for example a new data store client or a message bus adapter. Also, each application built on top of the platform is basically just a set of application-specific plugins.

Directories

Path Synopsis
Package config contains helper functions for parsing of configuration files.
Package config contains helper functions for parsing of configuration files.
Package core manages the lifecycle of all plugins (start, graceful shutdown) and defines the core lifecycle SPI.
Package core manages the lifecycle of all plugins (start, graceful shutdown) and defines the core lifecycle SPI.
Package datasync defines the datasync API, which abstracts the data transport between app plugins and backend data sources.
Package datasync defines the datasync API, which abstracts the data transport between app plugins and backend data sources.
grpcsync
Package grpcsync implements (in ALPHA VERSION) the gRPC client and server that satisfies the datasync API (see the definition of the service in ../syncbase/msg package).
Package grpcsync implements (in ALPHA VERSION) the gRPC client and server that satisfies the datasync API (see the definition of the service in ../syncbase/msg package).
kvdbsync
Package kvdbsync defines a key-value data store client API for unified access among key-value datastore servers.
Package kvdbsync defines a key-value data store client API for unified access among key-value datastore servers.
kvdbsync/local
Package local implements DB Transactions for the local "in memory" transport.
Package local implements DB Transactions for the local "in memory" transport.
msgsync
Package msgsync propagates protobuf messages to a particular topic.
Package msgsync propagates protobuf messages to a particular topic.
restsync
Package restsync implements (in ALPHA VERSION) the datasync API for the HTTP/REST transport.
Package restsync implements (in ALPHA VERSION) the datasync API for the HTTP/REST transport.
resync
Package resync implements the mechanism to notify previously registered plugins that the resync procedure needs to start.
Package resync implements the mechanism to notify previously registered plugins that the resync procedure needs to start.
syncbase
Package syncbase defines common structures used in multiple datasync transports.
Package syncbase defines common structures used in multiple datasync transports.
syncbase/msg
Package msg is a generated protocol buffer package.
Package msg is a generated protocol buffer package.
db
Package db is the parent package for the Data Broker client implementations for various key-value and SQL data stores.
Package db is the parent package for the Data Broker client implementations for various key-value and SQL data stores.
keyval
Package keyval provides an abstraction of a key-value data store and defines the keyval data broker API.
Package keyval provides an abstraction of a key-value data store and defines the keyval data broker API.
keyval/etcdv3
Package etcdv3 implements the key-value Data Broker client API for the etcdv3 key-value data store.
Package etcdv3 implements the key-value Data Broker client API for the etcdv3 key-value data store.
keyval/etcdv3/mocks
Package mocks implements an embedded etcdv3 mock used in unit & integration tests.
Package mocks implements an embedded etcdv3 mock used in unit & integration tests.
keyval/kvproto
Package kvproto provides a wrapper that simplifies the storing and retrieving of proto-modelled data into/from a key-value data store.
Package kvproto provides a wrapper that simplifies the storing and retrieving of proto-modelled data into/from a key-value data store.
keyval/plugin
Package plugin contains a keyval plugin skeleton used in various key-value data store clients (e.g.
Package plugin contains a keyval plugin skeleton used in various key-value data store clients (e.g.
keyval/redis
Package redis is the implementation of the key-value Data Broker client API for the Redis key-value data store.
Package redis is the implementation of the key-value Data Broker client API for the Redis key-value data store.
sql
Package sql provides an abstraction of a data store that supports an SQL-like query language and defines the SQL data broker API.
Package sql provides an abstraction of a data store that supports an SQL-like query language and defines the SQL data broker API.
sql/cassandra
Package cassandra is the implementation of the SQL Data Broker client API for the Cassandra data store.
Package cassandra is the implementation of the SQL Data Broker client API for the Cassandra data store.
Package examples contains examples that demonstrate the usage of various cn-infra components.
Package examples contains examples that demonstrate the usage of various cn-infra components.
cassandra-lib
Example application "cassandra-lib" demonstrates the usage of the cassandra library in a plain go program.
Example application "cassandra-lib" demonstrates the usage of the cassandra library in a plain go program.
configs-plugin
Example program "configs-plugin" demonstrates the usage of PluginConfig API.
Example program "configs-plugin" demonstrates the usage of PluginConfig API.
datasync-plugin
Example application "datasync-plugin" demonstrates the usage of the datasync API.
Example application "datasync-plugin" demonstrates the usage of the datasync API.
etcdv3-lib
Package etcdv3lib contains a collection of examples demonstrating the usage of the etcdv3 library.
Package etcdv3lib contains a collection of examples demonstrating the usage of the etcdv3 library.
etcdv3-lib/editor
Example program "editor" demonstrates CREATE/UPDATE/DELETE operations with etcdv3 library.
Example program "editor" demonstrates CREATE/UPDATE/DELETE operations with etcdv3 library.
etcdv3-lib/model/phonebook
Package phonebook is a generated protocol buffer package.
Package phonebook is a generated protocol buffer package.
etcdv3-lib/view
Example program "view" shows how to read data from etcdv3 datastore.
Example program "view" shows how to read data from etcdv3 datastore.
etcdv3-lib/watcher
Example program *watcher* shows how to watch for changes in etcdv3 datastore.
Example program *watcher* shows how to watch for changes in etcdv3 datastore.
flags-lib
Example program "flags-lib" demonstrates how to register CLI flags and how to display their runtime values.
Example program "flags-lib" demonstrates how to register CLI flags and how to display their runtime values.
grpc-plugin
Package grpcplugin contains grpc server & grpc client example.
Package grpcplugin contains grpc server & grpc client example.
grpc-plugin/grpc-server
Example demonstrates the use of GRPC plugin API.
Example demonstrates the use of GRPC plugin API.
kafka-lib
Package kafkalib contains multiple examples with the kafka library.
Package kafkalib contains multiple examples with the kafka library.
kafka-lib/asyncproducer
Asyncproducer is a simple command line tool for sending messages to Kafka using an asynchronous Kafka producer.
Asyncproducer is a simple command line tool for sending messages to Kafka using an asynchronous Kafka producer.
kafka-lib/consumer
Consumer is a simple command line tool for consuming a Kafka topic and printing the received messages to the stdout.
Consumer is a simple command line tool for consuming a Kafka topic and printing the received messages to the stdout.
kafka-lib/mux
Mux is a simple command line tool demonstrating the usage of Kafka multiplexer API.
Mux is a simple command line tool demonstrating the usage of Kafka multiplexer API.
kafka-lib/syncproducer
Syncproducer is simple command line tool for sending messages to Kafka using a synchronous Kafka producer.
Syncproducer is simple command line tool for sending messages to Kafka using a synchronous Kafka producer.
kafka-lib/utils
Package utils implements various tools used in examples for Kafka library.
Package utils implements various tools used in examples for Kafka library.
kafka-plugin
Package kafkaplugin contains multiple examples with the kafka plugin.
Package kafkaplugin contains multiple examples with the kafka plugin.
kafka-plugin/hash-partitioner
Example "hash-partitioner" demonstrates the use of Kafka plugin API with automatic (hash-based) partitioning.
Example "hash-partitioner" demonstrates the use of Kafka plugin API with automatic (hash-based) partitioning.
kafka-plugin/manual-partitioner
Example "manual-partitioner" demonstrates the use of Kafka plugin API with a focus on manual partitioning.
Example "manual-partitioner" demonstrates the use of Kafka plugin API with a focus on manual partitioning.
kafka-plugin/post-init-consumer
Example "later-stage" demonstrates the use partition watcher initialized and used after multiplexer Init()
Example "later-stage" demonstrates the use partition watcher initialized and used after multiplexer Init()
logs-lib
Package logslib contains multiple examples showing how to use the logger library with a focus on switching between the log levels.
Package logslib contains multiple examples showing how to use the logger library with a focus on switching between the log levels.
logs-lib/basic
Example "basic" presents the most commonly used APIs of the logger library.
Example "basic" presents the most commonly used APIs of the logger library.
logs-lib/custom
Example "custom" shows how to customize the logger.
Example "custom" shows how to customize the logger.
logs-lib/http
Example "http" demonstrates how to change the log level remotely via REST.
Example "http" demonstrates how to change the log level remotely via REST.
logs-plugin
Example "logs-plugin" shows how to use the logger library from a simple plugin.
Example "logs-plugin" shows how to use the logger library from a simple plugin.
model
Package etcdexample explains how to generate Golang structures from protobuf-formatted data.
Package etcdexample explains how to generate Golang structures from protobuf-formatted data.
redis-lib
Package redislib contains a collection of examples demonstrating the usage of the redis library.
Package redislib contains a collection of examples demonstrating the usage of the redis library.
redis-lib/airport
A bit more advanced example demonstrating the redis library API.
A bit more advanced example demonstrating the redis library API.
redis-lib/airport/model
Package flight defines protobuf-formatted data as used by the airport example.
Package flight defines protobuf-formatted data as used by the airport example.
redis-lib/simple
A simple example demonstrating the redis library API.
A simple example demonstrating the redis library API.
redis-plugin
Example demonstrates creating a Redis plugin
Example demonstrates creating a Redis plugin
simple-agent
Example simple-agent shows how easily a set of CN-infra based plugins can be turned into an application.
Example simple-agent shows how easily a set of CN-infra based plugins can be turned into an application.
Package flavors contains predefined cn-infra flavors.
Package flavors contains predefined cn-infra flavors.
connectors
Package connectors defines "Connectors" flavor, which is a combination of all plugins that allow connectivity to external database/messaging.
Package connectors defines "Connectors" flavor, which is a combination of all plugins that allow connectivity to external database/messaging.
local
Package local defines "Local" flavor, which combines logging with StatusCheck.
Package local defines "Local" flavor, which combines logging with StatusCheck.
rpc
Package rpc defines "RPC" flavor, which combines all plugins that provide RPC-like access.
Package rpc defines "RPC" flavor, which combines all plugins that provide RPC-like access.
Package health contains plugins for monitoring of the health status of the agent.
Package health contains plugins for monitoring of the health status of the agent.
probe
Package probe implements Liveness/Readiness/Prometheus health/metrics HTTP handlers.
Package probe implements Liveness/Readiness/Prometheus health/metrics HTTP handlers.
statuscheck
Package statuscheck defines the status report API for other CN-Infra plugins and implements the health status aggregator/exporter.
Package statuscheck defines the status report API for other CN-Infra plugins and implements the health status aggregator/exporter.
statuscheck/model/status
Package status is a generated protocol buffer package.
Package status is a generated protocol buffer package.
statuscheck/pluginstatusmap
Package pluginstatusmap implements specialization of idxmap used to store plugin status by plugin name.
Package pluginstatusmap implements specialization of idxmap used to store plugin status by plugin name.
Package idxmap defines a mapping structure which supports mapping change notifications and retrieval of items by fields in the value structure.
Package idxmap defines a mapping structure which supports mapping change notifications and retrieval of items by fields in the value structure.
mem
Package mem provides in-memory implementation of the mapping with multiple indexes.
Package mem provides in-memory implementation of the mapping with multiple indexes.
Package logging defines the logging API.
Package logging defines the logging API.
logmanager
Package logmanager implements the log manager that allows users to set log levels at run-time via a REST API.
Package logmanager implements the log manager that allows users to set log levels at run-time via a REST API.
logroot
Package logroot contains the default global logger.
Package logroot contains the default global logger.
logrus
Package logrus implements the logging API based on the Logrus logger.
Package logrus implements the logging API based on the Logrus logger.
Package messaging is the parent package for implementations of various messaging clients, e.g.
Package messaging is the parent package for implementations of various messaging clients, e.g.
kafka
Package kafka implements a client for the Kafka broker.
Package kafka implements a client for the Kafka broker.
kafka/client
Package client implements the synchronous and asynchronous kafka Producers and the kafka Consumer.
Package client implements the synchronous and asynchronous kafka Producers and the kafka Consumer.
kafka/mux
Package mux implements the session multiplexer that allows multiple plugins to share a single connection to a Kafka broker.
Package mux implements the session multiplexer that allows multiple plugins to share a single connection to a Kafka broker.
rpc
Package rpc contains plugins providing RPC-like access.
Package rpc contains plugins providing RPC-like access.
grpc
Package grpc implements the GRPC netListener through which plugins can expose their services/API to the outside world.
Package grpc implements the GRPC netListener through which plugins can expose their services/API to the outside world.
rest
Package rest implements the HTTP server through which plugins can expose their REST APIs to the outside world.
Package rest implements the HTTP server through which plugins can expose their REST APIs to the outside world.
rest/mock
Package mock implements a mock HTTP server.
Package mock implements a mock HTTP server.
Package servicelabel provides support for creating/retrieving an identifier (a service label) for a CN-Infra based app.
Package servicelabel provides support for creating/retrieving an identifier (a service label) for a CN-Infra based app.
Package tests contains CN-infra unit tests.
Package tests contains CN-infra unit tests.
go
Package go contains unit tests written in Golang.
Package go contains unit tests written in Golang.
go/itest
Package itest contains unit tests targeting dependency injection.
Package itest contains unit tests targeting dependency injection.
Package utils contains multiple utilities and helper functions.
Package utils contains multiple utilities and helper functions.
addrs
Package addrs contains helpers for ip and mac address conversion.
Package addrs contains helpers for ip and mac address conversion.
runtimeutils
Package runtimeutils contains utilities for inspecting Go's runtime system.
Package runtimeutils contains utilities for inspecting Go's runtime system.
safeclose
Package safeclose provides utilities for safe closing of I/O resources.
Package safeclose provides utilities for safe closing of I/O resources.
structs
Package structs provides reflection utilities to inspect structures.
Package structs provides reflection utilities to inspect structures.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL