Dataset Service
Dataset Service is responsible for managing deployment of log collection components (i.e. log writer and observation service), organizing the collected data, and generating dataset from the collected data.
API Specification
Dataset Service API is defined as protobuf.
The service exposes the API both as gRPC API and REST API.
The REST API documentation can be found in dataset_service.swagger.json.
Getting Started
To get started, install following software:
Setup all dependencies
make setup
To build dataset service binary execute
make build
To perform test
make test
To setup local development environment. This command will create a k3d cluster and install dependency services (MLP and Kafka) in the cluster.
make dev-env
Once the local development environment is ready, you can run dataset-service locally using following command.
The command will use local.yaml configuration, which is preconfigured to target the k3d cluster for deployment of log writer and observation service.
make run
And send following requests to deploy observation-service or log writer
# Deploy observation service
curl --location --request POST 'localhost:8080/v1/projects/1/observation_services' \
--header 'Content-Type: application/json' \
--data-raw '{
"observation_service" : {
"project_id" : 1,
"name" : "my-observation",
"source" : {
"type": "OBSERVATION_SERVICE_SOURCE_TYPE_KAFKA",
"kafka" : {
"brokers": "kafka.mlp.svc.cluster.local",
"topic" : "sample"
}
}
}
}'
# Deploy router log writer
curl --location --request POST 'localhost:8080/v1/projects/1/log_writers' \
--header 'Content-Type: application/json' \
--data-raw '{
"log_writer" : {
"project_id" : 1,
"name" : "my-router-log",
"source" : {
"type": "LOG_WRITER_SOURCE_TYPE_ROUTER_LOG",
"router_log_source" : {
"router_id": 1,
"router_name" : "my-router",
"kafka" : {
"brokers": "kafka.mlp.svc.cluster.local",
"topic" : "my-router-log"
}
}
}
}
}'
# Deploy prediction log writer
curl --location --request POST 'localhost:8080/v1/projects/1/log_writers' \
--header 'Content-Type: application/json' \
--data-raw '{
"log_writer" : {
"project_id" : 1,
"name" : "my-prediction-log",
"source" : {
"type": "LOG_WRITER_SOURCE_TYPE_PREDICTION_LOG",
"prediction_log_source" : {
"model_id": 1,
"model_name" : "my-model",
"kafka" : {
"brokers": "kafka.mlp.svc.cluster.local",
"topic" : "my-prediction-log"
}
}
}
}
}'
You can check all available commands using
make