mnist

README

Call chainer MNIST sample code from SensorBee.

Setup

1. link python

sensorbee/py plugin is required to link SensorBee and Python Use pkg-config, detail: Set up to link python

sensorbee/py supports only Python 2.x .

2. install python library

• chainer
  • support 1.5.x or later

3. install build_sensorbee

cd $GOPATH/src/gopkg.in/sensorbee/sensorbee.v0/cmd/build_sensorbee
go install

4. make working directory

• uds: For saving/loading SensorBee's shared state, which includes chainer's model.
• data: For MNIST train/test files, and SensorBee puts log file.

cd $GOPATH/src/gopkg.in/sensorbee/pymlstate.v0/example/mnist/sensorbee
mkdir uds
mkdir data

Put MNIST binary data on "data".

• train-images-idx3-ubyte
• train-labels-idx1-ubyte
• t10k-images-idx3-ubyte
• t10k-labels-idx1-ubyte

5. build sensorbee

build_sensorbee --download-plugins=false

build.yaml

--- # SensorBee plug-in list
plugins:
- gopkg.in/sensorbee/py.v0/pystate/plugin
- gopkg.in/sensorbee/pymlstate.v0/plugin
- gopkg.in/sensorbee/pymlstate.v0/example/mnist/plugin

Training

./sensorbee runfile -c conf.yaml -s ml_mnist -t mnist train.bql

"uds/mnist-ml_mnist-default.state" will be created. loss values are logged in "data/trained.jsonl" .

Test

./sensorbee runfile -c conf.yaml -s ml_mnist -t mnist test.bql

accuracy values are logged in "data/acc_log.jsonl" .

[Experiment] Online test

./sensorbee run -c conf.yaml

curl -v -H 'Accept: application/json' -H 'Content-type: application/json' -X POST -d '{"queries":"EVAL pymlstate_predict(\"ml_mnist\", [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.7490196078431373,1,1,0.18823529411764706,0,0,0,0.4980392156862745,1,0.25098039215686274,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.5019607843137255,1,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0.25098039215686274,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0.7490196078431373,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.7490196078431373,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0.996078431372549,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.7411764705882353,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.7490196078431373,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.3333333333333333,1,1,0.25098039215686274,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0.5019607843137255,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.4980392156862745,0.996078431372549,1,1,1,1,1,0.5019607843137255,1,1,1,1,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0.4745098039215686,0.5019607843137255,0.3215686274509804,0,0,0,0,0,0,0,0,0.48627450980392156,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]);"}' http://localhost:15601/api/v1/topologies/mnist/queries

Will be returned {"result":2}

Documentation

Index

• func CreateBucketStoreUDSF(ctx *core.Context, decl udf.UDSFDeclarer, stream, bucketName string) (udf.UDSF, error)
• func CreateEpochUDSF(ctx *core.Context, decl udf.UDSFDeclarer, stream, arrayKey string, ...) (udf.UDSF, error)
• func NewBucketState(ctx *core.Context, params data.Map) (core.SharedState, error)
• type DataSourceCreator
  • func (s *DataSourceCreator) CreateSource(ctx *core.Context, ioParams *bql.IOParams, params data.Map) (core.Source, error)

Functions

func CreateBucketStoreUDSF

func CreateBucketStoreUDSF(ctx *core.Context, decl udf.UDSFDeclarer, stream,
	bucketName string) (udf.UDSF, error)

CreateBucketStoreUDSF returns UDSF to store tuple data to the target bucket.

stream: target stream name bucketName: target bucket (shared state) name

func CreateEpochUDSF

func CreateEpochUDSF(ctx *core.Context, decl udf.UDSFDeclarer, stream,
	arrayKey string, batchSize, epoch int, random bool) (udf.UDSF, error)

CreateEpochUDSF returns bucket to batch stream function.

stream: target stream name arrayKey: key name of splitting array batchSize: splitting size epoch: loop count number random randomize before streaming

func NewBucketState

func NewBucketState(ctx *core.Context, params data.Map) (core.SharedState, error)

NewBucketState creates a shared state, which manages a bucket to train & predict.

bucket_size: a bucket size, required

Types

type DataSourceCreator

type DataSourceCreator struct{}

DataSourceCreator is a creator for MNIST data source.

func (*DataSourceCreator) CreateSource

func (s *DataSourceCreator) CreateSource(ctx *core.Context, ioParams *bql.IOParams,
	params data.Map) (core.Source, error)

CreateSource returns a source which generate MNIST data stream. The MNIST data spec is depended on "THE MNIST DATABASE of handwritten digits", see http://yann.lecun.com/exdb/mnist/

MNIST data is ubyte data, and parse when the source is created. Returns an error when cannot load MNIST file or parsing error.

WITH parameters

images_file_name: MNIST images nbytes file path [required]

labels_file_name: MNIST labels nbytes file path [required]

data_size: MNIST data size [required]

image_element_size: MNIST image element size (default: 784=28*28)