mnist

README

MNIST

MNIST is a simple computer vision dataset that consists of images of handwritten digits.

Some examples:

It also includes labels for each image, which we use to train our example models.

The mnist library

This package includes the following functionality:

• Download the dataset from storage.googleapis.com/cvdf-datasets/mnist,
• Create a Dataset object to iterate over it, use for training and evaluation.
• A linear and a CNN model demo.
• A command-line demo (in the demo sub-directory).

Reference

• Tensorflow "Deep MNIST for Experts"
• Exploring Different Loss Functions on MNIST (Kaggle)
• MNIST in Wikipedia

Documentation

Index

• Constants
• Variables
• func CnnEmbeddings(ctx *context.Context, images *Node) *Node
• func CnnModelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node
• func CreateDatasets(backend backends.Backend, config *DatasetsConfiguration) (trainDS, trainEvalDS, validationEvalDS train.Dataset)
• func CreateDefaultContext() *context.Context
• func Download(baseDir string) error
• func LinearModelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node
• func NewDataset(backend backends.Backend, name, baseDir, mode string, dtype dtypes.DType) (ds *data.InMemoryDataset, err error)
• func TrainModel(ctx *context.Context, dataDir, checkpointPath string, paramsSet []string) error
• type ContextFn
• type DatasetsConfiguration
  • func NewDatasetsConfigurationFromContext(ctx *context.Context, dataDir string) *DatasetsConfiguration
• type Image
  • func (img Image) At(x, y int) color.Color
  • func (img Image) Bounds() image.Rectangle
  • func (img Image) ColorModel() color.Model
  • func (img *Image) Set(x, y int, v byte)
• type Label

Constants

const (
	DownloadURL         = "https://storage.googleapis.com/cvdf-datasets/mnist"
	TrainImagesFilename = "train-images-idx3-ubyte.gz"
	TrainLabelsFilename = "train-labels-idx1-ubyte.gz"
	TestImagesFilename  = "t10k-images-idx3-ubyte.gz"
	TestLabelsFilename  = "t10k-labels-idx1-ubyte.gz"
	TrainImagesHash     = "440fcabf73cc546fa21475e81ea370265605f56be210a4024d2ca8f203523609"
	TrainLabelsHash     = "3552534a0a558bbed6aed32b30c495cca23d567ec52cac8be1a0730e8010255c"
	TestImagesHash      = "8d422c7b0a1c1c79245a5bcf07fe86e33eeafee792b84584aec276f5a2dbc4e6"
	TestLabelsHash      = "f7ae60f92e00ec6debd23a6088c31dbd2371eca3ffa0defaefb259924204aec6"

	Width         = 28
	Height        = 28
	Depth         = 3
	NumClasses    = 10
	TrainExamples = 60000
	TestExamples  = 10000

	ImageMagic = 0x00000803
	LabelMagic = 0x00000801
)

const (
	ImageFileType fileType = iota
	LabelFileType
)

Variables

var ModelList = []string{"linear", "cnn"}

Functions

func CnnEmbeddings

func CnnEmbeddings(ctx *context.Context, images *Node) *Node

func CnnModelGraph

func CnnModelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node

CnnModelGraph builds the CNN model for our demo. It returns the logit, not the predictions, which works with most losses with shape `[batch_size, NumClasses]`. inputs: only one tensor, with shape `[batch_size, width, height, depth]`.

func CreateDatasets

func CreateDatasets(backend backends.Backend, config *DatasetsConfiguration) (trainDS, trainEvalDS, validationEvalDS train.Dataset)

CreateDatasets used for training and evaluation.

func CreateDefaultContext

func CreateDefaultContext() *context.Context

func Download

func Download(baseDir string) error

Download MNIST Dataset to baseDir, unzips it

func LinearModelGraph

func LinearModelGraph(ctx *context.Context, spec any, inputs []*Node) []*Node

LinearModelGraph builds a simple model logistic model It returns the logit, not the predictions, which works with most losses with shape `[batch_size, NumClasses]`. inputs: only one tensor, with shape `[batch_size, width, height, depth]`.

func NewDataset

func NewDataset(backend backends.Backend, name, baseDir, mode string, dtype dtypes.DType) (ds *data.InMemoryDataset, err error)

NewDataset creates a train.Dataset that yields images from MNIST Dataset.

It takes the following arguments:

• name:
• baseDir:
• mode: choose between 'train' and 'test'

func TrainModel

func TrainModel(ctx *context.Context, dataDir, checkpointPath string, paramsSet []string) error

TrainModel based on configuration and flags.

Types

type ContextFn

type ContextFn func(ctx *context.Context) *context.Context

type DatasetsConfiguration

type DatasetsConfiguration struct {
	// DataDir, where downloaded and generated data is stored.
	DataDir string

	// BatchSize for training and evaluation batches.
	BatchSize, EvalBatchSize int

	// UseParallelism when using Dataset.
	UseParallelism bool

	// BufferSize used for data.ParallelDataset, to cache intermediary batches. This value is used
	// for each dataset.
	BufferSize int

	Dtype dtypes.DType
}

DatasetsConfiguration holds various parameters on how to transform the input images.

func NewDatasetsConfigurationFromContext

func NewDatasetsConfigurationFromContext(ctx *context.Context, dataDir string) *DatasetsConfiguration

NewDatasetsConfigurationFromContext create a preprocessing configuration based on hyperparameters set in the context.

type Image

type Image [Width * Height]byte

Image represents a MNIST image. It is a array a bytes representing the color. 0 is black (the background) and 255 is white (the digit color).

var (
	AssertImageIsImageImage *Image
)

func (Image) At

func (img Image) At(x, y int) color.Color

At implements the image.Image interface.

func (Image) Bounds

func (img Image) Bounds() image.Rectangle

Bounds implements the image.Image interface.

func (Image) ColorModel

func (img Image) ColorModel() color.Model

ColorModel implements the image.Image interface.

func (*Image) Set

func (img *Image) Set(x, y int, v byte)

Set modifies the pixel at (x,y).

type Label

type Label = int8

Label is the digit label from 0 to 9.