core

package
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 11, 2020 License: MIT Imports: 14 Imported by: 1

Documentation

Overview

Package tfdata provides interfaces to interact with TFRecord files and TFExamples.

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Package tfdata provides interfaces to interact with TFRecord files and TFExamples.

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Package tfdata provides interfaces to interact with TFRecord files and TFExamples.

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Package tfdata provides interface to interact with TFRecord files and TExamples

Copyright (c) 2020, NVIDIA CORPORATION. All rights reserved.

Index

Constants

View Source 
const KeyEntry = "__key__"

Variables

View Source 
var FeatureType = &TFFeatureEnum{}

Functions

This section is empty.

Types

type Sample 

type Sample struct {
	Entries map[string]interface{}
}

func NewSample 

func NewSample(entries ...map[string]interface{}) *Sample

type SampleChannel 

type SampleChannel struct {
	// contains filtered or unexported fields
}

func NewSampleChannel 

func NewSampleChannel(bufSize int) *SampleChannel

func (*SampleChannel) Close 

func (c *SampleChannel) Close()

func (*SampleChannel) Read 

func (c *SampleChannel) Read() (*Sample, error)

func (*SampleChannel) Write 

func (c *SampleChannel) Write(sample *Sample) error

type SampleReadWriter 

type SampleReadWriter interface {
	SampleReader
	SampleWriter
}

type SampleReader 

type SampleReader interface {
	Read() (sample *Sample, err error)
}

SampleReader / SampleWriter Returns io.EOF if there's nothing left to be read

type SampleWriter 

type SampleWriter interface {
	Write(s *Sample) error
	Close()
}

type TFExample 

type TFExample struct {
	proto.Example
}

TFExample is a wrapper over proto.Example struct generated by protoc from TensorFlow tf.Example proto files. It is a golang representation of tf.Example datastructure. It includes functions for adding elements to tf.Example.Features.

func NewTFExample 

func NewTFExample() *TFExample

NewTFExample initializes empty TFExample and returns it.

func (*TFExample) AddBytes 

func (e *TFExample) AddBytes(name string, bs ...[]byte)

func (*TFExample) AddBytesList 

func (e *TFExample) AddBytesList(name string, bs [][]byte)

func (*TFExample) AddFloat 

func (e *TFExample) AddFloat(name string, floats ...float32)

func (*TFExample) AddFloatList 

func (e *TFExample) AddFloatList(name string, floats []float32)

func (*TFExample) AddImage 

func (e *TFExample) AddImage(name string, img image.Image) error

func (*TFExample) AddInt 

func (e *TFExample) AddInt(name string, ints ...int)

func (*TFExample) AddInt64 

func (e *TFExample) AddInt64(name string, ints ...int64)

func (*TFExample) AddInt64List 

func (e *TFExample) AddInt64List(name string, ints []int64)

func (*TFExample) AddIntList 

func (e *TFExample) AddIntList(name string, ints []int)

func (*TFExample) GetBytesList 

func (e *TFExample) GetBytesList(name string) []byte

func (*TFExample) GetFeature 

func (e *TFExample) GetFeature(name string) *proto.Feature

func (*TFExample) GetFloat 

func (e *TFExample) GetFloat(name string) float32

func (*TFExample) GetFloatList 

func (e *TFExample) GetFloatList(name string) []float32

func (*TFExample) GetImage 

func (e *TFExample) GetImage(name string) (image.Image, error)

GetImage decodes an image from TFExample from JPEG, PNG or GIF

func (*TFExample) GetInt64 

func (e *TFExample) GetInt64(name string) int64

func (*TFExample) GetInt64List 

func (e *TFExample) GetInt64List(name string) []int64

func (*TFExample) HasFeature 

func (e *TFExample) HasFeature(name string) bool

func (*TFExample) SetFeature 

func (e *TFExample) SetFeature(name string, feature *proto.Feature)

type TFExampleChannel 

type TFExampleChannel struct {
	// contains filtered or unexported fields
}

func NewTFExampleChannel 

func NewTFExampleChannel(bufSize int) *TFExampleChannel

func (*TFExampleChannel) Close 

func (c *TFExampleChannel) Close()

func (*TFExampleChannel) Read 

func (c *TFExampleChannel) Read() (*TFExample, error)

func (*TFExampleChannel) Write 

func (c *TFExampleChannel) Write(example *TFExample) error

type TFExampleReadWriter 

type TFExampleReadWriter interface {
	TFExampleReader
	TFExampleWriter
}

type TFExampleReader 

type TFExampleReader interface {
	Read() (ex *TFExample, err error)
}

TFExampleReader / TFExampleWriter Returns io.EOF if there's nothing left to be read

type TFExampleWriter 

type TFExampleWriter interface {
	Write(ex *TFExample) error
	Close()
}

type TFFeatureEnum 

type TFFeatureEnum struct {
	INT64       *cmn.TFFeatureInt
	INT64LIST   *cmn.TFFeatureIntList
	FLOAT32     *cmn.TFFeatureFloat
	FLOAT32LIST *cmn.TFFeatureFloatList
	BYTES       *cmn.TFFeatureBytes
	BYTESLIST   *cmn.TFFeatureBytesList
}

type TFFeatureType 

type TFFeatureType interface {
	FeatureType() int
}

type TFRecordReader 

type TFRecordReader struct {
	// contains filtered or unexported fields
}

TFRecordReader implements TFRecordReader interface It reads objects from reader r and verify checksums with c

func NewTFRecordReader 

func NewTFRecordReader(r io.Reader) *TFRecordReader

NewTFRecordReader creates and initializes TFRecordReader with writer w and CRC checksumming method. Returns pointer to created TFRecordReader

func (*TFRecordReader) Read 

func (r *TFRecordReader) Read() (*TFExample, error)

func (*TFRecordReader) ReadAllExamples 

func (r *TFRecordReader) ReadAllExamples(expectedSize ...int) ([]*TFExample, error)

ReadAllExamples reads examples from TFRecord until EOF and loads them into memory If error occurred it terminates immediately without reading subsequent samples Returns slice of examples and error if occurred.

func (*TFRecordReader) ReadExamples 

func (r *TFRecordReader) ReadExamples(writer TFExampleWriter) error

ReadExamples reads and puts into ch examples from TFRecord one-by-one It error occurred, ReadExamples terminates immediately, without processing subsequent samples. ReadExamples closes ch upon termination.

func (*TFRecordReader) ReadNext 

func (r *TFRecordReader) ReadNext(message protobuf.Message) error

ReadNext reads next message from reader and stores it in provided message If error occurred ReadNext terminates immediately. If read bytes are not in TFRecord format ReadNext terminates with error.

type TFRecordReaderInterface 

type TFRecordReaderInterface interface {
	ReadNext(protobuf.Message) error
	Read() (*TFExample, error)

	ReadAllExamples(expectedSize ...int) ([]*TFExample, error)
	ReadExamples(writer TFExampleWriter) error
}

TFRecordReaderInterface is an interface which reads objects from TFRecord format TFRecord format is described here: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfrecords_format_details

type TFRecordWriter 

type TFRecordWriter struct {
	// contains filtered or unexported fields
}

TFRecordWriter implements TFRecordWriter interface It writes objects into writer w with checksums provided by c

func NewTFRecordWriter 

func NewTFRecordWriter(w io.Writer) *TFRecordWriter

NewTFRecordWriter creates and initializes TFRecordWriter with writer w and CRC checksumming method. Returns pointer to created TFRecordWriter

func (*TFRecordWriter) Write 

func (w *TFRecordWriter) Write(p []byte) (n int, err error)

Write writes p into writer into format specified in https://www.tensorflow.org/tutorials/load_data/tfrecord#tfrecords_format_details. If any of underlying writes to internal writer fails, number or already written bytes and error is returned Write is not atomic, meaning that underlying write error might leave internal writer in invalid TFRecord state Returns total number of written bytes and error if occurred

func (*TFRecordWriter) WriteExample 

func (w *TFRecordWriter) WriteExample(example *TFExample) (n int, err error)

func (*TFRecordWriter) WriteExamples 

func (w *TFRecordWriter) WriteExamples(ch <-chan *TFExample) error

WriteExamples behaves the same as WriteMessages but operates on channel of TFExamples

func (*TFRecordWriter) WriteMessage 

func (w *TFRecordWriter) WriteMessage(message protoreflect.ProtoMessage) (n int, err error)

WriteMessage marshals message and writes it to TFRecord

func (*TFRecordWriter) WriteMessages 

func (w *TFRecordWriter) WriteMessages(reader TFExampleReader) error

WriteMessages reads and writes to TFRecord messages one-by-one from ch and terminates when ch is closed and empty. WriteMessages doesn't close ch itself. Returns error immediately if occurred, without processing subsequent messages

func (*TFRecordWriter) WriteMessagesAsync 

func (w *TFRecordWriter) WriteMessagesAsync(reader TFExampleReader, numWorkers int) error

Reads TFExamples from reader asynchronously and writes synchronously to w.Writer TFRecordWriter is last element of a pipeline, if it makes Reads asynchronously All underlying readers will be called asynchronously. They all should be async-safe Almost all of transformations

type TFRecordWriterInterface 

type TFRecordWriterInterface interface {
	io.Writer
	WriteMessage(protobuf.Message) (n int, err error)
	WriteExample(*TFExample) (n int, err error)

	WriteMessages(reader TFExampleReader) error
}

TFRecordWriterInterface is an interface which writes objects in TFRecord format. TFRecord format is described here: https://www.tensorflow.org/tutorials/load_data/tfrecord#tfrecords_format_details

type TypesMap 

type TypesMap map[string]TFFeatureType

Defines types mapping of Sample entries to TFExample. TypesMap key corresponds to Sample key and TFFeatureType is one of: TFFeatureType.INT64, TFFeatureType.INT64LIST, TFFeatureType.FLOAT32, TFFeatureType.FLOAT32LIST, TFFeatureType.FLOAT32, TFFeatureType.FLOAT32LIST, TFFeatureType.BYTES, TFFeatureType.BYTESLIST

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.