cdf

Documentation

Overview

Package CDF provides facilities to read and write files in NetCDF 'classic' (V1 or V2) format. The HDF based NetCDF-4 format is not supported.

The data model and the classic file format are documented at

http://www.unidata.ucar.edu/software/netcdf/docs/tutorial.html
http://www.unidata.ucar.edu/software/netcdf/docs/classic_format_spec.html

A NetCDF file contains an immutable header (this library does not support modifying it) that defines the layout of the data section and contains metadata. The data can be read, written and, if there exists a record dimension, appended to.

To create a new file, first create a header, e.g.:

h := cdf.NewHeader([]string{"time", "x", "y", "z"}, []int{0, 10, 10, 10})
h.AddVariable("psi", []string{"time", "x"}, float32(0))
h.AddAttribute("", "comment", "This is a test file")
h.AddAttribute("psi", "description", "The value of psi as a function of time and x")
h.AddAttribute("psi", "interesting_value", float32(42))
h.Define()
ff, _ := os.Create("/path/to/file")
f, _ := cdf.Create(ff, h)   // writes the header to ff

To use an existing file:

ff, _ := os.Open("/path/to/file")
f, _ := cdf.Open(ff)

The Header field of f is now usable for inspection of dimensions, variables and attributes, but should not be modified (obvious ways of doing this will cause panics).

To read data from the file, use

r := f.Reader("psi", nil, nil)
buf := r.Zero(100)      // a []T of the right T for the variable.
n, err := r.Read(buf)   // similar to io.Read, but reads T's instead of bytes.

And similar for writing.

Index

• func UpdateNumRecs(f *os.File) error
• type File
  • func Create(rw ReaderWriterAt, h *Header) (*File, error)
  • func Open(rw ReaderWriterAt) (*File, error)
  • func (f *File) Fill(v string) error
  • func (f *File) FillRecord(r int) error
  • func (f *File) Reader(v string, begin, end []int) Reader
  • func (f *File) Writer(v string, begin, end []int) Writer
• type Header
  • func NewHeader(dims []string, lengths []int) *Header
  • func ReadHeader(r io.Reader) (*Header, error)
  • func (h *Header) AddAttribute(v, a string, val interface{})
  • func (h *Header) AddVariable(v string, dims []string, val interface{})
  • func (h *Header) Attributes(v string) []string
  • func (h *Header) Check() (errs []error)
  • func (h *Header) Define()
  • func (h *Header) Dimensions(v string) []string
  • func (h *Header) FillValue(v string) interface{}
  • func (h *Header) GetAttribute(v, a string) interface{}
  • func (h *Header) IsRecordVariable(v string) bool
  • func (h *Header) Lengths(v string) []int
  • func (h *Header) NumRecs(fsize int64) int64
  • func (h *Header) String() string
  • func (h *Header) Variables() []string
  • func (h *Header) WriteHeader(w io.Writer) error
  • func (h *Header) ZeroValue(v string, n int) interface{}
• type Reader
• type ReaderWriterAt
• type Writer

Functions

func UpdateNumRecs

func UpdateNumRecs(f *os.File) error

UpdateNumRecs determines the number of record from the file size and writes it into the file's header as the 'numrecs' field.

Any incomplete trailing record will not be included in the count.

Only valid headers will be updated. After a succesful call f's filepointer will be left at the end of the file.

This library does not use the numrecs header field but updating it enables full bit for bit compatibility with other libraries. There is no need to call this function until after all updates by the program, and it is rather costly because it reads, parses and checks the entire header.

Types

type File

type File struct {
	Header *Header
	// contains filtered or unexported fields
}

func Create

func Create(rw ReaderWriterAt, h *Header) (*File, error)

Create writes the header to a storage rw and returns a File usable for reading and writing.

The header should not be mutable, and may be shared by multiple Files. Note that at every Create the headers numrec field will be reset to -1 (STREAMING).

func Open

func Open(rw ReaderWriterAt) (*File, error)

Open reads the header from an existing storage rw and returns a File usable for reading or writing (if the underlying rw permits).

func (*File) Fill

func (f *File) Fill(v string) error

Fill overwrites the data for non-record variable named v with its fill value. Fill panics if v does not name a non-record variable. If the variable has a scalar attribute '_FillValue' of the same data type as the variable, it will be used, otherwise the type's default fill value will be used.

func (*File) FillRecord

func (f *File) FillRecord(r int) error

FillRecord overwrites the data for all record variables in the r'th slab with their fill values.

func (*File) Reader

func (f *File) Reader(v string, begin, end []int) Reader

Create a reader that starts at the corner begin, ends at end. If begin is nil, it defaults to the origin (0, 0, ...). If end is nil, it defaults to the f.Header.Lengths(v).

func (*File) Writer

func (f *File) Writer(v string, begin, end []int) Writer

Create a writer that starts at the corner begin, ends at end. If begin is nil, it defaults to the origin (0, 0, ...). If end is nil and the variable is a record variable, writing can proceed past EOF and the underlying file will be extended.

type Header

type Header struct {
	// contains filtered or unexported fields
}

A CDF file contains a header and a data section. The header defines the layout of the data section.

The serialized header layout is specified by "The NetCDF Classic Format Specification"

http://www.unidata.ucar.edu/software/netcdf/docs/classic_format_spec.html

A header read with ReadHeader can not be modified. A header created with NewHeader can be modified with AddVariable and AddAttribute until the call to Define.

The NetCDF defined 'numrecs' field is ignored on reading and set to -1 ('STREAMING') on writing of the header, but can be read and written separately.

func NewHeader

func NewHeader(dims []string, lengths []int) *Header

Newheader constructs a new CDF header.

dims and lengths specify the names and lengths of the dimensions. Invalid dimension or size specifications, repeated dimension names, as well as the occurence of more than 1 record dimension (size == 0) lead to panics.

Until the call to h.Define() the version of the header will not be set, and the header will mutable, meaning it can be modified by AddAttribute or AddVariable.

func ReadHeader

func ReadHeader(r io.Reader) (*Header, error)

readHeader decodes the CDF header from the io.Reader at the current position. On success readHeader returns a header struct and a nil error. If an error occurs that prevents further reading, the reader is left at the error position and err is set to badMagic, badVersion, badTag, badLenght or badAttributeType, or the error from the underlying call to binary.Read. The returned header is immutable, meaning it may not be modified with AddVariable or AddAttribute.

func (*Header) AddAttribute

func (h *Header) AddAttribute(v, a string, val interface{})

AddAttribute adds an attribute named a to a variable named v, or to the global attributes if v is the empty string.

Use of a nonexistent variable name or an existent attribute name leads to a panic. The value can be of type []uint8, string, []int16, []int32, []float32 or []float64, and will be stored as NetCDF type BYTE, CHAR, SHORT, INT, FLOAT, DOUBLE resp. The header must be mutable, i.e. created by NewHeader, not by ReadHeader.

func (*Header) AddVariable

func (h *Header) AddVariable(v string, dims []string, val interface{})

AddVariable adds a variable of given type with the named dimensions to the header.

Use of an existing variable name, or a nonexistent dimension name leads to a panic, as does use of the record dimension for any other than the first.

The datatype is determined from the dynamic type of val, which may be one of []uint8, string, []int16, []int32, []float32 or []float64. Any other type will lead to a panic. The contents of val are ignored.

The header must be mutable, i.e. created by NewHeader, not by ReadHeader.

func (*Header) Attributes

func (h *Header) Attributes(v string) []string

Variables returns a slice with the names of all attributes defined in the header, for variable v. If v is the empty string, returns all global attributes.

func (*Header) Check

func (h *Header) Check() (errs []error)

Check verifies the integrity of the header:

- at most one record dimension

- no duplicate dimension names

- no duplicate attribute names

- no duplicate variable names

- variable dimensions valid

- only the first dimension can be the record dimension

- offsets of non-variable records increasing, large enough and all before variable records

- offset of variable records also increasing, large enough

func (*Header) Define

func (h *Header) Define()

Define makes a mutable header immutable by calculating the variable offsets and setting the version number to V1 or V2, depending on whether the layout requires 64-bit offsets or not.

func (*Header) Dimensions

func (h *Header) Dimensions(v string) []string

Dimensions returns a slice with the names of the dimensions for variable v, all dimensions if v == "", or nil if v is not a valid variable.

May panic on un-Check-ed headers.

func (*Header) FillValue

func (h *Header) FillValue(v string) interface{}

Return the fill value for the variable v. If the variable has a scalar attribute '_FillValue' of the same data type as the variable, it will be used, otherwise the type's default fill value will be used.

func (*Header) GetAttribute

func (h *Header) GetAttribute(v, a string) interface{}

GetAttribute returns the value of the attribute a of variable v or the global attribute a if v == "". The returned value is of type []uint8, string, []int16, []int32, []float32 or []float64 and should not be modified by the caller, as it is shared by all callers.

func (*Header) IsRecordVariable

func (h *Header) IsRecordVariable(v string) bool

IsRecordVariable returns true iff a variable named v exists and its outermost dimension is the header's (unique) record dimension.

func (*Header) Lengths

func (h *Header) Lengths(v string) []int

Lengths returns a slice with the lengths of the dimensions for variable v, all dimensions if v == "", or nil if v is not a valid variable.

May panic on un-Check-ed headers.

func (*Header) NumRecs

func (h *Header) NumRecs(fsize int64) int64

numRecs computes the number or records from the filesize, returns the real number of records. For fsize < 0, returns -1.

func (*Header) String

func (h *Header) String() string

String returns a summary dump of the header, suitable for debugging.

func (*Header) Variables

func (h *Header) Variables() []string

Variables returns a slice with the names of all variables defined in the header.

func (*Header) WriteHeader

func (h *Header) WriteHeader(w io.Writer) error

writeHeader encodes the CDF header to the io.Writer at the current position. If an error occurs that prevents further writing, the writer is left at the erroring position and err is set to the error from the underlying call to binary.Write.

func (*Header) ZeroValue

func (h *Header) ZeroValue(v string, n int) interface{}

ZeroValue returns a zeroed slice of the type of the variable v of length n. If the named variable does not exist in h, Zero returns nil. For type CHAR, Zero returns an empty string.

type Reader

type Reader interface {
	// Read reads len(values.([]T)) elements from the underlying file into values.
	//
	// Values must be a slice of int{8,16,32} or float{32,64},
	// corresponding to the type of the variable, with one
	// exception: A variable of NetCDF type CHAR must be read into
	// a []byte.  Read returns the number of elements actually
	// read.  if n < len(values.([]T)), err will be set.
	Read(values interface{}) (n int, err error)

	// Zero returns a slice of the appropriate type for Read
	// if n < 0, the slice will be of the length
	// that can be read contiguously.
	Zero(n int) interface{}
}

A reader is an object that can read values from a CDF file.

type ReaderWriterAt

type ReaderWriterAt interface {
	io.ReaderAt
	io.WriterAt
}

A ReaderWriterAt is the underlying storage for a NetCDF file, providing {Read,Write}At([]byte, int64) methods. Since {Read,Write}At are required to not modify the underlying file pointer, one instance may be shared by multiple Files, although the documentation of io.WriterAt specifies that it only has to guarantee non-concurrent calls succeed.

type Writer

type Writer interface {
	// Write writes len(values.([]T)) elements from values to the underlying file.
	//
	// Values must be a slice of int{8,16,32} or float{32,64} or a
	// string, according to the type of the variable.  if n <
	// len(values.([]T)), err will be set.
	Write(values interface{}) (n int, err error)
}

A writer is an object that can write values to a CDF file.