npy

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 27, 2026 License: GPL-3.0 Imports: 14 Imported by: 0

README

gozeronpy

A fast, zero-copy Go library for reading and writing NumPy .npy files.

The array's shape and element type are detected automatically from the file — just like numpy.load — and on the common little-endian case the raw file bytes are reinterpreted directly as a Go slice with no per-element copy.

arr, _ := npy.Open("data.npy")
fmt.Println(arr.Shape)   // e.g. [2 3] — discovered from the file
xs, _ := arr.Float64()   // []float64 view of the data

Why it's fast

  • Zero-copy reads. When the file byte order matches the host (little-endian on amd64/arm64), the data region is reinterpreted as a Go slice via unsafe rather than parsed element by element. Reads run at memory bandwidth (~12 GB/s on an M-series laptop).
  • RAM-aware access modes. Auto (default) compares the data size to available system memory: small files are read into memory; files large relative to RAM are memory-mapped so you can work with arrays bigger than RAM without loading them.
  • Single aligned allocation. The in-memory path allocates one 8-byte aligned buffer for the whole array; the mmap path allocates almost nothing.
BenchmarkLoadInMemory-10    12360 MB/s     17 allocs/op   (64 MiB float64 array)
BenchmarkLoadMmap-10         9380 MB/s     16 allocs/op   (incl. page faults)
BenchmarkOpenDynamic-10     12319 MB/s     15 allocs/op
BenchmarkSave-10             3275 MB/s     17 allocs/op

Install

go get github.com/siddarth99/gozeronpy

import npy "github.com/siddarth99/gozeronpy"

Reading

Dynamic (NumPy-like)

The element type is whatever the file contains; Data holds the matching Go slice ([]float64, []int32, …).

arr, err := npy.Open("data.npy")
if err != nil { /* ... */ }
defer arr.Close() // releases the mapping if the array was memory-mapped

fmt.Println(arr.Shape)          // []int, auto-detected
fmt.Println(arr.Dtype)          // e.g. <f8
fmt.Println(arr.Dtype.GoType()) // "float64"

switch v := arr.Data.(type) {
case []float64:
    _ = v
case []int32:
    _ = v
}

// Or pull a typed view directly (errors if the dtype doesn't match):
xs, err := arr.Float64()
ys, err := npy.Values[int32](arr)

// Or convert any numeric dtype to float64 (copying):
f, err := arr.AsFloat64()
Typed (fastest, no runtime assertions)

If you know the element type at compile time, ask for it directly:

nd, err := npy.Load[float64]("data.npy")
if err != nil { /* ... */ }
defer nd.Close()

_ = nd.Values // []float64
_ = nd.Shape  // []int

Load returns an error if the file's element kind/size doesn't match T (byte order is converted automatically).

From a stream
arr, err := npy.Decode(r)        // dynamic, reads into memory
nd,  err := npy.Read[float64](r) // typed, reads into memory

Writing

Shape is optional — pass nil for a 1-D array, or give an explicit shape whose product matches the data length.

npy.Save("out.npy", []float64{1, 2, 3, 4}, []int{2, 2})
npy.Save("vec.npy", []int32{1, 2, 3}, nil)                  // 1-D
npy.Save("col.npy", data, []int{2, 2}, npy.WithFortran(true))

// To a stream:
npy.Write(w, []float64{1, 2, 3, 4}, []int{2, 2})

Archives (.npz)

A .npz file is a ZIP archive of .npy entries. Array names are the entry names without the .npy suffix, matching numpy.load(...).files.

Reading
arc, err := npy.OpenZip("data.npz")
if err != nil { /* ... */ }
defer arc.Close()

fmt.Println(arc.Names())          // []string in storage order

arr, _ := arc.Array("x")          // dynamic
nd,  _ := npy.ZipValues[float64](arc, "x") // typed
all, _ := arc.All()               // map[string]*npy.Array

// From an io.ReaderAt instead of a path:
arc, _ = npy.ReadZip(readerAt, size)

Like Open, OpenZip can memory-map the archive (WithMode): uncompressed entries are then read zero-copy directly from the mapping, and compressed entries are inflated into memory.

Writing
// All at once (uncompressed, like numpy.savez):
npy.SaveZip("out.npz", []npy.NamedArray{
    {Name: "x", Data: []float64{1, 2, 3, 4}, Shape: []int{2, 2}},
    {Name: "y", Data: []int32{5, 6, 7}},                  // nil shape => 1-D
})

// Compressed (like numpy.savez_compressed):
npy.SaveZip("out.npz", arrays, npy.WithCompression(true))

// Incrementally, without holding every array in memory at once:
zw, _ := npy.CreateZip("out.npz")
npy.AddTyped(zw, "a", []float64{1, 2, 3}, nil)
zw.Add("b", []int32{4, 5, 6}, []int{3})
zw.Close()

Access modes

npy.Open("big.npy", npy.WithMode(npy.Auto))               // default
npy.Open("big.npy", npy.WithMode(npy.InMemory))           // always read into RAM
npy.Load[float64]("big.npy", npy.WithMode(npy.Mmap))      // memory-map
npy.Open("big.npy", npy.WithMaxRAMFraction(0.25))         // Auto threshold
  • Auto — memory-map when the file is larger than MaxRAMFraction × system memory (default 0.5); otherwise read into memory. When system memory can't be determined, falls back to a 512 MiB threshold.
  • InMemory — always read the whole array into a heap buffer.
  • Mmap — memory-map the file (transparently falls back to InMemory on platforms without mmap, e.g. Windows).

Lifetime: for a memory-mapped array, Data/Values alias the mapping, so call Close() when you're done. A finalizer unmaps as a safety net, but explicit Close is preferred. In-memory arrays own their data; Close is a no-op.

Supported dtypes

NumPy Go NumPy Go
f4 / f8 float32/64 u1..u8 uint8..64
i1..i8 int8..64 b1 bool
c8 / c16 complex64/128 (le & be) byte-swapped on read

Both little- and big-endian files are read correctly (big-endian is converted to host order). Fortran (column-major) order is detected and exposed via the Fortran flag; the data is returned in its on-disk storage order. Structured / record dtypes, float16, datetime and object arrays are not supported.

Correctness

The library is cross-validated against real NumPy: a test generates .npy files with NumPy across every dtype, both byte orders, C/Fortran order and 0-D…3-D shapes (122 cases), reads them in Go, and checks shapes, dtypes and values match — and writes files in Go that NumPy then reads back. The same is done for .npz archives, both uncompressed and compressed.

go test ./...              # pure-Go tests; NumPy tests run if python3+numpy present
go test -bench=. ./...     # benchmarks

License

GPL-3.0 (see LICENSE).

Documentation

Overview

Package npy reads and writes NumPy .npy files.

It is designed to be fast: when the on-disk byte order matches the host (the common little-endian case) the raw file bytes are reinterpreted directly as a Go slice with no per-element copy. For files that are large relative to available RAM it can memory-map the data instead of loading it.

There are two API tiers:

  • Dynamic, Python-like:

    arr, err := npy.Open("data.npy") fmt.Println(arr.Shape) // shape auto-detected xs, _ := arr.Float64() // typed view of the data

  • Generic, fastest (no runtime type assertions):

    nd, err := npy.Load[float64]("data.npy") _ = nd.Values // []float64 _ = nd.Shape // []int

Writing mirrors this: 

npy.Save("out.npy", []float64{1, 2, 3, 4}, []int{2, 2})
npy.Write(w, []int32{1, 2, 3}, nil) // nil shape => 1-D

The .npy format is documented at https://numpy.org/devdocs/reference/generated/numpy.lib.format.html

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func AddTyped 

func AddTyped[T Element](zw *ZipWriter, name string, data []T, shape []int) error

AddTyped writes a statically-typed slice to the archive under name.

func Save 

func Save(path string, data any, shape []int, opts ...Option) error

Save writes a typed slice to a .npy file. data must be one of the supported element slice types (for example []float64 or []int32). If shape is nil the array is written as 1-D with length len(data); otherwise the product of shape must equal len(data). 

npy.Save("out.npy", []float64{1, 2, 3, 4}, []int{2, 2})

func SaveZip 

func SaveZip(path string, arrays []NamedArray, opts ...Option) error

SaveZip writes a .npz archive containing the given named arrays. By default entries are stored uncompressed (like numpy.savez); pass WithCompression(true) for DEFLATE (like numpy.savez_compressed). 

npy.SaveZip("out.npz", []npy.NamedArray{
    {Name: "x", Data: []float64{1, 2, 3, 4}, Shape: []int{2, 2}},
    {Name: "y", Data: []int32{5, 6, 7}},
})

func Values 

func Values[T Element](a *Array) ([]T, error)

Values returns the array's data as a typed slice. It returns an error if T does not match the array's on-disk dtype.

func Write 

func Write[T Element](w io.Writer, data []T, shape []int, opts ...Option) error

Write writes a statically-typed slice to a .npy stream. A nil shape means a 1-D array of length len(data).

Types

type AccessMode 

type AccessMode int

AccessMode controls how a file's data region is brought into the process. 

const (
	// Auto picks between InMemory and Mmap based on the data size relative to
	// available system memory. This is the default.
	Auto AccessMode = iota
	// InMemory reads the whole file into a heap buffer.
	InMemory
	// Mmap memory-maps the file (falling back to InMemory where unsupported).
	Mmap
)

type Archive 

type Archive struct {
	// contains filtered or unexported fields
}

Archive is a read handle for a NumPy .npz file (a ZIP archive of .npy entries). Array names are the entry names with the ".npy" suffix removed, matching numpy.load(...).files. 

arc, _ := npy.OpenZip("data.npz")
defer arc.Close()
for _, name := range arc.Names() {
    arr, _ := arc.Array(name)
    fmt.Println(name, arr.Shape)
}

func OpenZip 

func OpenZip(path string, opts ...Option) (*Archive, error)

OpenZip opens a .npz archive from a file path. Like Open, it may memory-map the file (honouring WithMode / WithMaxRAMFraction) so that uncompressed entries can be read zero-copy. Call Close when done.

func ReadZip 

func ReadZip(r io.ReaderAt, size int64) (*Archive, error)

ReadZip reads a .npz archive from r. size is the total length of the archive. Entries are read into memory (no mmap).

func (*Archive) All 

func (a *Archive) All() (map[string]*Array, error)

All reads every array in the archive into a name-keyed map.

func (*Archive) Array 

func (a *Archive) Array(name string) (*Array, error)

Array reads the named array into a dynamically-typed Array. For a memory-mapped archive the data may alias the mapping, so the returned Array must not be used after the Archive is closed.

func (*Archive) Close 

func (a *Archive) Close() error

Close releases the archive's resources (the file and/or mapping).

func (*Archive) Has 

func (a *Archive) Has(name string) bool

Has reports whether the archive contains an array with the given name.

func (*Archive) Names 

func (a *Archive) Names() []string

Names returns the array names in the archive, in storage order.

type Array 

type Array struct {
	Shape   []int // dimensions, as in NumPy
	Fortran bool  // true if the data is in Fortran (column-major) order
	Dtype   DType // the on-disk element type
	Data    any   // the typed element slice, e.g. []float64
	// contains filtered or unexported fields
}

Array is the dynamically-typed result of reading a .npy file. The element type is discovered from the file and stored in Data as the matching Go slice type (for example []float64 or []int32).

func Decode 

func Decode(r io.Reader) (*Array, error)

Decode reads a .npy stream into a dynamically-typed Array. The whole array is read into memory.

func Open 

func Open(path string, opts ...Option) (*Array, error)

Open reads a .npy file, returning a dynamically-typed Array whose shape and element type are taken from the file. This is the most NumPy-like entry point: 

arr, err := npy.Open("data.npy")
fmt.Println(arr.Shape)

For memory-mapped arrays, call Close when done.

func (*Array) AsFloat64 

func (a *Array) AsFloat64() ([]float64, error)

AsFloat64 returns a copy of the data converted to float64, regardless of the underlying numeric dtype. Boolean and complex arrays are not supported.

func (*Array) Bool 

func (a *Array) Bool() ([]bool, error)

func (*Array) Close 

func (a *Array) Close() error

Close releases resources associated with the array. It is only meaningful for memory-mapped arrays; for in-memory arrays it is a no-op. After Close the Data slice of a memory-mapped array must not be used.

func (*Array) Complex64 

func (a *Array) Complex64() ([]complex64, error)

func (*Array) Complex128 

func (a *Array) Complex128() ([]complex128, error)

func (*Array) Float32 

func (a *Array) Float32() ([]float32, error)

func (*Array) Float64 

func (a *Array) Float64() ([]float64, error)

func (*Array) Int8 

func (a *Array) Int8() ([]int8, error)

func (*Array) Int16 

func (a *Array) Int16() ([]int16, error)

func (*Array) Int32 

func (a *Array) Int32() ([]int32, error)

func (*Array) Int64 

func (a *Array) Int64() ([]int64, error)

func (*Array) Len 

func (a *Array) Len() int

Len returns the total number of elements (the product of Shape).

func (*Array) Uint8 

func (a *Array) Uint8() ([]uint8, error)

func (*Array) Uint16 

func (a *Array) Uint16() ([]uint16, error)

func (*Array) Uint32 

func (a *Array) Uint32() ([]uint32, error)

func (*Array) Uint64 

func (a *Array) Uint64() ([]uint64, error)

type DType 

type DType struct {
	Kind     byte             // 'f' float, 'i' int, 'u' uint, 'b' bool, 'c' complex
	ItemSize int              // bytes per element
	Order    binary.ByteOrder // on-disk byte order; nil for single-byte types
	Descr    string           // the original descr string, e.g. "<f8"
}

DType describes the element type of an array, mirroring a NumPy dtype.

func (DType) GoType 

func (d DType) GoType() string

GoType returns the name of the Go type used to represent this dtype.

func (DType) String 

func (d DType) String() string

String returns the NumPy-style descr, e.g. "<f8".

type Element 

type Element interface {
	~int8 | ~int16 | ~int32 | ~int64 |
		~uint8 | ~uint16 | ~uint32 | ~uint64 |
		~float32 | ~float64 |
		~complex64 | ~complex128 |
		~bool
}

Element is the set of array element types this package can read and write. It mirrors the numeric (and boolean) dtypes NumPy stores in .npy files.

type NDArray 

type NDArray[T Element] struct {
	Values  []T   // the element data
	Shape   []int // dimensions
	Fortran bool  // true if the data is in Fortran (column-major) order
	// contains filtered or unexported fields
}

NDArray is the statically-typed result of reading a .npy file with a known element type. It is the fastest API: there are no runtime type assertions.

func Load 

func Load[T Element](path string, opts ...Option) (*NDArray[T], error)

Load reads a .npy file into a statically-typed NDArray[T]. T must match the file's element kind and size (byte order is converted automatically). This is the fastest entry point. For memory-mapped arrays, call Close when done.

func Read 

func Read[T Element](r io.Reader) (*NDArray[T], error)

Read reads a .npy stream into a statically-typed NDArray[T].

func ZipValues 

func ZipValues[T Element](a *Archive, name string) (*NDArray[T], error)

ZipValues reads the named array from the archive into a statically-typed NDArray[T]. T must match the entry's element kind and size.

func (*NDArray[T]) Close 

func (n *NDArray[T]) Close() error

Close releases resources for memory-mapped arrays; otherwise a no-op. After Close the Values slice of a memory-mapped array must not be used.

func (*NDArray[T]) Len 

func (n *NDArray[T]) Len() int

Len returns the total number of elements.

type NamedArray 

type NamedArray struct {
	Name  string
	Data  any   // a supported element slice, e.g. []float64
	Shape []int // nil means a 1-D array of length len(Data)
}

NamedArray pairs an array name with its data and shape for SaveZip.

type Option 

type Option func(*config)

Option customises read and write behaviour.

func WithCompression 

func WithCompression(compress bool) Option

WithCompression enables DEFLATE compression of .npz archive entries (like numpy.savez_compressed). It has no effect on single .npy files. The default is no compression (like numpy.savez).

func WithFortran 

func WithFortran(fortran bool) Option

WithFortran requests Fortran (column-major) ordering when writing.

func WithMaxRAMFraction 

func WithMaxRAMFraction(f float64) Option

WithMaxRAMFraction sets the fraction of system memory above which Auto mode switches from reading into memory to memory-mapping. The default is 0.5.

func WithMode 

func WithMode(m AccessMode) Option

WithMode selects the access mode used when reading from a file path.

type ZipWriter 

type ZipWriter struct {
	// contains filtered or unexported fields
}

ZipWriter incrementally writes arrays to a .npz archive. This is useful when arrays are produced one at a time and need not all be held in memory. 

zw, _ := npy.CreateZip("out.npz")
npy.AddTyped(zw, "a", []float64{1, 2, 3}, nil)
zw.Add("b", []int32{4, 5, 6}, []int{3})
zw.Close()

func CreateZip 

func CreateZip(path string, opts ...Option) (*ZipWriter, error)

CreateZip creates a .npz archive at path for incremental writing.

func (*ZipWriter) Add 

func (zw *ZipWriter) Add(name string, data any, shape []int) error

Add writes a dynamically-typed array to the archive under name.

func (*ZipWriter) Close 

func (zw *ZipWriter) Close() error

Close finalises the archive and closes the underlying file.

Source Files

View all Source files

Directories

Path Synopsis
Command example demonstrates reading and writing .npy files.
 Command example demonstrates reading and writing .npy files.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.