zarr

README

zarr-go

A pure-Go implementation of the Zarr chunked, compressed, N-dimensional array format. Read/write for v3, read for v2, with pluggable codecs and storage backends.

• Typed arrays. Array[T] is generic over the element type — no interface{} round-trips on the hot path.
• Pluggable codecs. bytes, transpose, gzip, zlib, zstd, blosc, crc32c, and sharding_indexed (with partial decode and range coalescing).
• Pluggable stores. Local filesystem (with optional mmap fast path), in-memory, HTTP, S3, GCS, Azure Blob, plus an LRU cache decorator.
• Concurrent I/O. Per-chunk fetch+decode goroutines, bounded by user options.
• Safety by default. Atomic creates, decompression-bomb cap, fill-aware partial decode, lossless metadata round-trip.

Install

go get github.com/exergy-dev/zarr-go

Requires Go 1.25 or newer.

The S3, GCS and Azure Blob stores are separate modules, so their SDKs only enter your build if you use them. Add the ones you need:

go get github.com/exergy-dev/zarr-go/store/s3
go get github.com/exergy-dev/zarr-go/store/gcs
go get github.com/exergy-dev/zarr-go/store/azblob

The import paths are the same as the module paths. Every other store (fs, mem, http, cache) ships in the root module.

Quick start

Read

package main

import (
	"context"
	"fmt"

	zarr "github.com/exergy-dev/zarr-go"
	"github.com/exergy-dev/zarr-go/store/fs"
)

func main() {
	ctx := context.Background()

	st, err := fs.New("/data/example.zarr")
	if err != nil { panic(err) }

	a, err := zarr.OpenArray[float32](ctx, st, "")
	if err != nil { panic(err) }

	// Read the top-left 1024×1024 tile of a 2-D array.
	sel := zarr.Selection{{Start: 0, Stop: 1024}, {Start: 0, Stop: 1024}}
	out := make([]float32, sel.NumElements())
	if err := a.Read(ctx, sel, out); err != nil {
		panic(err)
	}
	fmt.Println("first element:", out[0])
}

A Selection has one [Start, Stop) range per dimension; Step strides (0 and 1 both mean every element). zarr.All(shape) covers the whole array. Open (instead of OpenArray[T]) returns a Node to type-switch on when the dtype is only known at run time, and Array.Stream yields chunks one at a time when the selection would not fit in memory.

Write

package main

import (
	"context"
	"math"

	zarr "github.com/exergy-dev/zarr-go"
	"github.com/exergy-dev/zarr-go/codec/zstd"
	"github.com/exergy-dev/zarr-go/store/fs"
)

func main() {
	ctx := context.Background()
	st, err := fs.New("/data/out.zarr") // created if missing
	if err != nil { panic(err) }

	shape := []int64{4096, 4096}
	chunk := []int64{512, 512}

	a, err := zarr.CreateArray[float32](ctx, st, "", shape, chunk,
		zarr.WithCodecs(zstd.Spec(3)),
		zarr.WithFillValue(float32(math.NaN())),
	)
	if err != nil { panic(err) }

	data := make([]float32, shape[0]*shape[1])
	// ... fill data ...
	if err := a.Write(ctx, zarr.All(shape), data); err != nil {
		panic(err)
	}
}

WithCodecs takes the pipeline in metadata order; when it has no array→bytes codec, a little-endian bytes codec is inserted, so a bare compressor is enough. WithFillValue takes a Go value of the element kind (NaN and ±Inf included) or a json.RawMessage in the v3 form. More runnable examples are in example_test.go.

Stores

Backend	Package	Notes
Local filesystem	store/fs	fs.WithMmap() for zero-copy reads
In-memory	store/mem	Tests, transient pipelines
HTTP (read-only)	store/http	Range GETs
S3	store/s3 (own module)	aws-sdk-go-v2; If-None-Match for atomic create
Google Cloud Storage	store/gcs (own module)
Azure Blob	store/azblob (own module)
LRU cache decorator	store/cache	Bytes + entry caps, coalesces concurrent misses

Every store satisfies store.Store, the read side: Get, GetRange, GetSuffix (the last n bytes, which is how a sharded read fetches a shard's index), Exists and List. Writable backends also satisfy store.Writer (Set, Delete, SetIfNotExists); CreateArray, CreateGroup and Array.Write return zarr.ErrReadOnly (the same value as store.ErrReadOnly) for a store that isn't one, such as the HTTP store. Keys a backend refuses to address (empty, or escaping its root) fail with store.ErrInvalidKey.

cache.New returns a read-only cache; cache.NewWriter wraps a store.Writer, writes through, and invalidates the written keys.

import (
    "github.com/exergy-dev/zarr-go/store/cache"
    "github.com/exergy-dev/zarr-go/store/s3"
)

base, _ := s3.New(ctx, s3.Config{Bucket: "my-bucket", Prefix: "datasets/"})
st := cache.NewWriter(base, cache.Config{
    MaxBytes:   512 << 20, // 512 MiB
    MaxEntries: 4096,
})

Codecs

All v3 core codecs plus the optional blosc codec (CGO, opt-in) ship in sub-packages of codec/, each with a Spec constructor for WithCodecs:

Codec	Spec	Kind
transpose	transpose.Spec(order ...int)	array→array
bytes	bytes.Spec(bytes.Big) (little is implied)	array→bytes
sharding_indexed	sharding.Spec(innerChunks, codecs...)	array→bytes
gzip	gzip.Spec(level)	bytes→bytes
zlib	zlib.Spec(level)	bytes→bytes
zstd	zstd.Spec(level)	bytes→bytes
crc32c	crc32c.Spec()	bytes→bytes
blosc	codec.Spec{Name: "blosc", Configuration: ...}	bytes→bytes

// 1024×1024 shards of 64×64 zstd-compressed inner chunks, each checksummed.
zarr.CreateArray[float32](ctx, st, "", shape, []int64{1024, 1024},
    zarr.WithCodecs(sharding.Spec([]int64{64, 64}, zstd.Spec(3), crc32c.Spec())))

Any codec.Spec works, so a configuration a constructor doesn't expose (zstd checksums, a shard index at the start) is one literal away. Register a custom codec with codec.Register(name, factory); see codec_register.go for the defaults.

Reading a sub-region of a sharded array fetches only the inner chunks it intersects (codec.PartialDecoder), coalescing nearby byte ranges into one request.

Decompression cap

A global cap on decompressed output guards gzip, zlib, zstd, and blosc against decompression bombs — a small encoded chunk that declares (or expands to) an enormous decoded size. Default 1 GiB; tune with:

zarr.SetMaxDecodedBytes(8 << 30) // 8 GiB
zarr.SetMaxDecodedBytes(0)       // disable

Concurrency

Read and Write each process the chunks they touch concurrently, and Read is safe to call from multiple goroutines. Write does not lock at the chunk level — two concurrent Write calls that touch the same chunk can lose updates, and a concurrent read may observe a half-written chunk. Serialize at the application layer if needed (one writer per array, or disjoint chunk ranges per writer).

Tunables:

zarr.OpenArray[float32](ctx, st, "",
    zarr.WithConcurrency(8),    // max concurrent chunk decodes / encodes
    zarr.WithIOConcurrency(16), // max in-flight store requests
    zarr.WithRangeGap(4 << 20), // sharded range-coalescing gap
)

These are OpenOptions, and every OpenOption is also a CreateOption, so the same calls can be passed to CreateArray to tune the array it returns.

Error sentinels

Sentinel	Meaning
store.ErrKeyNotFound	Key absent (wraps os.ErrNotExist).
store.ErrAlreadyExists	CreateArray / CreateGroup lost a race (wraps os.ErrExist). Pass WithOverwrite() to clobber.
crc32c.ErrChecksumMismatch	Stored CRC32C trailer disagrees with body.

All work with errors.Is.

Versions

• v3 read/write. Full pipeline, sharding (both index_location=end and index_location=start), CRC32C, dimension_names, attribute round-trip, unknown-field preservation in ArrayMeta.Extra / GroupMeta.Extra.
• v2 read. .zarray / .zgroup / .zattrs are honored when no zarr.json is present at the path.

Testing

go build ./...
go test ./...
go test -race ./...

./... stops at module boundaries, so the commands above skip the nested store/s3, store/gcs and store/azblob modules. make test, make race, make vet and make lint run over all four modules.

Each cloud backend's offline tests run against an in-process fake of its object API. Its integration tests are skipped unless the backend's credential env vars are set; see each store/<backend>/*_test.go.

The Python conformance fixtures under testdata/gen/ are generated with zarr-python (testdata/gen/generate.py for v3, generate_v2.py for v2).

Contributing

PRs welcome. Please run go test ./... and go vet ./... before submitting, and prefer adding a test that demonstrates the bug or the new behavior. Changes are recorded in CHANGELOG.md; the tagging procedure for the four modules is in RELEASING.md.

License

MIT.

Documentation

Overview

Package zarr reads and writes Zarr arrays: chunked, compressed, N-dimensional arrays stored as key/value objects. It implements Zarr v3 (read and write) and reads Zarr v2.

An array lives in a store.Store: a directory (store/fs), memory (store/mem), an HTTP server (store/http), or an object store (store/s3, store/gcs, store/azblob, each its own module). OpenArray opens an existing array as an Array whose element type T matches its dtype; CreateArray makes a new one:

st, err := fs.New("/data/out.zarr")
a, err := zarr.CreateArray[float32](ctx, st, "", []int64{4096, 4096}, []int64{512, 512},
	zarr.WithCodecs(zstd.Spec(3)))
err = a.Write(ctx, zarr.All(a.Shape()), data)

Array.Read and Array.Write move a Selection, a strided hyperrectangle, between the array and a flat row-major slice, fetching or storing the chunks it touches concurrently. Array.Stream yields chunks one at a time for data that does not fit in memory.

Compression and other codecs are configured by WithCodecs from the Spec constructor in each codec package (gzip.Spec, zstd.Spec, sharding.Spec, ...). Every codec in the v3 specification ships in a subpackage of codec and is registered by importing this package.

Example

Create a compressed array, write it, and read a window back.

package main

import (
	"context"
	"fmt"
	"log"
	"math"

	zarr "github.com/exergy-dev/zarr-go"
	"github.com/exergy-dev/zarr-go/codec/zstd"
	"github.com/exergy-dev/zarr-go/store/mem"
)

func main() {
	ctx := context.Background()
	st := mem.New() // or fs.New(dir) for a directory on disk

	shape, chunks := []int64{4, 6}, []int64{2, 3}
	a, err := zarr.CreateArray[float32](ctx, st, "temperature", shape, chunks,
		zarr.WithCodecs(zstd.Spec(3)),
		zarr.WithFillValue(float32(math.NaN())),
		zarr.WithDimensionNames([]string{"y", "x"}),
	)
	if err != nil {
		log.Fatal(err)
	}

	data := make([]float32, 4*6)
	for i := range data {
		data[i] = float32(i)
	}
	if err := a.Write(ctx, zarr.All(shape), data); err != nil {
		log.Fatal(err)
	}

	b, err := zarr.OpenArray[float32](ctx, st, "temperature")
	if err != nil {
		log.Fatal(err)
	}
	// Rows 1-2, columns 2-4.
	sel := zarr.Selection{{Start: 1, Stop: 3}, {Start: 2, Stop: 5}}
	out := make([]float32, sel.NumElements())
	if err := b.Read(ctx, sel, out); err != nil {
		log.Fatal(err)
	}
	fmt.Println(b.Shape(), b.Chunks(), b.Dtype(), b.DimensionNames())
	fmt.Println(out)
}

Output:
[4 6] [2 3] float32 [y x]
[8 9 10 14 15 16]

Index

• Variables
• func ChunksIntersecting(arrayShape, chunkShape []int64, sel Selection) iter.Seq[ChunkCoord]
• func Float16ToFloat32(h Float16) float32
• func MaxDecodedBytes() int64
• func SetMaxDecodedBytes(n int64)
• type Array
  • func CreateArray[T Numeric](ctx context.Context, s store.Store, path string, shape, chunks []int64, ...) (*Array[T], error)
  • func OpenArray[T Numeric](ctx context.Context, s store.Store, path string, opts ...OpenOption) (*Array[T], error)
  • func (a *Array[T]) Attrs() map[string]any
  • func (a *Array[T]) Chunks() []int64
  • func (a *Array[T]) DimensionNames() []string
  • func (a *Array[T]) Dtype() Dtype
  • func (a *Array[T]) FillValue() T
  • func (a *Array[T]) Path() string
  • func (a *Array[T]) Read(ctx context.Context, sel Selection, dst []T) error
  • func (a *Array[T]) Shape() []int64
  • func (a *Array[T]) Stream(ctx context.Context, sel Selection) iter.Seq2[Chunk[T], error]
  • func (a *Array[T]) Write(ctx context.Context, sel Selection, src []T) error
• type Chunk
• type ChunkCoord
  • func (c ChunkCoord) Clone() ChunkCoord
• type CreateOption
  • func WithAttributes(attrs map[string]any) CreateOption
  • func WithChunkKeySeparator(sep string) CreateOption
  • func WithCodecs(specs ...codec.Spec) CreateOption
  • func WithDimensionNames(names []string) CreateOption
  • func WithFillValue(v any) CreateOption
  • func WithOverwrite() CreateOption
  • func WithV2ChunkKeys() CreateOption
• type DecodeError
  • func (e *DecodeError) Error() string
  • func (e *DecodeError) Unwrap() error
• type DimRange
• type Dtype
  • func DtypeOf[T Numeric]() Dtype
  • func ParseDtype(name string) (Dtype, error)
  • func (d Dtype) Size() int
  • func (d Dtype) String() string
• type Float16
• type Group
  • func CreateGroup(ctx context.Context, s store.Store, path string, opts ...CreateOption) (*Group, error)
  • func (g *Group) Attrs() map[string]any
  • func (g *Group) CreateGroup(ctx context.Context, name string, opts ...CreateOption) (*Group, error)
  • func (g *Group) Path() string
• type Node
  • func Open(ctx context.Context, s store.Store, path string, opts ...OpenOption) (Node, error)
• type Numeric
• type OpenOption
  • func WithConcurrency(n int) OpenOption
  • func WithIOConcurrency(n int) OpenOption
  • func WithRangeGap(b int64) OpenOption
• type Selection
  • func All(shape []int64) Selection
  • func (s Selection) NumElements() int64
  • func (s Selection) Validate(shape []int64) error

Examples

• Package
• Array.Stream
• CreateGroup
• Open

Variables

var (
	// ErrCodecRequiresCGO is raised at array-open time when the metadata
	// references a codec the current build can't handle (e.g., Blosc on
	// the default no-cgo build). It aliases codec.ErrRequiresCGO so callers
	// can errors.Is against either form.
	ErrCodecRequiresCGO = codec.ErrRequiresCGO

	// ErrShapeMismatch is returned when the caller's destination slice is
	// the wrong size for the requested selection.
	ErrShapeMismatch = errors.New("shape mismatch")

	// ErrUnsupportedDtype is returned when metadata declares a dtype that
	// isn't in the Numeric constraint set.
	ErrUnsupportedDtype = errors.New("unsupported dtype")

	// ErrCorruptMetadata is returned when a zarr.json/.zarray/.zgroup
	// document parses as JSON but doesn't satisfy the spec (e.g., a chunk
	// shape with a zero dimension, a missing required field, an
	// unrecognized must_understand extension). Open/OpenArray wrap the
	// underlying internal/metadata error with this sentinel.
	ErrCorruptMetadata = errors.New("corrupt metadata")

	// ErrUnknownCodec is returned at pipeline construction when metadata
	// references a codec name that isn't in the registry. It aliases
	// codec.ErrUnknownCodec so callers can errors.Is against either form.
	ErrUnknownCodec = codec.ErrUnknownCodec

	// ErrReadOnly is returned by Array.Write, CreateArray, and CreateGroup
	// when the store is not a store.Writer. It aliases store.ErrReadOnly.
	ErrReadOnly = store.ErrReadOnly
)

Sentinel errors. Wrap with fmt.Errorf("...: %w", ...) at boundaries.

Functions

func ChunksIntersecting

func ChunksIntersecting(arrayShape, chunkShape []int64, sel Selection) iter.Seq[ChunkCoord]

ChunksIntersecting yields each chunk coordinate whose chunk overlaps the selection. Coordinates are emitted in row-major (C) order.

The yielded slice is reused across iterations; clone it if retained.

func Float16ToFloat32

func Float16ToFloat32(h Float16) float32

Float16ToFloat32 expands a half-precision float to single precision.

func MaxDecodedBytes

func MaxDecodedBytes() int64

MaxDecodedBytes returns the cap currently set by SetMaxDecodedBytes.

func SetMaxDecodedBytes

func SetMaxDecodedBytes(n int64)

SetMaxDecodedBytes sets the maximum number of bytes any single codec invocation (gzip, zlib, zstd, blosc, ...) may decompress a chunk or shard index trailer into. This guards against decompression bombs: a small compressed payload whose header declares a huge decoded size. Pass 0 to disable the cap. The default is 1 GiB. Safe to call concurrently; the new value applies to subsequent codec invocations, except that a zstd codec fixes its decoder memory limit when built, at array open (see zstd.New).

Types

type Array

type Array[T Numeric] struct {
	// contains filtered or unexported fields
}

Array is the typed view of a Zarr array. Construct via OpenArray[T].

Concurrency: Array[T].Read is internally concurrent (it dispatches per-chunk fetch+decode goroutines bounded by WithConcurrency / WithIOConcurrency) and is safe to call from multiple goroutines on the same Array.

Array[T].Write performs read-modify-write on each touched chunk and does NOT take a chunk-level lock. Concurrent Writes that touch the same chunk can lose updates; concurrent Read+Write may observe a torn intermediate state. Callers that need either pattern must serialize at the application layer (one goroutine writing per array, or a per-chunk mutex).

func CreateArray

func CreateArray[T Numeric](
	ctx context.Context,
	s store.Store,
	path string,
	shape, chunks []int64,
	opts ...CreateOption,
) (*Array[T], error)

CreateArray creates a new v3 array under path with the given shape and chunk shape. shape/chunks may both be empty to create a rank-0 (scalar) array. The store must implement store.Writer.

func OpenArray

func OpenArray[T Numeric](ctx context.Context, s store.Store, path string, opts ...OpenOption) (*Array[T], error)

OpenArray opens the array at path and asserts its dtype matches T. Detects v3 first; falls back to v2 if zarr.json is absent.

func (*Array[T]) Attrs

func (a *Array[T]) Attrs() map[string]any

Attrs returns a copy of the array's user attributes (v3 "attributes", or the v2 .zattrs sidecar), or nil if it has none.

func (*Array[T]) Chunks

func (a *Array[T]) Chunks() []int64

Chunks returns the array's chunk shape.

func (*Array[T]) DimensionNames

func (a *Array[T]) DimensionNames() []string

DimensionNames returns a copy of the array's v3 dimension_names, or nil if it has none (always nil for v2 arrays).

func (*Array[T]) Dtype

func (a *Array[T]) Dtype() Dtype

Dtype returns the array's element type.

func (*Array[T]) FillValue

func (a *Array[T]) FillValue() T

FillValue returns the value an unwritten chunk reads as.

func (*Array[T]) Path

func (a *Array[T]) Path() string

Path returns the array's store path ("" for an array at the store root).

func (*Array[T]) Read

func (a *Array[T]) Read(ctx context.Context, sel Selection, dst []T) error

Read materializes the selection into dst. dst must hold sel.NumElements() elements; layout is row-major (C order).

func (*Array[T]) Shape

func (a *Array[T]) Shape() []int64

Shape returns the array's logical shape.

func (*Array[T]) Stream

func (a *Array[T]) Stream(ctx context.Context, sel Selection) iter.Seq2[Chunk[T], error]

Stream yields decoded chunks intersecting sel, one whole chunk at a time, without assembling the full output buffer. Chunks are fetched and decoded sequentially; a missing chunk is yielded filled with the fill value.

Each yielded Chunk has its own freshly allocated Data, which the iterator never touches again: callers may retain it without copying.

Example

Stream an array chunk by chunk instead of materializing the selection.

package main

import (
	"context"
	"fmt"
	"log"

	zarr "github.com/exergy-dev/zarr-go"
	"github.com/exergy-dev/zarr-go/store/mem"
)

func main() {
	ctx := context.Background()
	st := mem.New()
	shape, chunks := []int64{5}, []int64{2}
	a, err := zarr.CreateArray[int32](ctx, st, "", shape, chunks, zarr.WithFillValue(-1))
	if err != nil {
		log.Fatal(err)
	}
	// Only the first chunk is written; the others read as the fill value.
	if err := a.Write(ctx, zarr.Selection{{Start: 0, Stop: 2}}, []int32{10, 11}); err != nil {
		log.Fatal(err)
	}

	for c, err := range a.Stream(ctx, zarr.All(shape)) {
		if err != nil {
			log.Fatal(err)
		}
		fmt.Println(c.Coord, c.Shape, c.Data)
	}
}

Output:
[0] [2] [10 11]
[1] [2] [-1 -1]
[2] [1] [-1]

func (*Array[T]) Write

func (a *Array[T]) Write(ctx context.Context, sel Selection, src []T) error

Write encodes the elements in src and writes them to the chunks intersecting sel. src must hold sel.NumElements() elements; layout is row-major.

Chunks are written concurrently, bounded like Read by WithConcurrency (encode) and WithIOConcurrency (store traffic). Each touched chunk is read-modify-written: if the existing chunk exists it is fetched and merged with src in the intersecting region, otherwise a new chunk filled with the array's fill value is created. Chunks that the selection covers exactly (the whole chunk shape) skip the read.

type Chunk

type Chunk[T Numeric] struct {
	Coord []int64
	Shape []int64
	Data  []T
}

Chunk is one decoded chunk yielded from Stream. Shape is clipped to the array bounds and Data holds exactly product(Shape) elements in row-major order. Data is freshly allocated for each chunk and owned by the caller.

type ChunkCoord

type ChunkCoord []int64

ChunkCoord identifies one chunk in the chunk grid.

func (ChunkCoord) Clone

func (c ChunkCoord) Clone() ChunkCoord

Clone returns a copy of c so the caller can retain it across iterator steps.

type CreateOption

type CreateOption interface {
	// contains filtered or unexported methods
}

CreateOption customizes CreateArray and CreateGroup. WithAttributes and WithOverwrite apply to both; every other option only means something for an array, and CreateGroup returns an error naming it rather than silently ignoring it.

Every OpenOption is also a CreateOption: passed to CreateArray, it configures the returned Array exactly as it would for OpenArray.

func WithAttributes

func WithAttributes(attrs map[string]any) CreateOption

WithAttributes attaches user attributes to the new node. The map is shallow-copied so later mutations by the caller don't alter the node's in-memory or already-persisted metadata.

func WithChunkKeySeparator

func WithChunkKeySeparator(sep string) CreateOption

WithChunkKeySeparator sets the separator used between chunk-coordinate components. The default chunk_key_encoding's default separator is "/"; WithV2ChunkKeys defaults it to ".". Call this after WithV2ChunkKeys in the options list to override that default.

func WithCodecs

func WithCodecs(specs ...codec.Spec) CreateOption

WithCodecs sets the array's codec pipeline, in the order the v3 metadata lists it: array→array codecs (transpose), then the array→bytes codec (bytes or sharding_indexed), then bytes→bytes codecs (compression, crc32c). Each codec package has a Spec constructor:

zarr.WithCodecs(zstd.Spec(3))
zarr.WithCodecs(sharding.Spec([]int64{64, 64}, zstd.Spec(3), crc32c.Spec()))

When the list has no array→bytes codec, a little-endian bytes codec is inserted in front of the first bytes→bytes codec, so a bare compressor is enough. The default pipeline is that bytes codec alone (no compression).

func WithDimensionNames

func WithDimensionNames(names []string) CreateOption

WithDimensionNames labels the array's dimensions.

func WithFillValue

func WithFillValue(v any) CreateOption

WithFillValue sets the value a chunk that was never written reads as. The default is zero (false for bool). v is a Go value of the array's element kind: a bool, any integer type, any float type (NaN and ±Inf included), Float16, or a complex type; or a json.RawMessage holding the fill value in its v3 metadata form, such as a hex bit pattern ("0x7fc00000") for a specific NaN payload. The value must be representable in the array's dtype: a fractional or out-of-range value is reported by CreateArray.

func WithOverwrite

func WithOverwrite() CreateOption

WithOverwrite makes CreateArray / CreateGroup unconditionally overwrite any existing zarr.json at the target path. Without it, concurrent creates use SetIfNotExists and the loser receives store.ErrAlreadyExists.

For CreateArray, an overwrite also deletes every existing key under path (chunks included) before the new metadata is written, so re-creating an array with a different shape/dtype/chunking at the same path can't leave stale chunk bytes that the new metadata's codec pipeline would decode as garbage.

func WithV2ChunkKeys

func WithV2ChunkKeys() CreateOption

WithV2ChunkKeys selects the v3 "v2"-compatible chunk_key_encoding (e.g. "arr/0.1" instead of the default "arr/c/0/1"), matching the key layout Zarr v2 stores use, and defaults the separator to "." (v2's convention). Call WithChunkKeySeparator after this option to use a different separator.

type DecodeError

type DecodeError struct {
	Coord []int64
	Err   error
}

DecodeError is returned from chunk decode failures with enough context to debug a single bad chunk in a 10M-chunk array. Err names the codec that failed (e.g. `codec "gzip": ...`) when the failure came from one.

func (*DecodeError) Error

func (e *DecodeError) Error() string

func (*DecodeError) Unwrap

func (e *DecodeError) Unwrap() error

type DimRange

type DimRange = codec.DimRange

DimRange is the half-open range [Start, Stop) along one dimension, with a non-negative Step. Step 0 is equivalent to Step 1 (every element).

type Dtype

type Dtype uint8

Dtype identifies the on-wire element type of a Zarr array.

const (
	DtypeInvalid Dtype = iota
	DtypeBool
	DtypeInt8
	DtypeInt16
	DtypeInt32
	DtypeInt64
	DtypeUint8
	DtypeUint16
	DtypeUint32
	DtypeUint64
	DtypeFloat16
	DtypeFloat32
	DtypeFloat64
	DtypeComplex64
	DtypeComplex128
)

func DtypeOf

func DtypeOf[T Numeric]() Dtype

DtypeOf returns the Dtype matching the Go type T.

func ParseDtype

func ParseDtype(name string) (Dtype, error)

ParseDtype maps a v3 spec dtype name to a Dtype.

func (Dtype) Size

func (d Dtype) Size() int

Size returns the byte size of one element of the dtype.

func (Dtype) String

func (d Dtype) String() string

String returns the v3 spec name for the dtype.

type Float16

type Float16 uint16

Float16 is the IEEE 754 binary16 type, stored as its raw bit pattern. Use Float16ToFloat32 to convert.

type Group

type Group struct {
	// contains filtered or unexported fields
}

Group is a group node (v3 zarr.json or v2 .zgroup).

func CreateGroup

func CreateGroup(ctx context.Context, s store.Store, path string, opts ...CreateOption) (*Group, error)

CreateGroup creates a new v3 group under path. The store must implement store.Writer. By default this is an atomic create: if a zarr.json already exists at the target the call returns store.ErrAlreadyExists. Pass WithOverwrite() to clobber an existing entry, and WithAttributes to attach user attributes. Any other (array-only) option is an error.

Example

Groups hold arrays and other groups; paths are slash-separated.

package main

import (
	"context"
	"fmt"
	"log"

	zarr "github.com/exergy-dev/zarr-go"
	"github.com/exergy-dev/zarr-go/store/mem"
)

func main() {
	ctx := context.Background()
	st := mem.New()
	g, err := zarr.CreateGroup(ctx, st, "experiment", zarr.WithAttributes(map[string]any{"run": 7}))
	if err != nil {
		log.Fatal(err)
	}
	a, err := zarr.CreateArray[uint8](ctx, st, g.Path()+"/mask", []int64{2, 2}, []int64{2, 2})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(g.Path(), g.Attrs()["run"], a.Path())
}

Output:
experiment 7 experiment/mask

func (*Group) Attrs

func (g *Group) Attrs() map[string]any

Attrs returns the group's user attributes.

func (*Group) CreateGroup

func (g *Group) CreateGroup(ctx context.Context, name string, opts ...CreateOption) (*Group, error)

CreateGroup adds a sub-group named name under g.

func (*Group) Path

func (g *Group) Path() string

Path returns the group's store path.

type Node

type Node interface {
	// contains filtered or unexported methods
}

Node is the union of Group and *Array[T] returned by Open.

Callers typically use OpenArray[T] when the dtype is known at compile time and avoid the type-switch entirely.

func Open

func Open(ctx context.Context, s store.Store, path string, opts ...OpenOption) (Node, error)

Open returns the Node (Group or Array) at path. Detects v3 by reading zarr.json; falls back to v2 (.zarray / .zgroup) if zarr.json is absent.

Example

Open an array whose dtype is not known at compile time.

package main

import (
	"context"
	"fmt"
	"log"

	zarr "github.com/exergy-dev/zarr-go"
	"github.com/exergy-dev/zarr-go/store/mem"
)

func main() {
	ctx := context.Background()
	st := mem.New()
	if _, err := zarr.CreateArray[int16](ctx, st, "counts", []int64{3}, []int64{3}); err != nil {
		log.Fatal(err)
	}

	node, err := zarr.Open(ctx, st, "counts")
	if err != nil {
		log.Fatal(err)
	}
	switch n := node.(type) {
	case *zarr.Array[int16]:
		fmt.Println("int16 array of shape", n.Shape())
	case *zarr.Array[float32]:
		fmt.Println("float32 array of shape", n.Shape())
	case *zarr.Group:
		fmt.Println("group at", n.Path())
	}
}

Output:
int16 array of shape [3]

type Numeric

type Numeric interface {
	bool |
		int8 | int16 | int32 | int64 |
		uint8 | uint16 | uint32 | uint64 |
		Float16 | float32 | float64 |
		complex64 | complex128
}

Numeric is the type constraint for all dtypes the typed Array[T] supports.

bool is included to match the Zarr v3 spec; one byte per element.

Deliberately exact (no ~): DtypeOf switches on T's concrete type, so a defined type like `type MyF32 float32` would satisfy an approximate constraint but map to DtypeInvalid. The exact constraint makes that a compile error instead.

type OpenOption

type OpenOption func(*arrayOptions)

OpenOption customizes Open / OpenArray. It is also a CreateOption, so the same options configure the Array that CreateArray returns.

func WithConcurrency

func WithConcurrency(n int) OpenOption

WithConcurrency caps the number of chunks being decoded (Read) or encoded (Write) at once. The default is the number of CPUs.

func WithIOConcurrency

func WithIOConcurrency(n int) OpenOption

WithIOConcurrency caps the number of in-flight store requests a Read or Write issues. The default is four times the number of CPUs.

func WithRangeGap

func WithRangeGap(b int64) OpenOption

WithRangeGap sets the maximum gap (in bytes) between adjacent ranges that will be coalesced into a single GetRange call. Used by the sharded partial-decode path: inner chunks intersecting a read's region are usually NOT contiguous in the shard body (unneeded inner chunks lie between needed rows), so paying for a few KB of extra payload is usually cheaper than paying for the latency of another round trip on a high-RTT object store. 0 disables coalescing beyond exact adjacency.

type Selection

type Selection []DimRange

Selection identifies a strided hyperrectangle of an array: one DimRange per dimension, so len(sel) must equal the array rank. A zero Step selects every element in [Start, Stop), as does Step 1:

sel := zarr.Selection{{Start: 0, Stop: 1024}, {Start: 0, Stop: 1024}}

All returns the selection covering a whole array.

func All

func All(shape []int64) Selection

All returns a Selection that covers the entire array of the given shape.

func (Selection) NumElements

func (s Selection) NumElements() int64

NumElements returns the count of elements covered by the selection. A rank-0 selection (nil or empty — only valid for a rank-0 array; Validate enforces len(sel) == array rank) covers exactly the one scalar element.

func (Selection) Validate

func (s Selection) Validate(shape []int64) error

Validate checks the selection against the given shape.