hdf5

package module
v0.0.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 20, 2026 License: MIT Imports: 16 Imported by: 0

README

# HDF5 Go 库

HDF5 文件格式的纯 Go 实现 - 无需 CGo

一个现代化的纯 Go 库，用于读取和写入 HDF5 文件，无需 CGo 依赖。兼容 HDF5 2.0.0，生产就绪的读写支持。

✨ 特性

  • ✅ 纯 Go - 无 CGo，无 C 依赖，跨平台
  • ✅ 现代设计 - 使用 Go 1.25+ 最佳实践构建
  • ✅ HDF5 2.0.0 兼容性 - 读写：v0、v2、v3 超级块 | 格式规范 v4.0 带校验和验证
  • ✅ 完整数据集读取 - 紧凑、连续、分块布局，支持 GZIP 压缩
  • ✅ 丰富的数据类型 - 整数、浮点数、字符串（固定/可变长度）、复合类型
  • ✅ 内存高效 - 缓冲区池和智能内存管理
  • ✅ 生产就绪 - 读取支持功能完整
  • ✍️ 全面的写入支持 - 数据集、组、属性 + 智能重新平衡！

🚀 快速开始

安装

go get github.com/huangzhengshun/hdf5

基本用法

package main

import (
    "fmt"
    "log"
    "github.com/huangzhengshun/hdf5"
)

func main() {
    // 打开 HDF5 文件
    file, err := hdf5.Open("data.h5")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    // 遍历文件结构
    file.Walk(func(path string, obj hdf5.Object) {
        switch v := obj.(type) {
        case *hdf5.Group:
            fmt.Printf("📁 %s (%d 个子节点)\n", path, len(v.Children()))
        case *hdf5.Dataset:
            fmt.Printf("📊 %s\n", path)
        }
    })
}

输出:

📁 / (2 个子节点)
📊 /temperature
📁 /experiments/ (3 个子节点)

更多示例 →

📚 文档

入门指南

参考文档

高级主题

⚡ 性能调优

当删除大量属性时，B-tree 可能会变得稀疏（浪费磁盘空间，搜索变慢）。本库提供 4 种重新平衡策略：

1. 默认（无重新平衡）

删除速度快，但 B-tree 可能变得稀疏

// 无选项 = 无重新平衡（类似于 HDF5 C 库）
fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)

适用场景: 仅追加工作负载，小文件（<100MB）

2. 延迟重新平衡（比立即重新平衡快 10-100 倍）

批量处理：达到阈值时重新平衡

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),         // 5% 下溢时触发
        hdf5.LazyMaxDelay(5*time.Minute), // 5 分钟后强制重新平衡
    ),
)

适用场景: 批量删除工作负载，中/大文件（100-500MB）

性能: ~2% 开销，偶尔 100-500ms 暂停

3. 增量重新平衡（零暂停）

后台处理：在后台 goroutine 中重新平衡

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(),  // 前提条件！
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalBudget(100*time.Millisecond),
        hdf5.IncrementalInterval(5*time.Second),
    ),
)
defer fw.Close()  // 停止后台 goroutine

适用场景: 大文件（>500MB），连续操作，TB 级数据

性能: ~4% 开销，零用户可见暂停

4. 智能重新平衡（自动驾驶）

自动调优：库检测工作负载并选择最佳模式

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithSmartRebalancing(
        hdf5.SmartAutoDetect(true),
        hdf5.SmartAutoSwitch(true),
    ),
)

适用场景: 未知工作负载，混合操作，研究环境

性能: ~6% 开销，自动适应

性能对比

模式 删除速度 暂停时间 适用场景
默认 100% (基准) 无 仅追加，小文件
延迟 95%（比立即重新平衡快 10-100 倍！） 100-500ms 批量 批量删除
增量 92% 无（后台） 大文件，连续操作
智能 88% 可变 未知工作负载

了解更多:

🎯 当前状态

HDF5 2.0.0 就绪，库覆盖率 88%+！ 🎉

✅ 完全实现

  • 文件结构:

    • 超级块解析（v0、v2、v3），带校验和验证（CRC32）
    • 对象头 v1（传统 HDF5 < 1.8），带续体
    • 对象头 v2（现代 HDF5 >= 1.8），带续体
    • 组（传统符号表 + 现代对象头）
    • B-tree（大文件的叶子节点 + 非叶子节点）
    • 本地堆（字符串存储）
    • 全局堆（可变长度数据）
    • 分形堆（密集属性的直接块） ✨ 新增

  • 数据集读取:

    • 紧凑布局（数据在对象头中）
    • 连续布局（顺序存储）
    • 带 B-tree 索引的分块布局
    • GZIP/Deflate 压缩
    • LZF 压缩（与 h5py/PyTables 兼容） ✨ 新增
    • 压缩数据的过滤器管道

  • 数据类型（读取 + 写入）:

    • 基本类型: int8-64, uint8-64, float32/64
    • AI/ML 类型: FP8（E4M3、E5M2），bfloat16 - 符合 IEEE 754 标准 ✨ 新增
    • 字符串: 固定长度（null/空格/null 填充），可变长度（通过全局堆）
    • 高级类型: 数组、枚举、引用（对象/区域）、不透明类型
    • 复合类型: 类似结构体，支持嵌套成员

  • 属性:

    • 紧凑属性（在对象头中） ✨ 新增
    • 密集属性（分形堆基础） ✨ 新增
    • 组和数据集的属性读取 ✨ 新增
    • 完整属性 API（Group.Attributes(), Dataset.Attributes()） ✨ 新增

  • 导航: 通过 Walk() 进行完整文件树遍历

  • 代码质量:

    • 测试覆盖率：88%+ 库包（目标：>70%） ✅
    • Lint 问题：0（34+ 个 linters） ✅
    • TODO 项：0（全部已解决） ✅
    • 官方 HDF5 测试套件：433 个文件，100% 通过 ✅

  • 安全性 ✨ 新增:

    • 4 个 CVE 已修复（CVE-2025-7067、CVE-2025-6269、CVE-2025-2926、CVE-2025-44905） ✅
    • 全面的溢出保护（SafeMultiply、缓冲区验证） ✅
    • 安全限制：1GB 块、64MB 属性、16MB 字符串 ✅
    • 39 个安全测试用例，全部通过 ✅

✍️ 写入支持 - 功能完整！

生产就绪的写入支持，包含所有功能！ ✅

数据集操作:

  • ✅ 创建数据集（所有布局：连续、分块、紧凑）
  • ✅ 写入数据（所有数据类型，包括复合类型）
  • ✅ 数据集调整大小，支持无限维度
  • ✅ 可变长度数据类型：字符串、不规则数组
  • ✅ 压缩（GZIP、Shuffle、Fletcher32）
  • ✅ 数组和枚举数据类型
  • ✅ 引用和不透明类型
  • ✅ 属性写入（密集 & 紧凑存储）
  • ✅ 属性修改/删除

链接:

  • ✅ 硬链接（完全支持）
  • ✅ 软链接（符号引用 - 完全支持）
  • ✅ 外部链接（跨文件引用 - 完全支持）

读取增强:

  • ✅ Hyperslab 选择（数据切片）- 快 10-250 倍！
  • ✅ 高效的部分数据集读取
  • ✅ 步长和块支持
  • ✅ 块感知读取（仅读取需要的块）
  • ✅ ChunkIterator API - 内存高效的大数据集迭代

验证:

  • ✅ 官方 HDF5 测试套件：100% 通过（378/378 文件）
  • ✅ 生产质量已确认

未来增强:

  • ✅ LZF 过滤器（读取 + 写入，纯 Go） ✨ 新增
  • ✅ BZIP2 过滤器（只读，标准库）
  • ⚠️ SZIP 过滤器（存根 - 需要 libaec）
  • ⚠️ 带互斥锁的线程安全 + SWMR 模式
  • ⚠️ 并行 I/O

❌ 计划中的功能

下一步 - 请参阅 ROADMAP.md 获取完整的时间线和版本策略。

🔧 开发

要求

  • Go 1.25 或更高版本
  • 库无外部依赖

构建

# 克隆仓库
git clone https://github.com/huangzhengshun/hdf5.git
cd hdf5

# 运行测试
go test ./...

# 构建示例
go build ./examples/...

# 构建工具
go build ./cmd/...

测试

# 运行所有测试
go test ./...

# 使用竞态检测器运行
go test -race ./...

# 运行覆盖率测试
go test -coverprofile=coverage.out ./...
go tool cover -html=coverage.out

🤝 贡献

欢迎贡献！这是一个早期阶段的项目，我们非常欢迎您的帮助。

贡献前:

  1. 阅读 CONTRIBUTING.md - Git 工作流程和开发指南
  2. 查看 open issues
  3. 查看 Architecture Overview

贡献方式:

  • 🐛 报告 bug
  • 💡 建议功能
  • 📝 改进文档
  • 🔧 提交 pull requests
  • ⭐ 给项目点赞

🗺️ 与其他库的比较

特性 本库 gonum/hdf5 go-hdf5/hdf5
纯 Go ✅ 是 ❌ CGo 包装器 ✅ 是
读取 ✅ 完整 ✅ 完整 ❌ 有限
写入 ✅ 完整 ✅ 完整 ❌ 无
HDF5 1.8+ ✅ 是 ⚠️ 有限 ❌ 无
高级数据类型 ✅ 全部 ✅ 是 ❌ 无
测试套件验证 ✅ 100% (378/378) ⚠️ 未知 ❌ 无
维护状态 ✅ 活跃 ⚠️ 缓慢 ❌ 不活跃
线程安全 ⚠️ 用户必须同步* ⚠️ 有条件 ❌ 无

* 不同的 File 实例是独立的。对同一 File 的并发访问需要用户同步（标准 Go 实践）。完整的线程安全（带互斥锁 + SWMR 模式）计划在未来版本中实现。

📖 HDF5 资源

📄 许可证

本项目采用 MIT 许可证 - 详见 LICENSE 文件。

🙏 致谢

  • HDF Group 提供的 HDF5 格式规范
  • gonum/hdf5 提供的灵感
  • 本项目的所有贡献者

特别感谢

Ancha Baranova 教授 - 如果没有她宝贵的帮助和支持，这个项目不可能完成。她的协助对于将这个库变为现实至关重要。

📞 支持

状态: 稳定 - 兼容 HDF5 2.0.0，带安全加固

由 HDF5 Go 社区用 ❤️ 构建 获得 HDF Group Forum 认可 ⭐

Documentation

Overview

Package hdf5 provides a pure Go implementation for reading HDF5 files. It supports HDF5 format versions 0, 2, and 3, with capabilities for reading datasets, groups, attributes, and various data layouts.

Index

Constants

View Source 
const (
	// SuperblockV0 (legacy format) - Maximum compatibility with older HDF5 tools.
	// Use this if you need files to be readable by h5dump, older Python h5py, or legacy C library.
	// This format doesn't have checksums but works with all HDF5 tools.
	SuperblockV0 = core.Version0

	// SuperblockV2 (modern format) - Default. Includes checksums for data integrity.
	// This is the recommended format for new files. Supported by HDF5 1.10+.
	SuperblockV2 = core.Version2

	// SuperblockV3 (latest format) - Future format, not yet implemented for writing.
	SuperblockV3 = core.Version3
)

Superblock version constants for file creation.

View Source 
const (
	// KB represents kilobyte size for smart configuration.
	KB = 1024
	// MB represents megabyte size for smart configuration.
	MB = 1024 * KB
	// GB represents gigabyte size for smart configuration.
	GB = 1024 * MB
)
View Source 
const (
	// MaxCompactAttributes is the threshold for transitioning to dense storage.
	// When an object has 8+ attributes, dense storage (Fractal Heap + B-tree)
	// is more efficient than compact storage (object header messages).
	MaxCompactAttributes = 8
)

Attribute storage threshold.

View Source 
const (
	SignatureSNOD = "SNOD" // Symbol table node signature.
)

HDF5 signature constants.

View Source 
const Unlimited uint64 = 0xFFFFFFFFFFFFFFFF

Unlimited represents unlimited dimension size for resizable datasets. Use with WithMaxDims option to allow dimension to grow indefinitely.

Variables

This section is empty.

Functions

This section is empty.

Types

type ChunkIterator 

type ChunkIterator struct {
	// contains filtered or unexported fields
}

ChunkIterator provides memory-efficient iteration over dataset chunks. It reads one chunk at a time, allowing processing of datasets larger than available memory.

Usage: 

iter, err := dataset.ChunkIterator()
if err != nil {
    log.Fatal(err)
}
for iter.Next() {
    chunk, err := iter.Chunk()
    if err != nil {
        log.Fatal(err)
    }
    processChunk(chunk)
}
if err := iter.Err(); err != nil {
    log.Fatal(err)
}

The iterator follows the Go scanner pattern (bufio.Scanner). Only chunked datasets are supported; compact and contiguous datasets should use Read() or ReadSlice() directly.

func (*ChunkIterator) Chunk 

func (it *ChunkIterator) Chunk() (interface{}, error)

Chunk returns the data for the current chunk. Must be called after Next() returns true. Returns the chunk data as interface{} (typically []float64).

func (*ChunkIterator) ChunkCoords 

func (it *ChunkIterator) ChunkCoords() []uint64

ChunkCoords returns the scaled coordinates of the current chunk. These are chunk indices, not element indices. For element indices, multiply by chunk dimensions.

func (*ChunkIterator) ChunkDims 

func (it *ChunkIterator) ChunkDims() []uint64

ChunkDims returns the chunk dimensions.

func (*ChunkIterator) DatasetDims 

func (it *ChunkIterator) DatasetDims() []uint64

DatasetDims returns the dataset dimensions.

func (*ChunkIterator) Err 

func (it *ChunkIterator) Err() error

Err returns any error that occurred during iteration. Should be checked after Next() returns false.

func (*ChunkIterator) Next 

func (it *ChunkIterator) Next() bool

Next advances to the next chunk. Returns false when iteration is complete or an error occurred. Check Err() after iteration to distinguish.

func (*ChunkIterator) OnProgress 

func (it *ChunkIterator) OnProgress(fn func(current, total int))

OnProgress sets a callback function that is called after each Next(). The callback receives the current chunk index (1-based) and total count.

Example: 

iter.OnProgress(func(current, total int) {
    fmt.Printf("Processing chunk %d/%d\n", current, total)
})

func (*ChunkIterator) Progress 

func (it *ChunkIterator) Progress() (current, total int)

Progress returns the current chunk index and total chunk count. Useful for progress reporting.

func (*ChunkIterator) Reset 

func (it *ChunkIterator) Reset()

Reset resets the iterator to the beginning, allowing re-iteration.

func (*ChunkIterator) Total 

func (it *ChunkIterator) Total() int

Total returns the total number of chunks in the dataset.

type CreateMode 

type CreateMode int

CreateMode specifies how to create a new HDF5 file. 

const (
	// CreateTruncate creates a new file, overwriting if it exists.
	// This is the default mode, equivalent to os.Create() behavior.
	CreateTruncate CreateMode = iota

	// CreateExclusive creates a new file, failing if it already exists.
	// Useful when you want to ensure a file doesn't get accidentally overwritten.
	CreateExclusive
)

type Dataset 

type Dataset struct {
	// contains filtered or unexported fields
}

Dataset represents an HDF5 dataset containing multidimensional array data.

func (*Dataset) Address 

func (d *Dataset) Address() uint64

Address returns the object header address (for internal/debugging use).

func (*Dataset) Attributes 

func (d *Dataset) Attributes() ([]*core.Attribute, error)

Attributes returns all attributes attached to this dataset.

func (*Dataset) ChunkIterator 

func (d *Dataset) ChunkIterator() (*ChunkIterator, error)

ChunkIterator returns an iterator for reading dataset chunks one at a time. This is memory-efficient for large chunked datasets.

Returns an error if the dataset is not chunked (compact or contiguous layout). For non-chunked datasets, use Read() or ReadSlice() instead.

func (*Dataset) ChunkIteratorWithContext 

func (d *Dataset) ChunkIteratorWithContext(ctx context.Context) (*ChunkIterator, error)

ChunkIteratorWithContext returns an iterator with context support for cancellation. The context is checked before each Next() call, allowing graceful cancellation.

Example: 

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

iter, err := dataset.ChunkIteratorWithContext(ctx)
for iter.Next() {
    // Process chunk...
}

func (*Dataset) Info 

func (d *Dataset) Info() (string, error)

Info returns metadata about the dataset without reading actual values.

func (*Dataset) ListAttributes 

func (d *Dataset) ListAttributes() ([]string, error)

ListAttributes returns the names of all attributes attached to this dataset.

func (*Dataset) Name 

func (d *Dataset) Name() string

Name returns the dataset's name.

func (*Dataset) Read 

func (d *Dataset) Read() ([]float64, error)

Read reads the dataset values and returns them as float64 array. Currently supports float64, float32, int32, int64 datatypes. All values are converted to float64 for convenience.

func (*Dataset) ReadAttribute 

func (d *Dataset) ReadAttribute(name string) (interface{}, error)

ReadAttribute reads a single attribute by name.

func (*Dataset) ReadCompound 

func (d *Dataset) ReadCompound() ([]core.CompoundValue, error)

ReadCompound reads compound dataset values and returns them as array of maps. Each map represents one compound structure instance with field names as keys. Supports nested compound types, numeric types, and fixed-length strings.

func (*Dataset) ReadHyperslab 

func (d *Dataset) ReadHyperslab(selection *HyperslabSelection) (interface{}, error)

ReadHyperslab reads data with full hyperslab parameters including stride and block. This provides complete control over the selection pattern, allowing strided and blocked selections.

Parameters:

  • selection: The hyperslab selection specification

The selection is validated against the dataset's dimensions before reading.

Example (read every 2nd element in 2D): 

sel := &HyperslabSelection{
    Start:  []uint64{100, 200},
    Count:  []uint64{25, 25},   // 25 blocks
    Stride: []uint64{2, 2},      // Every 2nd element
    Block:  []uint64{1, 1},      // 1x1 blocks
}
data, err := dataset.ReadHyperslab(sel)

Returns:

  • interface{}: The selected data in the dataset's native type
  • error: Error if selection is invalid or reading fails

func (*Dataset) ReadSlice 

func (d *Dataset) ReadSlice(start, count []uint64) (interface{}, error)

ReadSlice reads a rectangular block from the dataset using simple start/count parameters. This is a convenience method for the common case of reading a contiguous rectangular region.

Parameters:

  • start: Starting coordinates in each dimension (0-based)
  • count: Number of elements to read in each dimension

The number of dimensions in start and count must match the dataset's dimensionality.

Example (2D dataset): 

// Read 50x50 block starting at position (100, 200)
data, err := dataset.ReadSlice([]uint64{100, 200}, []uint64{50, 50})

Returns:

  • interface{}: The selected data in the dataset's native type ([]float64, []int32, etc.)
  • error: Error if selection is invalid or reading fails

func (*Dataset) ReadStrings 

func (d *Dataset) ReadStrings() ([]string, error)

ReadStrings reads string dataset values and returns them as string array. Supports fixed-length strings (null-terminated, null-padded, space-padded). Variable-length strings are not yet supported.

func (*Dataset) ReadVLenBytes 

func (d *Dataset) ReadVLenBytes() ([][]byte, error)

ReadVLenBytes reads a variable-length dataset and returns values as [][]byte. Each element in the outer slice corresponds to one dataset element; each inner slice contains the raw bytes of that variable-length sequence.

This works for any VLen datatype (VLenUint8, VLenInt32, VLenString, etc.). For typed sequences the caller must interpret the returned bytes according to the base element type and byte order.

type DatasetOption 

type DatasetOption func(*datasetConfig)

DatasetOption is a functional option for customizing dataset creation.

func WithArrayDims 

func WithArrayDims(dims []uint64) DatasetOption

WithArrayDims sets the dimensions for Array datatypes. This is required when creating an Array dataset.

Array datatypes are fixed-size collections of a base type. The dimensions specify the shape of each array element.

Example: 

// Dataset of shape [10] where each element is [3]int32
ds, _ := fw.CreateDataset("/vectors", hdf5.ArrayInt32, []uint64{10}, hdf5.WithArrayDims([]uint64{3}))

// Dataset of shape [5] where each element is [2][3]float64 (2D array)
ds, _ := fw.CreateDataset("/matrices", hdf5.ArrayFloat64, []uint64{5}, hdf5.WithArrayDims([]uint64{2, 3}))

func WithChunkDims 

func WithChunkDims(dims []uint64) DatasetOption

WithChunkDims enables chunked storage with specified chunk dimensions. When specified, the dataset will use chunked layout instead of contiguous.

Chunk dimensions must match dataset rank and be > 0 in all dimensions. Chunks should be chosen for optimal I/O patterns (typical: 10KB-1MB per chunk).

Example: 

// 2D dataset 1000x2000, chunked as 100x200
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{1000, 2000}, hdf5.WithChunkDims([]uint64{100, 200}))

func WithEnumValues 

func WithEnumValues(names []string, values []int64) DatasetOption

WithEnumValues sets the name-value mappings for Enum datatypes. This is required when creating an Enum dataset.

Enum datatypes map integer values to symbolic names. Both names and values slices must have the same length.

Example: 

// Create enum for days of week (0=Monday, 1=Tuesday, etc.)
names := []string{"Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday", "Sunday"}
values := []int64{0, 1, 2, 3, 4, 5, 6}
ds, _ := fw.CreateDataset("/days", hdf5.EnumInt8, []uint64{100}, hdf5.WithEnumValues(names, values))

func WithFletcher32 

func WithFletcher32() DatasetOption

WithFletcher32 enables Fletcher32 checksum for data integrity verification. This option is only valid for chunked datasets (requires WithChunkDims).

The Fletcher32 filter adds a 4-byte checksum to each chunk, allowing detection of data corruption during storage or transmission.

Overhead:

  • Storage: +4 bytes per chunk (minimal)
  • CPU: Low (faster than CRC32)

Use when:

  • Data integrity is critical
  • Detecting corruption is more important than preventing it
  • Working with unreliable storage or network

Example: 

// Create dataset with compression and checksum
ds, _ := fw.CreateDataset("/data", hdf5.Int32, []uint64{1000},
    hdf5.WithChunkDims([]uint64{100}),
    hdf5.WithGZIPCompression(6),
    hdf5.WithFletcher32())

func WithGZIPCompression 

func WithGZIPCompression(level int) DatasetOption

WithGZIPCompression enables GZIP compression with specified level (1-9). This option is only valid for chunked datasets (requires WithChunkDims).

Compression levels: 

1 = fastest compression, larger files
6 = balanced (default if invalid level)
9 = best compression, slower

GZIP compression reduces storage size but adds CPU overhead during read/write. Best used with repetitive or structured data.

Example: 

// Create compressed dataset with level 6 compression
ds, _ := fw.CreateDataset("/data", hdf5.Int32, []uint64{1000},
    hdf5.WithChunkDims([]uint64{100}),
    hdf5.WithGZIPCompression(6))

func WithMaxDims 

func WithMaxDims(maxDims []uint64) DatasetOption

WithMaxDims sets maximum dimensions for resizable datasets. Use hdf5.Unlimited (0xFFFFFFFFFFFFFFFF) for unlimited dimensions. Requires chunked layout (use WithChunkDims).

The maxDims slice must have the same length as the dataset dimensions. Each maxDim value must be >= the corresponding dimension, or Unlimited.

Example: 

// 1D dataset with unlimited dimension
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{10},
    hdf5.WithChunkDims([]uint64{5}),
    hdf5.WithMaxDims([]uint64{hdf5.Unlimited}))

// 2D dataset with one unlimited dimension
ds2, _ := fw.CreateDataset("/matrix", hdf5.Float64, []uint64{10, 20},
    hdf5.WithChunkDims([]uint64{5, 10}),
    hdf5.WithMaxDims([]uint64{hdf5.Unlimited, 20}))  // Rows unlimited, cols fixed

func WithOpaqueTag 

func WithOpaqueTag(tag string, size uint32) DatasetOption

WithOpaqueTag sets the tag and size for Opaque datatypes. This is required when creating an Opaque dataset.

Opaque datatypes are uninterpreted byte sequences with a descriptive tag. The tag describes the content (e.g., "JPEG image", "binary blob"). The size specifies the number of bytes per element.

Example: 

// Dataset of 10 JPEG images, each 1MB
ds, _ := fw.CreateDataset("/images", hdf5.Opaque, []uint64{10}, hdf5.WithOpaqueTag("JPEG image", 1024*1024))

func WithShuffle 

func WithShuffle() DatasetOption

WithShuffle enables byte shuffle filter (improves compression). This option is only valid for chunked datasets (requires WithChunkDims).

The shuffle filter reorders bytes to group similar values, significantly improving compression ratios for numeric data (typically 2-10x better).

Shuffle should be combined with compression (e.g., GZIP) to be effective. It's automatically placed before compression in the filter pipeline.

Best for:

  • Integer arrays with slowly changing values
  • Floating-point arrays with similar magnitudes
  • Multi-dimensional arrays with spatial locality

Example: 

// Create dataset with shuffle+compression for best compression
ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{1000},
    hdf5.WithChunkDims([]uint64{100}),
    hdf5.WithShuffle(),
    hdf5.WithGZIPCompression(9))

func WithStringSize 

func WithStringSize(size uint32) DatasetOption

WithStringSize sets the fixed string size for String datasets. This is required when creating a String dataset.

Example: 

ds, _ := fw.CreateDataset("/names", hdf5.String, []uint64{10}, hdf5.WithStringSize(32))

type DatasetWriter 

type DatasetWriter struct {
	// contains filtered or unexported fields
}

DatasetWriter provides write access to a dataset.

func (*DatasetWriter) Close 

func (dw *DatasetWriter) Close() error

Close closes the dataset writer. For MVP, this is a no-op (no per-dataset resources to release).

func (*DatasetWriter) DeleteAttribute 

func (ds *DatasetWriter) DeleteAttribute(name string) error

DeleteAttribute removes an attribute by name from the dataset.

This method supports both compact and dense attribute storage: - Compact storage (0-7 attributes): Removes message from object header - Dense storage (8+ attributes): Removes from B-tree and fractal heap

Parameters:

  • name: Attribute name to delete

Returns:

  • error: If attribute not found or deletion fails

Reference: H5Adelete.c - H5A__delete(), H5Adense.c - H5A__dense_remove().

func (*DatasetWriter) RebalanceAttributeBTree 

func (ds *DatasetWriter) RebalanceAttributeBTree() error

RebalanceAttributeBTree manually triggers B-tree rebalancing for this dataset's dense attribute storage.

Use this when:

  • You know this specific dataset needs rebalancing
  • More efficient than RebalanceAllBTrees() for targeted optimization
  • After batch deletions with rebalancing disabled

Performance (for current MVP with single-leaf B-trees):

  • Instant (< 1ms) - no-op for single-leaf trees

Future (when multi-level B-trees implemented):

  • Small (<1000 attrs): <10ms
  • Medium (1000-10000 attrs): 10-100ms
  • Large (10000+ attrs): 100ms-1s

Returns:

  • error: if dataset doesn't use dense storage or rebalancing fails

Example: 

fw.DisableRebalancing()
for i := 0; i < 1000; i++ {
    ds.DeleteAttribute(fmt.Sprintf("temp_%d", i))  // Fast deletions
}
ds.RebalanceAttributeBTree()  // Rebalance this dataset only

Reference: Similar to per-object rebalancing in HDF5 (hypothetical - not exposed in C API).

func (*DatasetWriter) Resize 

func (dw *DatasetWriter) Resize(newDims []uint64) error

Resize changes the dimensions of a dataset. The dataset must have been created with maxDims (using WithMaxDims option). Requires chunked layout. newDims must be <= maxDims for each dimension.

When extending (growing), new space is initialized with zeros. When shrinking, data beyond new dimensions is lost.

Example: 

ds, _ := fw.CreateDataset("/data", hdf5.Float64, []uint64{10},
    hdf5.WithChunkDims([]uint64{5}),
    hdf5.WithMaxDims([]uint64{hdf5.Unlimited}))
ds.Resize([]uint64{20})  // Extend to 20 elements

func (*DatasetWriter) Write 

func (dw *DatasetWriter) Write(data interface{}) error

Write writes data to the dataset. The data must match the dataset's datatype and dimensions.

Parameters:

  • data: Data to write (type must match dataset datatype)

Supported types:

  • []int8, []int16, []int32, []int64
  • []uint8, []uint16, []uint32, []uint64
  • []float32, []float64
  • []string (for fixed-length string datasets)

For multi-dimensional datasets, data should be flattened in row-major order.

Example: 

// 1D dataset
ds, _ := fw.CreateDataset("/data", hdf5.Int32, []uint64{5})
ds.Write([]int32{1, 2, 3, 4, 5})

// 2D dataset (3x4 matrix)
ds2, _ := fw.CreateDataset("/matrix", hdf5.Float64, []uint64{3, 4})
// Flatten row-major: [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
ds2.Write([]float64{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12})

func (*DatasetWriter) WriteAttribute 

func (ds *DatasetWriter) WriteAttribute(name string, value interface{}) error

WriteAttribute writes an attribute to a dataset.

Storage strategy (automatic):

  • 0-7 attributes: Compact storage (object header messages)
  • 8+ attributes: Dense storage (Fractal Heap + B-tree v2)

Supported value types:

  • Scalars: int8, int16, int32, int64, uint8, uint16, uint32, uint64, float32, float64
  • Arrays: []int32, []float64, etc. (1D arrays only)
  • Strings: string (fixed-length, converted to byte array)
  • String arrays: []string (variable-length strings via Global Heap)

Parameters:

  • name: Attribute name (ASCII, no null bytes)
  • value: Attribute value (Go scalar, slice, or string)

Returns:

  • error: If attribute cannot be written

Example: 

ds, _ := fw.CreateDataset("/temperature", Float64, []uint64{10})
ds.WriteAttribute("units", "Celsius")
ds.WriteAttribute("sensor_id", int32(42))
ds.WriteAttribute("calibration", []float64{1.0, 0.0})
ds.WriteAttribute("topics", []string{"camera", "lidar", "imu"})

Limitations:

  • No compound types
  • Attributes cannot be modified after creation (write-once)
  • No attribute deletion

func (*DatasetWriter) WriteRaw 

func (dw *DatasetWriter) WriteRaw(data []byte) error

WriteRaw writes raw bytes directly to the dataset without type conversion. This is useful for advanced use cases like compound datatypes where the user has already prepared the binary representation.

Parameters:

  • data: Raw bytes to write (must match dataset size exactly)

Returns:

  • error: If write fails or size mismatch

Example for compound datatype: 

// Write pre-encoded compound struct data
data := []byte{/* encoded struct bytes */}
err := ds.WriteRaw(data)

type Datatype 

type Datatype int

Datatype represents HDF5 datatype for creating datasets. 

const (
	// Int8 represents 8-bit signed integer type.
	Int8 Datatype = iota
	// Int16 represents 16-bit signed integer type.
	Int16
	// Int32 represents 32-bit signed integer type.
	Int32
	// Int64 represents 64-bit signed integer type.
	Int64
	// Uint8 represents 8-bit unsigned integer type.
	Uint8
	// Uint16 represents 16-bit unsigned integer type.
	Uint16
	// Uint32 represents 32-bit unsigned integer type.
	Uint32
	// Uint64 represents 64-bit unsigned integer type.
	Uint64
	// Float32 represents 32-bit floating point type.
	Float32
	// Float64 represents 64-bit floating point type.
	Float64
	// String represents fixed-length string type (use with WithStringSize option).
	String

	// ArrayInt8 represents array of 8-bit signed integers.
	ArrayInt8 Datatype = 100 + iota
	// ArrayInt16 represents array of 16-bit signed integers.
	ArrayInt16
	// ArrayInt32 represents array of 32-bit signed integers.
	ArrayInt32
	// ArrayInt64 represents array of 64-bit signed integers.
	ArrayInt64
	// ArrayUint8 represents array of 8-bit unsigned integers.
	ArrayUint8
	// ArrayUint16 represents array of 16-bit unsigned integers.
	ArrayUint16
	// ArrayUint32 represents array of 32-bit unsigned integers.
	ArrayUint32
	// ArrayUint64 represents array of 64-bit unsigned integers.
	ArrayUint64
	// ArrayFloat32 represents array of 32-bit floating point values.
	ArrayFloat32
	// ArrayFloat64 represents array of 64-bit floating point values.
	ArrayFloat64

	// EnumInt8 represents enumeration based on 8-bit signed integer.
	EnumInt8 Datatype = 200 + iota
	// EnumInt16 represents enumeration based on 16-bit signed integer.
	EnumInt16
	// EnumInt32 represents enumeration based on 32-bit signed integer.
	EnumInt32
	// EnumInt64 represents enumeration based on 64-bit signed integer.
	EnumInt64
	// EnumUint8 represents enumeration based on 8-bit unsigned integer.
	EnumUint8
	// EnumUint16 represents enumeration based on 16-bit unsigned integer.
	EnumUint16
	// EnumUint32 represents enumeration based on 32-bit unsigned integer.
	EnumUint32
	// EnumUint64 represents enumeration based on 64-bit unsigned integer.
	EnumUint64

	// ObjectReference represents reference to an object (group/dataset).
	// Value type: ObjectRef (uint64 - 8-byte object address).
	ObjectReference Datatype = 300

	// RegionReference represents reference to a dataset region.
	// Value type: RegionRef ([12]byte - 8-byte object addr + 4-byte region info).
	RegionReference Datatype = 301

	// Opaque represents opaque datatype (uninterpreted bytes with tag).
	// Example: JPEG image, binary blob, etc.
	Opaque Datatype = 400

	// VLenString represents variable-length string (most common vlen type!).
	// Each element can have different length.
	// Go type: []string
	// Example: []string{"short", "very long string"}.
	VLenString Datatype = 500

	// VLenInt32 represents variable-length int32 sequences (ragged arrays).
	// Each element can have different number of values.
	// Go type: [][]int32
	// Example: [][]int32{{1,2}, {3,4,5}, {6}}.
	VLenInt32 Datatype = 501

	// VLenInt64 represents variable-length int64 sequences.
	// Go type: [][]int64.
	VLenInt64 Datatype = 502

	// VLenFloat32 represents variable-length float32 sequences.
	// Go type: [][]float32.
	VLenFloat32 Datatype = 503

	// VLenFloat64 represents variable-length float64 sequences.
	// Go type: [][]float64.
	VLenFloat64 Datatype = 504

	// VLenUint32 represents variable-length uint32 sequences.
	// Go type: [][]uint32.
	VLenUint32 Datatype = 505

	// VLenUint64 represents variable-length uint64 sequences.
	// Go type: [][]uint64.
	VLenUint64 Datatype = 506

	// VLenUint8 represents variable-length uint8 sequences (byte arrays).
	// Go type: [][]byte.
	VLenUint8 Datatype = 507
)

type File 

type File struct {
	// contains filtered or unexported fields
}

File represents an open HDF5 file with its metadata and root group.

func Create 

func Create(filename string, mode CreateMode) (*File, error)

Create creates a new HDF5 file with a minimal structure. The file will contain:

  • Superblock v2 (48 bytes at offset 0)
  • Minimal root group (empty, with Link Info message)

The created file is a valid, minimal HDF5 file that can be:

  • Reopened with Open() for reading
  • Validated with h5dump
  • Extended with groups, datasets, and attributes (in future versions)

Parameters:

  • filename: Path to the file to create
  • mode: Creation mode (truncate or exclusive)

Returns:

  • *File: Handle to the created file (in read-only mode for MVP)
  • error: If file creation or initialization fails

Example: 

f, err := hdf5.Create("myfile.h5", hdf5.CreateTruncate)
if err != nil {
    return err
}
defer f.Close()

For MVP (v0.11.0-beta):

  • File is created but returned in read-only mode
  • Write operations (datasets, groups, attributes) are not yet supported
  • The returned File can only be used for reading the structure

func Open 

func Open(filename string) (*File, error)

Open opens an HDF5 file for reading and returns a File handle. The file must be a valid HDF5 file with a supported format version.

func (*File) Close 

func (f *File) Close() error

Close closes the HDF5 file and releases associated resources. It is safe to call Close multiple times.

func (*File) Reader 

func (f *File) Reader() io.ReaderAt

Reader returns the underlying file reader for low-level access.

func (*File) Root 

func (f *File) Root() *Group

Root returns the root group of the HDF5 file.

func (*File) Superblock 

func (f *File) Superblock() *core.Superblock

Superblock returns the file's superblock metadata structure.

func (*File) SuperblockVersion 

func (f *File) SuperblockVersion() uint8

SuperblockVersion returns the HDF5 superblock format version (0, 2, or 3).

func (*File) Walk 

func (f *File) Walk(fn func(path string, obj Object))

Walk traverses the entire file structure, calling fn for each object. Objects are visited in depth-first order starting from the root group.

type FileWriteConfig 

type FileWriteConfig struct {
	SuperblockVersion uint8 // HDF5 superblock version (0, 2, or 3)
	BTreeRebalancing  bool  // Enable B-tree rebalancing after deletions (default: true)
}

FileWriteConfig holds configuration for file creation.

type FileWriter 

type FileWriter struct {
	// contains filtered or unexported fields
}

FileWriter represents an HDF5 file opened for writing. It wraps a File handle and provides write operations.

func CreateForWrite 

func CreateForWrite(filename string, mode CreateMode, opts ...interface{}) (*FileWriter, error)

CreateForWrite creates a new HDF5 file for writing. Unlike Create(), this keeps the file open in write mode.

Parameters:

  • filename: Path to the file to create
  • mode: Creation mode (truncate or exclusive)
  • opts: Optional configuration (WithSuperblockVersion, etc.)

Returns:

  • *FileWriter: Handle for writing datasets
  • error: If creation fails

Example (default - modern format): 

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
if err != nil {
    return err
}
defer fw.Close()

Example (legacy format for h5dump compatibility): 

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithSuperblockVersion(core.Version0))

func OpenForWrite 

func OpenForWrite(filename string, mode OpenMode, opts ...WriteOption) (*FileWriter, error)

OpenForWrite opens an existing HDF5 file for modification. This function enables read-modify-write operations on existing files.

Supported operations:

  • Adding attributes to datasets with existing dense storage
  • Creating new datasets in existing files
  • Creating new groups (when group write support is added)

Parameters:

  • filename: Path to existing HDF5 file
  • mode: Open mode (OpenReadOnly or OpenReadWrite)

Returns:

  • *FileWriter: Handle for modifying the file
  • error: If file doesn't exist or isn't a valid HDF5 file

Example: 

// Reopen file to add more attributes
fw, err := hdf5.OpenForWrite("data.h5", hdf5.OpenReadWrite)
if err != nil {
    return err
}
defer fw.Close()

// Open existing dataset
ds, err := fw.OpenDataset("/temperature")
if err != nil {
    return err
}

// Add more attributes to existing dense storage
ds.WriteAttribute("calibration_date", "2025-11-01")
ds.WriteAttribute("sensor_location", "Lab A")

func (*FileWriter) Close 

func (fw *FileWriter) Close() error

Close closes the file writer and flushes all data to disk.

This method automatically stops any running incremental rebalancing goroutines, preventing goroutine leaks even if user forgets to call StopIncrementalRebalancing().

Best practice: Still call defer fw.StopIncrementalRebalancing() explicitly after EnableIncrementalRebalancing() for clarity, but Close() provides a safety net.

func (*FileWriter) CreateCompoundDataset 

func (fw *FileWriter) CreateCompoundDataset(name string, compoundType *core.DatatypeMessage, dims []uint64, opts ...DatasetOption) (*DatasetWriter, error)

CreateCompoundDataset creates a dataset with a compound (struct-like) datatype. This is an advanced method for creating datasets with complex structured data.

Parameters:

  • name: Dataset path (e.g., "/data" or "/group/dataset")
  • compoundType: Pre-configured compound datatype (use core.CreateCompoundTypeFromFields)
  • dims: Dataset dimensions (e.g., []uint64{10} for 1D, []uint64{3, 4} for 2D)
  • opts: Optional configuration (chunking, compression, etc.)

Returns:

  • *DatasetWriter: Dataset writer for writing data with WriteRaw()
  • error: If creation fails

Example: 

// Define compound type: struct { int32 id; float32 value }
int32Type, _ := core.CreateBasicDatatypeMessage(core.DatatypeFixed, 4)
float32Type, _ := core.CreateBasicDatatypeMessage(core.DatatypeFloat, 4)
fields := []core.CompoundFieldDef{
    {Name: "id", Offset: 0, Type: int32Type},
    {Name: "value", Offset: 4, Type: float32Type},
}
compoundType, _ := core.CreateCompoundTypeFromFields(fields)

// Create dataset
fw, _ := hdf5.CreateForWrite("file.h5", hdf5.CreateTruncate)
ds, _ := fw.CreateCompoundDataset("/data", compoundType, []uint64{100})

// Write raw struct data
data := []byte{/* encoded structs */}
ds.WriteRaw(data)

Reference: H5Dcreate2.c - H5D__create(), H5Tcompound.c - compound datatype handling.

func (*FileWriter) CreateDataset 

func (fw *FileWriter) CreateDataset(name string, dtype Datatype, dims []uint64, opts ...DatasetOption) (*DatasetWriter, error)

CreateDataset creates a new dataset in the HDF5 file. The dataset will use contiguous storage layout.

Parameters:

  • name: Dataset name (must start with "/" for root-level datasets)
  • dtype: Data type (Int32, Float64, etc.)
  • dims: Dimensions (e.g., []uint64{10} for 1D, []uint64{3,4} for 2D)

Returns:

  • *DatasetWriter: Handle for writing data to the dataset
  • error: If creation fails

Example: 

// Create file
fw, _ := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
defer fw.Close()

// Create 1D dataset
ds, _ := fw.CreateDataset("/temperature", hdf5.Float64, []uint64{100})

// Write data
data := make([]float64, 100)
// ... fill data ...
ds.Write(data)

Limitations for MVP (v0.11.0-beta):

  • Only contiguous layout (no chunking)
  • No compression
  • Dataset must be in root group (no nested groups yet)
  • Resizable datasets require chunked layout (use WithMaxDims with WithChunkDims)

func (*FileWriter) CreateDenseGroup 

func (fw *FileWriter) CreateDenseGroup(name string, links map[string]string) error

CreateDenseGroup creates new dense group (HDF5 1.8+ format).

Dense groups are more efficient for large numbers of links (>8). They use fractal heap + B-tree v2 instead of symbol table.

Parameters:

  • name: Group name (must start with "/")
  • links: Map of link_name → target_path

Returns:

  • error: Non-nil if creation fails

Example: 

err := fw.CreateDenseGroup("/large_group", map[string]string{
    "dataset1": "/data/dataset1",
    "dataset2": "/data/dataset2",
    // ... many links
})

Reference: H5Gcreate.c - H5Gcreate2(). 

func (fw *FileWriter) CreateExternalLink(linkPath, fileName, objectPath string) error

CreateExternalLink creates a link to an object in another HDF5 file. The link stores the external file path and object path within that file. Both files must exist when the external link is accessed (lazy resolution).

Parameters:

  • linkPath: Path where external link will be created (e.g., "/links/external1")
  • fileName: External HDF5 file name (absolute or relative path)
  • objectPath: Path to object within external file (e.g., "/dataset1")

Returns:

  • error: if validation fails or creation fails

Examples: 

fw.CreateExternalLink("/links/ext1", "other.h5", "/data/dataset1")
fw.CreateExternalLink("/links/ext2", "/absolute/path/file.h5", "/group1")

Behavior:

  • Validates all paths
  • Creates link message with external link type
  • Stores external file name and object path
  • Adds link entry in parent group's symbol table
  • No file existence check (lazy resolution)

Security:

  • Path traversal prevention (blocks ".." in file names)
  • File path stored as-is (absolute or relative)

Limitations:

  • Symbol table format only (dense groups not yet supported)
  • No external link resolution yet (reading external links not implemented)
  • No file caching or performance optimization

HDF5 Spec: Section IV.A.2.f "Link Message" - Type 64 (External Link) Reference: H5Lcreate_external() in H5L.c.

func (*FileWriter) CreateGroup 

func (fw *FileWriter) CreateGroup(path string) (*GroupWriter, error)

CreateGroup creates a new empty group in the HDF5 file. Groups organize datasets and other groups in a hierarchical structure.

This method creates an empty group using symbol table format (old HDF5 format). For groups with many links, consider using CreateDenseGroup() or CreateGroupWithLinks().

Parameters:

  • path: Group path (must start with "/", e.g., "/data" or "/data/experiments")

Returns:

  • *GroupWriter: Handle for writing attributes to the group
  • error: If creation fails

Example: 

fw, _ := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
defer fw.Close()

// Create root-level group
group, _ := fw.CreateGroup("/data")
group.WriteAttribute("description", "My data group")

// Create nested group
nested, _ := fw.CreateGroup("/data/experiments")
nested.WriteAttribute("MATLAB_class", "double")

Limitations for MVP (v0.11.0-beta):

  • Only symbol table structure (no indexed groups)
  • No link creation time tracking
  • Maximum 32 entries per group (symbol table node capacity)
  • Parent group must exist (create parents first)
func (fw *FileWriter) CreateGroupWithLinks(name string, links map[string]string) error

CreateGroupWithLinks creates group with automatic format selection.

This method automatically chooses the most efficient storage format:

  • Symbol table (old format) for ≤8 links (compact)
  • Dense format (new format) for >8 links (scalable)

This matches HDF5 1.8+ behavior: start compact, use dense when needed.

Parameters:

  • name: Group name (must start with "/")
  • links: Map of link_name → target_path (can be empty)

Returns:

  • error: Non-nil if creation fails

Example: 

// Small group (will use symbol table)
fw.CreateGroupWithLinks("/small", map[string]string{
    "data1": "/dataset1",
    "data2": "/dataset2",
})

// Large group (will use dense format)
largeLinks := make(map[string]string)
for i := 0; i < 100; i++ {
    largeLinks[fmt.Sprintf("link%d", i)] = fmt.Sprintf("/dataset%d", i)
}
fw.CreateGroupWithLinks("/large", largeLinks)

Reference: H5Gint.c - H5G_convert_to_dense(). 

func (fw *FileWriter) CreateHardLink(linkPath, targetPath string) error

CreateHardLink creates a hard link to an existing object.

Hard links are additional names for the same object. All hard links point to the same object header address. When one link is modified, changes are visible through all other links because they share the same data.

Parameters:

  • linkPath: Path where the new link will be created (e.g., "/group1/link_name")
  • targetPath: Path to the existing object to link to (e.g., "/group2/dataset1")

Returns:

  • error: Non-nil if link creation fails

Behavior:

  • Validates both paths exist and are properly formatted
  • Looks up target object's header address
  • Increments reference count on target object header
  • Creates link entry in parent group pointing to target address
  • Supports linking datasets and groups
  • Works with both symbol table and dense group formats

Example: 

fw, _ := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
defer fw.Close()

// Create dataset
fw.CreateDataset("/data/temperature", []float64{1.0, 2.0, 3.0})

// Create hard link to dataset
err := fw.CreateHardLink("/data/temp_link", "/data/temperature")
if err != nil {
    log.Fatal(err)
}
// Now /data/temperature and /data/temp_link point to the same dataset

Limitations (MVP v0.11.5-beta):

  • Target must exist before creating link
  • Parent group must exist before creating link
  • Reference count stored in object header (v1) or RefCount message (v2)
  • No link deletion support yet (DeleteLink not implemented)
  • No circular link detection

Reference: H5L.c - H5Lcreate_hard(). 

func (fw *FileWriter) CreateSoftLink(linkPath, targetPath string) error

CreateSoftLink creates a symbolic link to a path within the HDF5 file.

Soft links (symbolic links) store a path string that is resolved when accessed. Unlike hard links, soft links do not increment reference counts and can point to objects that don't exist yet (dangling links are allowed).

Parameters:

  • linkPath: Path where the soft link will be created (e.g., "/group1/link_to_dataset")
  • targetPath: Target path within file (e.g., "/group2/dataset1")

Returns:

  • error: Non-nil if link creation fails

Behavior:

  • Validates linkPath format (must be absolute path)
  • Target path does NOT need to exist (dangling links allowed)
  • Creates link message with soft link type
  • Adds link entry in parent group's symbol table
  • Link stores target path as string (not object address)
  • When accessed, target path is resolved dynamically

Example: 

fw, _ := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
defer fw.Close()

// Create dataset
fw.CreateDataset("/data/temperature", []float64{1.0, 2.0, 3.0})

// Create soft link (target exists)
err := fw.CreateSoftLink("/links/temp_link", "/data/temperature")
if err != nil {
    log.Fatal(err)
}

// Create dangling link (target doesn't exist yet)
err = fw.CreateSoftLink("/links/future_link", "/data/future_dataset")
// This is allowed - target can be created later

Limitations:

  • Symbol table format only (dense groups not yet supported)
  • No soft link resolution yet (reading soft links not implemented)
  • No circular link detection

HDF5 Spec: Section IV.A.2.f "Link Message" - Type 1 (Soft Link) Reference: H5L.c - H5Lcreate_soft().

func (*FileWriter) Delete 

func (fw *FileWriter) Delete(path string) error

Delete removes an object (dataset or empty group) from the HDF5 file.

This performs a full deletion:

  1. Unlinks the object from its parent group's symbol table
  2. Decrements the object's reference count (hard link count)
  3. If refcount reaches 0, performs cascade delete: - Frees contiguous data blocks - Frees chunked data blocks (walks chunk B-tree) - Frees the object header itself

Constraints:

  • Cannot delete the root group "/"
  • Cannot delete non-empty groups (delete children first)
  • Path must start with "/"
  • Object must exist

Parameters:

  • path: Absolute path to the object (e.g., "/dataset1", "/group1/data")

Returns:

  • error: If deletion fails

Example: 

fw, _ := hdf5.OpenForWrite("data.h5", hdf5.OpenReadWrite)
defer fw.Close()
fw.Delete("/old_dataset")       // Remove a dataset
fw.Delete("/empty_group")       // Remove an empty group

Reference: H5Ldelete.c, H5G_obj_remove(), H5O_link(adjust=-1), H5O_delete().

func (*FileWriter) DisableLazyRebalancing 

func (fw *FileWriter) DisableLazyRebalancing() error

DisableLazyRebalancing disables lazy rebalancing and triggers final batch rebalancing.

This ensures all pending deletions are properly rebalanced before continuing.

Returns:

  • error: if final rebalancing fails

func (*FileWriter) DisableRebalancing 

func (fw *FileWriter) DisableRebalancing()

DisableRebalancing temporarily disables B-tree rebalancing.

Use this to improve performance during batch delete operations. The B-tree may become sparse, but deletions will be faster.

Important: Call EnableRebalancing() when done, or RebalanceNow() to manually rebalance the tree.

Example - Batch deletions: 

fw.DisableRebalancing()
for i := 0; i < 100; i++ {
    ds.DeleteAttribute(fmt.Sprintf("temp_%d", i))
}
fw.EnableRebalancing()
fw.RebalanceNow() // Optional: manually rebalance

func (*FileWriter) EnableIncrementalRebalancing 

func (fw *FileWriter) EnableIncrementalRebalancing(config structures.IncrementalRebalancingConfig) error

EnableIncrementalRebalancing enables incremental background rebalancing for all B-trees.

This starts a background goroutine that performs rebalancing in small time slices, ensuring ZERO user-visible pause even for TB-scale datasets.

**CRITICAL: Resource Management**

  • Background goroutine runs until StopIncrementalRebalancing() called
  • ALWAYS call Stop() or defer it after Enable()
  • Failure to stop will leak goroutine!

**Prerequisites**:

  • Lazy rebalancing must be enabled first (EnableLazyRebalancing)
  • Incremental is built on top of lazy mode

**Use Cases**:

  • Files > 10GB
  • Real-time scientific data processing
  • Interactive applications (no freezing!)
  • TB-scale workflows

Parameters:

  • config: incremental rebalancing configuration

Returns:

  • error: if lazy mode not enabled or already running

Example: 

// Enable lazy first (required)
fw.EnableLazyRebalancing(structures.DefaultLazyConfig())

// Then enable incremental (zero-wait!)
config := structures.DefaultIncrementalConfig()
config.ProgressCallback = func(p structures.RebalancingProgress) {
    log.Printf("Rebalancing: %d nodes done, %d remaining, ETA: %v",
        p.NodesRebalanced, p.NodesRemaining, p.EstimatedRemaining)
}
fw.EnableIncrementalRebalancing(config)
defer fw.StopIncrementalRebalancing()  // CRITICAL!

// Delete millions of attributes - no pause!
for i := 0; i < 10000000; i++ {
    ds.DeleteAttribute(fmt.Sprintf("data_%d", i))
}
// Rebalancing happens in background, user sees no pause!

func (*FileWriter) EnableLazyRebalancing 

func (fw *FileWriter) EnableLazyRebalancing(config structures.LazyRebalancingConfig) error

EnableLazyRebalancing enables lazy rebalancing mode for all B-trees in the file.

Lazy rebalancing accumulates deletions and triggers batch rebalancing only when needed. This provides 10-100x performance improvement for deletion-heavy workloads.

**IMPORTANT: Use at your own risk!**

  • This is an advanced performance optimization
  • User must understand tradeoffs (temporary suboptimal tree structure)
  • Data integrity is always preserved

When to use:

  • Deleting thousands of attributes from large files (>1GB)
  • Batch deletion workflows
  • Scientific data processing pipelines

When NOT to use:

  • Small files (<100MB) - immediate rebalancing is fast enough
  • Read-heavy workloads - suboptimal tree structure may slow reads
  • If unsure - use immediate rebalancing (default)

Parameters:

  • config: lazy rebalancing configuration

Returns:

  • error: if configuration invalid or not supported

Example: 

config := structures.DefaultLazyConfig()
config.Threshold = 0.05 // Trigger at 5% underflow
fw.EnableLazyRebalancing(config)

See docs/guides/PERFORMANCE.md for tuning guidelines.

func (*FileWriter) EnableRebalancing 

func (fw *FileWriter) EnableRebalancing()

EnableRebalancing re-enables B-tree rebalancing after being disabled.

This restores the default behavior where deletions automatically trigger B-tree node merging and redistribution.

Example: 

fw.DisableRebalancing()
// ... batch operations ...
fw.EnableRebalancing()

func (*FileWriter) ForceBatchRebalance 

func (fw *FileWriter) ForceBatchRebalance() error

ForceBatchRebalance manually triggers batch rebalancing on all B-trees.

This is useful when:

  • User wants to optimize tree structure before critical read operations
  • Periodic maintenance (e.g., hourly)
  • Before closing file

**Safe to call anytime** - will only rebalance if lazy mode enabled.

Returns:

  • error: if rebalancing fails

Example: 

// Delete millions of attributes
for i := 0; i < 1000000; i++ {
    ds.DeleteAttribute(fmt.Sprintf("data_%d", i))
}
// Optimize tree before reads
fw.ForceBatchRebalance()

func (*FileWriter) GetIncrementalRebalancingProgress 

func (fw *FileWriter) GetIncrementalRebalancingProgress() (structures.RebalancingProgress, error)

GetIncrementalRebalancingProgress returns progress information for background rebalancing.

Returns:

  • progress: aggregated progress across all B-trees
  • error: if incremental rebalancing not enabled

Example: 

progress, err := fw.GetIncrementalRebalancingProgress()
if err == nil {
    fmt.Printf("Rebalanced: %d, Remaining: %d, ETA: %v\n",
        progress.NodesRebalanced, progress.NodesRemaining,
        progress.EstimatedRemaining)
}

func (*FileWriter) GetLazyRebalancingStats 

func (fw *FileWriter) GetLazyRebalancingStats() (totalUnderflow, totalPending int, oldestRebalance time.Duration)

GetLazyRebalancingStats returns statistics about lazy rebalancing across all B-trees.

Returns:

  • totalUnderflow: total number of underflow nodes across all B-trees
  • totalPending: total pending deletions across all B-trees
  • oldestRebalance: time since oldest rebalancing across all B-trees

func (*FileWriter) IsIncrementalRebalancingEnabled 

func (fw *FileWriter) IsIncrementalRebalancingEnabled() bool

IsIncrementalRebalancingEnabled checks if incremental rebalancing is active.

Returns:

  • bool: true if any B-tree has incremental rebalancing enabled

func (*FileWriter) IsLazyRebalancingEnabled 

func (fw *FileWriter) IsLazyRebalancingEnabled() bool

IsLazyRebalancingEnabled checks if lazy rebalancing is enabled.

Returns:

  • bool: true if any B-tree has lazy rebalancing enabled

func (*FileWriter) OpenDataset 

func (fw *FileWriter) OpenDataset(path string) (*DatasetWriter, error)

OpenDataset opens an existing dataset for modification. This enables read-modify-write operations on datasets.

Supported operations:

  • WriteAttribute(): Add attributes to existing dense storage
  • Write(): Overwrite dataset data (for contiguous layout)

Parameters:

  • path: Dataset path (e.g., "/temperature")

Returns:

  • *DatasetWriter: Handle for modifying the dataset
  • error: If dataset doesn't exist

Example: 

fw, _ := hdf5.OpenForWrite("data.h5", hdf5.OpenReadWrite)
defer fw.Close()

ds, _ := fw.OpenDataset("/temperature")
ds.WriteAttribute("units", "Celsius")  // Works with existing dense storage!

func (*FileWriter) RebalanceAllBTrees 

func (fw *FileWriter) RebalanceAllBTrees() error

RebalanceAllBTrees manually triggers B-tree rebalancing for all datasets with dense attribute storage.

Use cases:

  • After batch deletions with rebalancing disabled (performance optimization)
  • Periodic maintenance to optimize sparse B-trees
  • Before closing file to ensure optimal structure

Performance (for current MVP with single-leaf B-trees):

  • Small files (<10 datasets): <1ms (instant)
  • Medium files (10-100 datasets): 1-10ms
  • Large files (100+ datasets): 10-100ms

Future (when multi-level B-trees implemented):

  • Small datasets (<1000 attrs): <10ms per dataset
  • Medium datasets (1000-10000 attrs): 10-100ms per dataset
  • Large datasets (10000+ attrs): 100ms-1s per dataset

Note: This operation is I/O bound (reads/writes B-tree nodes to disk). For gigabyte-scale data, consider running during off-peak hours.

Example: 

fw.DisableRebalancing()
for i := 0; i < 10000; i++ {
    ds.DeleteAttribute(fmt.Sprintf("attr_%d", i))  // Fast, no rebalancing
}
fw.RebalanceAllBTrees()  // Rebalance once at end

Returns:

  • error: if rebalancing fails for any dataset

func (*FileWriter) RebalancingEnabled 

func (fw *FileWriter) RebalancingEnabled() bool

RebalancingEnabled returns true if B-tree rebalancing is currently enabled.

This can be used to check the current rebalancing state.

Returns:

  • bool: true if rebalancing is enabled, false otherwise

func (*FileWriter) StopIncrementalRebalancing 

func (fw *FileWriter) StopIncrementalRebalancing() error

StopIncrementalRebalancing stops all background rebalancing goroutines.

This method:

  1. Stops all background goroutines
  2. Waits for them to finish current session
  3. Performs final rebalancing of remaining nodes
  4. Cleans up resources

**CRITICAL**: Always call this before closing the file!

Returns:

  • error: if final rebalancing fails

Example: 

fw.EnableIncrementalRebalancing(config)
defer fw.StopIncrementalRebalancing()  // Ensures cleanup

type FileWriterOption 

type FileWriterOption func(*FileWriter) error

FileWriterOption configures a FileWriter during creation. This follows the Functional Options Pattern (Go standard 2025).

Example: 

fw := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),
    ),
)

func WithIncrementalRebalancing 

func WithIncrementalRebalancing(opts ...IncrementalOption) FileWriterOption

WithIncrementalRebalancing enables incremental (background) rebalancing mode.

Incremental rebalancing processes underflow nodes in the background using a goroutine with time budgets. This provides ZERO user-visible pause for TB-scale scientific data.

IMPORTANT: Requires lazy rebalancing to be enabled first (prerequisite).

Default configuration if no options provided:

  • Budget: 100ms per session
  • Interval: 5 seconds between sessions
  • ProgressCallback: nil

Example: 

fw := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(),  // Prerequisite!
    hdf5.WithIncrementalRebalancing(
        hdf5.IncrementalBudget(100*time.Millisecond),
        hdf5.IncrementalInterval(5*time.Second),
    ),
)
defer fw.Close()  // Automatically stops background goroutine

Reference: docs/dev/BTREE_PERFORMANCE_ANALYSIS.md lines 397-446.

func WithLazyRebalancing 

func WithLazyRebalancing(opts ...LazyOption) FileWriterOption

WithLazyRebalancing enables lazy (batch) rebalancing mode.

Lazy rebalancing accumulates deletions and triggers batch rebalancing when a threshold is reached. This is 10-100x faster than immediate rebalancing for deletion-heavy workloads.

Default configuration if no options provided:

  • Threshold: 0.05 (5% underflow)
  • MaxDelay: 5 minutes
  • BatchSize: 100 nodes

Example: 

fw := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithLazyRebalancing(
        hdf5.LazyThreshold(0.05),
        hdf5.LazyMaxDelay(5*time.Minute),
    ),
)

Reference: docs/dev/BTREE_PERFORMANCE_ANALYSIS.md.

func WithSmartRebalancing 

func WithSmartRebalancing(opts ...SmartOption) FileWriterOption

WithSmartRebalancing enables smart (auto-tuning) rebalancing mode.

Smart rebalancing automatically detects workload patterns and selects the optimal rebalancing mode (none, lazy, or incremental) based on:

  • File size
  • Operation patterns (delete ratio, batch size)
  • Resource constraints (CPU, memory limits)

This is the "auto-pilot" mode for scientific data workflows.

IMPORTANT: This is OPTIONAL and must be explicitly enabled. By default (no options), NO rebalancing is performed (like C library).

Example: 

fw := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithSmartRebalancing(
        hdf5.SmartAutoDetect(true),
        hdf5.SmartAutoSwitch(true),
        hdf5.SmartAllowedModes("lazy", "incremental"),
    ),
)

Reference: Phase 3 design (2025 best practices).

type Group 

type Group struct {
	// contains filtered or unexported fields
}

Group represents an HDF5 group that can contain other groups and datasets.

func (*Group) Attributes 

func (g *Group) Attributes() ([]*core.Attribute, error)

Attributes returns all attributes attached to this group. Note: For groups loaded via traditional format (SNOD), the address may be 0, and attributes cannot be retrieved (traditional format doesn't have attributes).

func (*Group) Children 

func (g *Group) Children() []Object

Children returns all child objects (groups and datasets) within this group.

func (*Group) Name 

func (g *Group) Name() string

Name returns the group's name.

type GroupMetadata 

type GroupMetadata struct {
	// contains filtered or unexported fields
}

GroupMetadata stores metadata for a group (symbol table format). Used for tracking non-root groups to enable nested dataset creation.

type GroupWriter 

type GroupWriter struct {
	// contains filtered or unexported fields
}

GroupWriter represents an HDF5 group opened for writing. Groups organize datasets and other groups in a hierarchical structure.

This type enables writing attributes to groups, similar to datasets. It provides a clean, object-oriented API consistent with DatasetWriter.

Example: 

fw, _ := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
defer fw.Close()

// Create group
group, _ := fw.CreateGroup("/mygroup")

// Write attributes to group
group.WriteAttribute("description", "My data group")
group.WriteAttribute("version", int32(1))

Note: This is a write-only handle. For reading group contents, use the file-level Walk() or Group() methods after reopening the file.

func (*GroupWriter) DeleteAttribute 

func (g *GroupWriter) DeleteAttribute(name string) error

DeleteAttribute removes an attribute by name from this group.

This method supports both compact and dense attribute storage:

  • Compact storage (0-7 attributes): Removes message from object header
  • Dense storage (8+ attributes): Removes from B-tree and fractal heap

Parameters:

  • name: Attribute name to delete

Returns:

  • error: If attribute not found or deletion fails

Example: 

group, _ := fw.CreateGroup("/mygroup")
group.WriteAttribute("temp", int32(42))
group.DeleteAttribute("temp") // Remove attribute

Reference: H5Adelete.c - H5A__delete().

func (*GroupWriter) Path 

func (g *GroupWriter) Path() string

Path returns the full path of this group.

This can be used to display the group's location in the file hierarchy or for debugging purposes.

Returns:

  • string: The group's path (e.g., "/mygroup" or "/data/experiments")

Example: 

group, _ := fw.CreateGroup("/mygroup")
fmt.Println(group.Path()) // Output: /mygroup

func (*GroupWriter) WriteAttribute 

func (g *GroupWriter) WriteAttribute(name string, value interface{}) error

WriteAttribute writes an attribute to this group.

Storage strategy (automatic):

  • 0-7 attributes: Compact storage (object header messages)
  • 8+ attributes: Dense storage (Fractal Heap + B-tree v2)

Supported value types:

  • Scalars: int8, int16, int32, int64, uint8, uint16, uint32, uint64, float32, float64
  • Arrays: []int32, []float64, etc. (1D arrays only)
  • Strings: string (fixed-length, converted to byte array)
  • String arrays: []string (variable-length strings via Global Heap)

Parameters:

  • name: Attribute name (ASCII, no null bytes)
  • value: Attribute value (Go scalar, slice, or string)

Returns:

  • error: If attribute cannot be written

Example: 

group, _ := fw.CreateGroup("/mygroup")
group.WriteAttribute("MATLAB_class", "double")
group.WriteAttribute("MATLAB_complex", uint8(1))
group.WriteAttribute("description", "Temperature measurements")
group.WriteAttribute("topics", []string{"camera", "lidar", "imu"})

Limitations:

  • No compound types
  • Attributes cannot be modified after creation (write-once)
  • No attribute deletion

type HeapID 

type HeapID struct {
	CollectionAddress uint64
	ObjectIndex       uint16
	SeqLen            uint32 // Number of elements in the VLen sequence
}

HeapID identifies a global heap object (collection address + object index). On-disk VLen format (C ref: H5Tvlen.c:300, H5Tvlen.c:876): 

seq_len (4 bytes) + heap_address (8 bytes) + object_index (4 bytes) = 16 bytes

SeqLen is the number of elements in the variable-length sequence. For VLen strings, SeqLen = string length in bytes (characters). For VLen sequences (e.g., []int32), SeqLen = number of elements.

func (HeapID) Encode 

func (hid HeapID) Encode() []byte

Encode encodes a heap ID to 16 bytes (HDF5 vlen on-disk format). Format (C ref: H5Tvlen.c:876, H5Tvlen.c:300): 

Bytes 0-3:  seq_len (uint32 LE) — number of elements in sequence
Bytes 4-11: heap_address (uint64 LE) — global heap collection address
Bytes 12-15: object_index (uint32 LE) — index within the collection

type HyperslabSelection 

type HyperslabSelection struct {
	Start  []uint64
	Count  []uint64
	Stride []uint64 // nil means all 1s (contiguous selection)
	Block  []uint64 // nil means all 1s (single element blocks)
}

HyperslabSelection represents a rectangular selection in N-dimensional space. It follows the HDF5 hyperslab specification with start, count, stride, and block parameters.

Parameters:

  • Start: Starting coordinates in each dimension (0-based indexing)
  • Count: Number of blocks to select in each dimension
  • Stride: Step between blocks in each dimension (nil = default to all 1s)
  • Block: Size of each block in each dimension (nil = default to all 1s)

The total number of elements selected is: product(Count[i] * Block[i]) for all dimensions.

Example 1 - Simple slice (start=100, count=50 in 1D array): 

sel := &HyperslabSelection{
    Start: []uint64{100},
    Count: []uint64{50},
}

Example 2 - Strided selection (every 2nd element): 

sel := &HyperslabSelection{
    Start:  []uint64{0, 0},
    Count:  []uint64{25, 25},  // 25 blocks in each dimension
    Stride: []uint64{2, 2},     // Every 2nd element
    Block:  []uint64{1, 1},     // Each block is 1x1
}

type IncrementalOption 

type IncrementalOption func(*structures.IncrementalRebalancingConfig)

IncrementalOption configures incremental rebalancing behavior.

func IncrementalBudget 

func IncrementalBudget(budget time.Duration) IncrementalOption

IncrementalBudget sets the time budget per rebalancing session.

The background goroutine will rebalance for this duration, then pause.

Smaller = less CPU impact, Larger = faster rebalancing Default: 100ms

Example: 

hdf5.IncrementalBudget(200*time.Millisecond)  // 200ms per session

func IncrementalInterval 

func IncrementalInterval(interval time.Duration) IncrementalOption

IncrementalInterval sets how often to run rebalancing sessions.

Smaller = more frequent rebalancing, Larger = more batching Default: 5 seconds

Example: 

hdf5.IncrementalInterval(10*time.Second)  // Every 10 seconds

func IncrementalProgressCallback 

func IncrementalProgressCallback(callback func(structures.RebalancingProgress)) IncrementalOption

IncrementalProgressCallback sets a callback for progress updates.

The callback is called after each rebalancing session with progress info. Optional: Can be nil for no progress reporting.

Example: 

hdf5.IncrementalProgressCallback(func(p structures.RebalancingProgress) {
    fmt.Printf("Rebalanced: %d, Remaining: %d, ETA: %v\n",
        p.NodesRebalanced, p.NodesRemaining, p.EstimatedRemaining)
})

type LazyOption 

type LazyOption func(*structures.LazyRebalancingConfig)

LazyOption configures lazy rebalancing behavior.

func LazyBatchSize 

func LazyBatchSize(size int) LazyOption

LazyBatchSize sets the number of nodes to rebalance per batch operation.

Larger batches = more work per rebalancing, but fewer total operations.

Default: 100 nodes

Example: 

hdf5.LazyBatchSize(200)  // Process 200 nodes per batch

func LazyMaxDelay 

func LazyMaxDelay(delay time.Duration) LazyOption

LazyMaxDelay sets the maximum time before forcing batch rebalancing.

Even if the threshold is not reached, rebalancing will trigger after this duration. This prevents indefinite delay in write-only workloads.

Default: 5 minutes

Example: 

hdf5.LazyMaxDelay(10*time.Minute)  // Force rebalance after 10 min

func LazyThreshold 

func LazyThreshold(threshold float64) LazyOption

LazyThreshold sets the underflow threshold for triggering batch rebalancing.

The threshold is a ratio of underflow nodes to total nodes. When (underflow_nodes / total_nodes) >= threshold, batch rebalancing triggers.

Range: 0.01 (1%) to 0.20 (20%) Default: 0.05 (5%)

Example: 

hdf5.LazyThreshold(0.10)  // Trigger at 10% underflow

type ModeDecision 

type ModeDecision struct {
	SelectedMode string             // Mode selected ("none", "lazy", "incremental")
	Reason       string             // Human-readable reason
	Confidence   float64            // Confidence level [0, 1]
	Factors      map[string]float64 // Factors that influenced decision
	Timestamp    time.Time          // When decision was made
}

ModeDecision explains why a rebalancing mode was selected.

This provides explainability for auto-tuning decisions.

type NamedDatatype 

type NamedDatatype struct {
	// contains filtered or unexported fields
}

NamedDatatype represents an HDF5 committed (named) datatype. A named datatype is a datatype stored as a first-class object in the file, allowing it to be shared by multiple datasets.

func (*NamedDatatype) Datatype 

func (n *NamedDatatype) Datatype() *core.DatatypeMessage

Datatype returns the underlying datatype definition.

func (*NamedDatatype) Name 

func (n *NamedDatatype) Name() string

Name returns the named datatype's name.

type Object 

type Object interface {
	Name() string
}

Object represents any HDF5 object (Group or Dataset) that can be accessed in the file structure.

type OpenMode 

type OpenMode int

OpenMode specifies how to open an existing HDF5 file. 

const (
	// OpenReadOnly opens the file for reading only.
	OpenReadOnly OpenMode = iota

	// OpenReadWrite opens the file for both reading and writing.
	// This enables read-modify-write operations like adding attributes
	// to existing dense storage.
	OpenReadWrite
)

type SmartOption 

type SmartOption func(*SmartRebalancingConfig)

SmartOption configures smart rebalancing behavior.

func SmartAllowedModes 

func SmartAllowedModes(modes ...string) SmartOption

SmartAllowedModes restricts which rebalancing modes can be auto-selected.

Modes: "none", "lazy", "incremental"

Example: 

hdf5.SmartAllowedModes("lazy", "incremental")  // Don't use "none"

func SmartAutoDetect 

func SmartAutoDetect(enabled bool) SmartOption

SmartAutoDetect enables automatic workload pattern detection.

func SmartAutoSwitch 

func SmartAutoSwitch(enabled bool) SmartOption

SmartAutoSwitch enables automatic mode switching based on detected patterns.

func SmartMinFileSize 

func SmartMinFileSize(size uint64) SmartOption

SmartMinFileSize sets the minimum file size for enabling auto-rebalancing.

Files smaller than this size will not trigger automatic rebalancing.

func SmartOnModeChange 

func SmartOnModeChange(callback func(ModeDecision)) SmartOption

SmartOnModeChange sets a callback for mode change notifications.

The callback receives a ModeDecision explaining the change.

Example: 

hdf5.SmartOnModeChange(func(d hdf5.ModeDecision) {
    log.Printf("Mode: %s (confidence: %.2f%%)", d.SelectedMode, d.Confidence*100)
    log.Printf("Reason: %s", d.Reason)
})

type SmartRebalancingConfig 

type SmartRebalancingConfig struct {
	// Auto-detection settings
	AutoDetect bool // Detect workload patterns automatically
	AutoSwitch bool // Automatically switch between modes

	// Constraints
	MinFileSize   uint64   // Minimum file size for auto-rebalancing
	AllowedModes  []string // Allowed rebalancing modes
	MaxCPUPercent int      // Maximum CPU usage percentage

	// Callbacks
	OnModeChange func(decision ModeDecision) // Called when mode changes

}

SmartRebalancingConfig configures smart (auto-tuning) rebalancing.

This will be fully implemented in Phase 3.

type WriteOption 

type WriteOption func(*FileWriteConfig)

WriteOption is a functional option for configuring file creation.

func WithBTreeRebalancing 

func WithBTreeRebalancing(enable bool) WriteOption

WithBTreeRebalancing enables or disables B-tree rebalancing after deletions.

When enabled (default):

  • Deleting attributes triggers B-tree node merging/redistribution
  • Maintains optimal B-tree structure (nodes ≥50% full)
  • Better performance for repeated deletions
  • Prevents tree from becoming sparse over time

When disabled:

  • Faster individual deletions (no rebalancing overhead)
  • B-tree may become sparse after many deletions
  • Useful for batch delete operations

Default: true (matches HDF5 C library behavior)

Example - Disable for batch deletions: 

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate,
    hdf5.WithBTreeRebalancing(false))
// ... perform many deletions ...
fw.RebalanceNow() // Optional: manually rebalance at end

Example - Default behavior (rebalancing enabled): 

fw, err := hdf5.CreateForWrite("data.h5", hdf5.CreateTruncate)
// Deletions automatically rebalance the tree

func WithSuperblockVersion 

func WithSuperblockVersion(version uint8) WriteOption

WithSuperblockVersion sets the HDF5 superblock version.

Available versions:

  • SuperblockV0: Legacy format, maximum compatibility with older tools (h5dump, etc.)
  • SuperblockV2: Modern format with checksums (default)
  • SuperblockV3: Latest format (not yet implemented for writing)

Default: SuperblockV2 (modern format)

Example for maximum compatibility: 

fw, err := hdf5.CreateForWrite("file.h5", hdf5.CreateTruncate,
    hdf5.WithSuperblockVersion(hdf5.SuperblockV0))

Source Files

View all Source files

Directories

Path Synopsis
cmd
dump_hdf5 command
Package main provides a command-line utility to dump HDF5 file contents.
 Package main provides a command-line utility to dump HDF5 file contents.
01-basic command
02-list-objects command
03-read-dataset command
04-vlen-strings command
05-comprehensive command
06-write-dataset command
Package main demonstrates how to create and write datasets to HDF5 files.
 Package main demonstrates how to create and write datasets to HDF5 files.
07-rebalancing/default command
07-rebalancing/incremental command
07-rebalancing/lazy command
07-rebalancing/smart command
internal
core
Package core provides HDF5 file format parsing and manipulation functionality.
 Package core provides HDF5 file format parsing and manipulation functionality.
rebalancing
Package rebalancing provides intelligent B-tree rebalancing strategies for HDF5 files.
 Package rebalancing provides intelligent B-tree rebalancing strategies for HDF5 files.
structures
Package structures provides parsers for HDF5 internal data structures.
 Package structures provides parsers for HDF5 internal data structures.
testing
Package testing provides test utilities for HDF5 library testing.
 Package testing provides test utilities for HDF5 library testing.
utils
Package utils provides utility functions for the HDF5 library.
 Package utils provides utility functions for the HDF5 library.
writer
Package writer provides HDF5 file writing infrastructure.
 Package writer provides HDF5 file writing infrastructure.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.