binpost

README

binpost

Binary file saving of struct slices

This is a simple go library to demonstrate the use of writing binary data to a file. This will be suitable for large data sets of ordered data where the data has a natural order like account postings by posting date.

Features:

• Data is a fixed size strct to allow random seeking of data
• be able to handle large data sets on file which are bigger than will fit into memory
• allow fast write aiming for ca 100K per second for a ca 25byte structure
• aim to use generic

Benchmark data:

This proves that this is fast enough and accurate enough.

The append method is about 50% slower as it verifies the total sum and reads the file back and recalculates.

goos: linux goarch: amd64 pkg: github.com/drummonds/binpost/binpost cpu: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz BenchmarkWriteRecords100k-8 14 85238195 ns/op BenchmarkReadRecords100k-8 16 69979147 ns/op BenchmarkWriteRecords1M-8 2 870476293 ns/op BenchmarkReadRecords1M-8 2 621415825 ns/op BenchmarkWriteRecords1M_Append-8 1 1855474531 ns/op

Documentation

Overview

Package binpost provides binary file saving of struct slices. Features: - Fixed size struct for random seeking - Handles large data sets bigger than memory - Fast write/read for ca 25-byte structures - Uses generics for flexibility

Index

• func OpenAppendFile[T any](filename string) (*os.File, func() error, error)
• func ReadRecords[T any](filename string) ([]T, error)
• func WriteOneRecord[T any](f *os.File, rec T) error
• func WriteRecords[T any](filename string, records []T) error
• type DataRecord

Functions

func OpenAppendFile

func OpenAppendFile[T any](filename string) (*os.File, func() error, error)

OpenAppendFile opens a file for appending (creates if not exists) and returns the file, a closer function, and an error.

func ReadRecords

func ReadRecords[T any](filename string) ([]T, error)

ReadRecords reads a slice of fixed-size structs from a binary file.

func WriteOneRecord

func WriteOneRecord[T any](f *os.File, rec T) error

WriteOneRecord writes a single record to the end of the given file.

func WriteRecords

func WriteRecords[T any](filename string, records []T) error

WriteRecords writes a slice of fixed-size structs to a binary file.

Types

type DataRecord

type DataRecord struct {
	// Example fields
	ID   int64
	Date int64
	Amt  int64
}

DataRecord is a placeholder for a fixed-size struct. Replace with your own struct.