hb

package module
v0.0.0-...-7e3c669 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 7, 2026 License: MIT Imports: 10 Imported by: 0

README

🏠 homebase-go-lib

Go Version License Go Report Card GitHub Stars

High-performance Go library for data processing, file I/O, and system utilities | Built for production workloads with automatic compression detection and format conversion

A comprehensive toolkit for building data pipelines, ETL workflows, and command-line tools in Go. Features universal file processing with 7 compression formats, structured data iteration (CSV, JSON Lines, Parquet, MsgPack), and production-ready utilities.

📢 Note: This library represents the beginning of open-sourcing minor components from parf's proprietary Homebase framework. The Homebase framework has been battle-tested in production environments processing petabytes of data. We're gradually sharing foundational utilities that can benefit the broader Go and data engineering community while keeping core business logic proprietary.

🛠️ Bundled Utilities

Command-line tools for data conversion and database import/export with comprehensive SQL support.

Utility Description
any2parquet Export data to Parquet format (recommended for analytics)
any2jsonl Export data to JSONL format (human-readable, debug-friendly)
any2csv Export data to CSV format (spreadsheet-compatible)
any2db Import data to MySQL/PostgreSQL databases with auto-schema

All utilities support:

  • 🔌 SQL queries from MySQL & PostgreSQL databases
  • 📦 Multiple file formats (Parquet, JSONL, CSV, MsgPack)
  • 🗜️ Compression (.gz, .zst, .lz4, .br, .xz)
  • 🔄 Stdout piping with - for data pipelines

📖 Full Documentation & Examples →

🌟 Key Features

📁 Universal File Processing
  • 7 compression formats with auto-detection
  • 5 structured formats: CSV, JSONL, Parquet, MsgPack, FlatBuffer
  • HTTP/HTTPS URL support for remote files
  • Streaming processing for large files
⚡ High Performance
  • Zero-copy operations where possible
  • Parallel processing support
  • Memory efficient streaming
  • Progress tracking built-in
🔧 Production Ready
  • Type-safe generic iterators
  • Error handling throughout
  • Battle-tested in production
  • Comprehensive tests
🎯 Developer Friendly
  • Simple API - consistent patterns
  • Auto-detection - no manual config
  • Rich examples included
  • Well documented

📦 Installation

go get github.com/parf/homebase-go-lib

🚀 Quick Start

Process Compressed Files Automatically
import "github.com/parf/homebase-go-lib/fileiterator"

// Works with .gz, .zst, .lz4, .br, .xz automatically!
fileiterator.IterateLines("access.log.gz", func(line string) error {
    // Process each line
    fmt.Println(line)
    return nil
})
Type-Safe JSON Lines Processing
type User struct {
    ID    int64  `json:"id"`
    Name  string `json:"name"`
    Email string `json:"email"`
}

// Generic type-safe iterator
fileiterator.IterateJSONLTyped("users.jsonl.zst", func(user User) error {
    fmt.Printf("User: %s <%s>\n", user.Name, user.Email)
    return nil
})
Universal Schema Support
// Read ANY schema from Parquet, CSV, JSONL, MsgPack
records, _ := fileiterator.ReadInput("data.parquet")
for _, record := range records {
    // record is map[string]any - works with any field structure!
    fmt.Printf("Record: %v\n", record)
}
Progress Tracking
import "github.com/parf/homebase-go-lib/clistat"

stat := clistat.New(10) // Report every 10 seconds
for i := 0; i < 1_000_000; i++ {
    stat.Hit()  // Auto-reports progress: "45.2K hits/sec (500K total)"
}
stat.Finish()   // Final summary

📋 Supported Formats

Structured Data Formats
Format Extensions Read Write Use Case Performance
📄 CSV .csv, .tsv ✅ ✅ Excel compatibility, human-readable Good
📝 JSON Lines .jsonl, .ndjson ✅ ✅ Debugging, wide support Moderate
📊 Apache Parquet .parquet, .pk ✅ ✅ Analytics, columnar queries Excellent
🔧 MessagePack .msgpack, .mp ✅ ✅ Binary efficiency, 2x smaller than JSON Very Good
⚡ FlatBuffer .fb ✅ ✅ Zero-copy, fastest reads (3x faster) Fastest
Compression Formats (Auto-Detected)
Compression Extension Speed Ratio Use Case
⚡ LZ4 .lz4 Fastest Good Real-time processing
🎯 Zstandard .zst Fast Excellent Recommended for most uses
📦 Gzip .gz Moderate Good Universal compatibility
🔥 Brotli .br Slow Best Maximum compression
❄️ XZ/LZMA .xz Very Slow Excellent Archive storage
📋 Zlib .zlib, .zz Moderate Good Legacy support

All formats work seamlessly with all compression types! For example: .jsonl.zst, .csv.gz, .parquet.lz4

💡 Use Cases

🔄 Data Pipeline Processing
// Convert between any formats with automatic compression
input, _ := fileiterator.ReadInput("raw-data.csv.gz")           // CSV + Gzip
fileiterator.WriteParquetAny("processed.parquet.zst", input)    // Parquet + Zstd
📊 Log Analysis
stat := clistat.New(5)
fileiterator.IterateLines("access.log.gz", func(line string) error {
    if strings.Contains(line, "ERROR") {
        // Process error logs
    }
    stat.Hit()
    return nil
})
stat.Finish()  // "Processed 2.5M lines in 3.2s (781K lines/sec)"
🗄️ Database ETL
// Extract from CSV, transform, load to database
fileiterator.IterateCSVMap("export.csv.zst", func(row map[string]string) error {
    // Transform data
    user := transformUser(row)

    // Load to database
    return db.Insert(user)
})
🚀 Batch Processing
// Process millions of records efficiently
fileiterator.IterateParquetAny("events.parquet", func(event map[string]any) error {
    // Process each event with automatic memory management
    return processEvent(event)
})

📚 Core Packages

📁 fileiterator - Universal File Processing

The heart of homebase-go-lib. Process any file format with automatic compression detection.

Key Functions
// Universal I/O
FUOpen(filename)              // Open any file/URL with auto-decompression
FUCreate(filename)            // Create file with auto-compression
ReadInput(filename)           // Read ANY schema to []map[string]any
WriteOutput(filename, data)   // Write ANY schema from []map[string]any

// Line-by-line Processing
IterateLines(filename, func(line string) error)

// Structured Data (Untyped)
IterateJSONL(filename, func(map[string]any) error)
IterateCSVMap(filename, func(map[string]string) error)
IterateMsgPack(filename, func(any) error)
IterateParquetAny(filename, func(map[string]any) error)

// Structured Data (Type-Safe Generics)
IterateJSONLTyped[T](filename, func(T) error)
IterateMsgPackTyped[T](filename, func(T) error)

// Binary Formats
IterateBinaryRecords(filename, recordSize, func([]byte) error)
IterateFlatBufferList(filename, func([]byte) error)

Features:

  • ✅ Automatic compression detection from file extension
  • ✅ HTTP/HTTPS URL support
  • ✅ Streaming for memory efficiency
  • ✅ Progress reporting integration
  • ✅ Error handling with context
📊 clistat - Real-Time Statistics

Track processing progress with automatic hit-rate reporting.

stat := clistat.New(10)  // Report every 10 seconds

for i := 0; i < 1_000_000; i++ {
    // Your processing logic
    stat.Hit()  // Automatically reports: "45.2K hits/sec (500K total)"
}

stat.Finish()  // Final: "Processed 1M items in 22.1s (45.2K/sec)"

Features:

  • ✅ Automatic progress reporting
  • ✅ Configurable intervals
  • ✅ Hits-per-second calculation
  • ✅ Total count tracking
  • ✅ Elapsed time reporting
🗄️ sql - Database Utilities

Efficient database operations with batch processing.

// Batch Insert
inserter := sql.NewBatchInserter(db, "users", []string{"id", "name", "email"}, 1000)
inserter.Add(1, "Alice", "alice@example.com")
inserter.Add(2, "Bob", "bob@example.com")
inserter.Flush()

// Query Iteration
sql.SqlIterator(db, "SELECT * FROM users WHERE active = true", func(row map[string]any) error {
    // Process each row
    return nil
})
💾 cache - Map-to-Parquet File Caching

Cache any Go map as a Parquet file with precise scalar type preservation.

import "github.com/parf/homebase-go-lib/cache"

data := make(map[string]any)
if !cache.Map("lookup.parquet", data) {
    data = expensiveComputation()
    if err := cache.WriteMap("lookup.parquet", data); err != nil {
        log.Printf("cache write failed: %v", err)
    }
}
// use data directly — no type assertion needed

Supports any map type — map[string]any, map[uint32]string, map[int]float64, etc.

Features:

  • ✅ Precise type preservation (uint32 stays uint32, float32 stays float32)
  • ✅ String-keyed maps (column-per-key) and numeric-keyed maps (key-value columns)
  • ✅ Best-effort caching — silent errors, corrupted files treated as misses
  • ✅ Internal Snappy compression

📖 Full Documentation →

🎯 Format Conversion Tools

Universal Converters (Included)

Located in cmd/ directory:

any2parquet - Convert to Apache Parquet
# Convert any format to Parquet
any2parquet data.jsonl                    # → data.parquet
any2parquet logs.csv.gz                   # → logs.parquet
any2parquet events.msgpack.zst            # → events.parquet
any2jsonl - Convert to JSON Lines
# Convert any format to human-readable JSONL
any2jsonl data.parquet                    # → data.jsonl
any2jsonl users.csv                       # → users.jsonl
any2jsonl metrics.parquet.zst             # → metrics.jsonl

Standalone Tool: any-to-parquet - Optimized Parquet converter

📈 Performance Benchmarks

Based on 1 million records:

Format File Size Read Time Write Time Compression Best For
Parquet 44 MB 0.15s 0.46s Excellent Everything 🏆
MsgPack.zst 38 MB 0.59s 0.61s Best Binary efficiency
JSONL.zst 43 MB 1.91s 0.84s Excellent Debugging
FlatBuffer.lz4 66 MB 0.06s 0.42s Good Ultra-fast reads
CSV.gz 52 MB 2.1s 1.2s Good Excel compatibility
Plain JSONL 156 MB 1.93s 1.38s None Human-readable

Winner: Parquet delivers the best balance of speed, compression, and compatibility.

Full Benchmark Results →

🧪 Examples & Tests

Running Examples
# File processing examples
cd examples/fileiterator
go run main.go

# Statistics tracking
cd examples/clistat
go run main.go

# Schema examples (5 different data structures)
cd cmd/examples/schemas
./test-all-schemas.sh
Test Different Schemas

The library works with ANY schema structure. See examples:

View All Schema Examples →

🛠️ Development

Prerequisites
  • Go 1.21 or higher
  • Make (optional)
Build
make build
Run Tests
make test
Test Coverage
make test-coverage
Format & Lint
make fmt
make lint

📁 Project Structure

homebase-go-lib/
├── 📦 fileiterator/       # File processing & format conversion
│   ├── parquet.go         # Apache Parquet support
│   ├── jsonl.go           # JSON Lines processing
│   ├── csv.go             # CSV with auto-detection
│   ├── msgpack.go         # MessagePack binary format
│   ├── genericio.go       # Universal I/O functions
│   └── compression.go     # 7 compression formats
│
├── 📊 clistat/            # Real-time statistics tracking
│   └── clistat.go
│
├── 🗄️ sql/                # Database utilities
│   ├── batch.go           # Batch insert operations
│   └── iterator.go        # Query iteration
│
├── 💾 cache/              # Map-to-Parquet file caching
│   └── mapcache.go        # Precise type-preserving cache
│
├── 🎯 cmd/                # Command-line tools
│   ├── any2parquet.go     # Universal → Parquet converter
│   ├── any2jsonl.go       # Universal → JSONL converter
│   └── examples/          # Usage examples
│       └── schemas/       # 5 different schema examples
│
├── 🧪 examples/           # Code examples
├── 📚 docs/               # Documentation
├── 🏗️ benchmarks/         # Performance benchmarks
└── 🧰 testdata/           # Test fixtures

🔑 Key Concepts

Automatic Compression Detection
// All these work automatically based on file extension:
fileiterator.IterateLines("file.txt")      // Plain text
fileiterator.IterateLines("file.txt.gz")   // Gzip compressed
fileiterator.IterateLines("file.txt.zst")  // Zstandard compressed
fileiterator.IterateLines("file.txt.lz4")  // LZ4 compressed
Universal Schema Support
// No schema definition needed - works with ANY structure!
records, _ := fileiterator.ReadInput("data.csv")
// records[0] might be: {"user_id": 1, "name": "Alice", "age": 28}

records2, _ := fileiterator.ReadInput("sensors.jsonl")
// records2[0] might be: {"sensor": "temp-01", "value": 23.5, "unit": "celsius"}
Type-Safe Generics
// Define your struct
type Product struct {
    ID    int     `json:"id"`
    Name  string  `json:"name"`
    Price float64 `json:"price"`
}

// Get type-safe iteration with Go generics
fileiterator.IterateJSONLTyped("products.jsonl", func(p Product) error {
    fmt.Printf("%s: $%.2f\n", p.Name, p.Price)
    return nil
})

🤝 Contributing

Contributions welcome! Please:

  1. 🍴 Fork the repository
  2. 🌿 Create a feature branch
  3. ✅ Add tests for new functionality
  4. 📝 Update documentation
  5. 🚀 Submit a pull request

Report Bug · Request Feature

📄 License

MIT License - see LICENSE file for details

🏷️ Keywords

go library, golang, file processing, data pipeline, ETL, compression, gzip, zstd, lz4, parquet, json lines, csv processing, msgpack, data engineering, batch processing, streaming, apache parquet, columnar format, data conversion, format converter, structured data, log processing, statistics tracking, progress reporting, database utilities, sql batch insert, type-safe iterators, go generics, high performance, production ready

⭐ Star History

If you find this library useful, please give it a star! ⭐

Star History Chart

Built with ❤️ for the Go and data engineering community

Documentation · Examples · Benchmarks

Documentation

Overview

Package hb provides the main functionality for the homebase library.

Index

Examples

Constants

View Source 
const Version = "0.1.0"

Version is the current version of the library

Variables

This section is empty.

Functions

func Any2uint32 

func Any2uint32(iii any) (r uint32, err error)

Any2uint32 converts various integer types to uint32

func DumpSortedMap 

func DumpSortedMap(m map[string]any)

DumpSortedMap prints a map in key-sorted order

func MemReport 

func MemReport(event string)

MemReport reports allocated memory to STDOUT

func Scale 

func Scale(nn uint32) byte

Scale returns a 0..9 scale value for uint32 numbers using logarithmic base-4. Returns 0 for nn=0, 1 for nn=1, then logBase4(nn)+1 capped at 9. This is a copy of HB::scale(nn, 4) method for int16 numbers range.

Example 
package main

import (
	"fmt"

	hb "github.com/parf/homebase-go-lib"
)

func main() {
	// Scale small values
	fmt.Println(hb.Scale(0))
	fmt.Println(hb.Scale(1))
	fmt.Println(hb.Scale(4))
	fmt.Println(hb.Scale(16))
	fmt.Println(hb.Scale(64))
	fmt.Println(hb.Scale(256))
	fmt.Println(hb.Scale(1024))
	fmt.Println(hb.Scale(4096))
	fmt.Println(hb.Scale(16384))
	fmt.Println(hb.Scale(100000))

}

Output:

0
1
2
3
4
5
6
7
8
9
Example (Ranges) 
package main

import (
	"fmt"

	hb "github.com/parf/homebase-go-lib"
)

func main() {
	// Demonstrate ranges
	values := []uint32{0, 1, 5, 17, 65, 257, 1025, 5000, 20000}
	for _, v := range values {
		fmt.Printf("%d -> scale %d\n", v, hb.Scale(v))
	}

}

Output:

0 -> scale 0
1 -> scale 1
5 -> scale 3
17 -> scale 4
65 -> scale 5
257 -> scale 6
1025 -> scale 7
5000 -> scale 8
20000 -> scale 9

func SysLogError 

func SysLogError(message string)

SysLogError writes error to syslog Usage: hb.SysLogError(fmt.Sprintf(...))

func SysLogNotice 

func SysLogNotice(message string)

SysLogNotice writes notice to syslog Usage: hb.SysLogNotice(fmt.Sprintf(...))

Types

type DebugFunction 

type DebugFunction func(level int, format string, a ...any)

DebugFunction is a function type for debug output

func Debug 

func Debug(prefix string, level int) DebugFunction

Debug creates a debug function that outputs to STDERR

func DebugLog 

func DebugLog(prefix string, level int) DebugFunction

DebugLog creates a debug function with timestamp prefix (uses log package)

type JobScheduler 

type JobScheduler struct {
	// contains filtered or unexported fields
}

JobScheduler represents a periodic job scheduler

func NewJobScheduler 

func NewJobScheduler(intervalSeconds int, jobFunc func()) *JobScheduler

NewJobScheduler creates a new job scheduler

func (*JobScheduler) IsRunning 

func (js *JobScheduler) IsRunning() bool

IsRunning returns whether the scheduler is currently running

func (*JobScheduler) Start 

func (js *JobScheduler) Start() error

Start begins the job scheduler

func (*JobScheduler) Stop 

func (js *JobScheduler) Stop() error

Stop halts the job scheduler

type ParallelRunner 

type ParallelRunner struct {
	// contains filtered or unexported fields
}

ParallelRunner runs tasks in parallel with performance tracking

func NewParallelRunner 

func NewParallelRunner() ParallelRunner

NewParallelRunner creates a new parallel task runner

func (*ParallelRunner) Finish 

func (p *ParallelRunner) Finish()

Finish waits for all parallel tasks to complete and logs final statistics

func (*ParallelRunner) Run 

func (p *ParallelRunner) Run(name string, f func())

Run starts a task in parallel and logs start/finish with elapsed time and memory total

type SequentialRunner 

type SequentialRunner struct {
	// contains filtered or unexported fields
}

SequentialRunner is a debug option to use instead of ParallelRunner

func NewSequentialRunner 

func NewSequentialRunner() SequentialRunner

NewSequentialRunner creates a drop-in debug replacement for ParallelRunner Usage: 

runner := hb.NewSequentialRunner()
runner.Run(func())
runner.Finish()

func (*SequentialRunner) Finish 

func (p *SequentialRunner) Finish()

Finish logs final statistics for the sequential runner

func (*SequentialRunner) Run 

func (p *SequentialRunner) Run(name string, f func())

Run executes a task sequentially and logs start/finish with elapsed time, memory diff & total

Source Files

View all Source files

Directories

Path Synopsis
cmd
examples command
examples
clistat command
compression command
scale command
sql command
internal
compression

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.