ioutils

README

IOUtils Package

Production-ready I/O utilities collection providing specialized tools for stream processing, resource management, progress tracking, and concurrent I/O operations with comprehensive testing and thread-safe implementations.

---

Table of Contents

• Overview
  • Design Philosophy
  • Key Features
• Architecture
  • Package Organization
• Performance
  • Benchmark Summary
  • Coverage Statistics
• Subpackages
  • aggregator
  • bufferReadCloser
  • delim
  • fileDescriptor
  • ioprogress
  • iowrapper
  • mapCloser
  • maxstdio
  • multi
  • nopwritecloser
• Root-Level Utilities
  • PathCheckCreate
• Use Cases
• Quick Start
  • Installation
  • Basic Examples
• Best Practices
• Testing
• Contributing
• Improvements & Security
• Resources
• AI Transparency
• License

---

Overview

The ioutils package is a comprehensive collection of I/O utilities for Go applications, providing 10 specialized subpackages plus root-level helper functions. Each subpackage addresses specific I/O challenges encountered in production environments: concurrent write aggregation, stream multiplexing, progress tracking, resource lifecycle management, and more.

Design Philosophy

1. Standard Interface Compliance: All implementations conform to Go's io package interfaces (Reader, Writer, Closer, ReadCloser, WriteCloser)
2. Thread Safety First: Atomic operations, proper locking, and race-free implementations verified with -race detector
3. Streaming-Oriented: Designed for continuous data flow with constant memory usage, not batch processing
4. Resource Management: Proper cleanup with defer, context cancellation support, and automatic lifecycle handling
5. Zero External Dependencies: Only standard library and internal golib packages
6. Production Hardened: 776 test specs, 90.4% average coverage, extensive documentation

Key Features

✅ Thread-Safe Operations: All subpackages safe for concurrent use
✅ Context Integration: Cancellation and deadline propagation throughout
✅ Comprehensive Testing: 776 specs, zero race conditions
✅ Rich Subpackage Ecosystem: 10 specialized packages for different I/O needs
✅ High Performance: Benchmarked and optimized for throughput and latency
✅ Well Documented: GoDoc comments, examples, README per subpackage

---

Architecture

Package Organization

ioutils/
├── tools.go                    Root-level utilities (PathCheckCreate)
│
├── aggregator/                 Thread-safe write aggregator that buffers and serializes concurrent writes
│   └── [115 specs, 84.9% coverage]
│
├── bufferReadCloser/           io.Closer wrappers for bytes.Buffer and bufio types
│   └── [44 specs, 100% coverage]
│
├── delim/                      Buffered reader for reading delimiter-separated data streams
│   └── [95 specs, 98.6% coverage]
│
├── fileDescriptor/             Cross-platform file descriptor limit management
│   └── [28 specs, 85.7% coverage]
│
├── ioprogress/                 Thread-safe I/O progress tracking wrappers
│   └── [54 specs, 84.7% coverage]
│
├── iowrapper/                  Flexible I/O wrapper for customization and interception
│   └── [88 specs, 100% coverage]
│
├── mapCloser/                  Thread-safe, context-aware manager for multiple io.Closer instances
│   └── [82 specs, 80.8% coverage]
│
├── maxstdio/                   Standard I/O limit management
│   └── [No specs - utility package]
│
├── multi/                      Thread-safe adaptive multi-writer extending io.MultiWriter
│   └── [112 specs, 80.8% coverage]
│
└── nopwritecloser/             io.WriteCloser wrapper with no-op Close()
    └── [54 specs, 100% coverage]

Total: 776 test specs, 90.4% average coverage

---

Performance

Benchmark Summary

Based on actual test execution (776 specs, ~41 seconds total):

Package	Specs	Coverage	Execution Time	Notable Metrics
aggregator	115	84.9%	~33.3s	Start: 10.7ms, Throughput: 5000-10000/s
bufferReadCloser	44	100%	~0.03s	Read: <1ms, Buffer: configurable
delim	95	98.6%	~6.68s	Read: <500µs, Scan: <1ms
fileDescriptor	28	85.7%	~0.01s	Limit check: <1µs
ioprogress	54	84.7%	~0.02s	Callback: <10µs overhead
iowrapper	88	100%	~0.08s	Wrap: <1µs, Pass-through
mapCloser	82	80.8%	~0.05s	Add/Remove: <1µs, Close all: <1ms
multi	112	80.8%	~0.69s	Write to N: O(N), Copy: <100µs
nopwritecloser	54	100%	~0.24s	No-op: <1ns
ioutils (root)	31	88.2%	~0.04s	PathCheckCreate: varies

Aggregate Performance:

• Total execution: ~41 seconds (including setup/teardown)
• Average spec time: ~53ms (dominated by aggregator's timing tests)
• Zero race conditions: All tests pass with -race detector
• Memory efficiency: Constant memory for streaming operations

Coverage Statistics

Overall Coverage:       90.4% (weighted average)
Packages at 100%:       3/10 (bufferReadCloser, iowrapper, nopwritecloser)
Packages >85%:          5/10
Lowest Coverage:        80.8% (mapCloser, multi - primarily error paths)

Coverage Breakdown:

Coverage Range	Count	Packages
100%	3	bufferReadCloser, iowrapper, nopwritecloser
85-99%	4	delim (98.6%), ioutils (88.2%), fileDescriptor (85.7%), ioprogress (84.7%)
80-89%	3	aggregator (84.9%), mapCloser (80.8%), multi (80.8%)

---

Subpackages

aggregator

Purpose: Thread-safe write aggregator that buffers and serializes concurrent write operations to a single writer function.

Key Features:

• Concurrent writes from multiple goroutines
• Configurable buffer for backpressure handling
• Optional periodic callbacks (async/sync)
• Real-time metrics (count and size based)
• Context integration for lifecycle management

Performance:

• Throughput: 5,000-10,000 writes/second (depends on FctWriter)
• Latency: Start 10.7ms, Stop 12.1ms, Write <1ms
• Metrics read: <5µs

Use Case: Socket server logging, database write pooling, network stream multiplexing

Documentation: aggregator/README.md

---

bufferReadCloser

Purpose: Lightweight wrappers around Go's standard buffered I/O types (bytes.Buffer, bufio.Reader, bufio.Writer, bufio.ReadWriter) that add io.Closer support with automatic resource cleanup and custom close callbacks.

Key Features:

• Wraps bytes.Buffer, bufio.Reader, bufio.Writer, bufio.ReadWriter
• Adds io.Closer interface to non-closeable buffers
• Automatic reset/flush on close
• Optional custom close callbacks
• Zero-copy passthrough to underlying buffers

Performance:

• Wrapper overhead: ~5.7ms per 10,000 operations
• I/O operations: 0-100ns (direct delegation)
• Memory: 24 bytes per wrapper

Use Case: Adding lifecycle management to standard buffers, automatic cleanup with defer, composable close operations

Documentation: bufferReadCloser/README.md

---

delim

Purpose: Buffered reader for reading delimiter-separated data streams.

Key Features:

• Read until any delimiter character
• Buffered scanning for efficiency
• Handles any byte delimiter
• Line and token reading

Performance:

• Read latency: <500µs
• Scan latency: <1ms
• Memory: constant per buffer

Use Case: CSV parsing, log file processing, protocol parsing

Documentation: delim/README.md

---

fileDescriptor

Purpose: Cross-platform utilities for managing file descriptor limits, offering a unified API for querying and modifying the maximum number of open files or I/O resources allowed for a process.

Key Features:

• Check system-wide FD limits
• Validate current usage
• Preemptive resource checks
• Cross-platform support

Performance:

• Limit check: <1µs
• No overhead on I/O operations

Use Case: High-concurrency servers, connection pooling, resource planning

Documentation: fileDescriptor/README.md

---

ioprogress

Purpose: Thread-safe I/O progress tracking wrappers for monitoring read and write operations in real-time through customizable callbacks.

Key Features:

• Reader/Writer progress callbacks
• Byte count tracking
• Real-time notification
• Progress bar integration

Performance:

• Callback overhead: <10µs per operation
• Thread-safe atomic counters
• Minimal allocation

Use Case: File uploads/downloads, progress bars, bandwidth monitoring, ETL pipelines

Documentation: ioprogress/README.md

---

iowrapper

Purpose: Flexible I/O wrapper that enables customization and interception of read, write, seek, and close operations on any underlying I/O object without modifying its implementation.

Key Features:

• Wrap readers and writers
• Add custom behavior
• Preserve interface semantics
• Zero-allocation pass-through

Performance:

• Wrap overhead: <1µs
• Pass-through: no measurable impact

Use Case: Logging wrappers, compression, encryption, transformation

Documentation: iowrapper/README.md

---

mapCloser

Purpose: Thread-safe, context-aware manager for multiple io.Closer instances.

Key Features:

• Manage multiple closers
• Automatic cleanup on context cancellation
• Error aggregation
• Add/remove closers dynamically

Performance:

• Add/Remove: <1µs
• Close all: <1ms (for moderate counts)
• Thread-safe operations

Use Case: Resource pools, connection management, cleanup coordination

Documentation: mapCloser/README.md

---

maxstdio

Purpose: Standard I/O (stdin/stdout/stderr) limit enforcement.

Key Features:

• Protect against excessive stdio usage
• Redirect overflow to alternatives
• Configurable thresholds

Use Case: Daemon processes, service wrappers, log management

Documentation: maxstdio/README.md

---

multi

Purpose: Thread-safe, adaptive multi-writer that extends Go's standard io.MultiWriter with advanced features including adaptive sequential/parallel execution, latency monitoring, and comprehensive concurrency support.

Key Features:

• Write to multiple writers atomically
• Dynamic writer addition/removal
• Error handling per writer
• Zero-allocation for single writer
• Copy to multiple outputs

Performance:

• Write to N writers: O(N) time
• Copy operation: <100µs
• Atomic operations

Use Case: Logging to multiple files, network fanout, tee operations

Documentation: multi/README.md

---

nopwritecloser

Purpose: Wrapper that implements io.WriteCloser for an io.Writer by adding a no-op Close() method.

Key Features:

• Implements io.WriteCloser for any io.Writer
• Write operations delegate to underlying writer unchanged
• Close() is no-op (always returns nil, never affects writer)
• Thread-safe if underlying writer is thread-safe

Performance:

• Write: Direct delegation (no overhead)
• Close: <1ns (no-op)
• Zero allocation

Use Case: Protecting stdout/stderr from closure, API compatibility (io.Writer → io.WriteCloser), testing with inspectable buffers after close, shared resource management

Documentation: nopwritecloser/README.md

---

Root-Level Utilities

PathCheckCreate

Function: PathCheckCreate(isFile bool, path string, permFile os.FileMode, permDir os.FileMode) error

Purpose: Ensures a file or directory exists with correct permissions.

Features:

• Creates files or directories as needed
• Creates parent directories automatically
• Validates and updates permissions
• Type checking (file vs directory)
• Atomic creation with proper error handling

Example:

// Ensure config directory exists
err := ioutils.PathCheckCreate(false, "/etc/app/config", 0644, 0755)

// Ensure log file exists
err := ioutils.PathCheckCreate(true, "/var/log/app.log", 0644, 0755)

Use Case: Application initialization, config file setup, log directory creation

---

Use Cases

1. High-Concurrency Logging

Problem: Multiple goroutines writing to a single log file (filesystem doesn't support concurrent writes).

Solution: Use aggregator to serialize writes.

logFile, _ := os.Create("app.log")
agg, _ := aggregator.New(ctx, aggregator.Config{
    BufWriter: 1000,
    FctWriter: func(p []byte) (int, error) {
        return logFile.Write(p)
    },
}, logger)

// All goroutines write through aggregator
for i := 0; i < 100; i++ {
    go func(id int) {
        agg.Write([]byte(fmt.Sprintf("[%d] Log message\n", id)))
    }(i)
}

2. Fan-Out Data Broadcasting

Problem: Send data to multiple destinations (files, network, stdout).

Solution: Use multi for write multiplexing.

mw := multi.New()
mw.AddWriter(os.Stdout)
mw.AddWriter(logFile)
mw.AddWriter(networkConn)

// One write reaches all destinations
mw.Write([]byte("Broadcast message\n"))

3. Upload Progress Tracking

Problem: Show upload progress to user during file transfer.

Solution: Use ioprogress wrapper.

file, _ := os.Open("large-file.dat")
progressReader := ioprogress.NewReader(file, func(bytes int64) {
    percent := float64(bytes) / float64(fileSize) * 100
    fmt.Printf("Uploaded: %.1f%%\r", percent)
})

http.Post(url, "application/octet-stream", progressReader)

4. Resource Management

Problem: Manage multiple connections/files with automatic cleanup.

Solution: Use mapCloser.

closer := mapcloser.New(ctx)

// Add resources
conn1, _ := net.Dial("tcp", "host1:port")
closer.Add("conn1", conn1)

conn2, _ := net.Dial("tcp", "host2:port")
closer.Add("conn2", conn2)

// Automatic cleanup on context cancel or explicit close
defer closer.Close()

5. Protocol Parsing

Problem: Parse delimited protocol messages efficiently.

Solution: Use delim scanner.

conn, _ := net.Dial("tcp", "server:port")
scanner := delim.NewScanner(conn, '\n')

for scanner.Scan() {
    message := scanner.Text()
    processMessage(message)
}

---

Quick Start

Installation

# Install entire package
go get github.com/nabbar/golib/ioutils

# Import specific subpackages as needed
import (
    "github.com/nabbar/golib/ioutils"
    "github.com/nabbar/golib/ioutils/aggregator"
    "github.com/nabbar/golib/ioutils/multi"
    "github.com/nabbar/golib/ioutils/ioprogress"
)

Basic Examples

Path Management:

import "github.com/nabbar/golib/ioutils"

// Create config directory
if err := ioutils.PathCheckCreate(false, "/etc/myapp", 0644, 0755); err != nil {
    log.Fatal(err)
}

// Create log file
if err := ioutils.PathCheckCreate(true, "/var/log/myapp.log", 0644, 0755); err != nil {
    log.Fatal(err)
}

Write Aggregation (aggregator):

import "github.com/nabbar/golib/ioutils/aggregator"

cfg := aggregator.Config{
    BufWriter: 100,
    FctWriter: func(p []byte) (int, error) {
        return logFile.Write(p)
    },
}

agg, _ := aggregator.New(ctx, cfg, logger)
agg.Start(ctx)
defer agg.Close()

// Write from multiple goroutines safely
for i := 0; i < 10; i++ {
    go func(id int) {
        agg.Write([]byte(fmt.Sprintf("Log from goroutine %d\n", id)))
    }(i)
}

Write Multiplexing (multi):

import "github.com/nabbar/golib/ioutils/multi"

// Create multi-writer
mw := multi.New()

// Add multiple destinations
file1, _ := os.Create("output1.txt")
file2, _ := os.Create("output2.txt")

mw.AddWriter(file1)
mw.AddWriter(file2)

// Write to both files at once
mw.Write([]byte("This goes to both files\n"))

Progress Tracking (ioprogress):

import "github.com/nabbar/golib/ioutils/ioprogress"

// Wrap reader with progress callback
reader := ioprogress.NewReader(file, func(bytes int64) {
    fmt.Printf("Read %d bytes so far\n", bytes)
})

// Read operations trigger callbacks
io.Copy(dest, reader)

Delimiter Reading (delim):

import "github.com/nabbar/golib/ioutils/delim"

scanner := delim.NewScanner(file, '\n')

for scanner.Scan() {
    line := scanner.Text()
    fmt.Println(line)
}

---

Best Practices

✅ DO

Choose the Right Subpackage:

// For concurrent writes → aggregator
// For broadcast writes → multi
// For progress tracking → ioprogress
// For resource management → mapCloser
// For delimiter parsing → delim

Proper Resource Cleanup:

// Always close resources
agg, _ := aggregator.New(ctx, cfg, logger)
defer agg.Close()

// Use context for lifecycle
ctx, cancel := context.WithCancel(parent)
defer cancel()

Buffer Sizing:

// Size buffers based on workload
cfg := aggregator.Config{
    BufWriter: writeRate * maxLatency * 1.5,  // Safety margin
    // ...
}

Error Handling:

// Check errors from I/O operations
if n, err := writer.Write(data); err != nil {
    log.Error("Write failed:", err)
    return err
}

Thread Safety:

// All subpackages are thread-safe
for i := 0; i < 10; i++ {
    go func() {
        agg.Write(data)  // Safe concurrent access
    }()
}

❌ DON'T

Don't Ignore Buffer Limits:

// ❌ BAD: No buffer with slow writer
cfg := aggregator.Config{
    BufWriter: 1,  // Will block immediately
    FctWriter: slowWriter,
}

// ✅ GOOD: Sized for throughput
cfg := aggregator.Config{
    BufWriter: 1000,
    FctWriter: slowWriter,
}

Don't Mix Interfaces:

// ❌ BAD: Writing directly bypasses aggregator
agg.Start(ctx)
logFile.Write(data)  // Bypasses serialization

// ✅ GOOD: All writes through aggregator
agg.Write(data)

Don't Forget Context:

// ❌ BAD: No cancellation
agg, _ := aggregator.New(context.Background(), cfg, logger)

// ✅ GOOD: Cancellable context
ctx, cancel := context.WithCancel(parent)
defer cancel()
agg, _ := aggregator.New(ctx, cfg, logger)

Don't Leave Resources Open:

// ❌ BAD: No cleanup
closer := mapcloser.New(ctx)
closer.Add("conn", conn)
// Program exits, resources leak

// ✅ GOOD: Explicit cleanup
defer closer.Close()

---

Testing

Comprehensive test suite with 776 specs across all subpackages.

Quick Test:

# Run all tests
go test ./...

# With race detector
CGO_ENABLED=1 go test -race ./...

# Coverage report
go test -cover ./...

Expected Results:

• 776 specs passed
• 90.4% average coverage
• Zero race conditions
• ~41 seconds execution time

See TESTING.md for detailed testing documentation including:

• Running tests (standard, race, coverage, profiling)
• Performance benchmarks per subpackage
• Writing new tests
• CI integration examples

---

Contributing

Contributions are welcome! Please follow these guidelines:

1. Code Quality

   • Follow Go best practices and idioms
   • Maintain or improve code coverage (target: >85%)
   • Pass all tests including race detector
   • Use gofmt and golint

2. AI Usage Policy

   • ❌ Do NOT use AI for implementing package functionality or core logic
   • ✅ AI may assist with:
     • Writing and improving tests
     • Documentation and comments
     • Debugging and troubleshooting
   • All AI-assisted contributions must be reviewed and validated by humans

3. Testing

   • Add tests for new features
   • Use Ginkgo v2 / Gomega for test framework
   • Ensure zero race conditions
   • Maintain coverage above 85%

4. Documentation

   • Update GoDoc comments for public APIs
   • Add examples for new features
   • Update README.md if adding subpackages
   • Update TESTING.md if changing test structure

5. Pull Request Process

   • Fork the repository
   • Create a feature branch
   • Write clear commit messages
   • Ensure all tests pass
   • Update documentation
   • Submit PR with description of changes

---

Improvements & Security

Current Status

The package is production-ready with no urgent improvements or security vulnerabilities identified across all subpackages.

Code Quality Metrics

• ✅ 90.4% average test coverage (target: >85%)
• ✅ Zero race conditions detected with -race flag
• ✅ Thread-safe implementations across all subpackages
• ✅ Memory-safe with proper resource cleanup
• ✅ Standard interfaces for maximum compatibility
• ✅ 776 comprehensive test specs ensuring reliability

Future Enhancements (Non-urgent)

The following enhancements could be considered for future versions:

New Subpackages:

1. iozip: Streaming compression/decompression wrappers
2. iocrypto: Encryption/decryption stream wrappers
3. ioratelimit: Bandwidth throttling and rate limiting
4. iocache: Write-through/write-back caching layers

Performance Optimizations:

1. SIMD-accelerated delimiter scanning (delim)
2. Lock-free queues for aggregator
3. Memory pool for buffer allocation
4. Zero-copy operations where possible

Monitoring Enhancements:

1. Prometheus metrics integration
2. OpenTelemetry tracing
3. Structured logging throughout
4. Performance profiling hooks

Advanced Features:

1. Async I/O support (io_uring on Linux)
2. Adaptive buffer sizing based on load
3. Priority queuing in aggregator
4. Circuit breaker patterns for reliability

These are optional improvements and not required for production use. The current implementation is stable, performant, and feature-complete for its intended use cases.

Suggestions and contributions are welcome via GitHub issues.

---

Resources

Internal Documentation

• GoDoc - Complete API documentation
• TESTING.md - Test suite documentation
• Individual subpackage READMEs (linked in Subpackages)

Related Packages

• github.com/nabbar/golib/logger - Logging interface used by subpackages
• github.com/nabbar/golib/runner/startStop - Lifecycle management
• github.com/nabbar/golib/socket/server - Socket server (uses aggregator)

External References

• Go io package - Standard library I/O interfaces
• Effective Go - Go best practices
• Go Concurrency Patterns - Official Go blog

---

AI Transparency

In compliance with EU AI Act Article 50.4: AI assistance was used for testing, documentation, and bug resolution under human supervision. All core functionality is human-designed and validated.

---

License

MIT License - See LICENSE file for details.

Copyright (c) 2025 Nicolas JUHEL

---

Maintained by: Nicolas JUHEL
Package: github.com/nabbar/golib/ioutils
Version: See releases for versioning

Documentation

Overview

Package ioutils provides a comprehensive suite of I/O utilities for Go applications, offering enhanced functionality for file operations, stream management, and data flow control.

Design Philosophy

The ioutils package is built around three core principles:

1. Safety: All operations include proper error handling and resource cleanup 2. Flexibility: Components can be combined to create complex I/O workflows 3. Performance: Optimized for both throughput and memory efficiency

Architecture

The package is organized into a root-level utility and several specialized subpackages:

Root Package (ioutils)
├── PathCheckCreate      - File/directory creation with permission management
│
├── aggregator          - Thread-safe write aggregator that buffers and serializes concurrent writes to a single writer function
├── bufferReadCloser    - io.Closer wrappers for bytes.Buffer and bufio types
├── delim               - Buffered reader for reading delimiter-separated data streams
├── fileDescriptor      - Cross-platform utilities for managing file descriptor limits
├── ioprogress          - Thread-safe I/O progress tracking wrappers for monitoring read and write operations
├── iowrapper           - Flexible I/O wrapper enabling customization and interception of I/O operations
├── mapCloser           - Thread-safe, context-aware manager for multiple io.Closer instances
├── maxstdio            - Standard I/O redirection and capture
├── multi               - Thread-safe, adaptive multi-writer extending io.MultiWriter with advanced features
└── nopwritecloser      - io.WriteCloser wrapper with no-op Close()

Each subpackage is designed to solve specific I/O challenges while maintaining compatibility with standard library interfaces.

Key Features

Path Management:

• Atomic file/directory creation
• Automatic parent directory creation
• Permission validation and correction
• Type checking (file vs directory)

Stream Processing:

• Buffered write aggregation
• Progress tracking
• Delimiter-based parsing
• Multiple source/destination handling

Resource Management:

• Automatic cleanup via defer patterns
• Closer collection management
• File descriptor lifecycle control

Performance Characteristics

The package is optimized for:

• Low memory overhead through buffer pooling
• Minimal allocations in hot paths
• Concurrent-safe operations where needed
• Efficient permission checking

Benchmarks show:

• PathCheckCreate: ~100ns for existing paths with correct permissions
• Aggregator: 10x throughput improvement for concurrent writes
• BufferReadCloser: 50% memory reduction vs bytes.Buffer for large streams

Limitations

1. Platform Dependencies:

• Permission handling varies between Unix and Windows
• File descriptor operations may require OS-specific handling
• Some permissions (e.g., write-only on Windows) have limited support

2. Concurrency:

• PathCheckCreate is not atomic for the same path across goroutines
• Subpackages provide their own concurrency guarantees

3. Error Handling:

• Does not use panic; all errors must be explicitly handled
• Some operations may partially succeed before returning an error

Common Use Cases

Application Initialization:

// Ensure application directories exist
if err := ioutils.PathCheckCreate(false, "/var/app/data", 0644, 0755); err != nil {
    return fmt.Errorf("data dir: %w", err)
}

// Create log file with proper permissions
if err := ioutils.PathCheckCreate(true, "/var/log/app.log", 0644, 0755); err != nil {
    return fmt.Errorf("log file: %w", err)
}

Configuration Management:

// Create config directory structure
configPaths := []string{
    "/etc/app/config",
    "/etc/app/config/plugins",
    "/etc/app/config/templates",
}

for _, path := range configPaths {
    if err := ioutils.PathCheckCreate(false, path, 0644, 0750); err != nil {
        return err
    }
}

Log File Rotation:

// Ensure log directory exists before rotation
logDir := filepath.Dir(logPath)
if err := ioutils.PathCheckCreate(false, logDir, 0644, 0755); err != nil {
    return err
}

// Create new log file
if err := ioutils.PathCheckCreate(true, logPath, 0644, 0755); err != nil {
    return err
}

See subpackage documentation for advanced I/O operations including write aggregation, progress tracking, and stream processing.

Thread Safety

The root PathCheckCreate function is safe to call from multiple goroutines for different paths. However, concurrent calls for the same path may result in race conditions. Use external synchronization if needed.

Each subpackage documents its own concurrency guarantees.

Error Handling

All functions return errors that can be inspected using standard error handling patterns:

if err := ioutils.PathCheckCreate(true, path, 0644, 0755); err != nil {
    if errors.Is(err, os.ErrPermission) {
        // Handle permission error
    } else if errors.Is(err, os.ErrNotExist) {
        // Handle path error
    } else {
        // Handle other errors
    }
}

Errors are wrapped to provide context about the operation that failed.

Best Practices

1. Always check errors - this package never uses panic 2. Use appropriate permissions - follow least privilege principle 3. Clean up resources - use defer for closers 4. Consider platform differences - test on target OS 5. Validate paths - ensure paths are absolute when needed

Migration Notes

When migrating from os package functions:

// Before
if err := os.MkdirAll(path, 0755); err != nil {
    return err
}

// After - also handles permission updates
if err := ioutils.PathCheckCreate(false, path, 0644, 0755); err != nil {
    return err
}

The PathCheckCreate function combines mkdir, permission checking, and type validation in a single call.

Related Packages

• os: Standard library file operations
• io: Standard library I/O interfaces
• filepath: Path manipulation utilities
• bufio: Buffered I/O operations

Subpackage Overview

For detailed documentation, see individual subpackage docs:

• aggregator: Thread-safe write aggregator that buffers and serializes concurrent write operations to a single writer function
• bufferReadCloser: io.Closer wrappers for bytes.Buffer and bufio types with cleanup
• delim: Buffered reader for reading delimiter-separated data streams
• fileDescriptor: Cross-platform utilities for managing file descriptor limits in Go applications
• ioprogress: Thread-safe I/O progress tracking wrappers for monitoring read and write operations in real-time through customizable callbacks
• iowrapper: Flexible I/O wrapper that enables customization and interception of read, write, seek, and close operations on any underlying I/O object
• mapCloser: Thread-safe, context-aware manager for multiple io.Closer instances
• maxstdio: Standard I/O stream redirection and capture
• multi: Thread-safe, adaptive multi-writer that extends Go's standard io.MultiWriter with advanced features
• nopwritecloser: No-op Close() wrapper for io.Writer (delegates Write unchanged)

Index

• func PathCheckCreate(isFile bool, path string, permFile os.FileMode, permDir os.FileMode) error

Examples

• PathCheckCreate (ApplicationInit)
• PathCheckCreate (BasicDirectory)
• PathCheckCreate (BasicFile)
• PathCheckCreate (ErrorHandling)
• PathCheckCreate (Idempotent)
• PathCheckCreate (LogRotation)
• PathCheckCreate (MultipleFiles)
• PathCheckCreate (NestedPath)
• PathCheckCreate (PermissionUpdate)
• PathCheckCreate (SecureConfig)

Functions

func PathCheckCreate

func PathCheckCreate(isFile bool, path string, permFile os.FileMode, permDir os.FileMode) error

PathCheckCreate ensures a file or directory exists at the given path with the correct permissions.

This function performs the following operations:

• Checks if the path exists
• Creates the path if it doesn't exist (file or directory based on isFile parameter)
• Creates parent directories if needed
• Validates that existing paths match the expected type (file vs directory)
• Updates permissions if they don't match the specified values

Parameters:

• isFile: true to ensure path is a file, false for a directory
• path: the filesystem path to check/create (should be absolute for predictable behavior)
• permFile: permissions to apply to files (e.g., 0644 for rw-r--r--)
• permDir: permissions to apply to directories (e.g., 0755 for rwxr-xr-x)

Returns:

• error: nil on success, or an error if:
• The path exists but is the wrong type (file when directory expected, or vice versa)
• Parent directory creation fails
• Permission updates fail
• File/directory creation fails
• Path is empty or invalid

Behavior:

• If path exists as expected type: validates and updates permissions if needed
• If path doesn't exist: creates it with specified permissions
• If parent directories don't exist: creates them with permDir permissions
• If path exists as wrong type: returns an error without modification

Implementation Details:

• Uses os.OpenRoot for atomic file creation on supported systems
• Permission comparison checks full mode, not just permission bits
• Recursively creates parent directories as needed
• Attempts to set permissions even if initial mode is close

Example:

// Ensure config directory exists with 0755 permissions
err := PathCheckCreate(false, "/etc/app/config", 0644, 0755)

// Ensure log file exists with 0644 permissions
err := PathCheckCreate(true, "/var/log/app.log", 0644, 0755)

Thread Safety:

This function is safe to call concurrently for different paths, but
concurrent calls for the same path may result in race conditions.
Use external synchronization if the same path may be accessed concurrently.

Example (ApplicationInit)

ExamplePathCheckCreate_applicationInit demonstrates a realistic use case: initializing application directories during startup.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	appRoot := filepath.Join(tmpDir, "myapp")

	// Define application directory structure
	dirs := []string{
		filepath.Join(appRoot, "data"),
		filepath.Join(appRoot, "config"),
		filepath.Join(appRoot, "logs"),
		filepath.Join(appRoot, "cache"),
		filepath.Join(appRoot, "tmp"),
	}

	// Create all directories
	for _, dir := range dirs {
		if err := ioutils.PathCheckCreate(false, dir, 0644, 0755); err != nil {
			log.Fatalf("Failed to create %s: %v", dir, err)
		}
	}

	// Create important files
	files := map[string]os.FileMode{
		filepath.Join(appRoot, "config", "app.conf"): 0640, // Restrictive config
		filepath.Join(appRoot, "logs", "app.log"):    0644, // Standard log
		filepath.Join(appRoot, "data", "state.json"): 0644, // Data file
	}

	for file, perm := range files {
		if err := ioutils.PathCheckCreate(true, file, perm, 0755); err != nil {
			log.Fatalf("Failed to create %s: %v", file, err)
		}
	}

	// Clean up
	defer os.RemoveAll(appRoot)

	// Verify structure
	allExist := true
	for _, dir := range dirs {
		if _, err := os.Stat(dir); err != nil {
			allExist = false
			break
		}
	}
	for file := range files {
		if _, err := os.Stat(file); err != nil {
			allExist = false
			break
		}
	}

	if allExist {
		fmt.Println("Application structure initialized")
	}

}

Output:

Application structure initialized

Example (BasicDirectory)

ExamplePathCheckCreate_basicDirectory demonstrates the simplest use case: creating a single directory with standard permissions.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	dirPath := filepath.Join(tmpDir, "example_dir")

	// Create a directory with standard permissions
	// isFile=false means we want a directory
	// 0644 is file permission (not used for directories)
	// 0755 is directory permission (rwxr-xr-x)
	err := ioutils.PathCheckCreate(false, dirPath, 0644, 0755)
	if err != nil {
		log.Fatalf("Failed to create directory: %v", err)
	}

	// Clean up
	defer os.RemoveAll(dirPath)

	// Verify it exists
	if info, err := os.Stat(dirPath); err == nil && info.IsDir() {
		fmt.Println("Directory created successfully")
	}

}

Output:

Directory created successfully

Example (BasicFile)

ExamplePathCheckCreate_basicFile demonstrates creating a single file with appropriate permissions.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	filePath := filepath.Join(tmpDir, "example_file.txt")

	// Create a file with standard permissions
	// isFile=true means we want a file
	// 0644 is file permission (rw-r--r--)
	// 0755 is directory permission for parent dirs
	err := ioutils.PathCheckCreate(true, filePath, 0644, 0755)
	if err != nil {
		log.Fatalf("Failed to create file: %v", err)
	}

	// Clean up
	defer os.Remove(filePath)

	// Verify it exists
	if info, err := os.Stat(filePath); err == nil && !info.IsDir() {
		fmt.Println("File created successfully")
	}

}

Output:

File created successfully

Example (ErrorHandling)

ExamplePathCheckCreate_errorHandling demonstrates proper error handling when path conflicts occur.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	path := filepath.Join(tmpDir, "example_conflict")

	// Create as directory first
	if err := ioutils.PathCheckCreate(false, path, 0644, 0755); err != nil {
		log.Fatalf("Failed to create directory: %v", err)
	}

	// Clean up
	defer os.RemoveAll(path)

	// Try to create as file - should fail
	err := ioutils.PathCheckCreate(true, path, 0644, 0755)
	if err != nil {
		fmt.Println("Expected error: path exists as directory")
	}

}

Output:

Expected error: path exists as directory

Example (Idempotent)

ExamplePathCheckCreate_idempotent demonstrates that PathCheckCreate can be called multiple times safely.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	dirPath := filepath.Join(tmpDir, "example_idempotent")

	// Clean up
	defer os.RemoveAll(dirPath)

	// Call multiple times - should not error
	for i := 0; i < 3; i++ {
		err := ioutils.PathCheckCreate(false, dirPath, 0644, 0755)
		if err != nil {
			log.Fatalf("Failed on iteration %d: %v", i, err)
		}
	}

	fmt.Println("Idempotent calls successful")

}

Output:

Idempotent calls successful

Example (LogRotation)

ExamplePathCheckCreate_logRotation demonstrates using PathCheckCreate for log file rotation scenarios.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	logDir := filepath.Join(tmpDir, "example_logs")
	currentLog := filepath.Join(logDir, "app.log")

	// Ensure log directory exists
	if err := ioutils.PathCheckCreate(false, logDir, 0644, 0755); err != nil {
		log.Fatalf("Failed to create log directory: %v", err)
	}

	// Create or verify current log file
	if err := ioutils.PathCheckCreate(true, currentLog, 0644, 0755); err != nil {
		log.Fatalf("Failed to create log file: %v", err)
	}

	// Simulate rotation - archive old log
	archivedLog := filepath.Join(logDir, "app.log.1")
	if err := os.Rename(currentLog, archivedLog); err == nil {
		// Create new log file
		if err := ioutils.PathCheckCreate(true, currentLog, 0644, 0755); err != nil {
			log.Fatalf("Failed to create new log file: %v", err)
		}
	}

	// Clean up
	defer os.RemoveAll(logDir)

	// Verify both files exist
	if _, err := os.Stat(currentLog); err == nil {
		if _, err := os.Stat(archivedLog); err == nil {
			fmt.Println("Log rotation successful")
		}
	}

}

Output:

Log rotation successful

Example (MultipleFiles)

ExamplePathCheckCreate_multipleFiles demonstrates creating multiple files efficiently in a batch operation.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	baseDir := filepath.Join(tmpDir, "example_batch")

	// Create base directory
	if err := ioutils.PathCheckCreate(false, baseDir, 0644, 0755); err != nil {
		log.Fatalf("Failed to create base directory: %v", err)
	}

	// Clean up
	defer os.RemoveAll(baseDir)

	// Create multiple files with different permissions
	files := []struct {
		path string
		perm os.FileMode
	}{
		{filepath.Join(baseDir, "public.txt"), 0644},
		{filepath.Join(baseDir, "private.txt"), 0600},
		{filepath.Join(baseDir, "executable.sh"), 0755},
		{filepath.Join(baseDir, "data", "nested.dat"), 0644},
	}

	for _, f := range files {
		if err := ioutils.PathCheckCreate(true, f.path, f.perm, 0755); err != nil {
			log.Fatalf("Failed to create %s: %v", f.path, err)
		}
	}

	// Verify all created
	allCreated := true
	for _, f := range files {
		if _, err := os.Stat(f.path); err != nil {
			allCreated = false
			break
		}
	}

	if allCreated {
		fmt.Println("Multiple files created successfully")
	}

}

Output:

Multiple files created successfully

Example (NestedPath)

ExamplePathCheckCreate_nestedPath demonstrates creating a file with automatic parent directory creation.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	filePath := filepath.Join(tmpDir, "example_nested", "deep", "path", "file.log")

	// Create file in nested directories - parent dirs created automatically
	err := ioutils.PathCheckCreate(true, filePath, 0644, 0755)
	if err != nil {
		log.Fatalf("Failed to create nested file: %v", err)
	}

	// Clean up
	defer os.RemoveAll(filepath.Join(tmpDir, "example_nested"))

	// Verify the entire path exists
	if _, err := os.Stat(filePath); err == nil {
		fmt.Println("Nested file structure created successfully")
	}

}

Output:

Nested file structure created successfully

Example (PermissionUpdate)

ExamplePathCheckCreate_permissionUpdate demonstrates how PathCheckCreate updates permissions if the path already exists.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	filePath := filepath.Join(tmpDir, "example_perm_update.txt")

	// Create file with restrictive permissions
	err := ioutils.PathCheckCreate(true, filePath, 0600, 0755)
	if err != nil {
		log.Fatalf("Failed to create file: %v", err)
	}

	// Clean up
	defer os.Remove(filePath)

	// Update permissions by calling again with different perms
	err = ioutils.PathCheckCreate(true, filePath, 0644, 0755)
	if err != nil {
		log.Fatalf("Failed to update permissions: %v", err)
	}

	// Verify new permissions
	if info, err := os.Stat(filePath); err == nil {
		if info.Mode()&0777 == 0644 {
			fmt.Println("Permissions updated successfully")
		}
	}

}

Output:

Permissions updated successfully

Example (SecureConfig)

ExamplePathCheckCreate_secureConfig demonstrates creating configuration files with restrictive permissions for security.

package main

import (
	"fmt"
	"log"
	"os"
	"path/filepath"

	"github.com/nabbar/golib/ioutils"
)

func main() {
	tmpDir := os.TempDir()
	configDir := filepath.Join(tmpDir, "example_secure_config")
	secretsFile := filepath.Join(configDir, "secrets.conf")

	// Create config directory with standard permissions
	if err := ioutils.PathCheckCreate(false, configDir, 0644, 0750); err != nil {
		log.Fatalf("Failed to create config directory: %v", err)
	}

	// Create secrets file with restrictive permissions (owner read/write only)
	if err := ioutils.PathCheckCreate(true, secretsFile, 0600, 0750); err != nil {
		log.Fatalf("Failed to create secrets file: %v", err)
	}

	// Clean up
	defer os.RemoveAll(configDir)

	// Verify restrictive permissions
	if info, err := os.Stat(secretsFile); err == nil {
		if info.Mode()&0777 == 0600 {
			fmt.Println("Secure configuration file created")
		}
	}

}

Output:

Secure configuration file created