fio

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 30, 2026 License: MIT Imports: 15 Imported by: 0

README

fio

Package fio provides streaming I/O utilities with session management, automatic resource cleanup, and flexible storage backends (memory/file).

Features

  • Write Once, Dynamic Storage - Same code works with memory or file storage; automatically switches based on data size
  • Type-safe Sources - Strongly typed input sources (files, URLs, bytes, readers, multipart)
  • Session Management - Automatic temp file cleanup via IoManager/IoSession
  • Dual Storage Backends - Memory or file-based storage with automatic spill-to-disk
  • Reusable Inputs - Reset and reuse input streams multiple times
  • ReaderAt Conversion - Convert any io.Reader to io.ReaderAt with memory/temp-file buffering
  • Memory-mapped I/O - Optional mmap support on Unix systems for fast file reading

Why fio?

Write once, run with any storage backend. Your code doesn't change whether data is stored in memory or files:

// Same code - storage is determined by configuration, not code changes
output, _ := fio.Copy(ctx, fio.PathSource("input.txt"), fio.Out(".txt"))

// Read result the same way regardless of storage type
data, _ := output.Bytes()      // works for both memory and file
reader, _ := output.OpenReader() // works for both memory and file

Automatic storage selection based on data size:

// Manager with auto-threshold: small data → memory, large data → file
mgr, _ := fio.NewIoManager("./temp", fio.Memory,
    fio.WithThreshold(10*1024*1024),  // Switch to file at 10MB
)

// Your code stays the same - fio decides storage automatically
output, _ := fio.Copy(ctx, source, fio.Out(".json"))
// 1KB file  → stored in memory
// 50MB file → stored in temp file

Benefits:

  • No if/else for memory vs file handling
  • Automatic cleanup of temp files via session
  • Consistent API regardless of storage backend
  • Optimal performance for each scenario

Installation

go get github.com/dreamph/fio

import "github.com/dreamph/fio"

Quick Start

Basic Copy
// Create an IoManager with memory storage
mgr, _ := fio.NewIoManager("./temp", fio.Memory)
defer mgr.Cleanup()

// Create a session
ses, _ := mgr.NewSession()
defer ses.Cleanup()

// Use session in context
ctx := fio.WithSession(context.Background(), ses)

// Copy from file to memory output
output, _ := fio.Copy(ctx, fio.PathSource("input.txt"), fio.Out(".txt"))

// Read the result
data, _ := output.Bytes()
Reading Files
// Read and process a file
result, err := fio.Read(ctx, fio.PathSource("data.json"), func(r io.Reader) (*MyData, error) {
    var data MyData
    if err := json.NewDecoder(r).Decode(&data); err != nil {
        return nil, err
    }
    return &data, nil
})
Processing with Output
// Transform input to output
output, err := fio.Process(ctx, fio.PathSource("input.txt"), fio.Out(".txt"),
    func(r io.Reader, w io.Writer) error {
        // Transform data from r to w
        _, err := io.Copy(w, r)
        return err
    })

Source Types

Create type-safe input sources:

// From file path
src := fio.PathSource("/path/to/file.txt")

// From URL (auto-downloads)
src := fio.URLSource("https://example.com/file.txt")

// From bytes
src := fio.BytesSource([]byte("hello world"))

// From io.Reader
src := fio.ReaderSource(reader)

// From io.ReadCloser
src := fio.ReadCloserSource(readCloser)

// From *os.File
src := fio.FileSource(file)

// From multipart file header
src := fio.MultipartSource(fileHeader)

// From existing Output
src := fio.OutputSource(output)

// From existing Input
src := fio.InputSource(input)

Session Management

IoManager

Manages temp directories and creates sessions:

// Create manager with file storage backend
mgr, err := fio.NewIoManager("./temp", fio.File,
    fio.WithThreshold(1024*1024),      // Auto-switch to file at 1MB
    fio.WithSpillThreshold(64<<20),    // Spill memory to file at 64MB
    fio.WithMaxPreallocate(1<<20),     // Cap pre-allocation at 1MB
    fio.WithMmap(true),                // Enable mmap on Unix
)
defer mgr.Cleanup()
IoSession

Represents a single operation scope with automatic cleanup:

ses, _ := mgr.NewSession()
defer ses.Cleanup() // Cleans up all temp files

// Create output within session
output, _ := ses.NewOut(fio.Out(".json"), 1024)
Context Integration
ctx := fio.WithSession(context.Background(), ses)

// Retrieve session from context
ses := fio.Session(ctx)

Output Configuration

Configure output behavior:

// Basic output with extension
out := fio.Out(".json")

// Force memory storage
out := fio.Out(".json", fio.Memory)

// Force file storage
out := fio.Out(".json", fio.File)

// With spill threshold
out := fio.Out(".json", fio.WithSpillThreshold(32<<20))

// With output reuse (for repeated operations)
var cached *fio.Output
out := fio.Out(".json", fio.OutReuse(&cached))

Reusable Inputs

Open a source once and read multiple times:

// Open as reusable
input, _ := fio.OpenIn(ctx, fio.PathSource("data.txt"), fio.Reusable())
defer input.Close()

// First read
io.Copy(w1, input.Reader)

// Reset and read again
input.Reset()
io.Copy(w2, input.Reader)

ReaderAt Conversion

Convert streaming readers to random-access:

// Auto-buffers in memory or spills to temp file
result, _ := fio.ToReaderAt(ctx, reader,
    fio.WithMaxMemoryBytes(8<<20),  // Buffer up to 8MB in memory
    fio.WithTempDir("./temp"),       // Temp dir for spill files
)
defer result.Cleanup()

ra := result.ReaderAt()
size := result.Size()

Scoped Operations

Read-only Scope (Do)
result, err := fio.Do(ctx, func(s *fio.Scope) (MyResult, error) {
    r, err := s.Use(fio.PathSource("input.txt"))  // Auto-cleanup on scope exit
    if err != nil {
        return MyResult{}, err
    }
    // Process r...
    return result, nil
})
Output Scope (DoOut)
output, err := fio.DoOut(ctx, fio.Out(".txt"),
    func(ctx context.Context, s *fio.OutScope, w io.Writer) error {
        r, err := s.Use(fio.PathSource("input.txt"))
        if err != nil {
            return err
        }
        _, err := io.Copy(w, r)
        return err
    })
Output with Result (DoOutResult)
output, metadata, err := fio.DoOutResult(ctx, fio.Out(".txt"),
    func(ctx context.Context, s *fio.OutScope, w io.Writer) (*Metadata, error) {
        r, size, err := s.UseSized(fio.PathSource("input.txt"))
        if err != nil {
            return nil, err
        }
        _, err := io.Copy(w, r)
        if err != nil {
            return nil, err
        }
        m := Metadata{Size: size}
        return &m, nil
    })

Utility Functions

Size Detection
// Get size from source (may open the source)
size, _ := fio.Size(ctx, src)

// Get size without opening (when possible)
size := fio.SizeFromStream(src)

// Get size from any type
size := fio.SizeAny(reader)
Line Reading
err := fio.ReadLines(ctx, fio.PathSource("file.txt"), func(line string) error {
    fmt.Println(line)
    return nil
})

// Shorthand for file path
err := fio.ReadFileLines(ctx, "file.txt", func(line string) error {
    return nil
})
Direct File Writing
// Write reader to file
n, err := fio.WriteFile(reader, "/path/to/output.txt")

// Write source to file
n, err := fio.WriteStreamToFile(src, "/path/to/output.txt")

Output Methods

// Get data as bytes
data, _ := output.Bytes()

// Get raw byte slice (memory storage only)
data := output.Data()

// Open reader
r, _ := output.OpenReader()
defer r.Close()

// Open writer
w, _ := output.OpenWriter(sizeHint)
defer w.Close()

// Write to io.Writer
n, _ := output.WriteTo(writer)

// Save to file
err := output.SaveAs("/path/to/file.txt")

// Keep file after session cleanup
output.Keep()

// Get file path (file storage only)
path := output.Path()

// Get storage type
st := output.StorageType()

// Get size
size := output.Size()

File Extension Constants

fio.Json  // ".json"
fio.Csv   // ".csv"
fio.Txt   // ".txt"
fio.Xml   // ".xml"
fio.Pdf   // ".pdf"
fio.Docx  // ".docx"
fio.Xlsx  // ".xlsx"
fio.Pptx  // ".pptx"
fio.Jpg   // ".jpg"
fio.Jpeg  // ".jpeg"
fio.Png   // ".png"
fio.Zip   // ".zip"

Helper Functions

// Convert MB to bytes
bytes := fio.MB(10) // 10485760

// Convert format to extension
ext := fio.ToExt("json") // ".json"

// Join cleanup functions
cleanup := fio.JoinCleanup(fn1, fn2, fn3)
defer cleanup()

// Safe close (ignores nil)
fio.SafeClose(closer)

Global Configuration

// Configure custom HTTP client (for URL sources)
fio.Configure(fio.NewConfig(&http.Client{
    Timeout: 60 * time.Second,
}))

Error Types

fio.ErrNilSource              // nil source provided
fio.ErrIoManagerClosed        // manager is closed
fio.ErrIoSessionClosed        // session is closed
fio.ErrDownloadFailed         // URL download failed
fio.ErrNoSession              // session is nil
fio.ErrFileStorageUnavailable // file storage requires directory
fio.ErrInvalidSessionType     // invalid session type
fio.ErrNilFunc                // function is nil
fio.ErrEmptyPath              // empty path string
fio.ErrEmptyURL               // empty URL string
fio.ErrOutputCleaned          // output already cleaned up
fio.ErrNilOutHandle           // nil OutHandle
fio.ErrNilOutScope            // nil out-scope
fio.ErrNewOutMultiple         // NewOut called more than once
fio.ErrOutReuseRequiresPtr    // OutReuse requires output pointer
fio.ErrCannotGetReaderAt      // reader does not support ReaderAt
fio.ErrInputNotReusable       // input is not reusable
fio.ErrCannotResetInput       // input reset failed
fio.ErrToReaderAtNilReader    // ToReaderAt called with nil reader

Use errors.Is to check wrapped errors:

if errors.Is(err, fio.ErrDownloadFailed) {
    // handle URL download failures
}

Platform Support

  • Memory-mapped I/O: Available on Darwin, Linux, FreeBSD, NetBSD, OpenBSD
  • Other platforms: Falls back to standard file I/O

Benchmark Comparison

Benchmark comparing fio and normal (standard library io.Copy) on Apple M2 Max.

Legend
Symbol Meaning
⚡ Fastest speed
💾 Lowest memory
🏆 Best overall
Bytes Source → Memory Storage
Size Method Speed Throughput Memory Allocs Notes
1KB normal 201 ns 5,084 MB/s 1,152 B 5 copies data
fio 130 ns 7,865 MB/s 249 B 3 🏆⚡💾 zero-copy
1MB normal 115 µs 9,139 MB/s 1.0 MB 5 copies data
fio 137 ns 7,655,321 MB/s 241 B 3 🏆⚡💾 zero-copy
10MB normal 604 µs 17,362 MB/s 10 MB 5 copies data
fio 144 ns 72,978,112 MB/s 241 B 3 🏆⚡💾 zero-copy
100MB normal 3.61 ms 29,010 MB/s 100 MB 5 copies data
fio 133 ns 788,153,403 MB/s 240 B 3 🏆⚡💾 zero-copy
Bytes Source → File Storage
Size Method Speed Throughput Memory Allocs Notes
1KB normal 117 µs 8.7 MB/s 743 B 9 ⚡
fio 139 µs 7.4 MB/s 727 B 11 💾
1MB normal 379 µs 2,765 MB/s 744 B 9
fio 268 µs 3,914 MB/s 720 B 11 🏆⚡💾
10MB normal 2.80 ms 3,751 MB/s 746 B 9
fio 2.18 ms 4,818 MB/s 709 B 11 🏆⚡💾
100MB normal 23.1 ms 4,540 MB/s 781 B 9
fio 22.8 ms 4,591 MB/s 718 B 11 🏆⚡💾
File Source → Memory Storage
Size Method Speed Throughput Memory Allocs Notes
1KB normal 20.7 µs 49.4 MB/s 34,096 B 8
fio 17.9 µs 57.2 MB/s 1,965 B 13 🏆⚡💾
1MB normal 797 µs 1,315 MB/s 2.0 MB 13
fio 108 µs 9,685 MB/s 1.0 MB 13 🏆⚡💾
10MB normal 2.66 ms 3,939 MB/s 32 MB 17
fio 1.33 ms 7,893 MB/s 10 MB 13 🏆⚡💾
100MB normal 16.7 ms 6,290 MB/s 256 MB 20
fio 16.3 ms 6,432 MB/s 100 MB 13 🏆⚡💾
File Source → File Storage
Size Method Speed Throughput Memory Allocs Notes
1KB normal 144 µs 7.1 MB/s 33,696 B 13 ⚡💾
fio 166 µs 6.2 MB/s 33,730 B 17
1MB normal 513 µs 2,043 MB/s 33,712 B 13 ⚡💾
fio 573 µs 1,830 MB/s 33,730 B 17
10MB normal 4.32 ms 2,425 MB/s 33,712 B 13 ⚡💾
fio 4.51 ms 2,326 MB/s 33,721 B 17
100MB normal 44.0 ms 2,385 MB/s 33,717 B 13
fio 41.1 ms 2,553 MB/s 33,723 B 17 🏆⚡
Bytes Source → Read Only (no output)
Size Method Speed Throughput Memory Allocs Notes
1KB normal 39.6 ns 25,862 MB/s 64 B 2 ⚡💾
fio 65.0 ns 15,755 MB/s 136 B 3
1MB normal 43.5 ns 24.1 TB/s 64 B 2 ⚡💾 zero-copy discard
fio 72.3 ns 14.5 TB/s 136 B 3
10MB normal 41.0 ns 256 TB/s 64 B 2 ⚡💾 zero-copy discard
fio 68.1 ns 154 TB/s 136 B 3
100MB normal 39.9 ns 2.6 PB/s 64 B 2 ⚡💾 zero-copy discard
fio 66.2 ns 1.6 PB/s 136 B 3
File Source → Read Only (no output)
Size Method Speed Throughput Memory Allocs Notes
1KB normal 15.7 µs 65.1 MB/s 240 B 4 ⚡💾
fio 16.5 µs 62.1 MB/s 553 B 9
1MB normal 98.6 µs 10,630 MB/s 240 B 4 ⚡💾
fio 99.7 µs 10,516 MB/s 553 B 9
10MB fio 839 µs 12,488 MB/s 554 B 9 ⚡
normal 844 µs 12,424 MB/s 244 B 4 💾
100MB normal 11.1 ms 9,407 MB/s 245 B 4 💾
fio 11.5 ms 9,100 MB/s 583 B 9
Summary
Scenario Winner Why
bytes → memory 🏆 fio Zero-copy, fastest in all sizes, minimal memory
bytes → file 🏆 fio Faster at 1MB/10MB/100MB (28-41% faster); slightly slower at 1KB
file → memory 🏆 fio Faster for all sizes with 50% less memory
file → file mixed normal faster at small sizes (4-12%); fio faster at 100MB (7%)
bytes → read-only normal Both near-instant; fio has minimal overhead (~1.6x)
file → read-only ~equal Comparable performance; fio slightly more allocations
Key Takeaways
  1. fio bytes→memory is zero-copy - constant-time regardless of data size
  2. fio bytes→file is optimized - 28-41% faster than normal for 1MB+ files
  3. fio file→memory is efficient - 7x faster at 1MB with 50% less memory
  4. file→file is competitive - fio slightly slower at small files (~10%), faster at large files
  5. read-only is near-instant for bytes - both normal and fio use zero-copy to io.Discard
  6. file read-only is I/O bound - ~10-12 GB/s throughput, both methods comparable
Run Benchmarks
# Basic benchmark
go test -bench=BenchmarkCompareFio -benchmem -benchtime=3s

# With mmap enabled (Unix only)
FIO_BENCH_USE_MMAP=true go test -bench=BenchmarkCompareFio -benchmem -benchtime=3s

Documentation

Index

Constants

View Source 
const (
	Json = ".json"
	Csv  = ".csv"
	Txt  = ".txt"
	Xml  = ".xml"
	Pdf  = ".pdf"
	Docx = ".docx"
	Xlsx = ".xlsx"
	Pptx = ".pptx"
	Jpg  = ".jpg"
	Jpeg = ".jpeg"
	Png  = ".png"
	Zip  = ".zip"
)
View Source 
const (
	KindFile      = "file"
	KindURL       = "url"
	KindMultipart = "multipart"
	KindMemory    = "memory"
	KindReader    = "reader"
	KindStream    = "stream"
)
View Source 
const DefaultBaseTempDir = "./temp"

Variables

View Source 
var (
	ErrNilSource              = errors.New("fio: nil source")
	ErrIoManagerClosed        = errors.New("fio: manager is closed")
	ErrIoSessionClosed        = errors.New("fio: session is closed")
	ErrDownloadFailed         = errors.New("fio: download failed")
	ErrNoSession              = errors.New("fio: session is nil")
	ErrFileStorageUnavailable = errors.New("fio: file storage requires directory")
	ErrInvalidSessionType     = errors.New("fio: invalid session type")
	ErrNilFunc                = errors.New("fio: fn is nil")
	ErrEmptyPath              = errors.New("fio: empty path")
	ErrEmptyURL               = errors.New("fio: empty url")
	ErrOutputCleaned          = errors.New("fio: output is cleaned up")
	ErrNilOutHandle           = errors.New("fio: nil OutHandle")
	ErrNilOutScope            = errors.New("fio: nil out-scope")
	ErrNewOutMultiple         = errors.New("fio: NewOut called more than once")
	ErrOutReuseRequiresPtr    = errors.New("fio: OutReuse requires out pointer")
	ErrCannotGetReaderAt      = errors.New("fio: cannot get ReaderAt")
	ErrInputNotReusable       = errors.New("fio: input is not reusable")
	ErrCannotResetInput       = errors.New("fio: cannot reset input")
	ErrToReaderAtNilReader    = errors.New("fio: ToReaderAt: nil reader")
	ErrNilInput               = errors.New("fio: nil input")
)

Functions

func Configure 

func Configure(config Config) error

Configure applies global configuration (call at app startup only).

func Do 

func Do[T any](ctx context.Context, fn func(s *Scope) (*T, error)) (*T, error)

Do: returns *T only (NO output possible; Scope has no NewOut)

func JoinCleanup 

func JoinCleanup(fns ...func() error) func() error

func MB 

func MB(size int64) int64

func Read 

func Read(ctx context.Context, src Source, fn func(r io.Reader) error) error

func ReadAt 

func ReadAt(ctx context.Context, src Source, fn func(ra io.ReaderAt, size int64) error, opts ...ToReaderAtOption) error

func ReadAtResult 

func ReadAtResult[T any](ctx context.Context, src Source, fn func(ra io.ReaderAt, size int64) (*T, error), opts ...ToReaderAtOption) (*T, error)

func ReadFileLines 

func ReadFileLines(ctx context.Context, path string, fn LineFunc) error

func ReadLines 

func ReadLines(ctx context.Context, src Source, fn LineFunc) error

func ReadList 

func ReadList(ctx context.Context, srcs []Source, fn func(readers []io.Reader) error) error

func ReadListResult 

func ReadListResult[T any](ctx context.Context, srcs []Source, fn func(readers []io.Reader) (*T, error)) (*T, error)

func ReadResult 

func ReadResult[T any](ctx context.Context, src Source, fn func(r io.Reader) (*T, error)) (*T, error)

func SafeClose 

func SafeClose(c io.Closer)

func Size 

func Size(ctx context.Context, src Source) (int64, error)

Size returns the size of a Source, or -1 if unknown.

func SizeAny 

func SizeAny(x any) int64

func SizeFromStream 

func SizeFromStream(src Source) int64

SizeFromStream returns size for a type-safe Source without opening when possible. Returns -1 if size cannot be determined.

func SizeFromStreamList 

func SizeFromStreamList(srcs []Source) int64

SizeFromStreamList sums sizes for known sources. Returns -1 if any size cannot be determined.

func ToExt 

func ToExt(format string) string

func WithMaxPreallocate 

func WithMaxPreallocate(bytes int64) maxPreallocateOption

WithMaxPreallocate caps memory pre-allocation when sizeHint is provided. Set to 0 to disable the cap.

func WithMmap 

func WithMmap(enabled bool) mmapOption

WithMmap enables or disables mmap for file-to-memory fast paths.

func WithSession 

func WithSession(ctx context.Context, ses IoSession) context.Context

func WithSpillThreshold 

func WithSpillThreshold(bytes int64) spillThresholdOption

WithSpillThreshold forces Memory to spill to File when sizeHint >= bytes. Set to 0 to disable spill-to-file behavior.

func WithThreshold 

func WithThreshold(bytes int64) thresholdOption

WithThreshold sets session auto file threshold (bytes). 0 = disabled.

func WriteFile 

func WriteFile(r io.Reader, path string) (int64, error)

func WriteStreamToFile 

func WriteStreamToFile(src Source, path string) (int64, error)

Types

type Config 

type Config struct {
	// contains filtered or unexported fields
}

func NewConfig 

func NewConfig(client *http.Client) Config

func (Config) WithClient 

func (c Config) WithClient(client *http.Client) Config

type DownloadReaderCloser 

type DownloadReaderCloser interface {
	io.Reader
	io.Closer
}

DownloadReaderCloser is a convenience interface for a read-only stream that also supports Close.

func NewDownloadReaderCloser 

func NewDownloadReaderCloser(src Source, cleanup ...func()) (DownloadReaderCloser, error)

NewDownloadReaderCloser wraps a Source as a DownloadReaderCloser.

type InOption 

type InOption func(*inConfig)

func DeleteAfterUse 

func DeleteAfterUse() InOption

DeleteAfterUse deletes local temp files for file/stream sources after read.

func Reusable 

func Reusable() InOption

type Input 

type Input struct {
	Reader io.ReadCloser
	Size   int64
	Kind   string
	Path   string
	// contains filtered or unexported fields
}

Input represents an opened input source with metadata. Supports reusable reset (optional).

func OpenIn 

func OpenIn(ctx context.Context, src Source, opts ...InOption) (*Input, error)

OpenIn opens a type-safe Source and returns an Input. If Reusable() is set, it will buffer (non-file) into memory or use ReaderAt for files.

func (*Input) Close 

func (in *Input) Close() error

func (*Input) IsReusable 

func (in *Input) IsReusable() bool

func (*Input) ReaderAt 

func (in *Input) ReaderAt() io.ReaderAt

func (*Input) Reset 

func (in *Input) Reset() error

type IoManager 

type IoManager interface {
	NewSession() (IoSession, error)
	Cleanup() error
}

func NewIoManager 

func NewIoManager(baseDir string, storageType StorageType, opts ...ManagerOption) (IoManager, error)

type IoSession 

type IoSession interface {
	NewOut(out OutConfig, sizeHint ...int64) (*Output, error)
	Cleanup() error
}

func Session 

func Session(ctx context.Context) IoSession

type LineFunc 

type LineFunc func(line string) error

type ManagerOption 

type ManagerOption interface {
	// contains filtered or unexported methods
}

type ManagerOptionFunc 

type ManagerOptionFunc func(*managerConfig)

type OutConfig 

type OutConfig struct {
	// contains filtered or unexported fields
}

func Out 

func Out(ext string, opts ...OutOption) OutConfig

func WithOut 

func WithOut(ext string, opts ...OutOption) OutConfig

func (OutConfig) AutoThreshold 

func (o OutConfig) AutoThreshold() *int64

func (OutConfig) Ext 

func (o OutConfig) Ext() string

func (OutConfig) StorageTypeVal 

func (o OutConfig) StorageTypeVal() *StorageType

type OutHandle 

type OutHandle struct {
	Writer io.WriteCloser
	// contains filtered or unexported fields
}

func NewOut 

func NewOut(ctx context.Context, out OutConfig, sizeHint ...int64) (*OutHandle, error)

func (*OutHandle) Cleanup 

func (h *OutHandle) Cleanup() error

func (*OutHandle) Finalize 

func (h *OutHandle) Finalize() (*Output, error)

type OutOption 

type OutOption interface {
	// contains filtered or unexported methods
}

func OutReuse 

func OutReuse(outPtr **Output, opts ...OutReuseOpt) OutOption

OutReuse configures output reuse for OutScope.NewOut.

func WithStorage 

func WithStorage(st StorageType) OutOption

type OutOptionFunc 

type OutOptionFunc func(*OutConfig)

type OutReuseOpt 

type OutReuseOpt interface {
	// contains filtered or unexported methods
}

func WithCleanupOld 

func WithCleanupOld(v bool) OutReuseOpt

func WithKeepMemCap 

func WithKeepMemCap(v bool) OutReuseOpt

func WithMaxMemCap 

func WithMaxMemCap(bytes int64) OutReuseOpt

type OutReuseOptFunc 

type OutReuseOptFunc func(*outReuseConfig)

type OutScope 

type OutScope struct {
	Scope
	// contains filtered or unexported fields
}

func (*OutScope) NewOut 

func (s *OutScope) NewOut(out OutConfig, sizeHint ...int64) (io.Writer, error)

NewOut creates output writer (only available in OutScope).

func (*OutScope) UseReaderAt 

func (s *OutScope) UseReaderAt(src Source, opts ...ToReaderAtOption) (io.ReaderAt, int64, error)

UseReaderAt opens a Source as ReaderAt and records size for output decisions.

func (*OutScope) UseSized 

func (s *OutScope) UseSized(src Source) (io.Reader, int64, error)

UseSized opens a Source and records size for output decisions.

type Output 

type Output struct {
	// contains filtered or unexported fields
}

func Copy 

func Copy(ctx context.Context, src Source, out OutConfig) (*Output, error)

func DoOut 

func DoOut(ctx context.Context, outCfg OutConfig, fn func(ctx context.Context, s *OutScope, w io.Writer) error) (*Output, error)

DoOut: returns *Output only (output-capable scope)

func DoOutResult 

func DoOutResult[T any](ctx context.Context, outCfg OutConfig, fn func(ctx context.Context, s *OutScope, w io.Writer) (*T, error)) (*Output, *T, error)

DoOutResult: returns *Output + *T (output-capable scope)

func Process 

func Process(ctx context.Context, src Source, out OutConfig, fn func(r io.Reader, w io.Writer) error) (*Output, error)

func ProcessAt 

func ProcessAt(ctx context.Context, src Source, out OutConfig, fn func(ra io.ReaderAt, size int64, w io.Writer) error, opts ...ToReaderAtOption) (*Output, error)

func ProcessAtResult 

func ProcessAtResult[T any](ctx context.Context, src Source, out OutConfig, fn func(ra io.ReaderAt, size int64, w io.Writer) (*T, error), opts ...ToReaderAtOption) (*Output, *T, error)

func ProcessList 

func ProcessList(ctx context.Context, srcs []Source, out OutConfig, fn func(readers []io.Reader, w io.Writer) error) (*Output, error)

func ProcessListResult 

func ProcessListResult[T any](ctx context.Context, srcs []Source, out OutConfig, fn func(readers []io.Reader, w io.Writer) (*T, error)) (*Output, *T, error)

func ProcessResult 

func ProcessResult[T any](ctx context.Context, src Source, out OutConfig, fn func(r io.Reader, w io.Writer) (*T, error)) (*Output, *T, error)

func (*Output) Bytes 

func (o *Output) Bytes() ([]byte, error)

func (*Output) Data 

func (o *Output) Data() []byte

func (*Output) Keep 

func (o *Output) Keep() *Output

func (*Output) OpenReader 

func (o *Output) OpenReader() (io.ReadCloser, error)

func (*Output) OpenWriter 

func (o *Output) OpenWriter(sizeHint ...int64) (io.WriteCloser, error)

func (*Output) Path 

func (o *Output) Path() string

func (*Output) SaveAs 

func (o *Output) SaveAs(path string) error

func (*Output) Size 

func (o *Output) Size() int64

func (*Output) StorageType 

func (o *Output) StorageType() StorageType

func (*Output) WriteTo 

func (o *Output) WriteTo(w io.Writer) (int64, error)

type ReaderAtResult 

type ReaderAtResult struct {
	// contains filtered or unexported fields
}

func ToReaderAt 

func ToReaderAt(ctx context.Context, r io.Reader, opts ...ToReaderAtOption) (*ReaderAtResult, error)

ToReaderAt converts any io.Reader into something that supports io.ReaderAt. - If the input already supports ReaderAt → returns it directly. - If it is small enough → keeps it in memory. - If too large → spills to temporary file.

func (*ReaderAtResult) Cleanup 

func (r *ReaderAtResult) Cleanup() error

func (*ReaderAtResult) ReaderAt 

func (r *ReaderAtResult) ReaderAt() io.ReaderAt

func (*ReaderAtResult) Size 

func (r *ReaderAtResult) Size() int64

func (*ReaderAtResult) Source 

func (r *ReaderAtResult) Source() string

type Scope 

type Scope struct {
	// contains filtered or unexported fields
}

func (*Scope) Use 

func (s *Scope) Use(src Source) (io.Reader, error)

Use opens a type-safe Source and returns reader. Cleanup is automatic.

func (*Scope) UseReaderAt 

func (s *Scope) UseReaderAt(src Source, opts ...ToReaderAtOption) (io.ReaderAt, int64, error)

UseReaderAt returns ReaderAt + size with options. Buffers into memory or spills to temp file based on options.

func (*Scope) UseSized 

func (s *Scope) UseSized(src Source) (io.Reader, int64, error)

type Source 

type Source interface {
	// contains filtered or unexported methods
}

Source is a type-safe input source.

func BytesSource 

func BytesSource(b []byte) Source

func FileSource 

func FileSource(f *os.File) Source

func In 

func In(src Source, opts ...InOption) Source

In keeps legacy call sites; DeleteAfterUse is best-effort in fio.

func InputSource 

func InputSource(in *Input) Source

func MultipartSource 

func MultipartSource(fh *multipart.FileHeader) Source

func OutputSource 

func OutputSource(o *Output) Source

func PathSource 

func PathSource(p string) Source

Constructors (type safe)

func ReadCloserSource 

func ReadCloserSource(rc io.ReadCloser) Source

func ReaderSource 

func ReaderSource(r io.Reader) Source

func URLSource 

func URLSource(u string) Source

type StorageType 

type StorageType int
const (
	File StorageType = iota
	Memory
)

func (StorageType) String 

func (s StorageType) String() string

type ToReaderAtOption 

type ToReaderAtOption func(*ToReaderAtOptions)

func WithMaxMemoryBytes 

func WithMaxMemoryBytes(n int64) ToReaderAtOption

func WithTempDir 

func WithTempDir(dir string) ToReaderAtOption

func WithTempPattern 

func WithTempPattern(p string) ToReaderAtOption

type ToReaderAtOptions 

type ToReaderAtOptions struct {
	// contains filtered or unexported fields
}

type Void 

type Void struct{}

Void replaces struct{} when you want “no value”.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.