extsort

README

extsort

A high-performance external sorting library for Go that enables sorting of arbitrarily large datasets that don't fit entirely in memory. The library operates on channels and uses temporary disk storage to handle datasets larger than available RAM.

Features

• Memory Efficient: Sorts datasets of any size using configurable memory limits
• High Performance: Optimized for throughput with parallel sorting and merging
• Generic Support: Modern Go generics for type-safe operations
• Legacy Compatible: Maintains backward compatibility with interface-based API
• Cross-Platform: Works on Unix, Linux, macOS, and Windows
• Channel-Based: Integrates seamlessly with Go's concurrency patterns

Installation

go get github.com/lanrat/extsort

Quick Start

Generic API (Recommended)

The modern generic API provides type safety and improved performance:

import (
    "context"
    "fmt"
    "math/rand"

    "github.com/lanrat/extsort"
)

func main() {
    // Create input channel with unsorted integers
    inputChan := make(chan int, 100)
    go func() {
        defer close(inputChan)
        for i := 0; i < 1000000; i++ {
            inputChan <- rand.Int()
        }
    }()

    // Sort using the generic API
    sorter, outputChan, errChan := extsort.Ordered(inputChan, nil)

    // Start sorting in background
    go sorter.Sort(context.Background())

    // Process sorted results
    for value := range outputChan {
        fmt.Println(value)
    }

    // Check for errors
    if err := <-errChan; err != nil {
        panic(err)
    }
}

String Sorting

For string data, use the optimized string sorter:

import (
    "context"
    "fmt"

    "github.com/lanrat/extsort"
)

func main() {
    words := []string{"zebra", "apple", "banana", "cherry"}

    inputChan := make(chan string, len(words))
    for _, word := range words {
        inputChan <- word
    }
    close(inputChan)

    sorter, outputChan, errChan := extsort.Strings(inputChan, nil)
    go sorter.Sort(context.Background())

    fmt.Println("Sorted words:")
    for word := range outputChan {
        fmt.Println(word)
    }

    if err := <-errChan; err != nil {
        panic(err)
    }
}

Custom Types with Generic API

import (
    "bytes"
    "context"
    "encoding/gob"
    "fmt"

    "github.com/lanrat/extsort"
)

type Person struct {
    Name string
    Age  int
}

func personToBytes(p Person) ([]byte, error) {
    var buf bytes.Buffer
    enc := gob.NewEncoder(&buf)
    err := enc.Encode(p)
    return buf.Bytes(), err
}

func personFromBytes(data []byte) (Person, error) {
    var p Person
    buf := bytes.NewReader(data)
    dec := gob.NewDecoder(buf)
    err := dec.Decode(&p)
    return p, err
}

func comparePersonsByAge(a, b Person) int {
    // Sort by age
    if a.Age != b.Age {
        if a.Age < b.Age {
            return -1
        }
        return 1
    }
    return 0
}

func main() {
    people := []Person{
        {"Alice", 30},
        {"Bob", 25},
        {"Charlie", 35},
    }

    inputChan := make(chan Person, len(people))
    for _, person := range people {
        inputChan <- person
    }
    close(inputChan)

    sorter, outputChan, errChan := extsort.Generic(
        inputChan,
        personFromBytes,
        personToBytes,
        comparePersonsByAge,
        nil,
    )

    go sorter.Sort(context.Background())

    fmt.Println("People sorted by age:")
    for person := range outputChan {
        fmt.Printf("%s (age %d)\n", person.Name, person.Age)
    }

    if err := <-errChan; err != nil {
        panic(err)
    }
}

Configuration

Customize sorting behavior with the Config struct:

config := &extsort.Config{
    ChunkSize:          500000,  // Records per chunk (default: 1M)
    NumWorkers:         4,       // Parallel sorting/merging workers (default: 2)
    ChanBuffSize:       10,      // Channel buffer size (default: 1)
    SortedChanBuffSize: 1000,    // Output channel buffer (default: 1000)
    TempFilesDir:       "/var/tmp",  // Temporary files directory (default: intelligent selection)
}

sorter, outputChan, errChan := extsort.Ordered(inputChan, config)

Temporary Directory Selection

When TempFilesDir is empty (default), the library intelligently selects temporary directories that prefer disk-backed locations over potentially memory-backed filesystems. On Linux systems where /tmp may be mounted as tmpfs (memory-backed), this helps prevent out-of-memory issues when sorting datasets larger than available RAM.

For production use with large datasets, it's recommended to explicitly set TempFilesDir to a known disk-backed directory (such as /var/tmp on Unix systems) to ensure optimal performance and avoid memory limitations.

Legacy Interface-Based API

The library maintains backward compatibility with the original interface-based API:

import (
    "context"
    "encoding/binary"
    "fmt"
    "math/rand"

    "github.com/lanrat/extsort"
)

type sortInt struct {
    value int64
}

func (s sortInt) ToBytes() []byte {
    buf := make([]byte, 8)
    binary.LittleEndian.PutUint64(buf, uint64(s.value))
    return buf
}

func sortIntFromBytes(data []byte) extsort.SortType {
    value := int64(binary.LittleEndian.Uint64(data))
    return sortInt{value: value}
}

func compareSortInt(a, b extsort.SortType) bool {
    return a.(sortInt).value < b.(sortInt).value
}

func main() {
    inputChan := make(chan extsort.SortType, 100)
    go func() {
        defer close(inputChan)
        for i := 0; i < 100000; i++ {
            inputChan <- sortInt{value: rand.Int63()}
        }
    }()

    sorter, outputChan, errChan := extsort.New(
        inputChan,
        sortIntFromBytes,
        compareSortInt,
        nil,
    )

    go sorter.Sort(context.Background())

    for item := range outputChan {
        fmt.Println(item.(sortInt).value)
    }

    if err := <-errChan; err != nil {
        panic(err)
    }
}

Diff Sub-Package

The diff sub-package provides functionality for comparing two sorted data streams and identifying differences. It's particularly useful for comparing large datasets efficiently.

Basic String Diff

import (
    "context"
    "fmt"

    "github.com/lanrat/extsort/diff"
)

func main() {
    // Create two sorted string channels
    streamA := make(chan string, 5)
    streamB := make(chan string, 5)

    // Populate stream A
    go func() {
        defer close(streamA)
        for _, item := range []string{"apple", "banana", "cherry", "elderberry"} {
            streamA <- item
        }
    }()

    // Populate stream B
    go func() {
        defer close(streamB)
        for _, item := range []string{"banana", "cherry", "date", "fig"} {
            streamB <- item
        }
    }()

    // Create error channels
    errA := make(chan error, 1)
    errB := make(chan error, 1)
    close(errA)
    close(errB)

    // Process differences
    result, err := diff.Strings(
        context.Background(),
        streamA, streamB,
        errA, errB,
        func(delta diff.Delta, item string) error {
            switch delta {
            case diff.OLD:
                fmt.Printf("Only in A: %s\n", item)
            case diff.NEW:
                fmt.Printf("Only in B: %s\n", item)
            }
            return nil
        },
    )

    if err != nil {
        panic(err)
    }

    fmt.Printf("Summary: %d items only in A, %d items only in B, %d common items\n",
        result.ExtraA, result.ExtraB, result.Common)
}

Generic Diff for Custom Types

import (
    "context"
    "fmt"

    "github.com/lanrat/extsort/diff"
)

func main() {
    // Create channels with integer data
    streamA := make(chan int, 5)
    streamB := make(chan int, 5)
    errA := make(chan error, 1)
    errB := make(chan error, 1)

    // Populate streams
    go func() {
        defer close(streamA)
        defer close(errA)
        for _, num := range []int{1, 3, 5, 7, 9} {
            streamA <- num
        }
    }()

    go func() {
        defer close(streamB)
        defer close(errB)
        for _, num := range []int{2, 4, 5, 6, 8} {
            streamB <- num
        }
    }()

    // Compare using generic diff
    compareFunc := func(a, b int) int {
        if a < b {
            return -1
        }
        if a > b {
            return 1
        }
        return 0
    }

    resultFunc := func(delta diff.Delta, item int) error {
        symbol := map[diff.Delta]string{diff.OLD: "<", diff.NEW: ">"}[delta]
        fmt.Printf("%s %d\n", symbol, item)
        return nil
    }

    result, err := diff.Generic(
        context.Background(),
        streamA, streamB,
        errA, errB,
        compareFunc,
        resultFunc,
    )

    if err != nil {
        panic(err)
    }

    fmt.Printf("Differences found: %d\n", result.ExtraA+result.ExtraB)
}

Parallel Diff Processing

import (
    "context"
    "fmt"
    "sync"

    "github.com/lanrat/extsort/diff"
)

func main() {
    streamA := make(chan string, 100)
    streamB := make(chan string, 100)
    errA := make(chan error, 1)
    errB := make(chan error, 1)

    // Populate streams with test data
    go func() {
        defer close(streamA)
        defer close(errA)
        for i := 0; i < 50; i += 2 {
            streamA <- fmt.Sprintf("item_%03d", i)
        }
    }()

    go func() {
        defer close(streamB)
        defer close(errB)
        for i := 1; i < 50; i += 2 {
            streamB <- fmt.Sprintf("item_%03d", i)
        }
    }()

    // Use channel-based result processing for parallel handling
    resultFunc, resultChan := diff.StringResultChan()

    var wg sync.WaitGroup
    wg.Add(1)

    // Process results in parallel
    go func() {
        defer wg.Done()
        for result := range resultChan {
            fmt.Printf("Difference: %s %s\n", result.D, result.S)
        }
    }()

    // Start diff operation
    go func() {
        defer close(resultChan)
        _, err := diff.Strings(
            context.Background(),
            streamA, streamB,
            errA, errB,
            resultFunc,
        )
        if err != nil {
            fmt.Printf("Diff error: %v\n", err)
        }
    }()

    wg.Wait()
    fmt.Println("Diff processing complete")
}

Performance Considerations

• Memory Usage: Configure ChunkSize based on available memory (larger chunks = less I/O, more memory)
• Parallelism: Increase NumWorkers on multi-core systems
• Temporary Storage:
  • Explicitly set TempFilesDir to a known disk-backed directory for large datasets
  • On Linux, prefer /var/tmp over /tmp (which may be tmpfs/memory-backed)
  • Use fast storage (SSD recommended) for temporary files
• Channel Buffers: Tune buffer sizes based on your producer/consumer patterns

Error Handling

The library uses Go's standard error handling patterns. Always check the error channel:

sorter, outputChan, errChan := extsort.Ordered(inputChan, nil)
go sorter.Sort(context.Background())

for item := range outputChan {
    // Process sorted item
}

if err := <-errChan; err != nil {
    // Handle error
    log.Fatal(err)
}

Limitations

• Not Stable: The sort is not stable (equal elements may change relative order)
• Disk Space: Requires temporary disk space approximately equal to input data size
• Memory: Minimum memory usage depends on chunk size configuration

License

This project is licensed under the Apache License - see the LICENSE file for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Documentation

Overview

Package extsort implements an unstable external sort for all the records in a chan or iterator

Index

• func NewComparisonError(cause interface{}, context string) error
• func NewDeserializationError(cause interface{}, dataSize int, context string) error
• func NewDiskError(err error, operation, path string) error
• func NewResourceError(err error, resource, context string) error
• func NewSerializationError(cause interface{}, context string) error
• func UniqStringChan(in chan string) chan string
• type CompareGeneric
• type CompareLessFunc
• type ComparisonError
  • func (e *ComparisonError) Error() string
  • func (e *ComparisonError) Unwrap() error
• type Config
  • func DefaultConfig() *Config
• type ConfigError
  • func (e *ConfigError) Error() string
• type DeserializationError
  • func (e *DeserializationError) Error() string
  • func (e *DeserializationError) Unwrap() error
• type FromBytes
• type FromBytesGeneric
• type GenericSorter
  • func Generic[E any](input <-chan E, fromBytes FromBytesGeneric[E], toBytes ToBytesGeneric[E], ...) (*GenericSorter[E], <-chan E, <-chan error)
  • func MockGeneric[E any](input <-chan E, fromBytes FromBytesGeneric[E], toBytes ToBytesGeneric[E], ...) (*GenericSorter[E], <-chan E, <-chan error)
  • func (s *GenericSorter[E]) Sort(ctx context.Context)
• type OrderedSorter
  • func Ordered[T cmp.Ordered](input <-chan T, config *Config) (*OrderedSorter[T], <-chan T, <-chan error)
  • func OrderedMock[T cmp.Ordered](input <-chan T, config *Config, n int) (*OrderedSorter[T], <-chan T, <-chan error)
• type SerializationError
  • func (e *SerializationError) Error() string
  • func (e *SerializationError) Unwrap() error
• type SortType
• type SortTypeSorter
  • func New(input <-chan SortType, fromBytes FromBytes, lessFunc CompareLessFunc, ...) (*SortTypeSorter, <-chan SortType, <-chan error)
  • func NewMock(input <-chan SortType, fromBytes FromBytes, lessFunc CompareLessFunc, ...) (*SortTypeSorter, <-chan SortType, <-chan error)
• type Sorter
• type StringSorter
  • func Strings(input <-chan string, config *Config) (*StringSorter, <-chan string, <-chan error)
  • func StringsMock(input <-chan string, config *Config, n int) (*StringSorter, <-chan string, <-chan error)
• type ToBytesGeneric

Functions

func NewComparisonError

func NewComparisonError(cause interface{}, context string) error

NewComparisonError creates a ComparisonError

func NewDeserializationError

func NewDeserializationError(cause interface{}, dataSize int, context string) error

NewDeserializationError creates a DeserializationError

func NewDiskError

func NewDiskError(err error, operation, path string) error

NewDiskError creates a DiskError wrapping the underlying I/O error

func NewResourceError

func NewResourceError(err error, resource, context string) error

NewResourceError creates a ResourceError wrapping the underlying error

func NewSerializationError

func NewSerializationError(cause interface{}, context string) error

NewSerializationError creates a SerializationError

func UniqStringChan

func UniqStringChan(in chan string) chan string

UniqStringChan returns a channel that filters out consecutive duplicate strings from the input. This function assumes the input channel provides strings in sorted order and uses string equality to detect duplicates. It preserves the first occurrence of each unique string while filtering out subsequent duplicates. The function is optimized for sorted inputs where duplicates appear consecutively, making it suitable for post-processing sorted results from external sort operations.

The returned channel will be closed when the input channel is closed. This function spawns a goroutine that will terminate when the input channel is closed.

Types

type CompareGeneric

type CompareGeneric[E any] func(a, b E) int

CompareGeneric is a function type for comparing two items of type E. It must implement a strict weak ordering: reflexivity, antisymmetry, and transitivity. Returns a negative integer if a should be ordered before b, zero if they are equal, and a positive integer if a should be ordered after b in the final sorted output. The function must be consistent and must handle any errors by panicking. This follows the same semantics as cmp.Compare and can be implemented using cmp.Compare[T] for ordered types.

type CompareLessFunc

type CompareLessFunc func(a, b SortType) bool

CompareLessFunc compares two SortType items and returns true if a is less than b

Deprecated: Use CompareGeneric[T] instead for new code. This type is maintained for backward compatibility.

type ComparisonError

type ComparisonError struct {
	// Cause is the original panic or error that occurred during comparison
	Cause interface{}
	// Context provides additional information about when the comparison failed
	Context string
}

ComparisonError represents an error that occurred during item comparison

func (*ComparisonError) Error

func (e *ComparisonError) Error() string

func (*ComparisonError) Unwrap

func (e *ComparisonError) Unwrap() error

type Config

type Config struct {
	// ChunkSize specifies the maximum number of records to store in each chunk
	// before writing to disk. Larger chunks use more memory but reduce I/O operations.
	// Default: 1,000,000 records. Must be > 1.
	ChunkSize int

	// NumWorkers controls the maximum number of goroutines used for parallel
	// chunk sorting and merging. More workers can improve CPU utilization on multi-core systems.
	// Default: 2 workers. Must be > 1.
	NumWorkers int

	// ChanBuffSize sets the buffer size for internal channels used during chunk merging.
	// Larger buffers can improve throughput but use more memory.
	// Default: 1. Must be >= 0.
	ChanBuffSize int

	// SortedChanBuffSize sets the buffer size for the output channel that delivers
	// sorted results. Larger buffers allow more decoupling between sorting and consumption.
	// Default: 1000. Must be >= 0.
	SortedChanBuffSize int

	// TempFilesDir specifies the directory for temporary files during sorting.
	// When empty (default), the library uses intelligent directory selection that
	// prefers disk-backed locations over potentially memory-backed filesystems
	// (like tmpfs on Linux). This helps prevent out-of-memory issues when sorting
	// datasets larger than available RAM.
	//
	// For production use with large datasets, it's recommended to explicitly set
	// this to a known disk-backed directory (such as "/var/tmp" on Unix systems)
	// to ensure optimal performance and avoid memory limitations. On Linux systems,
	// prefer "/var/tmp" over "/tmp" since "/tmp" may be mounted as tmpfs.
	//
	// Default: "" (intelligent selection).
	TempFilesDir string
}

Config holds configuration settings for external sorting operations. All fields have sensible defaults and can be left as zero values to use defaults.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns a Config with sensible default values optimized for general-purpose external sorting. These defaults balance memory usage, I/O efficiency, and parallelism for typical workloads.

type ConfigError

type ConfigError struct {
	// Field is the name of the configuration field that's invalid
	Field string
	// Value is the invalid value provided
	Value interface{}
	// Reason explains why the value is invalid
	Reason string
}

ConfigError represents an error in configuration parameters

func (*ConfigError) Error

func (e *ConfigError) Error() string

type DeserializationError

type DeserializationError struct {
	// Cause is the original panic or error that occurred during deserialization
	Cause interface{}
	// DataSize is the size of the data that failed to deserialize
	DataSize int
	// Context provides additional information about what was being deserialized
	Context string
}

DeserializationError represents an error that occurred during item deserialization (FromBytes)

func (*DeserializationError) Error

func (e *DeserializationError) Error() string

func (*DeserializationError) Unwrap

func (e *DeserializationError) Unwrap() error

type FromBytes

type FromBytes func([]byte) SortType

FromBytes unmarshal bytes to create a SortType when reading back the sorted items

Deprecated: Use FromBytesGeneric[T] instead for new code. This type is maintained for backward compatibility.

type FromBytesGeneric

type FromBytesGeneric[E any] func([]byte) (E, error)

FromBytesGeneric is a function type for deserializing bytes back to type E. It's used during the merge phase to reconstruct items from temporary storage. The function should be the inverse of the corresponding ToBytesGeneric function. It returns an error for any deserialization failures, which will be wrapped in a DeserializationError by the external sorter.

type GenericSorter

type GenericSorter[E any] struct {
	// contains filtered or unexported fields
}

GenericSorter implements external sorting for any type E using a divide-and-conquer approach. It reads input from a channel, splits data into chunks that fit in memory, sorts each chunk, saves them to temporary files, then merges all chunks back into a sorted output stream. The sorter uses configurable parallelism for both sorting and merging phases, and employs memory pools to reduce garbage collection pressure during operation.

func Generic

func Generic[E any](input <-chan E, fromBytes FromBytesGeneric[E], toBytes ToBytesGeneric[E], compareFunc CompareGeneric[E], config *Config) (*GenericSorter[E], <-chan E, <-chan error)

Generic creates a new external sorter for any type E and returns the sorter instance, output channel with sorted results, and error channel.

Parameters:

• input: Channel providing the data to be sorted
• fromBytes: Function to deserialize E from bytes when reading from disk
• toBytes: Function to serialize E to bytes when writing to disk
• compareFunc: Comparison function that returns negative/zero/positive for less/equal/greater
• config: Configuration options (nil uses defaults)

The sorting process:

1. Reads data from input channel into memory chunks
2. Sorts each chunk in parallel using the provided compareFunc
3. Saves sorted chunks to temporary files using toBytes serialization
4. Merges all chunks back into sorted order using fromBytes deserialization

Call Sort() on the returned sorter to begin the sorting process. Results are delivered via the output channel, errors via the error channel. On error or interruption, temporary files may remain in config.TempFilesDir.

func MockGeneric

func MockGeneric[E any](input <-chan E, fromBytes FromBytesGeneric[E], toBytes ToBytesGeneric[E], compareFunc CompareGeneric[E], config *Config, n int) (*GenericSorter[E], <-chan E, <-chan error)

MockGeneric creates an external sorter that uses in-memory storage instead of disk files. This is primarily useful for testing and benchmarking without filesystem I/O overhead. The parameter n specifies the initial capacity of the in-memory buffer. All other behavior is identical to Generic().

func (*GenericSorter[E]) Sort

func (s *GenericSorter[E]) Sort(ctx context.Context)

Sort sorts the Sorter's input chan and returns a new sorted chan, and error Chan Sort is a chunking operation that runs multiple workers asynchronously this blocks while sorting chunks and unblocks when merging NOTE: the context passed to Sort must outlive Sort() returning. Merge uses the same context and runs in a goroutine after Sort returns(). for example, if calling sort in an errGroup, you must pass the group's parent context into sort.

type OrderedSorter

type OrderedSorter[T cmp.Ordered] struct {
	GenericSorter[T]
	// contains filtered or unexported fields
}

OrderedSorter provides external sorting for types that implement cmp.Ordered. It embeds GenericSorter and adds optimized byte serialization using gob encoding with a sync.Pool for buffer reuse to reduce allocations.

func Ordered

func Ordered[T cmp.Ordered](input <-chan T, config *Config) (*OrderedSorter[T], <-chan T, <-chan error)

Ordered performs external sorting on a channel of cmp.Ordered types. It returns the sorter instance, output channel with sorted results, and error channel. Uses gob encoding for serialization and the < operator for comparison.

func OrderedMock

func OrderedMock[T cmp.Ordered](input <-chan T, config *Config, n int) (*OrderedSorter[T], <-chan T, <-chan error)

OrderedMock performs external sorting with a mock implementation that limits the number of items to sort (useful for testing). Takes the same parameters as Ordered plus n which limits the number of items processed.

type SerializationError

type SerializationError struct {
	// Cause is the original panic or error that occurred during serialization
	Cause interface{}
	// Context provides additional information about what was being serialized
	Context string
}

SerializationError represents an error that occurred during item serialization (ToBytes)

func (*SerializationError) Error

func (e *SerializationError) Error() string

func (*SerializationError) Unwrap

func (e *SerializationError) Unwrap() error

type SortType

type SortType interface {
	ToBytes() []byte // ToBytes used for marshaling
}

SortType defines the interface required by the extsort library to be able to sort the items

Deprecated: Use Generic() with custom types instead for new code. This interface is maintained for backward compatibility.

type SortTypeSorter

type SortTypeSorter struct {
	GenericSorter[SortType]
}

SortTypeSorter provides external sorting for types implementing the SortType interface, maintaining backward compatibility with the legacy interface-based API. It embeds GenericSorter[SortType] and adapts the interface methods to the generic implementation.

Deprecated: Use GenericSorter[T] instead for new code. This type is maintained for backward compatibility.

func New

func New(input <-chan SortType, fromBytes FromBytes, lessFunc CompareLessFunc, config *Config) (*SortTypeSorter, <-chan SortType, <-chan error)

New performs external sorting on a channel of SortType items using the legacy interface-based API. It takes a FromBytes function for deserialization and a CompareLessFunc for comparison. Returns the sorter instance, output channel with sorted items, and error channel. This function provides backward compatibility with the original extsort API.

Deprecated: Use Generic() instead for new code. This function is maintained for backward compatibility.

func NewMock

func NewMock(input <-chan SortType, fromBytes FromBytes, lessFunc CompareLessFunc, config *Config, n int) (*SortTypeSorter, <-chan SortType, <-chan error)

NewMock performs external sorting on SortType items with a mock implementation that limits the number of items to sort. Useful for testing with a controlled dataset size. The parameter n specifies the maximum number of items to process. Uses the same interface-based API as New for backward compatibility.

Deprecated: Use MockGeneric() instead for new code. This function is maintained for backward compatibility.

type Sorter

type Sorter interface {
	// Sort performs the complete external sorting operation.
	// It reads from the input channel, sorts the data using temporary storage,
	// and delivers results to the output channel. The operation can be cancelled
	// or timed out using the provided context.
	Sort(context.Context)
}

Sorter is the interface that all extsort sorter implementations must satisfy. It provides a single Sort method that performs the complete external sorting operation within the provided context, allowing for cancellation and timeout control.

type StringSorter

type StringSorter struct {
	GenericSorter[string]
}

StringSorter provides external sorting for strings, maintaining backward compatibility with the legacy string-specific API. It embeds GenericSorter[string] and uses simple byte slice conversion for serialization.

func Strings

func Strings(input <-chan string, config *Config) (*StringSorter, <-chan string, <-chan error)

Strings performs external sorting on a channel of strings using lexicographic ordering. Returns the sorter instance, output channel with sorted strings, and error channel. This function provides backward compatibility with the legacy string-specific API.

func StringsMock

func StringsMock(input <-chan string, config *Config, n int) (*StringSorter, <-chan string, <-chan error)

StringsMock performs external sorting on strings with a mock implementation that limits the number of strings to sort. Useful for testing with a controlled dataset size. The parameter n specifies the maximum number of strings to process.

type ToBytesGeneric

type ToBytesGeneric[E any] func(E) ([]byte, error)

ToBytesGeneric is a function type for serializing type E to bytes. It's used during chunk saving to store items in temporary files. The function should produce deterministic output that can be read back by the corresponding FromBytesGeneric function. It returns an error for any serialization failures, which will be wrapped in a SerializationError by the external sorter.