source

Documentation

Overview

CSV sampling: delimiter sniffing, header or synthesized names, null tokens, and strict rectangularity (encoding/csv reports the offending line when a row has the wrong field count).

JSONL sampling: one JSON object per line, keys become columns in first-seen order, values stay typed. Numbers arrive as json.Number so 64-bit integers never round-trip through float64.

Shared entry point: resolve format, dispatch to the CSV or JSONL reader, and return one uniform sample result.

Package source reads CSV and JSONL samples: format and delimiter sniffing, header handling, null-token policy, and streaming rows into infer columns without ever loading the whole file.

Index

• Variables
• func SniffDelimiter(sample []byte) rune
• type Format
  • func DetectFormat(path string, peek []byte) Format
• type Options
• type Result
  • func ReadSample(r io.Reader, path string, opts Options) (*Result, error)

Variables

var DefaultNullTokens = []string{"NULL", "null", `\N`}

DefaultNullTokens are the cell spellings treated as SQL NULL in CSV input, in addition to the empty cell. The set is deliberately small: "NA" is Namibia's country code and "None" is a valid category label, so neither is nulled unless you opt in with --null.

Functions

func SniffDelimiter

func SniffDelimiter(sample []byte) rune

SniffDelimiter inspects the first records of a CSV sample and picks the candidate delimiter that splits every sampled record into the same field count greater than one. Ties prefer more fields, then candidate order — a comma-delimited file with semicolons inside quoted cells still sniffs as comma because quoting is respected.

Types

type Format

type Format string

Format identifies the input encoding.

const (
	FormatCSV   Format = "csv"
	FormatJSONL Format = "jsonl"
)

func DetectFormat

func DetectFormat(path string, peek []byte) Format

DetectFormat picks CSV or JSONL from the file extension, falling back to content sniffing (peek at the first non-blank byte) for unknown extensions and stdin.

type Options

type Options struct {
	Format     Format // "", i.e. auto, resolves via DetectFormat
	Delimiter  rune   // 0 = sniff (CSV only)
	NoHeader   bool   // CSV only: synthesize column_N names
	NullTokens []string
	MaxRows    int // 0 = read every row
}

Options controls sampling.

type Result

type Result struct {
	Format    Format
	Delimiter rune // CSV only
	Header    bool
	Columns   []infer.Profile
	Rows      int
	Truncated bool // stopped at MaxRows with data left over
}

Result is a fully profiled sample.

func ReadSample

func ReadSample(r io.Reader, path string, opts Options) (*Result, error)

ReadSample profiles a CSV or JSONL stream. path is used only for format detection by extension; pass "-" or "" for stdin.