infer

Documentation

Overview

Column aggregation: fold a stream of classified values into a profile that a dialect can turn into a column definition, and a report can explain with evidence.

Package infer classifies sample values and aggregates them into per-column type profiles. It is pure: no I/O, no clocks, no globals — the same sample always yields the same profile.

Index

• type Column
  • func NewColumn(source string) *Column
  • func (c *Column) Add(key string, m ValueMeta)
  • func (c *Column) AddNull(token string)
  • func (c *Column) Profile() Profile
• type Kind
  • func (k Kind) String() string
• type Profile
  • func (p Profile) NonNull() int
  • func (p Profile) NullPercent() float64
• type ValueMeta
  • func ClassifyJSONNumber(raw string) ValueMeta
  • func ClassifyJSONString(s string) ValueMeta
  • func ClassifyString(s string) ValueMeta

Types

type Column

type Column struct {
	Name   string // sanitized identifier (set by the caller)
	Source string // original header or JSON key
	// contains filtered or unexported fields
}

Column accumulates classifications for one source column.

func NewColumn

func NewColumn(source string) *Column

NewColumn starts an empty profile for a source column.

func (*Column) Add

func (c *Column) Add(key string, m ValueMeta)

Add records one non-null value. key is the value's canonical text form, used for distinct/uniqueness tracking and for report samples.

func (*Column) AddNull

func (c *Column) AddNull(token string)

AddNull records a null. token is the raw spelling that meant NULL — "" for empty cells, JSON nulls and missing keys, or the matched null token. Load-command generators need this to know what COPY must translate.

func (*Column) Profile

func (c *Column) Profile() Profile

Profile freezes the accumulated evidence into a decision.

type Kind

type Kind uint16

Kind is a bitmask of column types a value is compatible with. A column's candidate set is the intersection (meet) of its values' sets; the final type is the most specific surviving bit.

const (
	Bool Kind = 1 << iota
	Int
	Decimal // exact decimal notation, no exponent
	Float   // needs binary floating point (exponent notation)
	Date
	Time
	Timestamp   // date+time without a UTC offset
	TimestampTZ // date+time with an explicit offset or Z
	UUID
	JSON
	Text // every value is at least text

	// AllKinds is the meet identity: a column starts compatible with
	// everything and each value narrows it down.
	AllKinds = Bool | Int | Decimal | Float | Date | Time |
		Timestamp | TimestampTZ | UUID | JSON | Text
)

func (Kind) String

func (k Kind) String() string

String names a single kind bit for reports and error messages.

type Profile

type Profile struct {
	Name   string
	Source string

	Rows    int // sampled rows, including nulls
	Nulls   int
	AllNull bool

	Decided Kind // single winning kind bit

	IntMin, IntMax int64
	IntOverflow    bool
	Precision      int // decimal: total significant digits
	Scale          int // decimal: digits after the point
	MaxLen         int // longest value, in runes

	Distinct       int
	DistinctCapped bool
	Unique         bool     // no duplicate among non-null values (tracker not capped)
	Values         []string // full sorted value set when low-cardinality
	Samples        []string // first few distinct values, in arrival order

	EmptyNullSeen bool     // at least one empty cell meant NULL
	NullTokens    []string // sorted non-empty spellings that meant NULL
}

Profile is the immutable inference result for one column.

func (Profile) NonNull

func (p Profile) NonNull() int

NonNull is the number of non-null sampled values.

func (Profile) NullPercent

func (p Profile) NullPercent() float64

NullPercent is the share of sampled rows that were null, in percent.

type ValueMeta

type ValueMeta struct {
	Kinds       Kind
	Int         int64 // valid when Kinds&Int != 0 and !IntOverflow
	IntOverflow bool  // integer notation wider than int64
	IntDigits   int   // significant digits before the decimal point
	Scale       int   // digits after the decimal point
	Len         int   // rune length of the raw text
}

ValueMeta is the classification of a single non-null value plus the metadata a column needs to size numeric types.

func ClassifyJSONNumber

func ClassifyJSONNumber(raw string) ValueMeta

ClassifyJSONNumber classifies a json.Number literal.

func ClassifyJSONString

func ClassifyJSONString(s string) ValueMeta

ClassifyJSONString classifies a JSON string value. Unlike CSV cells, numeric and boolean spellings are NOT promoted: when a producer writes "123" as a JSON string it chose a string, and round-tripping that choice into an integer column silently corrupts zero-padded identifiers. Temporal, UUID and embedded-JSON shapes are still recognized because JSON has no native syntax for them.

func ClassifyString

func ClassifyString(s string) ValueMeta

ClassifyString classifies one trimmed, non-null CSV cell.