README
¶
logidx
logidx is a batch CLI tool that turns mixed, free-form text logs into structured Parquet files. You describe how to parse each kind of log line as a regular expression rule in a YAML file, and logidx matches, type-converts, and writes one Parquet file per rule.
- Match log lines by regular expression, with built-in presets for common formats (Apache/nginx access logs, BSD/RFC5424 syslog)
- Convert matched fields to typed columns (
string/int/float/timestamp), with flexible timestamp parsing - Merge multiple input files, in chronological order, into one output per rule
- Handle multi-line log entries and embedded JSON/LTSV/logfmt payloads
- Transparently read gzip/xz/bzip2/zstd-compressed input
- Inspect, concatenate, and round-trip Parquet files (
info,cat,dump,restore)
Install
Prebuilt binaries — download from GitHub Releases (linux/darwin/windows, amd64/arm64; a wasm/wasip1 build is also published).
Docker — images are published to ghcr.io/wtnb75/logidx (see Packages for available tags; main tracks the main branch, version tags are published on tagged releases):
docker run --rm ghcr.io/wtnb75/logidx:main version
From source (requires Go 1.25+):
go install github.com/wtnb75/logidx/cmd/logidx@latest
or, from a checkout:
git clone https://github.com/wtnb75/logidx.git
cd logidx
go build -o bin/logidx ./cmd/logidx
(with go-task, task build does the same.)
Quick start
To get started quickly, run logidx scaffold > rules.yaml to print a minimal commented template, or write a rules file describing how to parse your logs. This example uses the built-in apache_clf preset for access logs:
# rules.yaml
rules:
- name: access_log
preset: apache_clf
Then convert one or more log files:
logidx import --rules rules.yaml --out ./out access.log
This writes ./out/access_log.parquet. Any line that doesn't match a rule is written to ./out/unmatched.txt instead of being silently dropped. Pass - as an input file to read from stdin, and pass multiple files to merge them into shared output.
Custom formats look like this — pattern is a regexp with named capture groups, and fields declares each group's output type:
rules:
- name: app_log
pattern: '^(?P<time>\S+) (?P<level>\S+) (?P<message>.*)$'
fields:
time:
type: timestamp
format: iso8601
level: string
message: string
See the reference for the full rules.yaml format: presets, timestamp formats, field value transforms, multi-line entries, embedded structured data, compression, sensitive data masking, and more.
Commands
| Command | Purpose |
|---|---|
logidx import --rules rules.yaml --out ./out <file|->... |
Match, convert, and merge logs into Parquet |
logidx info <file.parquet>... |
Show schema, compression, and row-count info |
logidx cat --output dst.parquet <file.parquet>... |
Concatenate same-schema Parquet files (merges by timestamp if present) |
logidx dump src.parquet dst.txt |
Export a Parquet file to JSON Lines |
logidx restore dst.txt restored.parquet |
Rebuild a Parquet file from a dump file |
logidx expand src.yaml dst.yaml |
Expand a rule's preset: into its pattern:/fields: |
logidx collapse src.yaml dst.yaml |
Collapse a rule's pattern:/fields: into preset: where it matches one exactly |
logidx scaffold |
Print a minimal rules.yaml template to start from |
logidx schema |
Print the JSON Schema for rules.yaml (for editor integration) |
logidx version |
Print version, commit, and build date |
Run logidx <command> --help for the full flag list of any command; see the reference for detailed behavior.
Editor integration
logidx schema prints a JSON Schema for rules.yaml that editors with a yaml-language-server integration (e.g. VS Code's YAML extension) can use for autocompletion and type checking. Point to it from the top of your rules.yaml:
# yaml-language-server: $schema=https://raw.githubusercontent.com/wtnb75/logidx/main/schema/rules.schema.json
The schema covers syntax and types only - semantic checks (e.g. a field name matching a named capture group in pattern) are still only caught by logidx import itself.
Development
task test # go test ./...
task lint # golangci-lint run ./...
task fmt # gofmt -l -w .
task build # go build -o bin/logidx ./cmd/logidx
License
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
logidx
command
|
|
|
internal
|
|
|
atomicfile
Package atomicfile writes a file's full contents to a temporary sibling first and only replaces the destination path with a single os.Rename once every write has succeeded, so an interrupted write (an error, or a panic unwinding through a deferred Abort) never truncates or corrupts a pre-existing file at the destination.
|
Package atomicfile writes a file's full contents to a temporary sibling first and only replaces the destination path with a single os.Rename once every write has succeeded, so an interrupted write (an error, or a panic unwinding through a deferred Abort) never truncates or corrupts a pre-existing file at the destination. |
|
compression
Package compression selects the Parquet page compression codec and level used when writing output files, and resolves that choice from CLI flags, the rules config file, and a built-in default, in that priority order.
|
Package compression selects the Parquet page compression codec and level used when writing output files, and resolves that choice from CLI flags, the rules config file, and a built-in default, in that priority order. |
|
convert
Package convert orchestrates processing of a single input file.
|
Package convert orchestrates processing of a single input file. |
|
decompress
Package decompress wraps an input reader in a decompressing reader based on the compressed file's extension, so logidx import can read gzip/xz/bzip2/zstd-compressed log files directly.
|
Package decompress wraps an input reader in a decompressing reader based on the compressed file's extension, so logidx import can read gzip/xz/bzip2/zstd-compressed log files directly. |
|
logging
Package logging sets up slog for structured logging.
|
Package logging sets up slog for structured logging. |
|
parse
Package parse handles line matching, normalization, and type conversion.
|
Package parse handles line matching, normalization, and type conversion. |
|
pqcat
Package pqcat concatenates one or more Parquet files sharing the exact same schema into a single Parquet file, optionally applying a different compression codec, level, or row group size.
|
Package pqcat concatenates one or more Parquet files sharing the exact same schema into a single Parquet file, optionally applying a different compression codec, level, or row group size. |
|
pqdump
Package pqdump converts a Parquet file to and from a human-readable text dump: one JSON header line describing the schema and compression settings, followed by one JSON object per row.
|
Package pqdump converts a Parquet file to and from a human-readable text dump: one JSON header line describing the schema and compression settings, followed by one JSON object per row. |
|
pqinfo
Package pqinfo reads a Parquet file's footer metadata and reports its schema, per-column compression, and row counts, without decoding any row data.
|
Package pqinfo reads a Parquet file's footer metadata and reports its schema, per-column compression, and row counts, without decoding any row data. |
|
rowgroup
Package rowgroup selects the Parquet row group row-count limit used when writing output files, and resolves that choice from CLI flags, the rules config file, and a built-in default (unlimited — parquet-go's own default), in that priority order.
|
Package rowgroup selects the Parquet row group row-count limit used when writing output files, and resolves that choice from CLI flags, the rules config file, and a built-in default (unlimited — parquet-go's own default), in that priority order. |
|
rules
Package rules loads and validates the YAML rule configuration used to match and structure log lines.
|
Package rules loads and validates the YAML rule configuration used to match and structure log lines. |
|
scaffold
Package scaffold holds the fixed-content template that `logidx scaffold` prints: a minimal, commented rules.yaml a new user can start editing immediately.
|
Package scaffold holds the fixed-content template that `logidx scaffold` prints: a minimal, commented rules.yaml a new user can start editing immediately. |
|
schema
Package schema derives Parquet schema from rule definitions.
|
Package schema derives Parquet schema from rule definitions. |
|
writer
Package writer handles type-specific Parquet writing and unmatched raw writing.
|
Package writer handles type-specific Parquet writing and unmatched raw writing. |
|
Package jsonschema embeds the JSON Schema (schema/rules.schema.json) that describes rules.yaml, for editor integration (yaml-language-server) and for `logidx schema` to print.
|
Package jsonschema embeds the JSON Schema (schema/rules.schema.json) that describes rules.yaml, for editor integration (yaml-language-server) and for `logidx schema` to print. |