README
¶
grubber
A data retrieval tool for Markdown and other text files with metadata. It just grubs fast through a big data field.
YAML code blocks in Markdown are usually used as code examples. grubber treats them as structured data records that live next to their context, across an entire directory of files.
Quick example
Input (examples/project-alpha.md):
---
title: Project Alpha
keywords: [project]
created: 2025-01-10
---
# Project Alpha
## Data
```yaml
type: project
name: "Project Alpha"
org: "Northwind Corp"
status: active
start: 2025-01-15
end: 2025-06-30
owner: Jane Smith
```
## Notes
Kickoff completed. First milestone due end of February.
Next review scheduled for March.
Output (grubber extract examples/project-alpha.md):
[
{
"_line": 11,
"_mtime": "2026-02-19T16:12:37Z",
"_note_file": "/home/you/grubber/examples/project-alpha.md",
"created": "2025-01-10",
"end": "2025-06-30",
"keywords": [
"project"
],
"name": "Project Alpha",
"org": "Northwind Corp",
"owner": "Jane Smith",
"start": "2025-01-15",
"status": "active",
"title": "Project Alpha",
"type": "project"
}
]
Frontmatter and YAML block are merged into one flat record, and grubber adds the file's path and modification time (_note_file, _mtime). A record from a YAML block also gets _line, the line of its opening fence, so _note_file and _line together say where the block stands. The prose stays in Markdown, untouched. Run this across a directory of 1,000 notes and you get a single JSON file with all your records.
Installation
Download a pre-built binary for macOS (Apple Silicon or Intel) or Linux (x86-64 or arm64) from Releases, or install with Go:
go install github.com/rhsev/grubber@latest
Or build from a checkout:
make install # builds and copies to /usr/local/bin
make install PREFIX=~/.local
Building needs Go 1.27+.
Quick start
grubber extract ~/notes
grubber extract ~/notes -f "type=project" --format tsv
grubber extract ~/notes/project.md
Why
Structured data and the context around it usually live in different places. A database for the fields, a wiki or folder for the notes. grubber keeps them together, as queryable YAML blocks inside Markdown files. The data stays where you read and write it.
- Standard Markdown. Any editor or renderer handles the format correctly. grubber adds a read layer on top.
- Fast enough to skip the database. 2,000 notes in under 30ms. No index, no daemon, no setup.
- Only structure what you query. Put queryable fields in YAML blocks. Everything else stays in plain Markdown. If you'd never filter by it, don't put it in a code block.
vs. databases
grubber is not a multi-dimensional database. But it covers many use cases: filtering, sorting, aggregating flat records across thousands of files. For personal data like contracts, contacts, inventory, or projects, that's often enough, and your data stays in plain text files.
vs. Dataview (Obsidian)
| Dataview | grubber | |
|---|---|---|
| Editor lock-in | Obsidian only | Any editor |
| Query language | Proprietary DQL | Standard tools (jq, nushell, miller) |
| Output | In-note rendering | JSON/TSV for any pipeline |
| Live updates | Yes | No (run on demand) |
| Formats | Markdown only | Markdown, Typst, vCard, extensible |
| Extensible | Plugin API | Shell scripts, any language |
grubber trades live updates for tool independence. There is no proprietary query language to learn. Output is queried with jq, nushell, or similar tools.
How it works
grubber scans files for metadata and structured data, merges everything into flat records, and outputs JSON or TSV. It only extracts. All logic like filtering, sorting, or aggregating happens downstream with standard tools.
The primary format is Markdown. grubber reads YAML frontmatter and fenced YAML code blocks and merges them into flat records. Multiple YAML blocks per file produce multiple records, each inheriting the frontmatter fields.
Beyond Markdown, grubber has a file-format registry that can handle other text files with metadata. Typst and vCard 3.0 are currently implemented. grubber reads #metadata((...)) and #set document(...) blocks from .typ files, and contact cards from .vcf files. Which file types are scanned is configurable via --extensions. Other formats with native metadata conventions, such as Org-mode, AsciiDoc, or plain YAML, are candidates for future parsers.
Usage
# Extract all records from a directory
grubber extract ~/notes
# Output as TSV
grubber extract ~/notes --format tsv
# Filter records
grubber extract ~/notes -f "type=contract"
grubber extract ~/notes -f "type=contract" -f "end^2025"
# Only YAML blocks (skip frontmatter-only notes)
grubber extract ~/notes --blocks-only
# Only frontmatter (ignore YAML blocks)
grubber extract ~/notes --frontmatter-only
# Write to file
grubber extract ~/notes -o data.json
# Use a config set
grubber extract --set contracts
# Report YAML that extract only handles by degrading it
grubber doctor ~/notes
Filter operators
| Operator | Meaning | Example |
|---|---|---|
= |
equals | type=contract |
~ |
contains | name~hosting |
^ |
starts with | end^2025 |
! |
not equals | status!archived |
Filters are case-insensitive and work on arrays (matches if any element matches). Comparisons are Unicode-normalized (NFC), so a value copied from a Finder file name matches a filter typed on the keyboard, although the two encode an umlaut differently. Output keeps the bytes as written.
Checking your notes (doctor)
Extract is deliberately tolerant. YAML that fails strict parsing falls back to a line-by-line scan, odd values (non-string map keys, NaN) are normalized away, and duplicate keys resolve last-wins. doctor reports every place where that tolerance kicked in. As corpus hygiene, it also reports invisible characters that web clippings drag into notes and that silently break search and pipelines (grep Geschäfte won't find Geschäfte with a soft hyphen inside):
grubber doctor ~/notes # report only
grubber doctor --set notes # same set resolution as extract
grubber doctor ~/notes --fix # remove invisible characters, in place
grubber doctor ~/notes --only yaml # just the hand-work findings
One line per finding (file:line[:col], category, message, tab-separated). Exit code 0 means clean, 1 means findings, so the command can run in CI or after bulk edits. The report goes to stdout and is never stored anywhere (grubber keeps no state between runs); redirect it if you want to keep it. That matters most with --fix, where the report is the record of what was removed:
grubber doctor ~/notes --fix > fix-log.txt
| Category | Meaning |
|---|---|
yaml-error |
block isn't valid YAML; line-by-line fallback was used |
unclosed-fence |
```yaml fence without a closing fence; block ignored |
unclosed-frontmatter |
first line is --- and no later line is ---; frontmatter ignored |
non-string-keys |
mapping with non-string keys (e.g. an unquoted {{...}}); degrades on extract |
non-finite |
NaN/Inf value; extracted as null |
duplicate-key |
duplicate key in one mapping; last value wins |
nested-mapping |
value is a nested mapping, more structure than a flat record wants |
leading-zero-number |
unquoted value resolved as a number and lost its leading zero (01711234890 → 1711234890); when all digits are octal-valid it becomes a different number entirely (0755 → 493) |
invisible-char |
soft hyphen, zero-width, bidi marks, BOM, C0/C1 controls; removed by --fix (ZWJ excepted, see below) |
suspect-char |
NBSP and ZWJ; may be intentional (typography, composed emoji), reported but never fixed |
invalid-utf8 |
byte that is not valid UTF-8; reported but never fixed, since the right repair depends on the original encoding |
crlf |
CRLF line endings; normalized to LF by --fix |
--only filters the report to given categories, comma-separated, with two class shorthands: yaml (the first eight categories in the table, fixed by hand in the note) and chars (the four character-level ones, of which --fix handles invisible-char and crlf). The exit code follows the filter, so doctor --only yaml in cron stays green while deliberate NBSPs remain in the corpus. --only never changes what --fix touches.
--fix rewrites only files with findings and is idempotent. It overwrites the file in place, so the file keeps its inode and extended attributes, and a symlinked note stays a link. The mtime is new, so sync tools and backups see the file as changed. While it writes, the new content is kept in .<name>.journal next to the note. If a write stops halfway, that file remains and later writes to the note refuse until it is removed. Tabs (TaskPaper semantics), NBSP and ZWJ (it holds composed emoji together) always survive. .jsonl files (e.g. collection indexes named in a set's from_jsonl) are scanned but never written; a finding there is a hint for the tool that owns them. Unicode normalization (NFC/NFD) is out of scope. INVISIBLES.md explains what problem each character class causes.
Piping to other tools
grubber outputs JSON by default, for piping into other tools. Filtering and sorting happen in the downstream tool:
# jq: contracts expiring in 2025
grubber extract ~/notes -f type=contract | jq '[.[] | select(.end | startswith("2025"))]'
# sql (duckdb): sort contacts by last interaction
grubber extract ~/notes -f type=person | duckdb -c "SELECT name, last_contact FROM read_json_auto('/dev/stdin') ORDER BY last_contact DESC"
# jsonl: stream into duckdb without buffering the full array
grubber extract ~/notes --format jsonl --no-fill | duckdb -c "SELECT name, status FROM read_jsonl_auto('/dev/stdin')"
# miller: TSV processing
grubber extract ~/notes --format tsv | mlr --tsv sort-by -nr amount
JSONL sources (--from-jsonl)
grubber can read pre-computed records from one or more .jsonl files and union them into the output alongside (or instead of) a fresh scan.
# Cache an expensive scan once …
grubber extract ~/notes --format jsonl -o cache.jsonl
# … then replay it with no Markdown read at all:
grubber extract --from-jsonl cache.jsonl
# … or merge a fresh scan of a small dir with a large cached baseline:
grubber extract ~/new-notes --from-jsonl cache.jsonl
# Multiple sources (files or directories of *.jsonl):
grubber extract --from-jsonl /path/to/cache1.jsonl --from-jsonl /path/to/dir/
--from-jsonl is repeatable. If a path is a directory, every *.jsonl file directly inside it is read (non-recursive, sorted by filename).
When the source is a fileregister index (collections/*.jsonl), the wire format is fixed by the shared contract in fileregister's NOTES-schema.md §JSONL contract (schema 3: id string, binder array, _-prefixed keys injected at read time; grubber's preserve-else-inject for _note_file/_mtime matches it).
Union semantics. By default source records are concatenated with scanned records, without deduplication. Scanned records come first, then sources in the order given.
Merging (--merge-on). When a JSONL index and scanned annotation files describe the same logical records in two layers, --merge-on KEYS collapses them:
# fileregister: collection index + Markdown annotations, one record per file
grubber extract ~/notes --from-jsonl ~/notes/collections/ --merge-on id,binder
A source record that matches a scanned record on all key fields is dropped after back-filling any fields the scanned record lacks (the scanned record wins; _note_file/_mtime are never touched). Unmatched source records pass through. The first key field is the primary identity. Records without it are never merge candidates; later keys default to "" when absent. Keys are compared Unicode-normalized like filters, but case-sensitive. Filters run after the merge, so they see back-filled fields. With --format jsonl the merge buffers instead of streaming.
Exploding (--explode FIELD). When the index keeps a single record per file with a field holding an array (e.g. fileregister stores binder: [project-a, reading]), --explode FIELD expands each such record into one row per element (the element as a scalar) before the merge. Per-binder Markdown blocks carry a single binder, so the exploded index rows then line up and collapse on (id, binder):
grubber extract ~/notes --from-jsonl ~/notes/collections/ --explode binder --merge-on id,binder -f binder=project-a
Scalar or absent values pass through unchanged; an empty array yields one row without the field (a binderless row). All other fields, including provenance, are copied to every row. Like --merge-on, filters run after the explode (so -f binder=project-a keeps only the matching membership), and --explode FIELD can live in defaults:/a set (an explicit --explode= disables it). For a per-file view, omit --explode, since a filter already matches a value inside an array (-f binder=reading matches [project-a, reading]).
merge_on can live in the config (defaults: or a set), so a schema you always use needs no flag; an explicit --merge-on= (empty) disables it for one run. Together with from_jsonl in a set, the whole database definition moves into config and the command shrinks to the query:
grubber extract -s notes -f binder=project-alpha
Provenance (_note_file / _mtime). grubber guarantees every emitted record carries _note_file:
- If a source record already has
_note_file(e.g., from a prior grubber run): preserved unchanged, so the original Markdown path survives a round trip. - If a source record has no
_note_file(tool-authored lines): grubber injects the source file path.
Filters and --array-fields apply to source records exactly as they do to scanned records. -b/-m (blocks-only / frontmatter-only) are scan-path concepts and do not filter source records.
The notes directory is optional when at least one --from-jsonl is given.
Configuration
Optional config file at ~/.config/grubber/config.yaml:
defaults:
blocks_only: true
array_fields: [keywords, category]
extensions: [.md, .typ]
merge_on: [id, binder]
filters: [status!archived]
sets:
contracts:
path: ~/notes
filters: [type=contract]
blocks_only: true
people:
path: ~/notes
filters: [type=person]
# a full database definition: scan + JSONL index + dedup
notes:
path: ~/notes
from_jsonl: ["~/notes/collections/"]
merge_on: [id, binder]
# blocks as records in their own right: no header field leaks into them
refs:
path: ~/notes
blocks_only: true
inherit: []
Use sets with grubber extract --set contracts.
Override hierarchy
CLI flags > config set > environment variables > config defaults > built-in defaults
A set named with --set outranks the environment variables. Filters are the exception to this order. Filters from defaults:, the set, and the command line are added together (duplicates dropped) instead of replacing each other.
Environment variables
| Variable | Purpose |
|---|---|
GRUBBER_NOTES |
Default notes directory |
GRUBBER_ARRAY_FIELDS |
Fields to normalize to arrays (comma-separated) |
GRUBBER_EXTENSIONS |
File extensions to scan (comma-separated, e.g. .md,.typ) |
Calling grubber from a program
A program that runs grubber to read notes should pass --no-config. The call then depends only on its arguments. grubber reads neither config.yaml nor the GRUBBER_* variables, and the directory (or --from-jsonl) must be given; without either, grubber exits with code 2. Without the switch, defaults.filters is added to the program's own filters and array_fields turns single values into lists. --no-config together with --set also exits with code 2.
grubber extract ~/notes --no-config -b --no-fill --inherit= -f type=ref
Options
-o, --output FILE Write to file instead of stdout
-s, --set NAME Load options from config set
--format FORMAT json (default), tsv, or jsonl
-b, --blocks-only Only extract YAML blocks
-m, --frontmatter-only Only extract frontmatter
-a, --all Extract frontmatter and blocks (undoes -b/-m, also from config)
--array-fields FIELDS Normalize fields to arrays (splits comma-separated values)
--extensions EXTS File extensions to scan (comma-separated, default: all registered)
--mmd Also read MultiMarkdown metadata headers
-d, --depth N Limit directory recursion depth (0 = no subdirectories)
--workers N Number of parallel workers (default: NumCPU)
--no-fill Skip nil-filling missing keys (useful for DuckDB)
--no-config Ignore config.yaml and GRUBBER_* variables (not with --set)
--inherit FIELDS Only these frontmatter fields reach a note's blocks (empty = none; default: all)
-f, --filter EXPR Filter records (repeatable)
--from-jsonl PATH Read records from JSONL file or directory; union into output (repeatable)
--merge-on KEYS Merge --from-jsonl records into scanned records sharing these key fields
--explode FIELD Expand a field's array value into one record per element (before merge)
-h, --help Show help
--array-fields normalizes string values to arrays. Comma-separated strings like a, b, c are split into ["a", "b", "c"]. YAML arrays are kept as-is. Single string values in array fields are wrapped in a one-element array. Values that contain commas as part of their meaning (e.g. "Smith, John") should be written as a YAML array instead.
How to structure your notes
Markdown
grubber reads two things from a Markdown file: YAML frontmatter and fenced YAML code blocks (```yaml). Everything else is ignored.
- Frontmatter holds note-level metadata (title, keywords, created date). These fields are merged into every record from that file.
- YAML code blocks hold structured data records. Only
```yamlblocks are read; other fenced blocks are ignored. A```yamlblock inside another code fence is an example and is not read. - Multiple YAML blocks in one note produce multiple records. Each inherits the frontmatter fields.
- Blocks keep document order. The records from one note come out in the order the blocks appear in it, in every output format. This is relative order, not position. A block that yields no record (empty, comments only, or not a mapping) is skipped, so the Nth record is not necessarily the Nth block.
- On field name collision, the YAML block wins over frontmatter.
- Notes without YAML blocks are extracted as frontmatter-only records (unless
--blocks-only).
Limiting inheritance (--inherit). Inheriting every header field is what a block wants when the note describes a group, like an album whose name sits in the frontmatter. It is wrong when each block is a record in its own right. A url: in the header then turns up in every block, as if each were a link. --inherit takes a list of the header fields that may pass:
grubber extract ~/notes --inherit=album,period # only these reach the blocks
grubber extract ~/notes --inherit= # none
grubber extract ~/notes # all, as without the flag
A block that sets a field itself keeps its own value either way. _note_file and _mtime always pass, being grubber's own provenance rather than frontmatter. A note without blocks, and every record under -m, is its frontmatter and stays complete. Filters see the result, so a filter on a header field (-f album~Safari) needs that field inherited. inherit can also live in the config, in defaults: or a set; inherit: [] there means none, and an explicit --inherit= overrides a configured list.
Typst
grubber reads metadata from two Typst constructs:
#metadata((key: "value", ...)) <label>: custom per-document metadata tuple, takes priority#set document(title: "...", author: "...", date: datetime(...)): standard Typst document metadata
Only the first matching block per file is read. datetime(year:, month:, day:) values are converted to YYYY-MM-DD strings. Arrays are supported: ("a", "b", "c") becomes a JSON array. A content block such as [My title] is read as its text.
vCard
grubber reads vCard 3.0 files (.vcf) as exported by Contacts.app. Each card becomes one record; a file may hold a single contact or a whole export.
The mapping is deliberately lossy. Cards are flattened for querying, not round-tripped:
FN→name,ORG→org(organization only),TITLE,NICKNAME,BDAY,NOTE→ same-named lowercase fieldsEMAIL,TEL,URL→ always arrays, even with a single value, so records type consistently across a corpusUID→uid, the card's identity. In a hand-maintained file, a short slug makes it the join key to a Markdown note carrying the sameuidin its YAML blockADR→ one joined string; TYPE parameters (CELL/HOME/WORK), photos, andX-properties are dropped
General
- A
_note_filefield is added automatically to every record for traceability. - A
_mtimefield (RFC3339 UTC) is added automatically with the file's last-modified time. - grubber scans directories recursively. Hidden directories (starting with
.) are skipped.
See docs/SCHEMA.md for an example schema.
YAML notes
grubber follows YAML 1.2. Values that look like numbers are parsed as numbers. If you want a value like a phone number or ID with a leading zero to remain a string, quote it explicitly:
phone: "01711234567" # string
phone_raw: 01711234567 # parsed as a number, and not the one you expect
The unquoted form loses its leading zero, and YAML 1.1 octal is still decoded
for compatibility. So when every digit is below 8, as above, the value is read
as octal and comes out as 254097783, a different number entirely. With an 8
or 9 somewhere it merely loses the zero (01711234890 → 1711234890).
grubber doctor reports both as leading-zero-number and says which one
happened. Quoting is a content decision, so --fix leaves them alone.
Design
- Extract only. grubber reads and outputs. No transforms, no joins, no computed fields.
- Valid Markdown. The format doesn't break any renderer. grubber adds a queryable layer on top.
- Dates are output as strings (
YYYY-MM-DD) for safe JSON serialization. A timestamp with a time of day is output as the note writes it. - Schema-agnostic. grubber extracts whatever YAML it finds. Field names and record types are up to you.
- JSON keys are sorted alphabetically for deterministic output. Record order is
fixed too, whatever the workers do, so the same corpus gives byte-identical
output on every run and diffs and syncs cleanly. Across notes,
jsonandtsvsort by filename (not path),jsonlfollows scan order; within one note, blocks always keep document order.
See also
Programs built on grubber that show or edit the text around a block use
commenter, a small companion command in this repository, or the Go package
entry it is built on. Both follow the same rules. grubber on its own does not
need either. docs/COMMENTER.md describes them. There is no
release binary. go install github.com/rhsev/grubber/cmd/commenter@latest
builds it.
matterbase is a terminal UI built on top of grubber. Its ARCHITECTURE.md also covers grubber's role in the overall stack.
License
MIT
Queried by matterbase and mark-twin. Part of a family of plain-text tools; the profile page has the map.
Documentation
¶
There is no documentation for this package.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
commenter
command
Command commenter reads or replaces the prose of one entry in a Markdown note: the text between a ```yaml block and the nearest boundary above it.
|
Command commenter reads or replaces the prose of one entry in a Markdown note: the text between a ```yaml block and the nearest boundary above it. |
|
Package entry knows where an entry's prose starts and ends in a Markdown note, reads it, and replaces it.
|
Package entry knows where an entry's prose starts and ends in a Markdown note, reads it, and replaces it. |