Documentation
¶
Overview ¶
Package notebook provides Go-native tooling for the Obsidian-style vault: incremental YAML state tracking, markdown task extraction, wiki analysis, and deterministic fan-out batch planning for the automated ingest/absorb pipelines. It is a port of the original Python scripts in tmp/Note/Scripts.
Index ¶
- Constants
- Variables
- func CollectCandidates(inputDir string, extensions []string) ([]string, error)
- func ComputeSHA1(path string) (string, error)
- func FindPending(candidates []string, state StateMap) ([]string, error)
- func FindTags(wikiDir, tagName string) ([]string, error)
- func GetTodoTasks(noteDir string, skipDirs []string) (string, error)
- func ListAllTags(wikiDir string) (map[string]int, error)
- func NeedsProcessing(path string, state StateMap) (bool, error)
- func ParseFrontmatter(content string) (map[string]interface{}, string)
- func PlanGroups(pendingFiles []string, journalBatchSize int) [][]string
- func RecordAbsorbSuccess(ctx context.Context, nctx *workflow.NodeContext) (string, error)
- func RecordFailure(path string, state StateMap)
- func RecordIngestSuccess(ctx context.Context, nctx *workflow.NodeContext) (string, error)
- func RecordSuccess(path string, state StateMap, status string, extra map[string]interface{}) error
- func SaveState(state StateMap, stateFile string) error
- func ScanAbsorbPending(ctx context.Context, nctx *workflow.NodeContext) (string, error)
- func ScanIngestPending(ctx context.Context, nctx *workflow.NodeContext) (string, error)
- func WriteFanoutItemsFile(groups [][]string, outputPath string) error
- type BacklinkResult
- type FileLock
- type SearchResult
- type StateItem
- type StateMap
- type TaskItem
Constants ¶
const ( FunctionScanIngestPending = "notebook.scan_ingest_pending" FunctionRecordIngestSuccess = "notebook.record_ingest_success" FunctionScanAbsorbPending = "notebook.scan_absorb_pending" FunctionRecordAbsorbSuccess = "notebook.record_absorb_success" )
Names under which the notebook functions are registered in the workflow default FunctionRegistry, so `type: function` nodes can invoke them via e.g. `function: notebook.scan_ingest_pending`.
const DefaultJournalBatchSize = 7
DefaultJournalBatchSize is the number of journal notes aggregated per absorption group, mirroring the original Python pipeline.
const MaxFailCount = 3
MaxFailCount is the failure retry ceiling: files whose recorded fail_count reached this value are skipped by NeedsProcessing to prevent infinite retry storms.
Variables ¶
var DefaultSkipDirs = []string{
".git",
".obsidian",
".venv",
".agents",
"tmp",
"99_Templates",
"node_modules",
}
DefaultSkipDirs lists directories excluded from task scanning by default.
Functions ¶
func CollectCandidates ¶
CollectCandidates recursively scans inputDir and returns the sorted list of files whose extension (case-insensitive) matches one of extensions. A missing inputDir yields an empty result.
func ComputeSHA1 ¶
ComputeSHA1 returns the hexadecimal SHA-1 digest of the file's content.
func FindPending ¶
FindPending filters candidates down to the files that need processing.
func FindTags ¶
FindTags returns the vault-relative paths of all notes carrying the tag, either in the frontmatter `tags` field (list or comma-separated string) or as an inline #tag in the body. Matching is case-insensitive.
func GetTodoTasks ¶
GetTodoTasks scans all markdown files under noteDir (skipping any path containing a skipDirs component, or DefaultSkipDirs when none are given) and formats the open tasks as `- [rel_path:line] task_text` lines. It returns "No pending tasks found." when nothing is pending.
func ListAllTags ¶
ListAllTags counts, per unique tag, how many wiki notes carry it in their frontmatter or body. Tags repeated within a single note count once.
func NeedsProcessing ¶
NeedsProcessing reports whether the file still requires processing: files unknown to the state or whose SHA-1 differs from the recorded one need processing, while files with FailCount >= MaxFailCount are skipped.
func ParseFrontmatter ¶
ParseFrontmatter extracts the YAML frontmatter and the markdown body from content. When the frontmatter is absent, malformed, or not a mapping, the full content is returned as the body with an empty metadata map.
func PlanGroups ¶
PlanGroups deterministically splits pending files into fan-out groups: journal notes (any path containing a Journal component) are sorted and batched journalBatchSize at a time (defaulting to DefaultJournalBatchSize when non-positive), while every other entity file forms a single-file group appended afterwards in input order.
func RecordAbsorbSuccess ¶
RecordAbsorbSuccess implements `notebook.record_absorb_success`: it reads the fan-out results from `${tmp_dir}/absorb_results.jsonl` (a missing file is tolerated as an empty result set). For SUCCEEDED groups it records SHA-1 checkpoints of every contained Raw file into `.state/absorb_state.yaml`; for FAILED groups it increments each file's fail_count. It returns the number of successfully settled files.
Note on concurrency: state settlement relies on the workflow engine's single-instance execution constraint per workflow to avoid concurrent read-modify-write races on `.state/absorb_state.yaml`.
func RecordFailure ¶
RecordFailure records a failed processing of path: the entry keeps only the path and status while incrementing FailCount. The SHA-1 is intentionally dropped so the file is retried on the next run until FailCount reaches MaxFailCount.
func RecordIngestSuccess ¶
RecordIngestSuccess implements `notebook.record_ingest_success`: it reads the fan-out results from `${tmp_dir}/ingest_results.jsonl` (a missing file is tolerated as an empty result set), records SHA-1 checkpoints for SUCCEEDED items into `.state/ingest_state.yaml`, increments fail_count for FAILED ones, and returns a settlement summary.
Note on concurrency: state settlement relies on the workflow engine's single-instance execution constraint per workflow to avoid concurrent read-modify-write races on `.state/ingest_state.yaml`.
func RecordSuccess ¶
RecordSuccess records a successful processing of path under its base name: it stores the computed SHA-1, status, today's date, resets FailCount and merges extra fields into the inline extras.
func SaveState ¶
SaveState atomically writes the state map to stateFile as YAML, creating parent directories as needed.
func ScanAbsorbPending ¶
ScanAbsorbPending implements `notebook.scan_absorb_pending`: under the `.state/absorb.lock` guard it scans `01_Raw/` markdown notes, filters them against `.state/absorb_state.yaml` (skipping entries at the failure ceiling), plans deterministic absorption groups via PlanGroups, and atomically writes the groups (one JSON array per line) to `${tmp_dir}/absorb_items.jsonl` for fan-out consumption. It returns a grouping summary.
func ScanIngestPending ¶
ScanIngestPending implements `notebook.scan_ingest_pending`: under the `.state/ingest.lock` guard it recursively scans `Data/` for supported file types, filters them against `.state/ingest_state.yaml` (skipping entries at the failure ceiling), and atomically writes the pending vault-relative paths (one per line) to `${tmp_dir}/ingest_items.jsonl` for fan-out consumption. It returns a one-line scan summary.
func WriteFanoutItemsFile ¶
WriteFanoutItemsFile atomically writes groups as a JSON Lines file: one JSON array of file paths per line, ready for fan-out consumption.
Types ¶
type BacklinkResult ¶
BacklinkResult is a single wikilink reference pointing at a target note.
func FindBacklinks ¶
func FindBacklinks(wikiDir, targetNote string) ([]BacklinkResult, error)
FindBacklinks returns every wiki note line referencing [[targetNote]], [[targetNote|alias]] or [[targetNote#anchor]] (case-insensitive). Paths are reported relative to the vault root (the parent of wikiDir).
type FileLock ¶
type FileLock struct {
// contains filtered or unexported fields
}
FileLock is a process-aware PID file lock preventing concurrent double runs. Locks held by dead processes are detected and taken over; malformed lock files are treated as stale.
func NewFileLock ¶
NewFileLock creates a FileLock guarding the given lock file path.
func (*FileLock) Acquire ¶
Acquire attempts to take the lock, returning false when a live process currently holds it. It uses O_CREATE|O_EXCL to ensure atomic, race-free acquisition.
type SearchResult ¶
SearchResult is a single content line matching a keyword search.
func SearchKeywords ¶
func SearchKeywords(wikiDir string, terms []string) ([]SearchResult, error)
SearchKeywords performs a case-insensitive, line-level content search for any of the terms, returning every matching line with its note path.
type StateItem ¶
type StateItem struct {
Path string `yaml:"path"`
SHA1 string `yaml:"sha1,omitempty"`
Status string `yaml:"status"`
Date string `yaml:"date,omitempty"`
FailCount int `yaml:"fail_count,omitempty"`
Extra map[string]interface{} `yaml:",inline"`
}
StateItem is the tracked processing state of a single vault file.
type StateMap ¶
StateMap is keyed by file base name (filepath.Base), fully compatible with the YAML state files produced by the original Python scripts.
type TaskItem ¶
TaskItem is a single open markdown todo item with its vault-relative location.
func ParseTasksFromFile ¶
ParseTasksFromFile scans a markdown file line by line and returns every open task item (lines starting with "- [ ]") together with its 1-based line number and slash-separated path relative to baseDir.
Source Files