Help us shape the future of Go! Take the 2026 Go Developer Survey

metabol

package module
v0.0.5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 17, 2026 License: MIT Imports: 20 Imported by: 0

README

metabol

Test Status Coverage Status MIT License PkgGoDev

metabol is a feed aggregator that archives articles locally as Markdown.

It is a stateless, configuration-driven orchestrator that collects feed items from a deterministic time window and asks mdhq to save the linked articles. It separates the scheduler's execution time from the logical window being processed, making scheduled runs repeatable and safe to retry.

Installation

# Install with Homebrew.
$ brew install Songmu/tap/metabol

# Install the latest version. (Install it into ./bin/ by default).
$ curl -sfL https://raw.githubusercontent.com/Songmu/metabol/main/install.sh | sh -s

# Specify the installation directory and version.
$ curl -sfL https://raw.githubusercontent.com/Songmu/metabol/main/install.sh |
    sh -s -- -b "$(go env GOPATH)/bin" vX.Y.Z

# Or install with Go.
$ go install github.com/Songmu/metabol/cmd/metabol@latest

Prerequisite: metabol invokes mdhq as an external command, so install it separately and ensure it is on PATH. For normal local use, install it globally:

$ npm install --global @songmu/mdhq

To use the mdhq version pinned and tested for this repository checkout, install the dependencies from the repository root and add the local binary directory to PATH:

$ npm ci
$ export PATH="$PWD/node_modules/.bin:$PATH"

Configuration

Create a starter configuration in the current directory:

$ metabol init

If the directory is not empty, metabol asks for confirmation and defaults to not creating the file. It never overwrites an existing metabol.yaml.

By default, metabol reads metabol.yaml from the current directory. An example configuration is:

# yaml-language-server: $schema=https://raw.githubusercontent.com/Songmu/metabol/main/schema.yaml
timezone: Asia/Tokyo # Optional, but recommended.

window:
  daily: "07:00"

sources:
  - https://example.com/feed.xml

A source may be a URL scalar or an object. These forms are equivalent:

sources:
  - https://example.com/feed

sources:
  - name: example # Reserved for future source metadata.
    url: https://example.com/feed

The object form also accepts an optional name reserved for future source metadata. It currently does not affect processing or output.

Feed fetching is provided by rssnip and supports RSS 2.0, Atom 1.0, RDF/RSS 1.0, and JSON Feed 1.0 and 1.1. Blog or site URLs are also accepted for feed discovery, but direct feed URLs are preferred.

window.daily defines a local-time boundary, not a schedule. Every processing window is a half-open interval, [start, end), calculated in timezone. Use metabol to run with the default config, or metabol --config path/to/file to read only the explicitly selected file.

Values are resolved in this order:

CLI flag > METABOL_* environment variable > configuration file > default value

The available settings are:

Configuration key CLI flag Environment variable Type Default Meaning
window.daily - - HH:MM string Required Local-time boundary for each daily processing window
window.count --window-count METABOL_WINDOW_COUNT Integer (1-366) 1 Number of consecutive processing windows
sources - - Non-empty list of URLs or source objects Required Feed sources to process
root --root METABOL_ROOT Path string Configuration file directory Markdown output directory
assets --assets METABOL_ASSETS Boolean false Download article assets
update --update METABOL_UPDATE Boolean false Re-evaluate existing Markdown
catchup --catchup METABOL_CATCHUP Boolean false Extend the newest selected window through the latest feed contents
timezone --timezone METABOL_TIMEZONE IANA timezone string Local timezone Timezone used to calculate processing windows
- --config METABOL_CONFIG Path string metabol.yaml Configuration file to read
- --at METABOL_AT RFC 3339 timestamp or YYYY-MM-DD Not set Select the window containing the specified date or time; when omitted, the most recently completed window is used

Relative root values in the configuration file are resolved from that file's directory. Relative values passed with --root or METABOL_ROOT are resolved from the command's working directory. After resolving assets, metabol explicitly passes either --assets or --no-assets to mdhq, so the resolved metabol setting overrides any value in mdhq configuration.

Time windows

Window boundaries are based on local calendar time in the configured timezone. A window that crosses a daylight-saving transition may therefore span 23 or 25 hours, while remaining contiguous and non-overlapping.

By default, metabol selects the most recently completed window unless --at is specified. For example, with a daily boundary of 07:00 in Asia/Tokyo, a run at 2026-09-13 19:00 JST processes:

[2026-09-12 07:00, 2026-09-13 07:00)

Runs at 08:00, 14:30, or 23:00 on September 13 therefore select the same logical window.

catchup, --catchup, or METABOL_CATCHUP removes the upper bound from the newest selected window. With the same boundary and execution time, catchup collects from 2026-09-12 07:00 JST through the latest items currently available from each feed:

$ metabol --catchup

This extends only the newest selected window. When window.count is greater than one, earlier windows remain bounded and are processed normally. Catchup is a one-shot fetch; it does not keep metabol running to watch for new items.

--at selects the [start, end) window containing the supplied value rather than the last complete window:

$ metabol --at 2026-09-13
$ metabol --at 2026-09-13T10:30:00+09:00

RFC 3339 timestamps and YYYY-MM-DD dates are accepted. A date without a time means 00:00 in the configured timezone. A value exactly on a boundary belongs to the window beginning at that boundary. Backfills are best effort because a feed may no longer contain old items.

If at and catchup are both set, at takes precedence. Metabol performs the bounded backfill and writes a warning to stderr that catchup was ignored. This allows scheduled configuration to keep catchup enabled while a manual run supplies at.

window.count, --window-count, or METABOL_WINDOW_COUNT selects multiple consecutive windows. The default is 1. The selected window is the newest; earlier windows are added by walking backward and all windows are processed from oldest to newest:

$ metabol --at 2026-09-13 --window-count 3

This processes the window containing September 13 and the two windows immediately before it. The count must be between 1 and 366.

Output and errors

Standard output format

metabol writes its results to stdout as JSON Lines (JSONL): one compact JSON object followed by a newline for each successfully processed article. The output uses the same camelCase field names and value semantics as the corresponding mdhq result fields, while omitting other mdhq fields.

{"requestedUrl":"https://example.com/a","sourceUrl":"https://example.com/a","path":"example.com/a.md","status":"saved"}
{"requestedUrl":"https://example.com/b","sourceUrl":"https://example.com/b","path":"example.com/b.md","status":"skipped"}

Every record contains these required, nonempty string fields:

Field Description
requestedUrl URL passed to mdhq for the article
sourceUrl Source URL reported by mdhq, which may differ after redirects or source resolution
path Path of the Markdown file relative to root
status Processing result: saved, updated, unchanged, or skipped

No record is emitted for a failed article. Logs, warnings, and diagnostics go to stderr, keeping stdout machine-readable.

If an individual feed or article fails, metabol continues processing the remaining work, preserves successful records on stdout, and exits nonzero after all possible work is complete. Configuration, validation, and window calculation errors fail before article processing.

GitHub Action

[!CAUTION] Run this workflow in a private repository. Archived articles may contain third-party copyrighted content, and publishing the repository may redistribute that content. Ensure that your use complies with applicable permissions, licenses, and terms.

Use the GitHub Action to collect articles from a workflow. The following example saves articles under articles, commits and pushes any changes, and uploads the processing results:

name: Collect feeds

on:
  schedule:
    - cron: "15 7 * * *"
      timezone: Asia/Tokyo
  workflow_dispatch:
    inputs:
      at:
        description: Date selecting the window to process (YYYY-MM-DD)
        type: string

jobs:
  collect:
    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - uses: actions/checkout@v7
      - id: metabol
        uses: Songmu/metabol@v0
        with:
          config: metabol.yaml
          root: articles
          timezone: Asia/Tokyo
          catchup: true
          at: ${{ inputs.at }}
      - name: Commit and push articles
        if: ${{ !cancelled() && steps.metabol.outputs.count > 0 }}
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add articles
          if git diff --cached --quiet; then
            exit 0
          fi
          git commit -m "Update articles"
          git push
      - uses: actions/upload-artifact@v7
        if: ${{ !cancelled() && steps.metabol.outputs.count > 0 }}
        with:
          name: metabol-results
          path: ${{ steps.metabol.outputs.results }}

Inputs are config, root, assets, update, catchup, timezone, at, and window-count.

The action exposes:

Output Description
results Absolute path to the captured JSONL processing results
count Number of result records

results points to the JSONL results file, and count is the number of successfully processed articles. Successful articles and results remain available even if another article fails.

Scheduled runs leave inputs.at empty, so the action omits at and processes the most recently completed window. The workflow execution time does not need to exactly match window.daily; a delayed or retried scheduled run selects the same logical window.

When manually starting the workflow, optionally enter a date in YYYY-MM-DD format, such as 2026-09-13, for at. This selects the window containing that date. If omitted, the most recently completed window is processed. In this example, catchup remains enabled for scheduled runs, but an explicitly supplied at takes precedence for manual backfills.

Agent Skill

The metabol binary includes an English Agent Skill that teaches compatible coding agents how to configure metabol, select deterministic processing windows, run backfills, and consume its JSON Lines output. The skill is managed through the bundled skillsmith subcommand and is never installed automatically.

# Inspect the skill bundled with this metabol release.
$ metabol skills list

# Install it for the current user under ~/.agents/skills.
$ metabol skills install

# Install it under the current repository's .agents/skills directory.
$ metabol skills install --scope repo

# Preview a change, check status, and apply an updated bundled version.
$ metabol skills update --dry-run
$ metabol skills status
$ metabol skills update

Use --prefix /path/to/skills to choose a custom installation directory. reinstall replaces managed skills even when the recorded version matches, while uninstall removes managed copies. New skill content ships with new metabol releases; installing a newer binary does not modify the user's skill directory until metabol skills update or reinstall is run.

Guarantees and non-goals

Given the same configuration and reference time, metabol selects the same logical window without an execution-history database. Re-running a window is designed to be idempotent through mdhq, but the contents of remote feeds and articles are outside metabol's reproducibility guarantee.

metabol is not a scheduler, state database, fetched-URL database, raw feed archive, or replacement for mdhq's storage layout.

Author

Songmu

Documentation

Index

Constants

View Source 
const DefaultConfigPath = "metabol.yaml"
View Source 
const MaxWindowCount = 366

MaxWindowCount limits one run to at most one leap year's worth of daily windows.

Variables

This section is empty.

Functions

func ParseAt 

func ParseAt(value string, location *time.Location) (time.Time, error)

ParseAt accepts RFC3339 timestamps or YYYY-MM-DD dates in location.

func ParseWindowAt 

func ParseWindowAt(value string, location *time.Location) (time.Time, error)

ParseWindowAt parses an RFC3339 instant or a YYYY-MM-DD date. Date-only values denote local midnight in location and use the same DST resolution rules as daily boundaries.

func Run 

func Run(ctx context.Context, argv []string, outStream, errStream io.Writer) error

Run executes metabol.

func ValidateArticleURL 

func ValidateArticleURL(rawURL string) error

ValidateArticleURL checks that rawURL is an absolute HTTP(S) URL without userinfo. Feed items are untrusted input that metabol passes to mdhq as a command-line argument, so values such as "--root=/tmp" or "mailto:x" must never reach the downstream command.

Types

type BoolValue 

type BoolValue struct {
	// contains filtered or unexported fields
}

BoolValue is a boolean flag value that distinguishes unset from false.

func (BoolValue) Get 

func (v BoolValue) Get() (bool, bool)

Get returns the value and whether it was explicitly set.

func (*BoolValue) IsBoolFlag 

func (*BoolValue) IsBoolFlag() bool

IsBoolFlag allows boolean options to mean true without an argument.

func (*BoolValue) Set 

func (v *BoolValue) Set(value string) error

func (*BoolValue) String 

func (v *BoolValue) String() string

type CLIValues 

type CLIValues struct {
	Config      StringValue
	Root        StringValue
	Assets      BoolValue
	Update      BoolValue
	Catchup     BoolValue
	Timezone    StringValue
	At          StringValue
	WindowCount IntValue
}

CLIValues contains configuration-related command-line values.

func (CLIValues) ConfigPath 

func (v CLIValues) ConfigPath(lookupEnv LookupEnvFunc) (path string, explicit bool, err error)

ConfigPath resolves CLI > METABOL_CONFIG > the default configuration path. Explicit reports whether CLI or the environment selected the path.

func (*CLIValues) RegisterFlags 

func (v *CLIValues) RegisterFlags(fs *flag.FlagSet)

RegisterFlags registers configuration-related flags on fs.

type CollectedURL 

type CollectedURL struct {
	URL       string
	SourceURL string
}

CollectedURL retains the feed that produced an article URL.

func CollectFeeds 

func CollectFeeds(
	ctx context.Context,
	fetcher FeedFetcher,
	sources []string,
	since time.Time,
	until *time.Time,
) ([]CollectedURL, error)

CollectFeeds collects unique article URLs in source and feed item order.

Collection continues after source failures. The returned error joins every source failure and every rejected article URL, while the returned URLs contain all successfully collected items. Article URLs that are not absolute HTTP(S) URLs are rejected rather than forwarded to mdhq, because feed items are untrusted input that becomes a command-line argument.

type CommandRunner 

type CommandRunner interface {
	Run(ctx context.Context, name string, args ...string) (stdout, stderr []byte, err error)
}

CommandRunner runs an external command and keeps its stdout and stderr separate.

type Config 

type Config struct {
	Root     string       `yaml:"root"`
	Assets   *bool        `yaml:"assets"`
	Update   *bool        `yaml:"update"`
	Catchup  *bool        `yaml:"catchup"`
	Timezone string       `yaml:"timezone"`
	Window   WindowConfig `yaml:"window"`
	Sources  []Source     `yaml:"sources"`
}

Config is the YAML configuration for metabol.

func LoadConfig 

func LoadConfig(path string) (*Config, error)

LoadConfig reads and validates exactly path; it does not search fallback paths.

func (Config) Validate 

func (c Config) Validate() error

Validate checks values that must be valid before runtime resolution.

type DailyTime 

type DailyTime struct {
	Hour   int
	Minute int
}

DailyTime is a parsed local-time daily boundary.

func ParseDailyTime 

func ParseDailyTime(value string) (DailyTime, error)

ParseDailyTime parses an exact 24-hour HH:MM value.

type DailyWindow 

type DailyWindow struct {
	// contains filtered or unexported fields
}

DailyWindow selects windows separated by a fixed local wall-clock time.

func NewDailyWindow 

func NewDailyWindow(location *time.Location, boundary string) (*DailyWindow, error)

NewDailyWindow creates a daily window selector in location.

func NewDailyWindowForTimezone 

func NewDailyWindowForTimezone(timezone, boundary string) (*DailyWindow, error)

NewDailyWindowForTimezone loads timezone and creates a daily window selector.

func NewDailyWindowFromTime 

func NewDailyWindowFromTime(location *time.Location, boundary DailyTime) (*DailyWindow, error)

NewDailyWindowFromTime creates a selector from an already parsed boundary.

func (*DailyWindow) Containing 

func (d *DailyWindow) Containing(instant time.Time) (Window, error)

Containing returns the half-open window containing instant.

func (*DailyWindow) LastComplete 

func (d *DailyWindow) LastComplete(reference time.Time) (Window, error)

LastComplete returns the last window whose end is not after reference.

func (*DailyWindow) Select 

func (d *DailyWindow) Select(reference time.Time, at *time.Time) (Window, error)

Select returns the window containing at when at is non-nil. Otherwise it returns the last window whose end is not after reference.

func (*DailyWindow) SelectMany 

func (d *DailyWindow) SelectMany(reference time.Time, at *time.Time, count int) ([]Window, error)

SelectMany returns count consecutive windows ending with the selected window, ordered from oldest to newest.

type ExecCommandRunner 

type ExecCommandRunner struct{}

ExecCommandRunner runs commands with os/exec.

func (ExecCommandRunner) Run 

func (ExecCommandRunner) Run(
	ctx context.Context,
	name string,
	args ...string,
) ([]byte, []byte, error)

Run implements CommandRunner.

type FeedFetcher 

type FeedFetcher interface {
	Fetch(ctx context.Context, sourceURL string, since time.Time, until *time.Time) ([]FeedItem, error)
}

FeedFetcher fetches feed items since a lower bound and, when non-nil, before an upper bound.

type FeedItem 

type FeedItem struct {
	URL string
}

FeedItem is the part of an rssnip item needed by the pipeline.

type IntValue 

type IntValue struct {
	// contains filtered or unexported fields
}

IntValue is an integer flag value that distinguishes unset from zero.

func (IntValue) Get 

func (v IntValue) Get() (int, bool)

Get returns the value and whether it was explicitly set.

func (*IntValue) Set 

func (v *IntValue) Set(value string) error

func (*IntValue) String 

func (v *IntValue) String() string

type LookupEnvFunc 

type LookupEnvFunc func(string) (string, bool)

LookupEnvFunc matches os.LookupEnv and is injectable for deterministic tests.

type MDHQ 

type MDHQ struct {
	Runner  CommandRunner
	Command string
}

MDHQ invokes the mdhq command once per URL.

func NewMDHQ 

func NewMDHQ(runner CommandRunner) *MDHQ

NewMDHQ constructs an mdhq client. A nil runner uses os/exec.

func (*MDHQ) Get 

func (m *MDHQ) Get(
	ctx context.Context,
	url string,
	options MDHQOptions,
) (MDHQResult, error)

Get fetches one URL and returns its attributed mdhq result.

type MDHQGetter 

type MDHQGetter interface {
	Get(ctx context.Context, url string, options MDHQOptions) (MDHQResult, error)
}

MDHQGetter saves one URL with mdhq.

type MDHQOptions 

type MDHQOptions struct {
	Root   string
	Assets bool
	Update bool
}

MDHQOptions controls mdhq storage behavior.

type MDHQResult 

type MDHQResult struct {
	RequestedURL string `json:"requestedUrl"`
	SourceURL    string `json:"sourceUrl"`
	Path         string `json:"path"`
	Status       string `json:"status"`
	Diagnostic   string `json:"-"`
}

MDHQResult is the stable subset of mdhq's JSON result emitted by metabol.

type Pipeline 

type Pipeline struct {
	Fetcher FeedFetcher
	MDHQ    MDHQGetter
}

Pipeline coordinates feed collection and mdhq.

func NewPipeline 

func NewPipeline(fetcher FeedFetcher, mdhq MDHQGetter) *Pipeline

NewPipeline constructs a pipeline with injectable dependencies.

func (*Pipeline) Run 

func (p *Pipeline) Run(
	ctx context.Context,
	request PipelineRequest,
	stdout, stderr io.Writer,
) error

Run executes the complete pipeline and writes compact successful results as JSON Lines to stdout. Diagnostics are written separately to stderr.

type PipelineRequest 

type PipelineRequest struct {
	Sources []string
	Since   time.Time
	Until   *time.Time
	Root    string
	Assets  bool
	Update  bool
}

PipelineRequest describes one feed collection and mdhq run. A nil Until leaves the feed collection upper bound open.

type RSSnipFetcher 

type RSSnipFetcher struct{}

RSSnipFetcher fetches feeds with the rssnip library.

func (RSSnipFetcher) Fetch 

func (RSSnipFetcher) Fetch(
	ctx context.Context,
	sourceURL string,
	since time.Time,
	until *time.Time,
) ([]FeedItem, error)

Fetch implements FeedFetcher.

type ResolvedConfig 

type ResolvedConfig struct {
	ConfigPath  string
	Root        string
	Assets      bool
	Update      bool
	Catchup     bool
	Location    *time.Location
	Daily       DailyTime
	WindowCount int
	At          *time.Time
	Sources     []Source
}

ResolvedConfig contains values after applying CLI, environment, YAML, and downstream/default fallbacks.

func LoadResolvedConfig 

func LoadResolvedConfig(cli CLIValues, lookupEnv LookupEnvFunc) (*ResolvedConfig, error)

LoadResolvedConfig selects, loads, and resolves the configuration.

func ResolveConfig 

func ResolveConfig(cli CLIValues, config *Config, lookupEnv LookupEnvFunc) (*ResolvedConfig, error)

ResolveConfig applies CLI > METABOL_* > YAML values. Callers must provide root through one of these sources because this entry point has no path fallback. Assets and update default to false, and timezone defaults to the local timezone.

type Source 

type Source struct {
	URL  string `yaml:"url"`
	Name string `yaml:"name,omitempty"`
}

Source describes an input feed.

func (*Source) UnmarshalYAML 

func (s *Source) UnmarshalYAML(data []byte) error

UnmarshalYAML accepts either a URL scalar or a source mapping.

func (Source) Validate 

func (s Source) Validate() error

Validate checks that the source has an absolute HTTP(S) URL without userinfo.

type StringValue 

type StringValue struct {
	// contains filtered or unexported fields
}

StringValue is a flag value that distinguishes an unset value from an explicitly supplied empty string.

func (StringValue) Get 

func (v StringValue) Get() (string, bool)

Get returns the value and whether it was explicitly set.

func (*StringValue) Set 

func (v *StringValue) Set(value string) error

func (*StringValue) String 

func (v *StringValue) String() string

type Window 

type Window struct {
	Start time.Time
	End   time.Time
}

Window is a half-open interval [Start, End).

func SelectDailyWindow 

func SelectDailyWindow(
	location *time.Location,
	boundary DailyTime,
	reference time.Time,
	at *time.Time,
) (Window, error)

SelectDailyWindow creates a selector from resolved configuration and selects either the last complete window or, when at is non-nil, its containing window.

func SelectDailyWindows 

func SelectDailyWindows(
	location *time.Location,
	boundary DailyTime,
	reference time.Time,
	at *time.Time,
	count int,
) ([]Window, error)

SelectDailyWindows creates a selector from resolved configuration and returns count consecutive windows in chronological order.

func (Window) Contains 

func (w Window) Contains(instant time.Time) bool

Contains reports whether instant belongs to the window.

type WindowConfig 

type WindowConfig struct {
	Daily string `yaml:"daily"`
	Count *int   `yaml:"count,omitempty"`
}

WindowConfig describes logical window selection.

Source Files

View all Source files

Directories

Path Synopsis
cmd
metabol command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.