mirror

Documentation

Index

• func RewriteLinks(content string, baseURL string) string
• type FilterFunc
  • func BuildFilter(include, exclude []string) FilterFunc
• type Mirror
  • func New(f fetcher.HTTPFetcher, s *store.Store, idx store.Indexer) *Mirror
  • func (m *Mirror) Sync(ctx context.Context, result *discovery.Result, filter FilterFunc) (*SyncResult, error)
• type SyncResult
• type Syncer

Functions

func RewriteLinks

func RewriteLinks(content string, baseURL string) string

RewriteLinks converts absolute URLs pointing to the same domain into relative local paths. External links are preserved as-is.

Types

type FilterFunc

type FilterFunc func(path string) bool

FilterFunc returns true if a path should be included in the sync.

func BuildFilter

func BuildFilter(include, exclude []string) FilterFunc

BuildFilter creates a FilterFunc from include/exclude glob patterns. If include is empty, all paths match. Exclude takes precedence. Supports ** for recursive directory matching (e.g. "/docs/**/*.md").

type Mirror

type Mirror struct {
	Fetcher fetcher.HTTPFetcher
	Store   *store.Store
	Index   store.Indexer
}

Mirror downloads discovered content and writes it to the store.

func New

func New(f fetcher.HTTPFetcher, s *store.Store, idx store.Indexer) *Mirror

New creates a Mirror.

func (*Mirror) Sync

func (m *Mirror) Sync(ctx context.Context, result *discovery.Result, filter FilterFunc) (*SyncResult, error)

Sync downloads all files from a discovery result and writes them to the store. If filter is non-nil, only paths that pass the filter are downloaded.

type SyncResult

type SyncResult struct {
	Domain    string   `json:"domain"`
	Added     []string `json:"added,omitempty"`
	Updated   []string `json:"updated,omitempty"`
	Unchanged []string `json:"unchanged,omitempty"`
	Skipped   []string `json:"skipped,omitempty"`
	Errors    []string `json:"errors,omitempty"`
}

SyncResult tracks what happened during a sync.

type Syncer

type Syncer interface {
	Sync(ctx context.Context, result *discovery.Result, filter FilterFunc) (*SyncResult, error)
}

Syncer downloads discovered content and writes it to the local store. The default implementation fetches via HTTP, converts HTML to markdown, rewrites links, and compares content for change detection. Alternative implementations can add content cleaning, chunking, or custom pipelines.