datasource

Documentation

Index

• type DataSource
• type DownloadRequest
• type Downloader
  • func NewDownloader(cacheDir string, logger *slog.Logger) *Downloader
  • func (d *Downloader) DownloadFile(ctx context.Context, req DownloadRequest, dst, tempPrefix string) error
  • func (d *Downloader) Fetch(ctx context.Context, req DownloadRequest) (string, error)
• type Manager
  • func NewManager(logger *slog.Logger) *Manager
  • func (m *Manager) EnsureAvailable(ctx context.Context, required ...string) error
  • func (m *Manager) Register(source DataSource)

Types

type DataSource

type DataSource interface {
	Name() string        // "wikidata", "moby", "cefrj"
	Path() string        // Resolved file/directory path
	DownloadURL() string // Source URL for download
	Exists() bool        // Check if file/dir exists
	Download(ctx context.Context) error
	Verify() error // Verify integrity after download
}

DataSource represents a pipeline data source with download capabilities

type DownloadRequest

type DownloadRequest struct {
	Source string
	URL    string
}

DownloadRequest describes a source download request.

type Downloader

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader encapsulates shared download/cache behavior for all data sources.

func NewDownloader

func NewDownloader(cacheDir string, logger *slog.Logger) *Downloader

func (*Downloader) DownloadFile

func (d *Downloader) DownloadFile(ctx context.Context, req DownloadRequest, dst, tempPrefix string) error

DownloadFile ensures the source URL is available locally and atomically writes it to dst.

func (*Downloader) Fetch

func (d *Downloader) Fetch(ctx context.Context, req DownloadRequest) (string, error)

Fetch returns a local artifact path for the requested source URL. Storage details (including cache) are internal to Downloader.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages pipeline data sources

func NewManager

func NewManager(logger *slog.Logger) *Manager

NewManager creates a new data source manager.

func (*Manager) EnsureAvailable

func (m *Manager) EnsureAvailable(ctx context.Context, required ...string) error

EnsureAvailable checks if required data sources are available, auto-downloading any that are missing.

func (*Manager) Register

func (m *Manager) Register(source DataSource)

Register adds a data source to the manager.