source

Documentation

Overview

Package source defines what a collector is and holds the registry of them.

A source declares itself, emits claims, and returns a checkpoint. It does not decide scope, allocate IDs, write files or construct network clients: those arrive with the request or belong to the pipeline. Adding a source therefore touches this package's registry and nothing else.

Index

• Constants
• func Fetch(ctx context.Context, c *http.Client, url string) (body []byte, sha256sum string, err error)
• func Register(s Source)
• func UnderRoot(host, root string) bool
• type Checkpoint
  • func Backoff(err error) Checkpoint
• type Claim
• type Collector
  • func (c *Collector) Add(host string, observedAt time.Time) error
  • func (c *Collector) Count() int
• type Descriptor
  • func (d Descriptor) Grant() netgate.Grant
• type Emitter
• type RateLimited
  • func AsRateLimited(err error) (*RateLimited, bool)
  • func (e *RateLimited) Error() string
• type Request
• type Source
  • func All() []Source
  • func Get(id string) (Source, bool)
  • func Select(ids []string) ([]Source, error)

Constants

const UserAgent = "canopy/0 (+https://github.com/anytool-sh/canopy)"

UserAgent identifies Canopy to providers. Several of them reject an empty one, and an operator reading their logs should be able to tell who called.

Functions

func Fetch

func Fetch(ctx context.Context, c *http.Client, url string) (body []byte, sha256sum string, err error)

Fetch performs one GET with the client the pipeline issued, and returns the body with its SHA-256 so a claim can cite the exact bytes it came from. The client is gate-issued: passing anything else is how a source escapes scope enforcement, so callers must use req.HTTP.

func Register

func Register(s Source)

Register adds a source to the registry. It panics on a duplicate ID, which can only be a programming error at init time.

func UnderRoot

func UnderRoot(host, root string) bool

UnderRoot reports whether host is the root or a name beneath it. A provider answer can carry names from unrelated domains that shared a certificate or a crawl; those are somebody else's asset and are not this run's to report.

Types

type Checkpoint

type Checkpoint struct {
	Cursor     string
	Done       bool
	RetryAfter time.Duration
}

Checkpoint is where a source stopped, so a later call can resume.

func Backoff

func Backoff(err error) Checkpoint

Backoff converts an error into the checkpoint to report with it, carrying the provider's retry delay so a later call can wait rather than guess.

type Claim

type Claim struct {
	Kind       core.Kind
	Hostname   string
	Value      string
	Query      string
	RawSHA256  string
	ObservedAt time.Time
}

Claim is one thing a source observed. The pipeline turns it into a core.Observation, assigning the ID and run ID so a source cannot forge them.

ObservedAt is the provider's date for the claim, such as a certificate's issuance time. A zero value means the source has no date of its own and the pipeline stamps the observation with the run clock.

type Collector

type Collector struct {
	Emit  Emitter
	Root  string
	Query string
	// SHA is the hash of the response the names came from.
	SHA string
	// contains filtered or unexported fields
}

Collector turns raw provider strings into claims. Every source needs the same four things — normalize, reject malformed names, keep only names under the requested root, and emit each name once — so they live here rather than being re-derived per provider with slightly different bugs.

func (*Collector) Add

func (c *Collector) Add(host string, observedAt time.Time) error

Add emits host as a hostname claim unless it is malformed, outside the root, or already seen in this run of the source. observedAt is the provider's date for the name; a zero value lets the pipeline stamp it with the run clock.

func (*Collector) Count

func (c *Collector) Count() int

Count is how many distinct hostnames have been emitted.

type Descriptor

type Descriptor struct {
	ID           string        `json:"id"`
	Activity     netgate.Class `json:"activity"`
	Capabilities []string      `json:"capabilities"`
	// Credential names the secret this source needs, empty when it needs none.
	// A source with a credential and no value is skipped, not failed.
	Credential string `json:"credential,omitempty"`
	// Endpoints are the provider hosts the source may reach.
	Endpoints []string `json:"endpoints,omitempty"`
	// RPS caps request rate against the provider. Zero means unlimited.
	RPS float64 `json:"rps,omitempty"`
}

Descriptor is a source's self-description, read before it runs so the pipeline can grant it the narrowest clients that let it work.

func (Descriptor) Grant

func (d Descriptor) Grant() netgate.Grant

Grant converts a descriptor into the authorization the gate enforces.

type Emitter

type Emitter func(Claim) error

Emitter is the only way to produce observations. It returns an error when the run is over budget or the ledger cannot be written, and a source must stop rather than retry.

type RateLimited

type RateLimited struct {
	Status     string
	RetryAfter time.Duration
}

RateLimited is a provider asking us to back off. It is separate from a generic failure because the run summary reports when a source may be retried rather than treating it as broken.

func AsRateLimited

func AsRateLimited(err error) (*RateLimited, bool)

AsRateLimited reports whether err was a provider backoff request.

func (*RateLimited) Error

func (e *RateLimited) Error() string

type Request

type Request struct {
	Root   string
	Cursor string
	// Limit caps hostnames per call, zero for the source's own default.
	Limit int
	// Credential is the secret for Descriptor.Credential, empty when none.
	Credential string
	HTTP       *http.Client
	DNS        *netgate.Resolver
}

Request is one call to a source. The clients are gate-issued: a source has no other way to reach the network, which is what makes the activity class it declares enforceable rather than advisory.

type Source

type Source interface {
	Describe() Descriptor
	Discover(ctx context.Context, req Request, emit Emitter) (Checkpoint, error)
}

Source collects hostnames from one provider.

func All

func All() []Source

All returns every registered source, ordered by ID.

func Get

func Get(id string) (Source, bool)

Get returns one registered source.

func Select

func Select(ids []string) ([]Source, error)

Select returns the named sources, reporting the first name that is unknown.