discovery

Documentation

Overview

Package discovery is a scope-aware content-discovery (forced-browse) engine: it brute-forces paths from a wordlist against a base URL, calibrates a per-directory soft-404 signature so it doesn't drown in false hits, optionally recurses into discovered directories, and streams results as it goes.

The engine is transport-agnostic: it sends through an injected Probe (so it is unit-testable without a network) and asks an injected scope predicate before touching any URL. The control layer wires a real HTTP client (honouring the upstream proxy and MITM TLS) and records found endpoints as flows.

Index

• func IsAPIPath(path string) bool
• type Engine
  • func New() *Engine
  • func (e *Engine) SetNotifier(fn func())
  • func (e *Engine) SetProbe(p Probe)
  • func (e *Engine) SetRecorder(fn Recorder)
  • func (e *Engine) SetScope(fn func(rawURL string) bool)
  • func (e *Engine) Start(spec Spec) error
  • func (e *Engine) State() State
  • func (e *Engine) Stop()
• type Outcome
• type Probe
• type Recorder
• type Result
• type Spec
• type State

Functions

func IsAPIPath

func IsAPIPath(path string) bool

IsAPIPath reports whether a URL path looks like an API endpoint rather than a static asset. It is conservative: obvious static assets (.css/.js/.png/…) are never tagged, and only high-signal patterns count as API — an "/api" or "/graphql" segment, a version segment like "/v1"/"/v2", or a data extension such as ".json"/".xml". The control layer uses it to auto-tag discovered API surface so operators can filter it from static files.

Types

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine runs content-discovery jobs (one at a time).

func New

func New() *Engine

New returns an idle engine. Wire a Probe (and optionally scope/notifier) before Start.

func (*Engine) SetNotifier

func (e *Engine) SetNotifier(fn func())

SetNotifier registers a callback fired (outside the lock) whenever results or running-state change, so the control layer can push SSE updates.

func (*Engine) SetProbe

func (e *Engine) SetProbe(p Probe)

SetProbe sets the request transport. Required before Start.

func (*Engine) SetRecorder

func (e *Engine) SetRecorder(fn Recorder)

SetRecorder registers an optional callback that persists each found result as a flow and returns its store id (0 when recording is off or fails).

func (*Engine) SetScope

func (e *Engine) SetScope(fn func(rawURL string) bool)

SetScope sets the in-scope predicate; URLs it rejects are never probed. A nil predicate (the default) treats everything as in scope.

func (*Engine) Start

func (e *Engine) Start(spec Spec) error

Start validates the spec and launches the run in a goroutine. It returns an error only for bad input; runtime/transport errors surface as Result.Error.

func (*Engine) State

func (e *Engine) State() State

State returns a copy-safe snapshot of the current/last run.

func (*Engine) Stop

func (e *Engine) Stop()

Stop cancels an in-flight run (no-op if idle).

type Outcome

type Outcome struct {
	Status      int
	Length      int64
	Body        []byte // a capped sample, used for word/line counts (optional)
	ContentType string
	Location    string // redirect target (for 3xx)
}

Outcome is what a Probe reports for a single URL.

type Probe

type Probe func(ctx context.Context, method, rawURL string, headers map[string]string) (Outcome, error)

Probe issues a single request and reports the outcome. It must be safe for concurrent use.

type Recorder

type Recorder func(Result) int64

Recorder is an optional hook the control layer uses to persist a found URL as a flow when recording is enabled. It runs outside the engine lock and may block.

type Result

type Result struct {
	URL         string `json:"url"`
	Path        string `json:"path"`
	Status      int    `json:"status"`
	Length      int64  `json:"length"`
	Words       int    `json:"words"`
	Lines       int    `json:"lines"`
	ContentType string `json:"contentType,omitempty"`
	Redirect    string `json:"redirect,omitempty"`
	Depth       int    `json:"depth"`
	Dir         bool   `json:"dir"`
	Error       string `json:"error,omitempty"`
	FlowID      int64  `json:"flowId,omitempty"` // set when the control layer records this hit as a flow
}

Result is one discovered (or notable) path.

type Spec

type Spec struct {
	BaseURL    string            // e.g. https://target/path/ — probes hang off here
	Words      []string          // wordlist (comments "#…" and blanks are ignored)
	Extensions []string          // e.g. [".php",".bak"] — each word is also tried bare
	Threads    int               // max concurrent probes (1–64)
	DelayMs    int               // delay between dispatching probes (throttle)
	Recursive  bool              // recurse into discovered directories
	MaxDepth   int               // recursion depth limit (0 = base level only)
	MatchCodes []int             // statuses considered "found" (empty = anything but 404/soft-404)
	HideCodes  []int             // statuses to suppress even if otherwise matched
	FilterLen  int64             // suppress results whose body length equals this (manual soft-404 filter)
	Headers    map[string]string // sent on every probe (auth cookies, tokens, …)
	MaxReq     int               // total request budget (0 = default)
	AutoTagAPI bool              // auto-tag recorded API-looking endpoints (see IsAPIPath)

	// DisableSoft404Calibration turns off the automatic soft-404 detection that
	// fires before each wordlist sweep. By default (false) the engine issues
	// calibProbes randomly-named requests per directory; if the server returns the
	// same status and a similar body length for all of them it is treated as a
	// soft-404 server and results matching that fingerprint are suppressed.
	// Set to true to skip calibration entirely (e.g. when the server is known to
	// return honest 404s or calibration is too expensive).
	DisableSoft404Calibration bool
}

Spec configures one discovery run.

type State

type State struct {
	Running   bool     `json:"running"`
	BaseURL   string   `json:"baseUrl"`
	Tried     int      `json:"tried"`
	Found     int      `json:"found"`
	Results   []Result `json:"results"`
	StartedMs int64    `json:"startedMs"`
	DoneMs    int64    `json:"doneMs"`
	Note      string   `json:"note,omitempty"`
}

State is a snapshot of a run for the API/UI.