Documentation
¶
Overview ¶
Package threatfeed is the local threat-feed manager: it downloads, persists, and provides instant offline lookups for known-malicious URL/domain lists. Zero external API dependency — all feeds are freely available without registration or rate limits.
Feeds:
- URLhaus (abuse.ch): malware distribution URLs https://urlhaus.abuse.ch/downloads/text/
- OpenPhish: phishing URLs https://openphish.com/feed.txt
The feed data is stored in a JSON file so the proxy survives restarts without waiting for a fresh download. A background goroutine re-syncs on the configured interval (default: 6 hours).
Extracted from package main per ADR-0002. The package is a pure leaf: the SSRF private-IP table comes from internal/ssrf, durable writes from internal/fileutil, and logging from the obs facade (including the new obs.Debugf for the sync-start debug line). package main keeps the globalThreatFeed singleton and the admin/cluster surfaces behind a type alias.
Index ¶
- func NormaliseURL(raw string) (norm, host string)
- type Feed
- func (tf *Feed) AddDomainAllowlist(domain string) error
- func (tf *Feed) AllowlistMaskedTotal() int64
- func (tf *Feed) CheckDomain(domain string) (malicious bool, source string)
- func (tf *Feed) CheckURL(rawURL string) (malicious bool, source string)
- func (tf *Feed) DomainAllowlist() []string
- func (tf *Feed) DomainAllowlisted(domain string) bool
- func (tf *Feed) Enabled() bool
- func (tf *Feed) ExportDomains() map[string]int64
- func (tf *Feed) ExportURLs() map[string]int64
- func (tf *Feed) ImportFeedData(urls map[string]int64, domains map[string]int64)
- func (tf *Feed) Init(dbPath string, syncInterval time.Duration)
- func (tf *Feed) RemoveDomainAllowlist(domain string) error
- func (tf *Feed) Save()
- func (tf *Feed) SeedForTest(urls, domains map[string]string)
- func (tf *Feed) SetDBPathForTest(path string) (old string)
- func (tf *Feed) SetDomainAllowlist(domains []string) error
- func (tf *Feed) SetEnabledForTest(enabled bool) (old bool)
- func (tf *Feed) Start(ctx context.Context)
- func (tf *Feed) Stats() (int64, time.Time, time.Duration)
- func (tf *Feed) Sync()
- func (tf *Feed) SyncStatus() (ok bool, lastSuccess time.Time, errSummary string)
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func NormaliseURL ¶
NormaliseURL parses a raw URL string into a canonical lookup key (scheme + host + path, no query or fragment) and the bare hostname. Returns ("", "") for invalid, private-IP, or non-HTTP(S) entries.
Types ¶
type Feed ¶
type Feed struct {
// contains filtered or unexported fields
}
Feed manages local copies of public threat intelligence lists. All methods are safe for concurrent use.
func New ¶
func New() *Feed
New returns an idle Feed with the same shape as the pre-extraction package-main literal: initialised maps and the 6-hour default sync interval. Init configures and enables it.
func (*Feed) AddDomainAllowlist ¶
AddDomainAllowlist adds a domain to the allowlist and persists.
func (*Feed) AllowlistMaskedTotal ¶ added in v1.0.47
AllowlistMaskedTotal returns the cumulative count of domain-level threat hits suppressed by the domain allowlist since process start. A rising value means the allowlist is actively overriding live threat intel — worth an operator's attention (the exemption may be too broad, or an allowlisted platform is now hosting malware at the domain level).
func (*Feed) CheckDomain ¶
CheckDomain looks up a bare hostname against the threat feed. Returns (isMalicious, sourceName).
func (*Feed) CheckURL ¶
CheckURL looks up a full URL against the threat feed. Returns (isMalicious, sourceName).
func (*Feed) DomainAllowlist ¶
DomainAllowlist returns the current allowlist entries sorted.
func (*Feed) DomainAllowlisted ¶
DomainAllowlisted reports whether a domain is on the threat-feed allowlist (domain-level blocking skipped; URL-level blocking still applies).
func (*Feed) ExportDomains ¶
ExportDomains returns a copy of the domain threat map as domain→unix-timestamp. Used by the Control Plane to include feed data in config sync.
Currently-allowlisted hosts are excluded: the ThreatFeedDomains wire field keeps its pre-masking meaning ("hosts a lookup may block") because DP nodes running an older binary have no lookup-time allowlist mask — exporting masked hosts would make a mixed-version rolling upgrade (new CP, old DPs) hard-block allowlisted platforms fleet-wide. New DPs lose nothing: their own allowlist (synced separately) masks these hosts at lookup anyway.
func (*Feed) ExportURLs ¶
ExportURLs returns a copy of the URL threat map as url→unix-timestamp. Used by the Control Plane to include feed data in config sync.
func (*Feed) ImportFeedData ¶
ImportFeedData replaces the in-memory URL and domain maps with data received from the Control Plane config sync. Each value is a unix timestamp (added_at). The source is set to "cluster-sync" to distinguish from direct feed downloads.
func (*Feed) Init ¶
Init configures the feed manager and loads any persisted DB from disk. dbPath may be "" to disable persistence (feed data lives in-memory only).
func (*Feed) RemoveDomainAllowlist ¶
RemoveDomainAllowlist removes a domain from the allowlist and persists.
func (*Feed) Save ¶
func (tf *Feed) Save()
Save is the caller-facing wrapper around saveToDisk. Used by paths like applyConfigSnapshot that mutate via ImportFeedData — which does NOT auto-persist (unlike SetDomainAllowlist / AddDomainAllowlist / RemoveDomainAllowlist which call saveToDisk internally). Errors are logged, not returned, to match the void-Save convention of the other Bucket-1 snapshot stores (policyStore, globalProfileStore, etc.).
func (*Feed) SeedForTest ¶
SeedForTest inserts URL and domain entries directly (value = source name), replacing the whitebox map pokes package main's integration tests used pre-extraction. AddedAt is stamped now; totalEntries tracks the URL count.
func (*Feed) SetDBPathForTest ¶
SetDBPathForTest swaps the persistence path and returns the previous one. Test support for main-side durability tests that point the process-wide feed at a temp dir; pair the restore with ImportFeedData(nil, nil) to clear seeded data.
func (*Feed) SetDomainAllowlist ¶
SetDomainAllowlist replaces the entire allowlist and persists to disk.
func (*Feed) SetEnabledForTest ¶ added in v1.0.47
SetEnabledForTest toggles the feed's enabled flag and returns the previous value. Test support for main-side tests that assert CheckURL/CheckDomain verdicts on the process-wide feed: outside tests the flag is only set by Init, so a test that needs positive verdicts must enable the feed itself (and restore the old value in cleanup) instead of depending on whether an earlier test happened to run Init — that ordering dependence is exactly what the shuffle/determinism gate flags.
func (*Feed) Start ¶
Start launches the background sync goroutine. An immediate sync is performed when the cache is empty or has never synced.
func (*Feed) Sync ¶
func (tf *Feed) Sync()
Sync downloads all configured feeds and atomically replaces the in-memory lookup tables. Safe to call concurrently; calls run sequentially.
func (*Feed) SyncStatus ¶ added in v1.0.33
SyncStatus reports whether the most recent sync fetched every feed cleanly, the time of the last fully-successful sync (which may lag lastSync from Stats if recent attempts have been failing), and a summary of the most recent failure (empty when the last sync succeeded). lastSync updates on every attempt regardless of outcome, so relying on Stats alone lets a persistently-failing feed hide behind a perpetually-fresh timestamp.
Source Files