Documentation
¶
Overview ¶
Package urldetrack strips tracking query parameters from links and unwraps redirector links to their embedded target URL, driven by one or more operator-supplied ClearURLs rules catalogs, a small supplement of redirector rules for corporate mail gateways upstream doesn't cover (data/mailgateways.json: Microsoft Defender Safe Links, Barracuda Link Protect), and code decoders for the two Proofpoint URL Defense encodings no declarative ruleset can express (see proofpoint.go).
No catalog is embedded. A tracking-parameter catalog is only as good as its last refresh, and a snapshot compiled into a release would keep stripping yesterday's parameters for as long as that binary is deployed while quietly missing everything added since. Requiring the operator to supply the file makes the refresh their visible, schedulable job (see `sanimail clearurls fetch`) rather than something that silently rots. The mail-gateway supplement is different and stays embedded: it is sanimail's own hand-written data, not a vendored copy of anyone's list, and it is always merged in.
It is entirely offline: redirectors are unwrapped only when the destination is embedded in the link itself (e.g. ...?url=https%3A%2F%2Freal.example%2F); opaque-token click trackers, whose destination only the tracking server knows (e.g. Mimecast), are left intact. Cleaning never makes a network request.
A Detracker cleans URLs from two sources: the HTML attributes a recipient navigates to — see Navigable — via the htmlpolicy.URLRewriter returned by Detracker.Rewriter (the same hook the image inliner uses), and bare URLs in text/plain bodies, via Detracker.DetrackText. No URL a client fetches on its own is touched: image src, CSS url(), @font-face and the rest pass through unchanged so signed/CDN URLs are never broken. That half of the URL space belongs to email/trackerblock and the inliner, and the two sets are disjoint by construction (see the invariant test in this package).
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Navigable ¶ added in v0.20.0
func Navigable(ctx htmlpolicy.URLContext) bool
Navigable reports whether ctx is a URL the recipient navigates to — the only URL contexts this package cleans. It gates Detracker.Rewriter and doubles as the filter for htmlpolicy.WithOriginalURLAttr, so pre-rewrite URL preservation covers exactly what de-tracking may change and nothing else.
It is the complement of gitlab.com/grepular/sanimail/email/urlctx.Classify's subresource set: that package classifies a URL a client fetches on view, and says of the rest that "neutralising those is email/urldetrack's job". This is that job. The two must stay disjoint — a URL cleaned here must never also be a URL --block-trackers may neutralise — which TestNavigableIsNotSubresource enforces rather than assumes.
href is decided by element and the rest by attribute alone, matching how urlctx splits them: href is navigation on <a> and <area> but a subresource on <link> and an SVG <image>, whereas action/formaction are navigation wherever they appear. <a ping> is deliberately absent: it is a fire-on-click beacon rather than a destination, so cleaning its parameters would leave the beacon firing and merely make it less informative.
Types ¶
type Config ¶
type Config struct {
// Documents holds the operator-supplied de-tracking rulesets, compiled
// in the order given and merged after the mail-gateway supplement.
//
// Each is either a ClearURLs data.minify.json catalog or an Adblock
// filter list, whose $removeparam rules are read (see removeparam.go);
// the format is detected per document, so the two may be mixed freely
// across repetitions of the flag. Together they must yield at least one
// provider or one $removeparam rule.
Documents [][]byte
}
Config supplies the ruleset. There is no default: nothing is embedded, so a caller that supplies no document gets an error rather than a detracker that silently strips nothing.
type Counts ¶ added in v0.22.0
type Counts struct {
// Providers is the number of ClearURLs providers the documents yielded,
// excluding the always-merged mail-gateway supplement.
Providers int
// Blocking is the number of Adblock $removeparam rules that strip
// something. It is counted apart from Exceptions because an exception on
// its own does nothing: it can only spare a parameter another rule would
// have stripped, so a list with none of these cleans nothing however many
// it has.
Blocking int
// Exceptions is the number of compiled @@ $removeparam rules.
Exceptions int
// Unsupported counts rules this compiler could not represent: a pattern
// RE2 rejects, an option it has never seen. That is a defect in the
// parser or new syntax upstream, and is what a fetch refuses a list for.
Unsupported int
// Inapplicable counts rules it read in full and can prove will never
// fire against a link in mail: a ClearURLs completeProvider (a domain to
// block wholesale, which a navigational link cannot be), a rule scoped
// to a subresource type or a first-party context, or — in a list serving
// both consumers — a plain blocking rule, which is
// email/trackerblock's to read. Reported, never a reason to refuse.
Inapplicable int
}
Counts reports what a set of de-tracking documents compiled to. It is what `sanimail clearurls fetch` and `sanimail adblock fetch` check a download against before writing it.
func Validate ¶ added in v0.20.0
Validate compiles documents and reports what they yield, without building a Detracker. It is what the ruleset fetch subcommands check a download against before writing it.
Yielding nothing is deliberately not an error here. `adblock fetch` runs this alongside [trackerblock.Validate] because one filter list may carry rules for both, so a document that gives this package nothing may still be a perfectly good blocking list; the caller has both halves and decides. New still refuses an empty ruleset, where nothing else can stand in.
type Detracker ¶
type Detracker struct {
// contains filtered or unexported fields
}
Detracker holds the compiled ruleset and per-message counters. One per message. Safe to use for both the HTML rewriter and text passes within a message; htmlpolicy invokes the rewriter synchronously and single-threaded and the text pass runs separately, so the plain-int counters are never touched concurrently.
func New ¶
New compiles the ruleset and returns a Detracker. The mail-gateway supplement (data/mailgateways.json) is always merged in ahead of the catalogs — it is sanimail behavior, not part of the operator-supplied data. A nil logger is replaced with a no-op (zero-value safe). An error is returned when a supplied document is not valid JSON, or when the supplied documents yield no providers at all. Cleaning is otherwise silently fail-safe — an unrecognised or unparseable URL is left unchanged — so there is no warning sink to wire.
func (*Detracker) CleanURL ¶
CleanURL cleans a single bare URL (tracking-parameter stripping and redirector unwrapping), updates the per-message counters, and returns the result. It is the property-value analogue of the <a href> rewriter and the text-body pass — same engine, same counters — for callers (e.g. the text/calendar pass) that hold a URL out of band. A URL it cannot parse or recognize is returned unchanged.
func (*Detracker) DetrackText ¶
DetrackText rewrites every bare http(s) URL found in a text/plain body and reports whether anything changed. Text outside URLs — including the punctuation immediately following a URL — is preserved verbatim.
When delSp is true (the part is RFC 3676 format=flowed with DelSp=yes), a URL that ends right at a soft line break is left untouched: with DelSp the break space is deleted on reconstruction, so the URL may continue on the next physical line, and cleaning only the visible fragment could corrupt it. For ordinary flowed text (DelSp=no, the default) the break space is retained, so a URL at a line wrap is complete and is cleaned like any other.
func (*Detracker) Rewriter ¶
func (c *Detracker) Rewriter() htmlpolicy.URLRewriter
Rewriter returns the URLRewriter callback for htmlpolicy. It cleans only the navigational URLs Navigable names; every other context (image src, CSS url(), @font-face, …) is returned unchanged so resource URLs are never altered.
type Stats ¶
type Stats struct {
// Changed is the number of URLs that ended up different after cleaning.
Changed int `json:"changed"`
// ParamsStripped is the total number of query parameters removed.
ParamsStripped int `json:"params_stripped"`
// RedirectsDecoded is the number of redirector links unwrapped.
RedirectsDecoded int `json:"redirects_decoded"`
// Skipped is the number of URLs left uncleaned because the per-message
// budget was spent (see maxURLsPerMessage).
Skipped int `json:"skipped"`
}
Stats holds per-message cleaning activity. Counts every cleaned URL occurrence (an HTML attribute deduplicated by htmlpolicy counts once; the same URL appearing twice in a text body counts twice).
Source Files