README
¶
webfilteringdatabase-go
Classification client for Go gateways, resolvers and proxies. Give Classify a hostname or URL and it returns the filtering category the service assigns, the per-domain counterpart of bulk domain classification. Use it next to the downloadable database: the file answers for domains you already have, and this client answers for the ones you do not.
go get github.com/explainableaixai/webfilteringdatabase-go
Call it
wf := webfilteringdatabase.New(os.Getenv("AQ_API_KEY"))
res, err := wf.Classify(ctx, "newly-registered-site.example")
res is a webfilteringdatabase.Result, a map[string]any decoded from the JSON reply. The central value is the filtering category, with a confidence score. The API reference on the product site lists every field. The client passes the JSON through untouched.
Fitting it into a filter
A filter built in Go usually has a hot path that must never block and a slow path that may. Classification belongs on the slow path. A workable shape:
type Filter struct {
local map[string]string // loaded from the licensed file
learned sync.Map // host -> category, filled in the background
pending sync.Map // host -> struct{}, avoids duplicate work
wf *webfilteringdatabase.Client
}
func (f *Filter) Category(host string) string {
if c, ok := f.local[host]; ok {
return c
}
if c, ok := f.learned.Load(host); ok {
return c.(string)
}
if _, busy := f.pending.LoadOrStore(host, struct{}{}); !busy {
go f.learn(host)
}
return "unclassified"
}
func (f *Filter) learn(host string) {
defer f.pending.Delete(host)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
r, err := f.wf.Classify(ctx, host)
if err != nil {
return
}
if c, ok := r["web_filtering_category"].(string); ok && c != "" {
f.learned.Store(host, c)
}
}
The first request for an unknown host gets the policy for unclassified. Every request after that gets the real category. Nothing waits on the network.
Policy for the unclassified window
What unclassified means is a policy decision, not a technical one:
| Environment | Typical choice |
|---|---|
| School or library | Block until classified |
| Corporate office | Allow and log |
| Guest Wi-Fi | Allow, but block known high-risk categories |
| Kiosk or locked-down device | Block |
Newly registered domains are over-represented in phishing and scams, so strict networks lose little by holding them for the minute a classification takes.
Mapping categories to actions
Keep categories and actions apart. A small table per tenant, loaded from configuration, is easier to audit than logic spread through code. The names below are placeholders, so use the exact category names from the API reference:
var policy = map[string]string{
"Adult": "block",
"Gambling": "block",
"Proxy": "block",
"Social": "allow",
"Streaming": "throttle",
}
Look up the category, fall back to a default action, and log both values. When someone asks why a site was blocked, the log answers in one line.
Errors and backpressure
*webfilteringdatabase.APIErrorfor any status of 400 or above. InspectStatus: 429 means ease off, while 401 and 403 mean check the key or quota.- Context and transport errors for timeouts and network faults.
- A plain error, before sending, when the key or value is empty.
In a background learner like the one above, drop the item on error and let the next request for that host try again. For 429, pause all learners briefly, because every goroutine will hit the same limit.
Configuration
New sets BaseURL to the hosted API and HTTPClient to a client with a 30 second timeout. Both are exported fields. Swap HTTPClient for one with your own transport if the gateway must egress through a proxy, and set BaseURL to a stub for integration tests. The key travels in the POST body over HTTPS, as the endpoint requires.
What gets sent
Only the string you pass to Classify, plus the key. The client adds no client IPs, user identities or headers from the filtered request. If you classify full URLs, strip query strings first, since they sometimes carry tokens or email addresses. The host is enough for a category.
MSPs and multi-tenant filters
When one gateway serves many customers, share the learned categories across all tenants and keep policy tables per tenant. A domain's category is a fact about the domain, while blocking it is a customer's choice. New tenants then benefit from everything the gateway has already learned.
AI and shadow IT
Generative AI rarely fits a traditional filtering scheme. Schools in particular can add K12 content filtering for AI tools next to the general database. To find out how much AI is already in use, the resolver logs you keep anyway reveal shadow AI hidden in DNS logs.
If you need domain categorization data you host yourself for analytics rather than enforcement, or want a domain categorization API for newly seen domains with the IAB taxonomy, those services use the same account model.
Compliance notes
US schools that receive E-rate discounts must filter under the Children's Internet Protection Act. The categories map directly onto the content types CIPA names. Enterprises often use the same categories to enforce acceptable-use policies and to cut exposure to malware and phishing sites.
Testing
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
fmt.Fprint(w, `{"web_filtering_category":"News","confidence":0.93}`)
}))
defer srv.Close()
wf := webfilteringdatabase.New("test")
wf.BaseURL = srv.URL
Observability
Export three counters from the learner: lookups started, lookups failed by status, and time to classify. A rising share of 429s means the learner pool is too eager. A growing backlog of pending hosts means traffic has shifted to many new domains, which is worth a look on its own, since sudden bursts of fresh domains often come from malware or a phishing campaign.
Same client, other stacks
There is a Dart and Flutter build, a Rust crate for the same endpoint, a PHP library on Packagist and a JavaScript package.
License
MIT
Documentation
Source Files
Directories