filter

package
v1.149.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 13, 2026 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package filter provides declarative, rule-based filtering for in-memory slices.

Problem

API endpoints frequently need to apply client-driven filters to server-side collections (for example, after parsing a `filter=` query parameter). Handwritten loop logic is repetitive, hard to validate, and difficult to keep consistent across data models.

Solution

This package evaluates structured Rule expressions against slice elements and filters the slice in place.

Rules are grouped as `[][]Rule` with boolean semantics:

  • outer slice: AND
  • inner slice: OR

So `[[A], [B, C], [D]]` evaluates as `A AND (B OR C) AND D`. Every group must hold at least one rule, and every rule must set all three fields; an empty group, an empty rule set, or a rule missing a field is rejected as a malformed filter.

Rules can be supplied directly or parsed from JSON via Processor.ParseJSON, and query parameter payloads can be loaded with Processor.ParseURLQuery. A JSON rule object must carry the "field", "type", and "value" keys (an empty "field" selects the whole element; a "value" of null is a nil reference); the JSON grammar is defined by filter_schema.json. Supplying rules directly as a [][]Rule in Go is not held to that shape — a Go caller may, for example, pass an empty rule set to match every element.

Key Features

  • JSON-friendly filtering grammar for client-provided filters.
  • Rich comparison operators: regexp, equality/equal-fold, prefix/suffix, contains, and numeric ordering (<, <=, >, >=) with a collection/string length fallback.
  • Optional negation prefix (`!`) for every operator.
  • Dot-path field selection for nested struct fields. By default selectors are matched against Go field names, case-sensitively (so `Address.Country`); use WithFieldNameTag to match a struct tag instead (so a JSON-style `address.country`).
  • Optional struct-tag based field lookup via WithFieldNameTag.
  • In-place filtering with pagination-style controls via Processor.ApplySubset (offset + length), plus total-match count.
  • Limits to constrain runtime and untrusted-input cost: rule/result counts (WithMaxRules, WithMaxResults), value length (WithMaxValueLength), payload size (WithMaxFilterBytes), and field-path depth (WithMaxFieldDepth).
  • Reflection-path caching for repeated evaluations on the same types.

Important Behavior

  • The slice argument for Processor.Apply / Processor.ApplySubset must be a pointer to a slice and is modified in place: matching elements are compacted to the front and the slice is shortened (its backing array is not zeroed beyond the new length).
  • A field selector that cannot be resolved against a concrete element type — it names no such field, descends into a non-struct, targets an unexported field, or is deeper than WithMaxFieldDepth — is a deterministic client error, rejected with ErrInvalidFilter before any element is touched. A field that is merely unreachable on a given element — a nil pointer along the path, or a field absent from the concrete type of an element in an interface-typed slice (e.g. []any), knowable only per element — makes that element a non-match (filtered out) rather than an error. A pointer or interface leaf is dereferenced to the value it holds, and a nil leaf compares as a nil operand.
  • Processor.ParseURLQuery returns nil rules when the configured query key is missing or empty.
  • The `!` prefix inverts an operator's result, including for operands the base operator cannot handle: `!==` against a type it cannot compare, or `!<` against a non-ordered operand, matches. For any element whose field is actually read, exactly one of a rule and its negation matches. They fail to partition the slice only when the field is *unreachable* on an element — a nil pointer along the path, or a field absent on a []any element — since such an element is a non-match for both the rule and its negation (the evaluator is never reached). Note that `!=` negates equal-fold (`=`), i.e. it is "not equal under case folding".
  • A Processor and a compiled [][]Rule are safe for concurrent use across goroutines. That safety does not extend to a target slice shared between concurrent Processor.Apply calls, since Apply mutates it in place.

Security

This package performs filtering only; it does NOT perform authorization. Filter expressions are untrusted client input, and the grammar lets a client select and compare any exported field of the elements (including nested fields). Callers MUST therefore apply it only to data the requesting user is already authorized to see: pre-filter the slice to that user's permitted records (and, where relevant, project away fields they may not read) before calling Processor.Apply. Passing records that contain fields the user is not entitled to would let a crafted filter probe those values through the match result and total-match count.

Regular-expression rules use Go's RE2 engine, which matches in linear time with no catastrophic backtracking, so a malicious pattern cannot cause exponential-time matching. Cost is instead driven by input size, which the Processor bounds by default: a rule's string value (including a regexp pattern) is limited to DefaultMaxValueLength bytes (WithMaxValueLength), the raw filter payload decoded by Processor.ParseJSON and Processor.ParseURLQuery is limited to DefaultMaxFilterBytes bytes (WithMaxFilterBytes), WithMaxRules caps how many rules may be applied, and WithMaxFieldDepth bounds field-selector nesting. Array and object rule values are rejected outright by the JSON parser. The number of distinct resolved field paths cached per Processor is also capped internally, so filtering a recursive element type with an unbounded variety of selectors cannot grow memory without limit. Note that Processor.Apply still evaluates every rule against every element regardless of the requested result window, so callers should also bound the size of the slice being filtered. Decode untrusted filter JSON with Processor.ParseJSON (or Processor.ParseURLQuery): these apply the payload-size and rule-count limits.

Errors caused by a malformed or disallowed client filter are wrapped with ErrInvalidFilter. A handler processing untrusted input can test errors.Is(err, ErrInvalidFilter) to return a generic rejection (for example an HTTP 400) and log the detail server-side, rather than returning the underlying message, which may echo client input or internal type names.

Example

The following pretty-printed JSON: 

[
  [
    {
      "field": "name",
      "type": "==",
      "value": "doe"
    },
    {
      "field": "age",
      "type": "<=",
      "value": 42
    }
  ],
  [
    {
      "field": "address.country",
      "type": "regexp",
      "value": "^EN$|^FR$"
    }
  ]
]

can be represented in one line as: 

[[{"field":"name","type":"==","value":"doe"},{"field":"age","type":"<=","value":42}],[{"field":"address.country","type":"regexp","value":"^EN$|^FR$"}]]

and URL-encoded as a query parameter: 

filter=%5B%5B%7B%22field%22%3A%22name%22%2C%22type%22%3A%22%3D%3D%22%2C%22value%22%3A%22doe%22%7D%2C%7B%22field%22%3A%22age%22%2C%22type%22%3A%22%3C%3D%22%2C%22value%22%3A42%7D%5D%2C%5B%7B%22field%22%3A%22address.country%22%2C%22type%22%3A%22regexp%22%2C%22value%22%3A%22%5EEN%24%7C%5EFR%24%22%7D%5D%5D

The equivalent logic is: 

((name==doe OR age<=42) AND (address.country match "EN" or "FR"))

These selectors (name, age, address.country) are JSON-style names, so the Processor must be built with WithFieldNameTag("json") to resolve them against the corresponding struct tags; see the Processor.Apply example. Without it, selectors resolve against Go field names (Name, Age, Address.Country) and a JSON-style name is rejected as an unknown field selector.

Available Rule Types

Supported rule types are:

  • `regexp` : matches the value against a reference regular expression (strings only).
  • `==` : Equal to - matches exactly the reference value. Numbers compare numerically (so an int field equals a float reference of the same value); two nils are equal.
  • `=` : Equal fold - for strings, matches when they are equal under simple Unicode case-folding, a more general form of case-insensitivity (for example `AB` matches `ab`); for numbers it behaves exactly like `==`.
  • `^=` : Starts with - (strings only) matches when the value begins with the reference string.
  • `=$` : Ends with - (strings only) matches when the value ends with the reference string.
  • `~=` : Contains - (strings only) matches when the reference string is a sub-string of the value.
  • `<` : Less than - matches when the value is less than the reference.
  • `<=` : Less than or equal to - matches when the value is less than or equal the reference.
  • `>` : Greater than - matches when the value is greater than reference.
  • `>=` : Greater than or equal to - matches when the value is greater than or equal the reference.

Rule types are matched case-insensitively, so `==`, `REGEXP` and `regexp` are all valid.

The ordering operators (<, <=, >, >=) require a numeric reference value (a non-numeric reference is a configuration error, ErrInvalidFilter, not a silent false). They compare numeric values directly; for strings, arrays, slices, and maps they compare the length of the value against the reference (not lexicographic order). Anything else evaluates to false.

The string operators (regexp, ^=, =$, ~=) act only on string values; a non-string reference is a configuration error, and a non-string value being tested is a non-match.

Every rule type can be prefixed with `!` to negate its result. `!==` matches values that are not equal, `!<` values that are not less than the reference, and so on. Negation inverts the whole result: a `!` rule also matches when the base operator could not apply at all (a type it cannot compare, or an operand with no ordering). For any element whose field is actually read, exactly one of a rule and its negation matches; they fail to partition the elements only when the field is unreachable on an element (a nil pointer along the path, or a field absent on a []any element), which is a non-match for both. Note that `!=` is the negation of `=` (equal-fold), i.e. "not equal under case folding", not a distinct operator.

Benefits

filter enables expressive, validated, and reusable filtering logic for API and application layers while minimizing repetitive per-type query code.

Index

Examples

Constants

View Source 
const (
	// MaxResults is the maximum number of results that can be returned.
	MaxResults = 1<<31 - 1 // math.MaxInt32

	// DefaultMaxResults is the default number of results for Apply.
	// Can be overridden with WithMaxResults().
	DefaultMaxResults = MaxResults

	// DefaultMaxRules is the default maximum number of rules.
	//
	// The limit bounds two counts: the number of AND groups (the outer slice), and the
	// total number of rules summed across all the OR groups (the inner slices). So
	// "name AND age AND (country==EN OR country==FR)" counts as 3 groups and 4 rules.
	//
	// Can be overridden with WithMaxRules().
	DefaultMaxRules = 8

	// DefaultURLQueryFilterKey is the default URL query key used by Processor.ParseURLQuery().
	// Can be customized with WithQueryFilterKey().
	DefaultURLQueryFilterKey = "filter"

	// DefaultMaxValueLength is the default maximum byte length of a rule's string
	// value (e.g. a regexp pattern). It bounds regexp compilation and matching cost
	// for untrusted filters. Can be overridden with WithMaxValueLength().
	DefaultMaxValueLength = 4096

	// DefaultMaxFilterBytes is the default maximum byte length of the raw filter payload
	// accepted by Processor.ParseJSON() (and hence Processor.ParseURLQuery()) before it is
	// JSON-decoded. It bounds parse-time cost for untrusted input. Can be overridden with
	// WithMaxFilterBytes().
	DefaultMaxFilterBytes = 1 << 16 // 64 KiB

	// DefaultMaxFieldDepth is the default maximum number of dot-separated segments in a
	// rule field selector (e.g. "a.b.c" has depth 3). It bounds field-resolution cost per
	// selector. Can be overridden with WithMaxFieldDepth().
	DefaultMaxFieldDepth = 32
)
View Source 
const (
	// TypePrefixNot is a prefix that can be added to any type to get the negated value (opposite match).
	TypePrefixNot = "!"

	// TypeRegexp is a filter type that matches the value against a reference regular expression.
	// The reference value must be a regular expression that can compile.
	// Works only with strings (anything else will evaluate to false).
	TypeRegexp = "regexp"

	// TypeEqual is a filter type that matches exactly the reference value.
	TypeEqual = "=="

	// TypeEqualFold is a filter type that matches when strings, interpreted as UTF-8, are equal under simple Unicode case-folding, which is a more general form of case-insensitivity. For example "AB" will match "ab".
	TypeEqualFold = "="

	// TypeHasPrefix is a filter type that matches when the value begins with the reference string.
	TypeHasPrefix = "^="

	// TypeHasSuffix  is a filter type that matches when the value ends with the reference string.
	TypeHasSuffix = "=$"

	// TypeContains  is a filter type that matches when the reference string is a sub-string of the value.
	TypeContains = "~="

	// TypeLT is a filter type that matches when the value is less than reference.
	TypeLT = "<"

	// TypeLTE is a filter type that matches when the value is less than or equal the reference.
	TypeLTE = "<="

	// TypeGT is a filter type that matches when the value is greater than reference.
	TypeGT = ">"

	// TypeGTE is a filter type that matches when the value is greater than or equal the reference.
	TypeGTE = ">="
)
View Source 
const (
	// FieldNameSeparator is the separator for Rule fields.
	FieldNameSeparator = "."
)

Variables

View Source 
var ErrInvalidFilter = errors.New("invalid filter")

ErrInvalidFilter wraps every error attributable to a malformed or disallowed client filter: invalid JSON, an oversized payload or value, too many rules, an unsupported or mistyped rule, an uncompilable regexp, or a field selector that cannot be resolved. Callers handling untrusted input can test errors.Is(err, ErrInvalidFilter) to reject the request generically (for example an HTTP 400) and log the detail server-side, instead of returning the underlying message, which may echo client input or internal type names.

Errors caused by misuse of the Apply/ApplySubset arguments by the calling code — a non-slice-pointer target, or an out-of-range length — are NOT wrapped with it.

Functions

This section is empty.

Types

type Option 

type Option func(p *Processor) error

Option configures a Processor instance.

func WithFieldNameTag 

func WithFieldNameTag(tag string) Option

WithFieldNameTag configures field lookup to use the given struct tag instead of field names. Evaluates the tag value before the first comma (e.g., "json:my_field,omitempty" → "my_field"). Returns error if tag is empty.

func WithMaxFieldDepth 

func WithMaxFieldDepth(maxdepth uint) Option

WithMaxFieldDepth sets the maximum number of dot-separated segments allowed in a rule's field selector (for example "a.b.c" has depth 3). Deeper selectors are rejected, bounding per-selector field-resolution cost. It bounds each selector's length, not the number of distinct selectors cached; that is capped separately and internally. Defaults to DefaultMaxFieldDepth. Returns error if maxdepth < 1.

func WithMaxFilterBytes 

func WithMaxFilterBytes(maxbytes uint) Option

WithMaxFilterBytes sets the maximum byte length of the raw filter payload accepted by Processor.ParseJSON() (and hence Processor.ParseURLQuery()) before it is JSON-decoded, bounding parse-time cost for untrusted input. Defaults to DefaultMaxFilterBytes. Returns error if maxbytes < 1.

func WithMaxResults 

func WithMaxResults(resmax uint) Option

WithMaxResults sets the maximum returned-element count for Apply() and ApplySubset(). Returns error if resmax < 1 or resmax > MaxResults.

func WithMaxRules 

func WithMaxRules(rulemax uint) Option

WithMaxRules sets the maximum permitted rule count to limit evaluation runtime cost.

The limit is applied to two counts: the number of AND groups (the outer slice), and the total number of rules summed across all the OR groups (the inner slices). So "name AND age AND (country==EN OR country==FR)" counts as 3 groups and 4 rules, and needs a limit of at least 4.

Defaults to DefaultMaxRules if not set. Returns error if rulemax < 1.

func WithMaxValueLength 

func WithMaxValueLength(maxlen uint) Option

WithMaxValueLength sets the maximum byte length allowed for a rule's string value, such as a regexp pattern or a string comparison operand. Rules whose string value exceeds the limit are rejected at compile time (by Processor.Apply / Processor.ApplySubset), bounding regexp compilation and matching cost for untrusted filters. The check applies to any string-kinded value, including a named string type. Defaults to DefaultMaxValueLength. Returns error if maxlen < 1.

func WithQueryFilterKey 

func WithQueryFilterKey(key string) Option

WithQueryFilterKey customizes the URL query parameter key that ParseURLQuery() searches for. Returns error if key is empty.

type Processor 

type Processor struct {
	// contains filtered or unexported fields
}

Processor provides the filtering logic and methods. Construct one with New; the zero Processor has all limits set to zero and rejects every input.

func New 

func New(opts ...Option) (*Processor, error)

New constructs a Processor for declarative rule-based filtering with custom options. The first rule level uses AND semantics; the second uses OR ("[a,[b,c],d]" → "a AND (b OR c) AND d"). Customizable via field-tag lookup, query-parameter keys, count limits, and the untrusted-input safety limits (value length, payload size, field-path depth).

func (*Processor) Apply 

func (p *Processor) Apply(rules [][]Rule, slicePtr any) (uint, uint, error)

Apply filters a slice by removing non-matching elements in place. Convenience method equivalent to ApplySubset with offset 0 and length the configured maximum (WithMaxResults). Returns filtered-slice length, total-match count, and any error.

Example 
package main

import (
	"fmt"
	"log"
	"net/url"

	"github.com/tecnickcom/nurago/pkg/filter"
)

// Address is an example structure type used to test nested structures.
type Address struct {
	Country string `json:"country"`
}

// ID is an example structure type.
type ID struct {
	Name string  `json:"name"`
	Age  int     `json:"age"`
	Addr Address `json:"address"`
}

func main() {
	// Simulate an encoded query passed in the http.Request of a http.Handler
	encodedJSONFilter := "%5B%5B%7B%22field%22%3A%22name%22%2C%22type%22%3A%22%3D%3D%22%2C%22value%22%3A%22doe%22%7D%2C%7B%22field%22%3A%22age%22%2C%22type%22%3A%22%3C%3D%22%2C%22value%22%3A42%7D%5D%2C%5B%7B%22field%22%3A%22address.country%22%2C%22type%22%3A%22regexp%22%2C%22value%22%3A%22%5EEN%24%7C%5EFR%24%22%7D%5D%5D"

	u, err := url.Parse("https://example.invalid/items?filter=" + encodedJSONFilter)
	if err != nil {
		log.Fatal(err)
	}

	// Initialize the filter with options
	// * WithFieldNameTag: to express the filter based on JSON tags and not the actual field names
	f, err := filter.New(
		filter.WithFieldNameTag("json"),
	)
	if err != nil {
		log.Fatal(err)
	}

	// The filter matches the following pretty printed json:
	//
	//	[
	//	  [
	//	    {
	//	      "field": "name",
	//	      "type": "==",
	//	      "value": "doe"
	//	    },
	//	    {
	//	      "field": "age",
	//	      "type": "<=",
	//	      "value": 42
	//	    }
	//	  ],
	//	  [
	//	    {
	//	      "field": "address.country",
	//	      "type": "regexp",
	//	      "value": "^EN$|^FR$"
	//	    }
	//	  ]
	//	]
	//
	// can be represented in one line as:
	//
	//	[[{"field":"name","type":"==","value":"doe"},{"field":"age","type":"<=","value":42}],[{"field":"address.country","type":"regexp","value":"^EN$|^FR$"}]]
	//
	// and URL-encoded as a query parameter:
	//
	//	filter=%5B%5B%7B%22field%22%3A%22name%22%2C%22type%22%3A%22%3D%3D%22%2C%22value%22%3A%22doe%22%7D%2C%7B%22field%22%3A%22age%22%2C%22type%22%3A%22%3C%3D%22%2C%22value%22%3A42%7D%5D%2C%5B%7B%22field%22%3A%22address.country%22%2C%22type%22%3A%22regexp%22%2C%22value%22%3A%22%5EEN%24%7C%5EFR%24%22%7D%5D%5D
	//
	// the equivalent logic is:
	//
	//	((name==doe OR age<=42) AND (address.country match "EN" or "FR"))
	rules, err := f.ParseURLQuery(u.Query())
	if err != nil {
		log.Fatal(err)
	}

	// Given this list, the last item will be filtered
	list := []ID{
		{
			Name: "doe",
			Age:  55,
			Addr: Address{
				Country: "EN",
			},
		},
		{
			Name: "dupont",
			Age:  42,
			Addr: Address{
				Country: "FR",
			},
		},
		{
			Name: "doe",
			Age:  41,
			Addr: Address{
				Country: "US",
			},
		},
	}

	// Filters the list in place
	sliceLen, totalMatches, err := f.Apply(rules, &list)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Println(sliceLen)
	fmt.Println(totalMatches)

	for _, id := range list {
		fmt.Println(id)
	}

}

Output:
2
2
{doe 55 {EN}}
{dupont 42 {FR}}

func (*Processor) ApplySubset 

func (p *Processor) ApplySubset(rules [][]Rule, slicePtr any, offset, length uint) (uint, uint, error)

ApplySubset filters a slice by removing non-matching elements in place, with pagination support. The slicePtr must be a pointer to a slice; offset and length control pagination within matches. Returns filtered-slice length within window, total-match count overall, and any error.

func (*Processor) ParseJSON 

func (p *Processor) ParseJSON(s string) ([][]Rule, error)

ParseJSON decodes a JSON filter rule set with the Processor's safety limits applied: it rejects payloads larger than the configured maximum (WithMaxFilterBytes) before decoding, and rejects rule sets exceeding WithMaxRules. The payload must match the shape of filter_schema.json — at least one non-empty AND group, and every rule object carrying the "field", "type", and "value" keys — otherwise it is rejected as a malformed filter. It is the only supported way to decode untrusted filter JSON; the unbounded decoder is internal. Filter-attributable errors are wrapped with ErrInvalidFilter.

func (*Processor) ParseURLQuery 

func (p *Processor) ParseURLQuery(q url.Values) ([][]Rule, error)

ParseURLQuery unmarshals the JSON filter rule set from a URL query parameter. Defaults to the "filter" key; override with WithQueryFilterKey. Returns nil rules when the key is missing or empty; otherwise it delegates to Processor.ParseJSON and applies the same limits (WithMaxFilterBytes, WithMaxRules). Filter-attributable errors are wrapped with ErrInvalidFilter.

type Rule 

type Rule struct {
	// Field is a dot-separated selector used to target a specific field of the evaluated value.
	//
	// * "Age" will select the Age field of a structure
	// * "Address.Country" will select the Country subfield of the Address structure
	// * "" will select the whole value (e.g. to filter a []string)
	Field string `json:"field"`

	// Type controls the evaluation to apply.
	// An invalid value causes an error when the rule is applied via [Processor.Apply].
	// See the Type* constants of this package for valid values.
	Type string `json:"type"`

	// Value is the reference value to evaluate against.
	// Its type should be accepted by the chosen Type.
	//
	// When a rule is decoded from JSON via [Processor.ParseJSON] or [Processor.ParseURLQuery],
	// Value is normalized to a concrete Go type: an integer literal becomes an int64 (or a
	// uint64 when it exceeds math.MaxInt64), any other number becomes a float64, and strings,
	// booleans, and null pass through as string, bool, and nil. Integer literals therefore
	// compare exactly, even beyond 2^53. A number too large for any of those types (e.g. 1e400)
	// and any array or object value are rejected by the parser. A Go caller may still set Value
	// to any type the chosen Type accepts.
	Value any `json:"value"`
}

Rule defines one filter expression evaluated against an input value.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.