jsonFilter

README

Overview

This library allows you to apply post processing filters to the Datastore/Firestore results.

The filter format is API oriented and designed to be provided by an API consumer in param to your its request.

Why to use this additional filter

Firestore and datastore have several query limitation for Firestore and for Datastore

• Only one array-contains-any or IN is allowed per query
• Only 10 elements are allowed in a IN or array-contains-any clause
• When you filter on range AND a field, you need to create a composite index beforehand
• You can search in the arrays that only contain values, and not object structure.

The library allows you to unlock these limitation:

• Allow to filter on any type and nested type (map, array, map of array/object, array of map/object,...)
• Allow to use several filters IN on the set of data
• Use more than 10 elements in a IN condition
• Allow to use several filters NOT IN on set of data
• Allow to compare several range with > and < operators
• Don't required any composite index creation

limitation

There is the known limitation of this library. These can be implemented -> Open a feature request!

• No wildcard like * to replace any JSON field name.
• No wildcard like * or regex to filter on values

Performance concern

The filters should be applied only on a small array of results and the filtering overhead is very small

Indeed, the documents have to be read for being filter. If you read thousand of document, you will pay a lot for nothing

In addition, your API response time will take more time because of the high number of documents to recover and the filtering duration.

Way of working

This library work with Go app and use reflection. It performs 3 things

• Check if the provided filter is valid 
• Compile the filter according with the data structure to filter -> Validate the filter against the structure to filter
• Apply the filter to the array of structure

See example for a practical implementation.

Filter format

The default filter format is the following

key1=val1,val2:key2.subkey=val3

Where:

• key1 is the JSON field name to filter. You can use composed filter to browse your JSON tree, like key2.subkey
• = is the operator. != > < are also available
• Val1, val2, val3 are the values to compare
• The tuple key + value(s) is named Filter

Behavior:

The filters are applied on an array of struct. Each element of the struct are evaluate against the filters

Each filter element must return OK for keeping the entry value. The behavior of the 4 operators are different:

• The equality, comparable to IN sql clause: at least one value must matches. Default operator is =
• The not equality, comparable to NOT IN sql clause: all values mustn't match. Default operator is !=
• The Greater Than: only one numeric can be compared. Default operator is >
• The Lower Than: only one numeric can be compared. Default operator is <

Customize filter format

The default filter format use these character

• Keys and values are separated by operator sign =,!=,<,> by default
• Filters are separated by colon : by default
• Values are separated by comma , by default
• Different fields value of a composed key is dot . by default

You can set an Options structure on filter to customize your filter like this

	o := &jsonFilter.Options{
		MaxDepth:                       4,
		EqualKeyValueSeparator:    		"=",
  		GreaterThanKeyValueSeparator: 	">",
		LowerThanKeyValueSeparator:   	"<",
		NotEqualKeyValueSeparator:    	"!=",
		ValueSeparator:                 ",",
		KeysSeparator:                  ":",
		ComposedKeySeparator:           "->",
	}
	
	filter.SetOptions(o)

If you don't define a part of the option, the default value is used for this part (a log message display this)

Max depth

You can also define the max depth of composed key. By default, this value is set to 0, which means infinite. You can override this value in the option structure.

Filter value type

You can filter on these simple types

• string
• int
• float
• bool

Complex type are supported

• pointer (invisible in JSON result but your structure can include filters)
• struct
• array
  • of simple types
  • of map
  • of array
  • of pointer
• map
  • of simple types
  • of map
  • of array
  • of pointer

Special filter on map

In JSON, the map representation is the following

{
    "mapsSimple":{
        "entryMap1":"value1",
        "entryMap2":"value2"
    },
    "mapsStruct":{
        "entryMap1": {
            "fieldName":"value1"
        },
        "entryMap2":{
            "fieldName":"value2"
        },
    },
    "mapsArray":{
        "entryMap1": [
            {
                "fieldName":"value1"
            }
        ],
        "entryMap2":[
            {
                "fieldName":"value2"
            },
        ]
    },

}

The filter key will be the following

• mapsSimple.entryMap1 if it's a simple map
• mapsStruct.entryMap1.fieldName if it's a map of structure
• mapsArray.entryMap1.fieldName if it's a map of Array. The array is invisible in the processing

Licence

This library is licensed under Apache 2.0. Full license text is available in LICENSE.

Documentation

Overview

Apply a post processing filters to the Datastore/Firestore results mapped in struct with json tag or not.

The filter format is designed to be passed in API param (query or path). The filters can express compound operation.

During the processing the values to filter (an array) is passed to the library to apply the filters. A filter can be composed to several part:

• Several filter elements
• Each filter element have a tree path into the JSON, name the key, an operator and the value(s) to compare

Each filter element must return OK for keeping the entry value. For this, 4 operators are allowed:

• The equality, comparable to IN sql clause: at least one value must matches. Default operator is `=`
• The not equality, comparable to NOT IN sql clause: all values mustn't match. Default operator is `!=`
• The Greater Than: only one numeric can be compared. Default operator is `>`
• The Lower Than: only one numeric can be compared. Default operator is `<`

It's possible to combine operators on the same key, for example k1 < 10 && k1 != 2. The same operator on the same key will raise an error.

The filters are applicable on this list types and structures (and combination possibles):

• simple types
• string
• int
• float
• bool
• Complex type
• pointer (invisible in JSON result but your structure can include filters)
• struct
• array
• of simple types
• of map
• of array
• of pointer
• map
• of simple types
• of map
• of array
• of pointer

This library works with Go app and use reflection. It performs 3 things

• Check if the provided filter is valid.
• Compile the filter according with the data structure to filter -> Validate the filter against the structure to filter.
• Apply the filter to the array of structure.

Index

• type Filter
  • func (f *Filter) ApplyFilter(e interface{}) (interface{}, error)
  • func (f *Filter) Init(v string, i interface{}) (err error)
  • func (f *Filter) SetOptions(o *Options)
• type Options

Types

type Filter

type Filter struct {
	// contains filtered or unexported fields
}

Filter structure to use for filtering. Init the default value like this

filter := jsonFilter.Filter{}

func (*Filter) ApplyFilter

func (f *Filter) ApplyFilter(e interface{}) (interface{}, error)

Apply the initialized Filter to a list (array) of struct. The type of array elements is the same as this one provided in the Init method. The entries must be an array.

Return an array with only the matching entries, else an error is returned.

Cast the return in the array type like this:

ret, err := filter.ApplyFilter(results)
if err != nil {
	// Perform error handling
	fmt.Println(err)
	return
}
// Cast here the return
results = ret.([]structExample)

func (*Filter) Init

func (f *Filter) Init(v string, i interface{}) (err error)

Initialize the filter with the requested filter and the struct on which to apply later the filter

The filter parsing and compilation are saved in the Filter struct.

Errors are returned in case of:

• Duplicated entry in the filter key name for the same operator
• Violation of filter format:
• No values for a key
• No key for a filter
• More than 1 value for Greater Than and Lower than operator
• Not a numeric (float compliant) value for Greater Than and Lower than operator
• Filter key not exist in the provided interface
• Struct field name not match the filter key
• Struct json tag not match the filter key

func (*Filter) SetOptions

func (f *Filter) SetOptions(o *Options)

Set the option to the filter.

If the option is nil, the default option will be used.

If there is some missing or incorrect value to the defined option, a warning message is displayed and the erroneous part is replace by the default ones.

To set option:

	filter := jsonFilter.Filter{}

	o := &jsonFilter.Options{
		MaxDepth:             			4,
		EqualKeyValueSeparator:    		"=",
  		GreaterThanKeyValueSeparator: 	">",
		LowerThanKeyValueSeparator:   	"<",
		NotEqualKeyValueSeparator:    	"!=",
		ValueSeparator:       			",",
		KeysSeparator:        			":",
		ComposedKeySeparator: 			"->",
	}

	filter.SetOptions(o)

type Options

type Options struct {
	// Limit the depth of the key search. In case of complex object, can limit the compute resources. 0 means infinite. Default is '0'
	MaxDepth int
	// Character(s) to separate key (filter name)  from values (value to compare) for an equal comparison. Default is '='
	EqualKeyValueSeparator string
	// Character(s) to separate key (filter name)  from values (value to compare) for a greater than comparison. Default is '>'
	GreaterThanKeyValueSeparator string
	// Character(s) to separate key (filter name)  from values (value to compare) for a lower than comparison. Default is '<'
	LowerThanKeyValueSeparator string
	// Character(s) to separate key (filter name)  from values (value to compare) for a not equal comparison. Default is '!='
	NotEqualKeyValueSeparator string
	//  Character(s) to separate values (value to compare). Default is ','
	ValueSeparator string
	// Character(s) to separate keys (filters name). Default is ':'
	KeysSeparator string
	// Character(s) to separate key part in case of composed key (filter.subfilter) . Default is '.'
	ComposedKeySeparator string
}

Structure to define the option of the Filter.

You can customize it if you want. Else the default values are applied