waf

README

go-libddwaf

This project's goal is to produce a higher level API for the go bindings to libddwaf: DataDog in-app WAF. It consists of 2 separate entities: the bindings for the calls to libddwaf, and the encoder which job is to convert any go value to its libddwaf object representation.

An example usage would be:

import waf "github.com/DataDog/go-libddwaf/v2"

//go:embed
var ruleset []byte

func main() {
    var parsedRuleset any

    if err := json.Unmarshal(ruleset, &parsedRuleset); err != nil {
        return 1
    }

    wafHandle, err := waf.NewHandle(parsedRuleset, "", "")
    if err != nil {
        return 1
    }

    defer wafHandle.Close()

    wafCtx := wafHandle.NewContext()
    defer wafCtx.Close()

    matches, actions := wafCtx.Run(RunAddressData{
        Persistent: map[string]any{
            "server.request.path_params": "/rfiinc.txt",
        },
    }, time.Minute)
}

The API documentation details can be found on pkg.go.dev.

Originally this project was only here to provide CGO Wrappers to the calls to libddwaf. But with the appearance of ddwaf_object tree like structure, but also with the intention to build CGO-less bindings, this project size has grown to be a fully integrated brick in the DataDog tracer structure. Which in turn made it necessary to document the project, to maintain it in an orderly fashion.

Supported platforms

This library currently support the following platform doublets:

OS	Arch
Linux	amd64
Linux	aarch64
OSX	amd64
OSX	arm64

This means that when the platform is not supported, top-level functions will return a WafDisabledError error including the purpose of it.

Note that:

• Linux support include for glibc and musl variants
• OSX under 10.9 is not supported
• A build tag named datadog.no_waf can be manually added to force the WAF to be disabled.

Design

The WAF bindings have multiple moving parts that are necessary to understand:

• Handle: a object wrapper over the pointer to the C WAF Handle
• Context: a object wrapper over a pointer to the C WAF Context
• Encoder: its goal is to construct a tree of Waf Objects to send to the WAF
• CGORefPool: Does all allocation operations for the construction of Waf Objects and keeps track of the equivalent go pointers
• Decoder: Transforms Waf Objects returned from the WAF to usual go objects (e.g. maps, arrays, ...)
• Library: The low-level go bindings to the C library, providing improved typing

flowchart LR

    START:::hidden -->|NewHandle| Handle -->|NewContext| Context

    Context -->|Encode Inputs| Encoder

    Handle -->|Encode Ruleset| Encoder
    Handle -->|Init WAF| Library
    Context -->|Decode Result| Decoder

    Handle -->|Decode Init Errors| Decoder

    Context -->|Run| Library
    Context -->|Store Go References| CGORefPool

    Encoder -->|Allocate Waf Objects| TempCGORefPool

    TempCGORefPool -->|Copy after each encoding| CGORefPool

    Library -->|Call C code| libddwaf

    classDef hidden display: none;

CGO Reference Pool

The cgoRefPool type is a pure Go pointer pool of ddwaf_object C values on the Go memory heap. the cgoRefPool go type is a way to make sure we can safely send Go allocated data to the C side of the WAF The main issue is the following: the WafObject uses a C union to store the tree structure of the full object, union equivalent in go are interfaces and they are not compatible with C unions. The only way to be 100% sure that the Go WafObject struct has the same layout as the C one is to only use primitive types. So the only way to store a raw pointer is to use the uintptr type. But since uintptr do not have pointer semantics (and are just basically integers), we need another method to store the value as Go pointer because the GC will delete our data if it is not referenced by Go pointers.

That's where the cgoRefPool object comes into play: all new WafObject elements are created via this API which is especially built to make sure there is no gap for the Garbage Collector to exploit. From there, since underlying values of the wafObject are either arrays of WafObjects (for maps, structs and arrays) or string (for all ints, booleans and strings), we can store 2 slices of arrays and use unsafe.KeepAlive in each code path to protect them from the GC.

All these objects stored in the reference pool need to live throughout the use of the associated Waf Context.

Typical call to Run()

Here is an example of the flow of operations on a simple call to Run():

• Encode input data into WAF Objects and store references in the temporary pool
• Lock the context mutex until the end of the call
• Store references from the temporary pool into the context level pool
• Call ddwaf_run
• Decode the matches and actions

CGO-less C Bindings

This library uses purego to implement C bindings without requiring use of CGO at compilation time. The high-level workflow is to embed the C shared library using go:embed, dump it into a file, open the library using dlopen, load the symbols using dlsym, and finally call them.

⚠ Keep in mind that purego only works on linux/darwin for amd64/arm64 and so does go-libddwaf.

Another requirement of libddwaf is to have a FHS filesystem on your machine and, for linux, to provide libc.so.6, libpthread.so.0, and libdl.so.2 as dynamic libraries.

Contributing pitfalls

• Cannot dlopen twice in the app lifetime on OSX. It messes with Thread Local Storage and usually finishes with a std::bad_alloc()
• keepAlive() calls are here to prevent the GC from destroying objects too early
• Since there is a stack switch between the Go code and the C code, usually the only C stacktrace you will ever get is from GDB
• If a segfault happens during a call to the C code, the goroutine stacktrace which has done the call is the one annotated with [syscall]
• GoLand does not support CGO_ENABLED=0 (as of June 2023)
• Keep in mind that we fully escape the type system. If you send the wrong data it will segfault in the best cases but not always!
• The structs in ctypes.go are here to reproduce the memory layout of the structs in include/ddwaf.h because pointers to these structs will be passed directly
• Do not use uintptr as function arguments or results types, coming from unsafe.Pointer casts of Go values, because they escape the pointer analysis which can create wrongly optimized code and crash. Pointer arithmetic is of course necessary in such a library but must be kept in the same function scope.
• GDB is available on arm64 but is not officially supported so it usually crashes pretty fast (as of June 2023)
• No pointer to variables on the stack shall be sent to the C code because Go stacks can be moved during the C call. More on this here

Documentation

Index

• Constants
• func Health() (bool, error)
• func Load() (ok bool, err error)
• func SupportsTarget() (bool, error)
• func Version() string
• type Context
  • func NewContext(handle *Handle) *Context
  • func NewContextWithBudget(handle *Handle, budget time.Duration) *Context
  • func (context *Context) Close()
  • func (context *Context) Run(addressData RunAddressData, _ time.Duration) (res Result, err error)
  • func (context *Context) Stats() Stats
  • func (context *Context) TotalRuntime() (uint64, uint64)
  • func (context *Context) TotalTimeouts() uint64
• type DiagnosticAddresses
• type DiagnosticEntry
• type Diagnostics
  • func (d *Diagnostics) TopLevelError() error
• type Handle
  • func NewHandle(rules any, keyObfuscatorRegex string, valueObfuscatorRegex string) (*Handle, error)
  • func (handle *Handle) Addresses() []string
  • func (handle *Handle) Close()
  • func (handle *Handle) Diagnostics() Diagnostics
  • func (handle *Handle) Update(newRules any) (*Handle, error)
• type Result
  • func (r *Result) HasActions() bool
  • func (r *Result) HasDerivatives() bool
  • func (r *Result) HasEvents() bool
• type RunAddressData
• type Stats
  • func (stats Stats) Metrics() map[string]any
• type TruncationReason
  • func (reason TruncationReason) String() string

Constants

const (
	AppsecFieldTag            = "ddwaf"
	AppsecFieldTagValueIgnore = "ignore"
)

const ErrTimeout = wafErrors.ErrTimeout

ErrTimeout is the error returned when the WAF times out while processing a request. Deprecated: use github.com/DataDog/go-libddwaf/errors.ErrTimeout instead.

Functions

func Health

func Health() (bool, error)

Health returns true if the waf is usable, false otherwise. At the same time it can return an error if the waf is not usable, but the error is not blocking if true is returned, otherwise it is. The following conditions are checked: - The Waf library has been loaded successfully (you need to call `Load()` first for this case to be taken into account) - The Waf library has not been manually disabled with the `datadog.no_waf` go build tag - The Waf library is not in an unsupported OS/Arch - The Waf library is not in an unsupported Go version

func Load

func Load() (ok bool, err error)

Load loads libddwaf's dynamic library. The dynamic library is opened only once by the first call to this function and internally stored globally, and no function is currently provided in this API to close the opened handle. Calling this function is not mandatory and is automatically performed by calls to NewHandle, the entrypoint of libddwaf, but Load is useful in order to explicitly check libddwaf's general health where calling NewHandle doesn't necessarily apply nor is doable. The function returns ok when libddwaf was successfully loaded, along with a non-nil error if any. Note that both ok and err can be set, meaning that libddwaf is usable but some non-critical errors happened, such as failures to remove temporary files. It is safe to continue using libddwaf in such case.

func SupportsTarget

func SupportsTarget() (bool, error)

SupportsTarget returns true and a nil error when the target host environment is supported by this package and can be further used. Otherwise, it returns false along with an error detailing why.

func Version

func Version() string

Version returns the version returned by libddwaf. It relies on the dynamic loading of the library, which can fail and return an empty string or the previously loaded version, if any.

Types

type Context

type Context struct {
	// contains filtered or unexported fields
}

Context is a WAF execution context. It allows running the WAF incrementally when calling it multiple times to run its rules every time new addresses become available. Each request must have its own Context.

func NewContext

func NewContext(handle *Handle) *Context

NewContext returns a new WAF context of to the given WAF handle. A nil value is returned when the WAF handle was released or when the WAF context couldn't be created. handle. A nil value is returned when the WAF handle can no longer be used or the WAF context couldn't be created.

func NewContextWithBudget

func NewContextWithBudget(handle *Handle, budget time.Duration) *Context

NewContextWithBudget returns a new WAF context of to the given WAF handle. A nil value is returned when the WAF handle was released or when the WAF context couldn't be created. handle. A nil value is returned when the WAF handle can no longer be used or the WAF context couldn't be created.

func (*Context) Close

func (context *Context) Close()

Close the underlying `ddwaf_context` and releases the associated internal data. Also decreases the reference count of the `ddwaf_hadnle` which created this context, possibly releasing it completely (if this was the last context created from this handle & it was released by its creator).

func (*Context) Run

func (context *Context) Run(addressData RunAddressData, _ time.Duration) (res Result, err error)

Run encodes the given addressData values and runs them against the WAF rules within the given timeout value. If a given address is present both as persistent and ephemeral, the persistent value takes precedence. It returns the matches as a JSON string (usually opaquely used) along with the corresponding actions in any. In case of an error, matches and actions can still be returned, for instance in the case of a timeout error. Errors can be tested against the RunError type. Struct fields having the tag `ddwaf:"ignore"` will not be encoded and sent to the WAF if the output of TotalTime() exceeds the value of Timeout, the function will immediately return with errors.ErrTimeout The second parameter is deprecated and should be passed to NewContextWithBudget instead.

func (*Context) Stats

func (context *Context) Stats() Stats

Stats returns the cumulative time spent in various parts of the WAF, all in nanoseconds and the timeout value used

func (*Context) TotalRuntime

func (context *Context) TotalRuntime() (uint64, uint64)

TotalRuntime returns the cumulated WAF runtime across various run calls within the same WAF context. Returned time is in nanoseconds. Deprecated: use Timings instead

func (*Context) TotalTimeouts

func (context *Context) TotalTimeouts() uint64

TotalTimeouts returns the cumulated amount of WAF timeouts across various run calls within the same WAF context.

type DiagnosticAddresses

type DiagnosticAddresses struct {
	Required []string
	Optional []string
}

DiagnosticAddresses stores the information - provided by the WAF - about the known addresses and whether they are required or optional. Addresses used by WAF rules are always required. Addresses used by WAF exclusion filters may be required or (rarely) optional. Addresses used by WAF processors may be required or optional.

type DiagnosticEntry

type DiagnosticEntry struct {
	Addresses *DiagnosticAddresses
	Errors    map[string][]string // Item-level errors (map of error message to entity identifiers or index:#)
	Error     string              // If the entire entry was in error (e.g: invalid format)
	Loaded    []string            // Successfully loaded entity identifiers (or index:#)
	Failed    []string            // Failed entity identifiers (or index:#)
}

DiagnosticEntry stores the information - provided by the WAF - about loaded and failed rules for a specific entry in the WAF ruleset

type Diagnostics

type Diagnostics struct {
	Rules          *DiagnosticEntry
	CustomRules    *DiagnosticEntry
	Exclusions     *DiagnosticEntry
	RulesOverrides *DiagnosticEntry
	RulesData      *DiagnosticEntry
	Processors     *DiagnosticEntry
	Scanners       *DiagnosticEntry
	Version        string
}

Diagnostics stores the information - provided by the WAF - about WAF rules initialization.

func (*Diagnostics) TopLevelError

func (d *Diagnostics) TopLevelError() error

TopLevelErrors returns the list of top-level errors reported by the WAF on any of the Diagnostics entries, rolled up into a single error value. Returns nil if no top-level errors were reported. Individual, item-level errors might still exist.

type Handle

type Handle struct {
	// contains filtered or unexported fields
}

Handle represents an instance of the WAF for a given ruleset.

func NewHandle

func NewHandle(rules any, keyObfuscatorRegex string, valueObfuscatorRegex string) (*Handle, error)

NewHandle creates and returns a new instance of the WAF with the given security rules and configuration of the sensitive data obfuscator. The returned handle is nil in case of an error. Rules-related metrics, including errors, are accessible with the `RulesetInfo()` method.

func (*Handle) Addresses

func (handle *Handle) Addresses() []string

Addresses returns the list of addresses the WAF rule is expecting.

func (*Handle) Close

func (handle *Handle) Close()

Close puts the handle in termination state, when all the contexts are closed the handle will be destroyed

func (*Handle) Diagnostics

func (handle *Handle) Diagnostics() Diagnostics

Diagnostics returns the rules initialization metrics for the current WAF handle

func (*Handle) Update

func (handle *Handle) Update(newRules any) (*Handle, error)

Update the ruleset of a WAF instance into a new handle on its own the previous handle still needs to be closed manually

type Result

type Result struct {
	// Events is the list of events the WAF detected, together with any relevant
	// details.
	Events []any

	// Derivatives is the set of key-value pairs generated by the WAF, and which
	// need to be reported on the trace to provide additional data to the backend.
	Derivatives map[string]any

	// Actions is the set of actions the WAF decided on when evaluating rules
	// against the provided address data.
	Actions []string

	// TimeSpent is the time the WAF self-reported as spent processing the call to ddwaf_run
	TimeSpent time.Duration
}

Result stores the multiple values returned by a call to ddwaf_run

func (*Result) HasActions

func (r *Result) HasActions() bool

HasActions return true if the result holds at least 1 action

func (*Result) HasDerivatives

func (r *Result) HasDerivatives() bool

HasDerivatives return true if the result holds at least 1 derivative

func (*Result) HasEvents

func (r *Result) HasEvents() bool

HasEvents return true if the result holds at least 1 event

type RunAddressData

type RunAddressData struct {
	// Persistent address data is scoped to the lifetime of a given Context, and subsquent calls to Context.Run with the
	// same address name will be silently ignored.
	Persistent map[string]any
	// Ephemeral address data is scoped to a given Context.Run call and is not persisted across calls. This is used for
	// protocols such as gRPC client/server streaming or GraphQL, where a single request can incur multiple subrequests.
	Ephemeral map[string]any
}

RunAddressData provides address data to the Context.Run method. If a given key is present in both RunAddressData.Persistent and RunAddressData.Ephemeral, the value from RunAddressData.Persistent will take precedence.

type Stats

type Stats struct {
	// Timers returns a map of metrics and their durations.
	Timers map[string]time.Duration

	// Timeout
	TimeoutCount uint64

	// Truncations provides details about truncations that occurred while
	// encoding address data for WAF execution.
	Truncations map[TruncationReason][]int
}

Stats stores the metrics collected by the WAF.

func (Stats) Metrics

func (stats Stats) Metrics() map[string]any

Metrics transform the stats returned by the WAF into a map of key value metrics for datadog backend

type TruncationReason

type TruncationReason uint8

TruncationReason is a flag representing reasons why some input was not encoded in full.

const (
	// StringTooLong indicates a string exceeded the maximum string length configured. The truncation
	// values indicate the actual length of truncated strings.
	StringTooLong TruncationReason = 1 << iota
	// ContainerTooLarge indicates a container (list, map, struct) exceeded the maximum number of
	// elements configured. The truncation values indicate the actual number of elements in the
	// truncated container.
	ContainerTooLarge
	// ObjectTooDeep indicates an overall object exceeded the maximum encoding depths configured. The
	// truncation values indicate an estimated actual depth of the truncated object. The value is
	// guaranteed to be less than or equal to the actual depth (it may not be more).
	ObjectTooDeep
)

func (TruncationReason) String

func (reason TruncationReason) String() string