libddwaf

README

go-libddwaf

This project's goal is to produce a higher level API for the go bindings to libddwaf: DataDog in-app WAF. It consists of 2 separate entities: the bindings for the calls to libddwaf, and the encoder which job is to convert any go value to its libddwaf object representation.

An example usage would be:

import waf "github.com/DataDog/go-libddwaf/v4"

//go:embed
var ruleset []byte

func main() {
    var parsedRuleset any

    if err := json.Unmarshal(ruleset, &parsedRuleset); err != nil {
        panic(err)
    }

    builder, err := waf.NewBuilder("", "")
    if err != nil {
        panic(err)
    }
    _, err := builder.AddOrUpdateConfig(parsedRuleset)
    if err != nil {
        panic(err)
    }

    wafHandle := builder.Build()
    if wafHandle == nil {
        panic("WAF handle is nil")
    }
    defer wafHandle.Close()

    wafCtx := wafHandle.NewContext(timer.WithUnlimitedBudget(), timer.WithComponent("waf", "rasp"))
    defer wafCtx.Close()

    matches, actions := wafCtx.Run(RunAddressData{
        Persistent: map[string]any{
            "server.request.path_params": "/rfiinc.txt",
        },
		TimerKey: "waf",
    })
}

The API documentation details can be found on pkg.go.dev.

Originally this project was only here to provide CGO Wrappers to the calls to libddwaf. But with the appearance of ddwaf_object tree like structure, but also with the intention to build CGO-less bindings, this project size has grown to be a fully integrated brick in the DataDog tracer structure. Which in turn made it necessary to document the project, to maintain it in an orderly fashion.

Supported platforms

This library currently support the following platform doublets:

OS	Arch
Linux	amd64
Linux	aarch64
OSX	amd64
OSX	arm64

This means that when the platform is not supported, top-level functions will return a WafDisabledError error including the purpose of it.

Note that:

• Linux support include for glibc and musl variants
• OSX under 10.9 is not supported
• A build tag named datadog.no_waf can be manually added to force the WAF to be disabled.

Design

The WAF bindings have multiple moving parts that are necessary to understand:

• Builder: an object wrapper over the pointer to the C WAF Builder
• Handle: an object wrapper over the pointer to the C WAF Handle
• Context: an object wrapper over a pointer to the C WAF Context
• Encoder: its goal is to construct a tree of Waf Objects to send to the WAF
• Decoder: Transforms Waf Objects returned from the WAF to usual go objects (e.g. maps, arrays, ...)
• Library: The low-level go bindings to the C library, providing improved typing

flowchart LR
    START:::hidden -->|NewBuilder| Builder -->|Build| Handle

    Handle -->|NewContext| Context

    Context -->|Encode Inputs| Encoder

    Handle -->|Encode Ruleset| Encoder
    Handle -->|Init WAF| Library
    Context -->|Decode Result| Decoder

    Handle -->|Decode Init Errors| Decoder

    Context -->|Run| Library
    Encoder -->|Allocate Waf Objects| runtime.Pinner

    Library -->|Call C code| libddwaf

    classDef hidden display: none;

runtime.Pinner

When passing Go values to the WAF, it is necessary to make sure that memory remains valid and does not move until the WAF no longer has any pointers to it. We do this by using a runtime.Pinner. Persistent address data is added to a Context-associated runtime.Pinner; while ephemeral address data is managed by a transient runtime.Pinner that only exists for the duration of the call.

Typical call to Run()

Here is an example of the flow of operations on a simple call to Run():

• Encode input data into WAF Objects and store references in the temporary pool
• Lock the context mutex until the end of the call
• Store references from the temporary pool into the context level pool
• Call ddwaf_run
• Decode the matches and actions

CGO-less C Bindings

This library uses purego to implement C bindings without requiring use of CGO at compilation time. The high-level workflow is to embed the C shared library using go:embed, dump it into a file, open the library using dlopen, load the symbols using dlsym, and finally call them. On Linux systems, using memfd_create(2) enables the library to be loaded without writing to the filesystem.

Another requirement of libddwaf is to have a FHS filesystem on your machine and, for Linux, to provide libc.so.6, libpthread.so.0, and libdl.so.2 as dynamic libraries.

⚠ Keep in mind that purego only works on linux/darwin for amd64/arm64 and so does go-libddwaf.

Contributing pitfalls

• Cannot dlopen twice in the app lifetime on OSX. It messes with Thread Local Storage and usually finishes with a std::bad_alloc()
• keepAlive() calls are here to prevent the GC from destroying objects too early
• Since there is a stack switch between the Go code and the C code, usually the only C stacktrace you will ever get is from GDB
• If a segfault happens during a call to the C code, the goroutine stacktrace which has done the call is the one annotated with [syscall]
• GoLand does not support CGO_ENABLED=0 (as of June 2023)
• Keep in mind that we fully escape the type system. If you send the wrong data it will segfault in the best cases but not always!
• The structs in ctypes.go are here to reproduce the memory layout of the structs in include/ddwaf.h because pointers to these structs will be passed directly
• Do not use uintptr as function arguments or results types, coming from unsafe.Pointer casts of Go values, because they escape the pointer analysis which can create wrongly optimized code and crash. Pointer arithmetic is of course necessary in such a library but must be kept in the same function scope.
• GDB is available on arm64 but is not officially supported so it usually crashes pretty fast (as of June 2023)
• No pointer to variables on the stack shall be sent to the C code because Go stacks can be moved during the C call. More on this here

Debugging

Debug-logging can be enabled for underlying C/C++ library by building (or testing) by setting the DD_APPSEC_WAF_LOG_LEVEL environment variable to one of: trace, debug, info, warn (or warning), error, off (which is the default behavior and logs nothing).

The DD_APPSEC_WAF_LOG_FILTER environment variable can be set to a valid (per the regexp package) regular expression to limit logging to only messages that match the regular expression.

Documentation

Index

• Constants
• func DecodeObject(obj *WAFObject) (any, error)
• func Load() (bool, error)
• func Usable() (bool, error)
• func Version() string
• type Builder
  • func NewBuilder(keyObfuscatorRegex string, valueObfuscatorRegex string) (*Builder, error)
  • func (b *Builder) AddDefaultRecommendedRuleset() (Diagnostics, error)
  • func (b *Builder) AddOrUpdateConfig(path string, fragment any) (Diagnostics, error)
  • func (b *Builder) Build() *Handle
  • func (b *Builder) Close()
  • func (b *Builder) ConfigPaths(filter string) []string
  • func (b *Builder) RemoveConfig(path string) bool
  • func (b *Builder) RemoveDefaultRecommendedRuleset() bool
• type Context
  • func (context *Context) Close()
  • func (context *Context) Run(addressData RunAddressData) (res Result, err error)
  • func (context *Context) Truncations() map[TruncationReason][]int
• type Diagnostics
  • func (d *Diagnostics) EachFeature(cb func(string, *Feature))
  • func (d *Diagnostics) TopLevelError() error
• type Encodable
• type EncoderConfig
• type Feature
• type Handle
  • func (handle *Handle) Actions() []string
  • func (handle *Handle) Addresses() []string
  • func (handle *Handle) Close()
  • func (handle *Handle) NewContext(timerOptions ...timer.Option) (*Context, error)
• type Result
  • func (r *Result) HasActions() bool
  • func (r *Result) HasDerivatives() bool
  • func (r *Result) HasEvents() bool
• type RunAddressData
• type TruncationReason
  • func (reason TruncationReason) String() string
• type WAFObject

Constants

const (
	AppsecFieldTag            = "ddwaf"
	AppsecFieldTagValueIgnore = "ignore"
)

const (
	// EncodeTimeKey is the key used to track the time spent encoding the address data reported in [Result.TimerStats].
	EncodeTimeKey timer.Key = "encode"
	// DurationTimeKey is the key used to track the time spent in libddwaf ddwaf_run C function reported in [Result.TimerStats].
	DurationTimeKey timer.Key = "duration"
	// DecodeTimeKey is the key used to track the time spent decoding the address data reported in [Result.TimerStats].
	DecodeTimeKey timer.Key = "decode"
)

Functions

func DecodeObject

func DecodeObject(obj *WAFObject) (any, error)

Deprecated: This is merely wrapping bindings.WAFObject.AnyValue, which should be used directly instead.

func Load

func Load() (bool, error)

Load loads libddwaf's dynamic library. The dynamic library is opened only once by the first call to this function and internally stored globally. No function is currently provided in this API to unload it.

This function is automatically called by NewBuilder, and most users need not explicitly call it. It is however useful in order to explicitly check for the status of the WAF library's initialization.

The function returns true when libddwaf was successfully loaded, along with an error value. An error might still be returned even though the WAF load was successful: in such cases the error is indicative that some non-critical features are not available; but the WAF may still be used.

func Usable

func Usable() (bool, error)

Usable returns true if the WAF is usable, false and an error otherwise.

If the WAF is usable, an error value may still be returned and should be treated as a warning (it is non-blocking).

The following conditions are checked:

• The WAF library has been loaded successfully (you need to call Load first for this case to be taken into account)
• The WAF library has not been manually disabled with the `datadog.no_waf` go build tag
• The WAF library is not in an unsupported OS/Arch
• The WAF library is not in an unsupported Go version

func Version

func Version() string

Version returns the version returned by libddwaf. It relies on the dynamic loading of the library, which can fail and return an empty string or the previously loaded version, if any.

Types

type Builder

type Builder struct {
	// contains filtered or unexported fields
}

Builder manages an evolving WAF configuration over time. Its lifecycle is typically tied to that of a remote configuration client, as its purpose is to keep an up-to-date view of the current coniguration with low overhead. This type is not safe for concurrent use, and users should protect it with a mutex or similar when sharing it across multiple goroutines. All methods of this type are safe to call with a nil receiver.

func NewBuilder

func NewBuilder(keyObfuscatorRegex string, valueObfuscatorRegex string) (*Builder, error)

NewBuilder creates a new Builder instance. Its lifecycle is typically tied to that of a remote configuration client, as its purpose is to keep an up-to-date view of the current coniguration with low overhead. Returns nil if an error occurs when initializing the builder. The caller is responsible for calling Builder.Close when the builder is no longer needed.

func (*Builder) AddDefaultRecommendedRuleset

func (b *Builder) AddDefaultRecommendedRuleset() (Diagnostics, error)

AddDefaultRecommendedRuleset adds the default recommended ruleset to the receiving Builder, and returns the Diagnostics produced in the process.

func (*Builder) AddOrUpdateConfig

func (b *Builder) AddOrUpdateConfig(path string, fragment any) (Diagnostics, error)

AddOrUpdateConfig adds or updates a configuration fragment to this Builder. Returns the Diagnostics produced by adding or updating this configuration.

func (*Builder) Build

func (b *Builder) Build() *Handle

Build creates a new Handle instance that uses the current configuration. Returns nil if an error occurs when building the handle. The caller is responsible for calling Handle.Close when the handle is no longer needed. This function may return nil.

func (*Builder) Close

func (b *Builder) Close()

Close releases all resources associated with this builder.

func (*Builder) ConfigPaths

func (b *Builder) ConfigPaths(filter string) []string

ConfigPaths returns the list of currently loaded configuration paths.

func (*Builder) RemoveConfig

func (b *Builder) RemoveConfig(path string) bool

RemoveConfig removes the configuration associated with the given path from this Builder. Returns true if the removal was successful.

func (*Builder) RemoveDefaultRecommendedRuleset

func (b *Builder) RemoveDefaultRecommendedRuleset() bool

RemoveDefaultRecommendedRuleset removes the default recommended ruleset from the receiving Builder. Returns true if the removal occurred (meaning the default recommended ruleset was indeed present in the builder).

type Context

type Context struct {
	// Timer registers the time spent in the WAF and go-libddwaf. It is created alongside the Context using the options
	// passed in to NewContext. Once its time budget is exhausted, each new call to Context.Run will return a timeout error.
	Timer timer.NodeTimer
	// contains filtered or unexported fields
}

Context is a WAF execution context. It allows running the WAF incrementally when calling it multiple times to run its rules every time new addresses become available. Each request must have its own Context. New Context instances can be created by calling Handle.NewContext.

func (*Context) Close

func (context *Context) Close()

Close disposes of the underlying `ddwaf_context` and releases the associated internal data. It also decreases the reference count of the Handle which created this Context, possibly releasing it completely (if this was the last Context created from it, and it is no longer in use by its creator).

func (*Context) Run

func (context *Context) Run(addressData RunAddressData) (res Result, err error)

Run encodes the given RunAddressData values and runs them against the WAF rules. Callers must check the returned Result object even when an error is returned, as the WAF might have been able to match some rules and generate events or actions before the error was reached; especially when the error is waferrors.ErrTimeout.

func (*Context) Truncations

func (context *Context) Truncations() map[TruncationReason][]int

Truncations returns the truncations that occurred while encoding address data for WAF execution. The key is the truncation reason: either because the object was too deep, the arrays where to large or the strings were too long. The value is a slice of integers, each integer being the original size of the object that was truncated. In case of the ObjectTooDeep reason, the original size can only be approximated because of recursive objects.

type Diagnostics

type Diagnostics struct {
	// Rules contains information about the loaded rules.
	Rules *Feature
	// CustomRules contains information about the loaded custom rules.
	CustomRules *Feature
	// Actions contains information about the loaded actions.
	Actions *Feature
	// Exclusions contains information about the loaded exclusions.
	Exclusions *Feature
	// RulesOverrides contains information about the loaded rules overrides.
	RulesOverrides *Feature
	// RulesData contains information about the loaded rules data.
	RulesData *Feature
	// ExclusionData contains information about the loaded exclusion data.
	ExclusionData *Feature
	// Processors contains information about the loaded processors.
	Processors *Feature
	// Scanners contains information about the loaded scanners.
	Scanners *Feature
	// Version is the version of the parsed ruleset if available.
	Version string
}

Diagnostics stores the information as provided by the WAF about WAF rules parsing and loading. It is returned by Builder.AddOrUpdateConfig.

func (*Diagnostics) EachFeature

func (d *Diagnostics) EachFeature(cb func(string, *Feature))

EachFeature calls the provided callback for each (non-nil) feature in this diagnostics object.

func (*Diagnostics) TopLevelError

func (d *Diagnostics) TopLevelError() error

TopLevelError returns the list of top-level errors reported by the WAF on any of the Diagnostics entries, rolled up into a single error value. Returns nil if no top-level errors were reported. Individual, item-level errors might still exist.

type Encodable

type Encodable interface {
	// Encode encodes the receiver as the WAFObject obj using the provided EncoderConfig and remaining depth allowed.
	// It returns a map of truncation reasons and their respective actual sizes. If the error returned is not nil,
	// it is greatly advised to return errors from the waferrors package error when it matters.
	// Outside of encoding the value, it is expected to check for truncations sizes as advised in the EncoderConfig
	// and to regularly call the EncoderConfig.Timer.Exhausted() method to check if the encoding is still allowed
	// and return waferrors.ErrTimeout if it is not.
	// This method is not expected or required to be safe to concurrently call from multiple goroutines.
	Encode(config EncoderConfig, obj *bindings.WAFObject, depth int) (map[TruncationReason][]int, error)
}

Encodable represent a type that can encode itself into a WAFObject. The encodable is responsible for using the pin.Pinner object passed in the EncoderConfig to pin the data referenced by the encoded bindings.WAFObject. The encoder must also use the timer.Timer passed in the EncoderConfig to make sure it doesn't spend too much time doing its job. The encoder must also respect the EncoderConfig limits and report truncations.

type EncoderConfig

type EncoderConfig struct {
	// Pinner is used to pin the data referenced by the encoded wafObjects.
	Pinner pin.Pinner
	// Timer makes sure the encoder doesn't spend too much time doing its job.
	Timer timer.Timer
	// MaxContainerSize is the maximum number of elements in a container (list, map, struct) that will be encoded.
	MaxContainerSize int
	// MaxStringSize is the maximum length of a string that will be encoded.
	MaxStringSize int
	// MaxObjectDepth is the maximum depth of the object that will be encoded.
	MaxObjectDepth int
}

type Feature

type Feature struct {
	// Errors is a map of parsing errors to a list of unique identifiers from the elements which
	// failed loading due to this specific error.
	Errors map[string][]string
	// Warnings is a map of parsing warnings to a list of unique identifiers from the elements which
	// resulted in this specific warning.
	Warnings map[string][]string
	// Error is the single error which prevented parsing this feature.
	Error string
	// Loaded is a list of the unique identifiers from successfully loaded elements.
	Loaded []string
	// Failed is a list of the unique identifiers from the elements which couldn't be loaded.
	Failed []string
	// Skipped is a list of the unique identifiers from the elements which were skipped.
	Skipped []string
}

Feature stores the information as provided by the WAF about loaded and failed rules for a specific feature of the WAF ruleset.

type Handle

type Handle struct {
	// contains filtered or unexported fields
}

Handle represents an instance of the WAF for a given ruleset. It is obtained from Builder.Build; and must be disposed of by calling Handle.Close once no longer in use.

func (*Handle) Actions

func (handle *Handle) Actions() []string

Actions returns the list of actions the WAF has been configured to monitor based on the input ruleset.

func (*Handle) Addresses

func (handle *Handle) Addresses() []string

Addresses returns the list of addresses the WAF has been configured to monitor based on the input ruleset.

func (*Handle) Close

func (handle *Handle) Close()

Close decrements the reference counter of this Handle, possibly allowing it to be destroyed and all the resources associated with it to be released.

func (*Handle) NewContext

func (handle *Handle) NewContext(timerOptions ...timer.Option) (*Context, error)

NewContext returns a new WAF context for the given WAF handle. An error is returned when the WAF handle was released or when the WAF context couldn't be created.

type Result

type Result struct {
	// Events is the list of events the WAF detected, together with any relevant
	// details. These are typically forwarded as opaque objects to the Datadog
	// backend.
	Events []any

	// Derivatives is the set of key-value pairs generated by the WAF, and which
	// need to be reported on the trace to provide additional data to the Datadog
	// backend.
	Derivatives map[string]any

	// Actions is the set of actions the WAF decided on when evaluating rules
	// against the provided address data. It maps action types to their dynamic
	// parameter values.
	Actions map[string]any

	// Timer returns the time spend in the different parts of the run. Keys can be found with the suffix [
	TimerStats map[timer.Key]time.Duration

	// Keep is true if the WAF instructs the trace should be set to manual keep priority.
	Keep bool
}

Result stores the multiple values returned by a call to Context.Run.

func (*Result) HasActions

func (r *Result) HasActions() bool

HasActions return true if the Result holds at least 1 action.

func (*Result) HasDerivatives

func (r *Result) HasDerivatives() bool

HasDerivatives return true if the Result holds at least 1 derivative.

func (*Result) HasEvents

func (r *Result) HasEvents() bool

HasEvents return true if the Result holds at least 1 event.

type RunAddressData

type RunAddressData struct {
	// Persistent address data is scoped to the lifetime of a given Context, and subsquent calls to
	// Context.Run with the same address name will be silently ignored.
	Persistent map[string]any
	// Ephemeral address data is scoped to a given Context.Run call and is not persisted across
	// calls. This is used for protocols such as gRPC client/server streaming or GraphQL, where a
	// single request can incur multiple subrequests.
	Ephemeral map[string]any

	// TimerKey is the key used to track the time spent in the WAF for this run.
	// If left empty, a new timer with unlimited budget is started.
	TimerKey timer.Key
}

RunAddressData provides address data to the Context.Run method. If a given key is present in both `Persistent` and `Ephemeral`, the value from `Persistent` will take precedence. When encoding Go structs to the WAF-compatible format, fields with the `ddwaf:"ignore"` tag are ignored and will not be visible to the WAF.

type TruncationReason

type TruncationReason uint8

TruncationReason is a flag representing reasons why some input was not encoded in full.

const (
	// StringTooLong indicates a string exceeded the maximum string length configured. The truncation
	// values indicate the actual length of truncated strings.
	StringTooLong TruncationReason = 1 << iota
	// ContainerTooLarge indicates a container (list, map, struct) exceeded the maximum number of
	// elements configured. The truncation values indicate the actual number of elements in the
	// truncated container.
	ContainerTooLarge
	// ObjectTooDeep indicates an overall object exceeded the maximum encoding depths configured. The
	// truncation values indicate an estimated actual depth of the truncated object. The value is
	// guaranteed to be less than or equal to the actual depth (it may not be more).
	ObjectTooDeep
)

func (TruncationReason) String

func (reason TruncationReason) String() string

type WAFObject

type WAFObject = bindings.WAFObject

WAFObject is the C struct that represents a WAF object. It is passed as-is to the C-world. It is highly advised to use the methods on the object to manipulate it and not set the fields manually.