uspclient

package module
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 8, 2024 License: Apache-2.0 Imports: 16 Imported by: 41

README

go-uspclient

Golang USP Client

Protocol

USP is based on a secure websocket connection to the relevant LimaCharlie datacenter and is formatted in JSON.

Establish Connection

Upon connecting, the client will issue a connection handshake to the server containing identifying information.

ConnectionHeader Format

  • OID: Organization ID this client belongs to.
  • IID: Installation Key ID authorizing this client.
  • Hostname: Hostname of the client.
  • Platform: name of the Platform sensors this client will create.
  • Architecture: name of the Architecture sensors this client will create.
  • Mapping: metadata indicating how to map fields coming from this client into JSON in LimaCharlie (see below).
  • Mappings: a list of Mapping (as defined above) that will be attempted, the first one to match on the ParsingRE field will be used.
  • SensorSeedKey: an arbitrary string used in generating the sensor IDs this client will create (see below).
  • IsCompressed: whether the cloud can expect the content shipped to it is compressed.
  • DataFormat: the rest of the connection can support msgpack or json.
  • InstanceID: an identifier unique to the instance of the adapter, used an indication the state of the adapter was reset.

Mapping Metadata

  • ParsingRE: regular expression with named capture groups. The name of each group will be used as the key in the converted JSON parsing.
  • SensorKeyPath: indicates which component of the events represent unique sensor identifiers.
  • SensorHostnamePath: indicates which component of the event represents the hostname of the resulting Sensor in LimaCharlie.
  • EventTypePath: indicates which component of the event represents the Event Type of the resulting event in LimaCharlie.
  • EventTimePath: indicates which component of the event represents the Event Time of the resulting event in LimaCharlie.
  • IsRenameOnly: if true, indicates the field mappings defined here should only be renamed fields from the original event (and not completely replacing the original event).
  • Mappings: a list of field remapping to be performed:
    • SourceField: the source component to remap.
    • DestinationField: what the SourceField should be mapped to in the final event.

References to field or path in the Mapping represent slash-separated (/) elements of a string indicating where a recursive JSON object a given value is. For example, the EventTypePath equal to metadata/original/action applied to the following event would result in foo being used as an Event Type:

{
    "metadata": {
        "original": {
            "action": "foo",
            "bar": "foobar"
        },
        "another": {
            "random": "field"
        }
    }
}

Indexing Metadata

  • EventsIncluded: an optional inclusion list of event types this index applies to.
  • EventsExcluded: an optional exclusion list of event types this index does not apply to.
  • Path: the path within the event who's value is to be indexed.
  • Regexp: the regular expression, with a single capture group, to extract the value to index from the Path.
  • IndexType: the type of index this should apply to.
Authorization

Clients are authorized to connect based on the Organization ID (OID) and Installation Key ID (IID). Deleting the IID will prevent the client from connecting with it.

Sensor IDs

USP Clients generate LimaCharlie Sensors at runtime. The ID of those sensors (SID) is generated based on the Organization ID (OID) and the Sensor Seed Key.

This implies that if want to re-key an IID (perhaps it was leaked), you may replace the IID with a new valid one. As long as you use the same OID and Sensor Seed Key, the generated SIDs will be stable despite the IID change.

Send Data

Once the handshake has been sent, the client can send events.

DataMessage

Receive Feedback

The server may then send control messages to the client.

ControlMessage

The Verb element of a ControlMessage indicates the type of feedback and content that can be expected in the rest of the message.

Flow Control

Starting with version 2 and up, USP supports a TCP-like flow control concept. The new ControlMessage verb of ControlMessageFLOW (fl) with a component WindowSize (win) which is an indicator from the LimaCharlie backend that a larger Window Size (max number of unacknowledged messages in transit) should be set in the client. This is a way for the LimaCharlie backend to bring up to speed a given client in a controlled way (without flooding the LimaCharlie backend and requiring aggressive backoff). In version 2, the client should begin execution with a max window size of 1 and wait for feedback from LimaCharlie for when to increase it and by how much to increase it.

Acknowledging DataMessage

DataMessage is expected to contain a Sequence Number monotonically increasing. A ControlMessage with a Verb value of ControlMessageACK is used by the server to indicate all DataMessages with a Sequence Number equal or lower has been received and processed.

Custom Formatting

Data sent via USP can be formatted in many different ways. Data is processed in a specific order as a pipeline:

  1. Regular Expression with named capture groups parsing a string into a JSON object.
  2. Built-in (in the cloud) LimaCharlie parsers that apply to specific Platform values (like carbon_black).
  3. The various "extractors" defined, like EventTypePath, EventTimePath, SensorHostnamePath and SensorKeyPath.
  4. Custom Mappings directives provided by the client.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var ErrorBufferFull = errors.New("buffer full")

Functions

This section is empty.

Types

type AckBuffer

type AckBuffer struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

func NewAckBuffer

func NewAckBuffer(o AckBufferOptions) (*AckBuffer, error)

func (*AckBuffer) Ack

func (b *AckBuffer) Ack(seq uint64) error

func (*AckBuffer) Add

func (b *AckBuffer) Add(e *protocol.DataMessage, timeout time.Duration) bool

func (*AckBuffer) Close

func (b *AckBuffer) Close()

func (*AckBuffer) GetCurrentCapacity

func (b *AckBuffer) GetCurrentCapacity() uint64

func (*AckBuffer) GetEmptyEvent

func (b *AckBuffer) GetEmptyEvent() *Event

func (*AckBuffer) GetNextToDeliver

func (b *AckBuffer) GetNextToDeliver(timeout time.Duration) *protocol.DataMessage

func (*AckBuffer) GetUnAcked

func (b *AckBuffer) GetUnAcked() ([]*protocol.DataMessage, error)

func (*AckBuffer) ResetDelivery

func (b *AckBuffer) ResetDelivery()

func (*AckBuffer) UpdateCapacity

func (b *AckBuffer) UpdateCapacity(newCapacity uint64)

type AckBufferOptions

type AckBufferOptions struct {
	OnBackPressure func() `json:"-" yaml:"-"`
	OnAck          func() `json:"-" yaml:"-"`
}

type Client

type Client struct {
	// contains filtered or unexported fields
}

func NewClient

func NewClient(o ClientOptions) (*Client, error)

func (*Client) Close

func (c *Client) Close() ([]*protocol.DataMessage, error)

func (*Client) Drain

func (c *Client) Drain(timeout time.Duration) error

func (*Client) GetLastError

func (c *Client) GetLastError() error

func (*Client) GetUnsent

func (c *Client) GetUnsent() ([]*protocol.DataMessage, error)

func (*Client) Reconnect

func (c *Client) Reconnect()

func (*Client) Ship

func (c *Client) Ship(message *protocol.DataMessage, timeout time.Duration) error

type ClientOptions

type ClientOptions struct {
	Identity      Identity                     `json:"identity" yaml:"identity"`
	Hostname      string                       `json:"hostname,omitempty" yaml:"hostname,omitempty"`
	Platform      string                       `json:"platform,omitempty" yaml:"platform,omitempty"`
	Architecture  string                       `json:"architecture,omitempty" yaml:"architecture,omitempty"`
	Mapping       protocol.MappingDescriptor   `json:"mapping,omitempty" yaml:"mapping,omitempty"`
	Mappings      []protocol.MappingDescriptor `json:"mappings,omitempty" yaml:"mappings,omitempty"`
	Indexing      []protocol.IndexDescriptor   `json:"indexing,omitempty" yaml:"indexing,omitempty"`
	BufferOptions AckBufferOptions             `json:"buffer_options,omitempty" yaml:"buffer_options,omitempty"`
	IsCompressed  bool                         `json:"is_compressed,omitempty" yaml:"is_compressed,omitempty"`

	SensorSeedKey string `json:"sensor_seed_key" yaml:"sensor_seed_key"`

	DebugLog  func(string) `json:"-" yaml:"-"`
	OnError   func(error)  `json:"-" yaml:"-"`
	OnWarning func(string) `json:"-" yaml:"-"`

	// Auto-detect if not specified (preferred).
	DestURL string        `json:"dest_url,omitempty" yaml:"dest_url,omitempty"`
	GenURL  func() string `json:"-" yaml:"-"`
}

func (ClientOptions) Validate

func (o ClientOptions) Validate() error

type Event

type Event struct {
	// contains filtered or unexported fields
}

func NewEvent

func NewEvent() *Event

func (*Event) Clear

func (e *Event) Clear()

func (*Event) IsSet

func (e *Event) IsSet() bool

func (*Event) Set

func (e *Event) Set()

func (*Event) Wait

func (e *Event) Wait()

func (*Event) WaitFor

func (e *Event) WaitFor(d time.Duration) bool

type Identity

type Identity struct {
	Oid             string `json:"oid" yaml:"oid"`
	InstallationKey string `json:"installation_key" yaml:"installation_key"`
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL