noisy-neighbor-nozzle

README

Noisy Neighbor Nozzle

The Noisy Neighbor Nozzle is a Loggregator Firehose nozzle and CLI Tool to help Operators identify applications producing a large amount of logs - i.e. "noise".

Getting Started & Authentication

In order to properly deploy the nozzle components you'll need to create a UAA client with the apropriate permisions. This can be done by adding the following client to your deoployment manifest and updating your deployment.

noisy-neighbor-nozzle:
  authorities: doppler.firehose,uaa.resource,cloud_controller.admin_read_only
  authorized-grant-types: client_credentials,refresh_token
  override: true
  scope: doppler.firehose
  secret: <secret>

Deploying

The easiest way to deploy is to use the deployer binary for your local OS included in our release package. If you add the flag --interactive you will be propted for all the information required to deploy the nozzle components. If you are interested in operationalizing the deployment experience you should use the Noisy Neighbor Nozzle Release.

Using the CLI

The easisest way to quickly check your platform for top log producers is to use the CLI tool. Download and install the binary for your local os using the command cf install-plugin.

To see the top 10 log producers in the last minute run:

cf log-noise

Integrating with the Noisy Neigbor Nozzle

The datadog-reporter is an optional component used for integrating with datadog. When deployed, it will request rates from the accumulator every minute and report the top 50 noisiest applications to Datadog

How it works

The nozzle will read logs (excluding router logs by default) from the Loggregator firehose keeping counts for the number of logs received for each application. The nozzle stores the last 60 minutes worth of this data in an in-memory cache.

The accumulator acts as a proxy for all the nozzles. When the accumulator receives an HTTP request it will forward the same request to all the nozzles. The accumulator then takes to rates from all the nozzles and sums them together, responding with the total rates.

Scaling

The nozzle can be scaled horizontally. We recommend having the same number of nozzles as you have Loggregator Traffic Controllers.

The accumulator and datadog-reporter should only be deployed with a single instance.

Accumulator API

GET /rates/{timestamp}

Headers

• Authorization - OAuth2 token, must have doppler.firehose scope.

Parameters

• timestamp - Unix timetamp truncated (to the minute) to the nozzles POLLING_INTERVAL.

Query Parameter

• truncate_timestamp - Optional query parameter to truncate the given timestamp to the by the configured RATE_INTERVAL (Default is 1 minute). If true timestamp will be truncated, otherwise it will not be modified.

Example

curl -H "Authorization: $AUTH_TOKEN" https://nn-accumulator.<app-domain>/rates/$(python -c 'import time; n=time.time(); print(int(n-n%60)-(5*60))')
{
    "timestamp": 1514042640,
    "counts": {
        "06d83ae4-7632-46b9-af96-5f90f56ba0c5/0": 6456,
        "0dbb1e16-9da6-4a31-b8b3-fdff5258e20b/0": 129,
        "14213570-140d-41df-9a4e-481f7e010a08/0": 7,
        ...
    }
}