firehose-to-syslog

command module
Version: v0.0.0-...-04a75a9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 23, 2015 License: MIT Imports: 11 Imported by: 0

README

Firehose-to-syslog

This nifty util aggregates all the events from the firehose feature in CloudFoundry.

./firehose-to-syslog \
          --api-address="https://api.10.244.0.34.xip.io" \
          --skip-ssl-validation \
          --debug
....
....
{"cf_app_id":"c5cb762b-b7bb-44b6-97d1-2b612d4baba9","cf_app_name":"lattice","cf_org_id":"fb5777e6-e234-4832-8844-773114b505b0","cf_org_name":"GWENN","cf_origin":"firehose","cf_space_id":"3c910823-22e7-41ff-98de-094759594398","cf_space_name":"GWENN-SPACE","event_type":"LogMessage","level":"info","message_type":"OUT","msg":"Lattice-app. Says Hello. on index: 0","origin":"rep","source_instance":"0","source_type":"APP","time":"2015-06-12T11:46:11+09:00","timestamp":1434077171244715915}

Options

usage: firehose-to-syslog --api-endpoint=API-ENDPOINT [<flags>]

Flags:
  --help              Show help (also see --help-long and --help-man).
  --debug             Enable debug mode. This disables forwarding to syslog
  --api-endpoint=API-ENDPOINT  
                      Api endpoint address. For bosh-lite installation of CF: https://api.10.244.0.34.xip.io
  --doppler-endpoint=DOPPLER-ENDPOINT  
                      Overwrite default doppler endpoint return by /v2/info
  --syslog-server=SYSLOG-SERVER  
                      Syslog server.
  --subscription-id="firehose"  
                      Id for the subscription.
  --user="admin"      Admin user.
  --password="admin"  Admin password.
  --skip-ssl-validation  
                      Please don't
  --events="LogMessage"  
                      Comma seperated list of events you would like. Valid options are LogMessage, ValueMetric, CounterEvent, Error, ContainerMetric, Heartbeat, HttpStart,
                      HttpStop, HttpStartStop
  --boltdb-path="my.db"  
                      Bolt Database path
  --cc-pull-time=60s  CloudController Pooling time in sec
  --version           Show application version.

#Endpoint definition

We use gocf-client which will call the CF endpoint /v2/info to get Auth., doppler endpoint.

But for doppler endpoint you can overwrite it with --doppler-address as we know some people use different endpoint.

Event documentation

See the dropsonde protocol documentation for details on what data is sent as part of each event.

Caching

We use boltdb for caching application name, org and space name.

We have 3 caching strategies:

  • Pull all application data on start.
  • Pull application data if not cached yet.
  • Pull all application data every "cc-pull-time".

To test and build

# Setup repo
go get github.com/cloudfoundry-community/firehose-to-syslog
cd $GOPATH/src/github.com/cloudfoundry-community/firehose-to-syslog

# Test
ginkgo -r .

# Build binary
godep go build

Deploy with Bosh

logsearch-for-cloudfoundry

Run agains a bosh-lite CF deployment

godep go run main.go \
	--debug \
	--skip-ssl-validation \
	--api-address="https://api.10.244.0.34.xip.io"

Parsing the logs with Logstash

logsearch-for-cloudfoundry

Devel

This is a Git Flow project. Please fork and branch your features from develop.

Contributors

  • Ed King - Added support to skip ssl validation.
  • Mark Alston - Added support for more events and general code cleaup.
  • Etourneau Gwenn - Added validation of selected events and general code cleanup, caching system..

Documentation

The Go Gopher

There is no documentation for this package.

Source Files

Directories

Path Synopsis
Godeps
_workspace/src/github.com/alecthomas/template
Package template implements data-driven templates for generating textual output.
Package template implements data-driven templates for generating textual output.
_workspace/src/github.com/alecthomas/template/parse
Package parse builds parse trees for templates as defined by text/template and html/template.
Package parse builds parse trees for templates as defined by text/template and html/template.
_workspace/src/github.com/alecthomas/units
Package units provides helpful unit multipliers and functions for Go.
Package units provides helpful unit multipliers and functions for Go.
_workspace/src/github.com/boltdb/bolt
Package bolt implements a low-level key/value store in pure Go.
Package bolt implements a low-level key/value store in pure Go.
_workspace/src/github.com/cloudfoundry/noaa/events
Package events is a generated protocol buffer package.
Package events is a generated protocol buffer package.
_workspace/src/github.com/gogo/protobuf/proto
Package proto converts data structures to and from the wire format of protocol buffers.
Package proto converts data structures to and from the wire format of protocol buffers.
_workspace/src/github.com/gorilla/websocket
Package websocket implements the WebSocket protocol defined in RFC 6455.
Package websocket implements the WebSocket protocol defined in RFC 6455.
_workspace/src/github.com/gorilla/websocket/examples/autobahn
Command server is a test server for the Autobahn WebSockets Test Suite.
Command server is a test server for the Autobahn WebSockets Test Suite.
_workspace/src/golang.org/x/net/context
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
Package context defines the Context type, which carries deadlines, cancelation signals, and other request-scoped values across API boundaries and between processes.
_workspace/src/golang.org/x/oauth2
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
Package oauth2 provides support for making OAuth2 authorized and authenticated HTTP requests.
_workspace/src/golang.org/x/oauth2/clientcredentials
Package clientcredentials implements the OAuth2.0 "client credentials" token flow, also known as the "two-legged OAuth 2.0".
Package clientcredentials implements the OAuth2.0 "client credentials" token flow, also known as the "two-legged OAuth 2.0".
_workspace/src/golang.org/x/oauth2/facebook
Package facebook provides constants for using OAuth2 to access Facebook.
Package facebook provides constants for using OAuth2 to access Facebook.
_workspace/src/golang.org/x/oauth2/github
Package github provides constants for using OAuth2 to access Github.
Package github provides constants for using OAuth2 to access Github.
_workspace/src/golang.org/x/oauth2/google
Package google provides support for making OAuth2 authorized and authenticated HTTP requests to Google APIs.
Package google provides support for making OAuth2 authorized and authenticated HTTP requests to Google APIs.
_workspace/src/golang.org/x/oauth2/jws
Package jws provides encoding and decoding utilities for signed JWS messages.
Package jws provides encoding and decoding utilities for signed JWS messages.
_workspace/src/golang.org/x/oauth2/jwt
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
Package jwt implements the OAuth 2.0 JSON Web Token flow, commonly known as "two-legged OAuth 2.0".
_workspace/src/golang.org/x/oauth2/linkedin
Package linkedin provides constants for using OAuth2 to access LinkedIn.
Package linkedin provides constants for using OAuth2 to access LinkedIn.
_workspace/src/golang.org/x/oauth2/odnoklassniki
Package odnoklassniki provides constants for using OAuth2 to access Odnoklassniki.
Package odnoklassniki provides constants for using OAuth2 to access Odnoklassniki.
_workspace/src/golang.org/x/oauth2/paypal
Package paypal provides constants for using OAuth2 to access PayPal.
Package paypal provides constants for using OAuth2 to access PayPal.
_workspace/src/golang.org/x/oauth2/vk
Package vk provides constants for using OAuth2 to access VK.com.
Package vk provides constants for using OAuth2 to access VK.com.
_workspace/src/gopkg.in/alecthomas/kingpin.v2
Package kingpin provides command line interfaces like this: $ chat usage: chat [<flags>] <command> [<flags>] [<args> ...] Flags: --debug enable debug mode --help Show help.
Package kingpin provides command line interfaces like this: $ chat usage: chat [<flags>] <command> [<flags>] [<args> ...] Flags: --debug enable debug mode --help Show help.
_workspace/src/gopkg.in/alecthomas/kingpin.v2/examples/curl
A curl-like HTTP command-line client.
A curl-like HTTP command-line client.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL