collector

package module
v0.5.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 20, 2015 License: Apache-2.0 Imports: 21 Imported by: 2

README

Banyan Collector: A framework to peek inside containers

Have you wondered what your container images really contain? If they have the very packages that are susceptible to all kinds of attacks? Or, if they have the configuration you expect when they are run? Banyan Collector provides a powerful, extensible framework to answer all these questions and more.

Banyan Collector is a light-weight, easy to use, and modular system that allows you to launch containers from a registry, run arbitrary scripts inside them, and gather useful information. This framework can be used to statically analyze images for several purposes including:

  • Collect specific information from all images (e.g., packages installed)
  • Enforce policies (e.g., no unauthorized user accounts, etc.)
  • Validate invariants (e.g., nginx.conf is present in the right directory, etc.)
  • and so on...

Getting started

The collector can be run in one of two modes: (a) as a standalone executable (b) in a container. The main requirement is to run the collector on a Docker Host (machine that has the Docker Daemon running). If you want to collect data from a private registry make sure you are logged into it (sudo docker login REGISTRY)

(a) To run it as a standalone executable, you need go in your environment (https://golang.org/doc/install). Once go is installed, just run the following on a Docker Host:

$ go get -u github.com/banyanops/collector/...
$ cd <COLLECTOR_SOURCE_DIR>; sudo COLLECTOR_DIR=$PWD $GOPATH/bin/collector <REGISTRY> <REPO>

where REGISTRY is either a private registry (e.g., http://reg.myorg.com) or Docker Hub (index.docker.io), and REPO is a repository for which you'd like to collect data. For a private registry (with search enabled), if no REPO is specified, data is collected from all the repositories.

(b) To run the collector in a container, please follow instructions on Docker Hub.

More generally, collector can be configured using several options (e.g., registry poll interval, remove images threshold, secure registry settings, etc.):

$ sudo collector [options] REGISTRY [REPO1 REPO2 ...]

For a list of all the options run:

$ collector -h

Why not just use shell scripts?

Shell scripts are great for quickly getting specific information from images. However, as we add more complexity, it becomes hard to write scripts that are easy to maintain, quickly extensible and portable to different environments.

For example, the complexity in managing registry connections, keeping track of repo/tag changes, cleaning up stale images, supporting arbitrary policies, etc. is too much to handle using simple scripts. Furthermore, packaging the collector in a container provides a portable framework that is not dependent on any particular host configuration, and can seamlessly run anywhere.

Tests

The go tests rely on write access to the Docker UNIX socket /var/run/docker.sock. One approach is to add the user to the "docker" group to enable write access to the socket. Alternatively, the tests can be run as "root", for example using "sudo", but this requires the root user to share the Go development environment ($GOPATH, etc.).

Another requirement to run the tests is to set environment variables $DOCKER_USER and $DOCKER_PASSWORD to the user's Docker Hub login credentials. Additionally, the tests emit a warning if $COLLECTOR_DIR is not set (but the warning can be safely ignored).

Once the environment has been correctly setup, go tests can be run using the standard command:

$ go test

More information

More details about Collector operation/architecture, etc. are available under docs.

For further details about how one might use this in an enterprise, please check out Banyan. This SAAS service offers deeper analysis of your data and provides a dashboard showing which of your images are compliant to your policies (e.g., which of your images have security vulnerabilities, etc.) along with real-time updates and email notifications.

Get involved

Collector is under active development. Fork the project and submit pull requests, or file issues or tweet us @banyanops.

License

Banyan Collector is distributed under Apache 2.0 License. More details in LICENSE.

Documentation

Overview

distro.go has functions for identifying Linux distribution type and version. Currently handles some CentOS, Ubuntu, and Debian versions.

imagedata.go has funcitons that deal with the contents of images, including Linux distribution identification and application package names, versions, and architectures.

metadata.go has functions to gather and save metadata about a Docker image, including its ID, Author, Parent, Creation time, etc.

registryauth.go has functions for Docker registry authentication and Docker Hub authentication and indexing.

Index

Constants

View Source 
const (

	// HTTPTIMEOUT is the time to wait for an HTTP request to complete before giving up.
	HTTPTIMEOUT = 32 * time.Second
	// TARGETCONTAINERDIR is the path in the target container where the exported binaries and scripts are located.
	TARGETCONTAINERDIR = "/banyancollector"
)
View Source 
const (
	PKGEXTRACTSCRIPT = "pkgextractscript.sh"
)

Variables

View Source 
var (
	// DockerTransport points to the http transport used to connect to the docker unix socket
	DockerTransport *http.Transport
	DockerTLSVerify = true
	DockerProto     = "unix"
	DockerAddr      = dummydomain
)
View Source 
var (
	ReposToProcess = make(map[RepoType]bool)
	ExcludeRepo    = func() map[RepoType]bool {
		excludeList := []RepoType{}
		m := make(map[RepoType]bool)
		for _, r := range excludeList {
			m[r] = true
		}
		return m
	}()
)
View Source 
var (
	// HubAPI indicates whether to use the Docker Hub API.
	HubAPI        bool
	HTTPSRegistry = flag.Bool([]string{"-registryhttps"}, true,
		"Set to false if registry does not need HTTPS (SSL/TLS)")
	AuthRegistry = flag.Bool([]string{"-registryauth"}, true,
		"Set to false if registry does not need authentication")
	RegistryProto = flag.String([]string{"-registryproto"}, "v1",
		"Select the registry protocol to use: v1, v2, quay")
	// registryspec is the host.domainname of the registry
	RegistrySpec string
	// registryAPIURL is the http(s)://[user:password@]host.domainname of the registry
	RegistryAPIURL string
	// XRegistryAuth is the base64-encoded AuthConfig object (for X-Registry-Auth HTTP request header)
	XRegistryAuth string
	// BasicAuth is the base64-encoded Auth field read from $HOME/.dockercfg
	BasicAuth string
	// DockerConfig is the name of the config file containing registry authentication information.
	DockerConfig string
)
View Source 
var (
	//userScriptsDir    = flag.String([]string{"userscriptsdir"}, config.BANYANDIR()+"/hosttarget/userscripts", "Directory with all user-specified scripts")
	UserScriptStore   = flag.String([]string{"u", "-userscriptstore"}, config.COLLECTORDIR()+"/data/userscripts", "Directory with all user-specified scripts")
	UserScriptsDir    = config.BANYANDIR() + "/hosttarget/userscripts"
	DefaultScriptsDir = config.BANYANDIR() + "/hosttarget/defaultscripts"
	BinDir            = config.BANYANDIR() + "/hosttarget/bin"
)
View Source 
var DistroMap = map[string]string{
	"Ubuntu 15.10": "UBUNTU-wily",
	"Ubuntu 15.04": "UBUNTU-vivid",
	"Ubuntu 14.10": "UBUNTU-utopic",
	"Ubuntu Utopic Unicorn (development branch)": "UBUNTU-utopic",
	"Ubuntu 14.04.2 LTS":                         "UBUNTU-trusty",
	"Ubuntu 14.04.1 LTS":                         "UBUNTU-trusty",
	"Ubuntu 14.04 LTS":                           "UBUNTU-trusty",
	"Ubuntu 12.04 LTS":                           "UBUNTU-precise",
	"Ubuntu precise (12.04.5 LTS)":               "UBUNTU-precise",
	"Ubuntu precise (12.04.4 LTS)":               "UBUNTU-precise",
	"Ubuntu precise (12.04.3 LTS)":               "UBUNTU-precise",
	"Ubuntu precise (12.04.2 LTS)":               "UBUNTU-precise",
	"Ubuntu precise (12.04.1 LTS)":               "UBUNTU-precise",
	"Ubuntu 10.04.1 LTS":                         "UBUNTU-lucid",
	"Ubuntu 10.04.2 LTS":                         "UBUNTU-lucid",
	"Ubuntu 10.04.3 LTS":                         "UBUNTU-lucid",
	"Ubuntu 10.04.4 LTS":                         "UBUNTU-lucid",
	"Ubuntu 13.10":                               "UBUNTU-saucy",
	"Ubuntu 13.04":                               "UBUNTU-raring",
	"Ubuntu 12.10":                               "UBUNTU-quantal",
	"Ubuntu 11.10":                               "UBUNTU-oneiric",
	"Ubuntu 11.04":                               "UBUNTU-natty",
	"Ubuntu 10.10":                               "UBUNTU-maverick",
	"Ubuntu 10.04":                               "UBUNTU-lucid",
	"CentOS Linux 7 (Core)":                      "REDHAT-7Server",
	"Debian GNU/Linux 7 (wheezy)":                "DEBIAN-wheezy",
	"Debian 6.0.10":                              "DEBIAN-squeeze",
	"Debian GNU/Linux 8 (jessie)":                "DEBIAN-jessie",
}

DistroMap is a reference that maps each pretty name to the corresponding distribution name.

View Source 
var (
	WriterList []Writer
)

Functions

func CopyDir 

func CopyDir(srcDir, destDir string)

copyDir copies all files from srcDir to destDir

func CopyDirTree 

func CopyDirTree(srcDir, destDir string)

CopyDirTree copies all files from srcDir to destDir

func CreateDirIfNotExist 

func CreateDirIfNotExist(dir string) (err error)

func GetImageAllData 

func GetImageAllData(pulledImages ImageSet) (outMapMap map[string]map[string]interface{})

GetImageAllData extracts content info from each pulled image. Currently it gets system package info.

func GetImageMetadata 

func GetImageMetadata(oldMetadataSet MetadataSet) (tagSlice []TagInfo, metadataSlice []ImageMetadataInfo)

GetImageMetadata returns repository/tag/image metadata queried from a Docker registry. If the user has specified the repositories to examine, then no other repositories are examined. If the user has not specified repositories, then the registry search API is used to get the list of all repositories in the registry.

func GetImageMetadataHub 

func GetImageMetadataHub(oldMetadataSet MetadataSet) (tagSlice []TagInfo, metadataSlice []ImageMetadataInfo)

GetImageMetadataHub returns repositories/tags/image metadata from the Docker Hub. The user must have specified a set of repositories of interest. The function queries Docker Hub as an index to the registries, and then retrieves information directly from the registries, using Docker Hub authentication tokens.

func GetImageToMDMap 

func GetImageToMDMap(imageMDs []ImageMetadataInfo) (imageToMDMap map[string][]ImageMetadataInfo)

GetImageToMDMap takes image metadata structs and produces a map of imageID to metadata struct.

func GetNewImageMetadata 

func GetNewImageMetadata(oldMetadataSet MetadataSet) (tagSlice []TagInfo,
	metadataSlice []ImageMetadataInfo, currentMetadataSet MetadataSet)

GetNewImageMetadata takes the set of existing images, queries the registry to find any changes, and then brings the Output Writer up to date by telling it the obsolete metadata to delete and the new metadata to add.

func GetRegistryURL 

func GetRegistryURL() (URL string, hubAPI bool, BasicAuth string, XRegistryAuth string)

GetRegistryURL determines the full URL, with or without HTTP Basic Auth, needed to access the registry or Docker Hub.

func NewDockerTransport 

func NewDockerTransport(proto, addr string) (tr *http.Transport, e error)

NewDockerTransport creates an HTTP transport to the Docker unix/tcp socket.

func NewTLSTransport 

func NewTLSTransport(hostpath string, certfile, cafile, keyfile string) (transport *http.Transport, err error)

func PullImage 

func PullImage(metadata ImageMetadataInfo)

PullImage performs a docker pull on an image specified by repo/tag. TODO: Detect if the pulled image has a different imageID than the value retrieved from metadata, and if so correct the metadata, or at least skip processing the image.

func RegAuth 

func RegAuth(registry string) (basicAuth, fullRegistry, authConfig string)

RegAuth takes as input the name of a registry, and it parses the contents of $HOME/.dockercfg or $HOME/.docker/config.json to return the user authentication info and registry URL. TODO: Change this to return authConfig instead of user&password, and then use X-Registry-Auth in the HTTP request header.

func RegistryQuery 

func RegistryQuery(client *http.Client, URL, basicAuth string) (response []byte, e error)

RegistryQuery performs an HTTP GET operation from the registry and returns the response.

func RegistryRequestWithToken 

func RegistryRequestWithToken(client *http.Client, URL string, basicAuth string, dockerToken string) (response []byte, e error)

RegistryRequestWithToken queries a Docker Registry that accepts Authorization: Token headers with token values obtained from a Docker Index, e.g., Docker Hub.

func RemoveImages 

func RemoveImages(PulledImages []ImageMetadataInfo, imageToMDMap map[string][]ImageMetadataInfo)

RemoveImages removes least recently pulled docker images from the local docker host.

func RemoveObsoleteMetadata 

func RemoveObsoleteMetadata(obsolete []ImageMetadataInfo)

RemoveObsoleteMetadata removes obsolete metadata from the Banyan service.

func SaveImageAllData 

func SaveImageAllData(outMapMap map[string]map[string]interface{})

SaveImageAllData saves output of all the scripts.

func SaveImageMetadata 

func SaveImageMetadata(metadataSlice []ImageMetadataInfo)

SaveImageMetadata saves image metadata to selected storage location (standard output, Banyan service, etc.).

func ValidRepoName 

func ValidRepoName(name string) bool

ValidRepoName verifies that the name of a repo is in a legal format.

Types

type AuthConfig 

type AuthConfig struct {
	Username      string `json:"username,omitempty"`
	Password      string `json:"password,omitempty"`
	Auth          string `json:"auth"`
	Email         string `json:"email"`
	ServerAddress string `json:"serveraddress,omitempty"`
}

AuthConfig is a Registry auth info type copied from docker package cliconfig config.go and needed to generate the Authorization header for the Docker Remote API.

type ByDateTime 

type ByDateTime []ImageMetadataInfo

ByDateTime is used to sort ImageMetadataInfo slices by image age from newest to oldest.

func (ByDateTime) Len 

func (a ByDateTime) Len() int

func (ByDateTime) Less 

func (a ByDateTime) Less(i int, j int) bool

func (ByDateTime) Swap 

func (a ByDateTime) Swap(i int, j int)

type Container 

type Container struct {
	User         string
	AttachStdin  bool
	AttachStdout bool
	AttachStderr bool
	Tty          bool
	Env          []string
	Cmd          []string
	Entrypoint   []string
	Image        string
	WorkingDir   string
	HostConfig   HostConfig
}

type DockerAuth 

type DockerAuth struct {
	Auth  string
	Email string
}

type DockerAuthSet 

type DockerAuthSet map[string]DockerAuth

DockerAuthSet contains authentication info parsed from $HOME/.dockercfg or $HOME/.docker/config.json

type DockerConfigJSON added in v0.5.2 

type DockerConfigJSON struct {
	Auths DockerAuthSet
}

DockerConfigJSON is used to decode $HOME/.docker/config.json

type FileWriter 

type FileWriter struct {
	// contains filtered or unexported fields
}

func (*FileWriter) AppendImageMetadata 

func (f *FileWriter) AppendImageMetadata(imageMetadata []ImageMetadataInfo)

AppendImageMetadata appends image metadata to file

func (*FileWriter) RemoveImageMetadata 

func (f *FileWriter) RemoveImageMetadata(imageMetadata []ImageMetadataInfo)

RemoveImageMetadata removes image metadata from file

func (*FileWriter) WriteImageAllData 

func (f *FileWriter) WriteImageAllData(outMapMap map[string]map[string]interface{})

WriteImageAllData writes image (pkg and other) data into file

type HTTPStatusCodeError 

type HTTPStatusCodeError struct {
	StatusCode int
	// contains filtered or unexported fields
}

func (*HTTPStatusCodeError) Error 

func (s *HTTPStatusCodeError) Error() string

type HostConfig 

type HostConfig struct {
	Binds       []string
	Links       []string
	Privileged  bool
	VolumesFrom []string
}

type HubInfo 

type HubInfo struct {
	Repo        RepoType
	DockerToken string
	RegistryURL string
}

HubInfo records the index and auth information provided by Docker Hub to access a repository.

type HubInfoMap 

type HubInfoMap map[RepoType]HubInfo

HubInfoMap maps repository name to the corresponding Docker Hub auth/index info.

func NewHubInfoMap 

func NewHubInfoMap() HubInfoMap

NewHubInfoMap is a constructor for HubInfoMap.

type ImageDataInfo 

type ImageDataInfo struct {
	Image        string //this has to be the first field (used in order by)
	DistroName   string //e.g., ubuntu 14.04.02 Trusty....
	DistroID     string //e.g., Trusty
	Pkg          string
	Version      string
	Architecture string
}

ImageDataInfo describes a package included in the contents of an image.

type ImageIDType 

type ImageIDType string

ImageIDType represents docker image IDs.

type ImageMetadataAndAction 

type ImageMetadataAndAction struct {
	Action        string
	ImageMetadata []ImageMetadataInfo
}

type ImageMetadataInfo 

type ImageMetadataInfo struct {
	Image    string    //this has to be the first field (used in order by)
	Datetime time.Time //created at
	Repo     string
	Tag      string
	Size     uint64
	Author   string
	Checksum string
	Comment  string
	Parent   string
}

ImageMetadataInfo records basic information about an image.

type ImageSet 

type ImageSet map[ImageIDType]bool

ImageSet is a set of image IDs.

func NewImageSet 

func NewImageSet() ImageSet

NewImageSet creates a new ImageSet.

type ImageStruct 

type ImageStruct struct {
	ID       string
	Parent   string
	Checksum string
	Created  string
	// Container string
	Author  string
	Size    uint64
	Comment string
}

ImageStruct records information returned by the registry to describe an image. This information gets copied to an object of type ImageMetadataInfo.

type ImageToMetadataMap 

type ImageToMetadataMap map[ImageIDType]ImageMetadataInfo

ImageToMetadataMap maps image IDs to ImageMetadataInfo structs.

func NewImageToMetadataMap 

func NewImageToMetadataMap() ImageToMetadataMap

NewImageToMetadataMap is a constructor for ImageToMetadataMap.

func (ImageToMetadataMap) Exists 

func (m ImageToMetadataMap) Exists(imageID ImageIDType) bool

Exists checks whether an image ID is present in an ImageToMetadataMap.

func (ImageToMetadataMap) Insert 

func (m ImageToMetadataMap) Insert(imageID ImageIDType, metadata ImageMetadataInfo)

Insert adds an image ID to an Image ID Map.

func (ImageToMetadataMap) Metadata 

func (m ImageToMetadataMap) Metadata(imageID ImageIDType) (metadata ImageMetadataInfo, e error)

Metadata returns the ImageMetadataInfo corresponding to an image ID if that image is present in the input ImageToMetadataMap.

type MetadataSet 

type MetadataSet map[ImageMetadataInfo]bool

MetadataSet is a set of Image Metadata Info structures.

func NewMetadataSet 

func NewMetadataSet() MetadataSet

NewMetadataSet creates a new MetadataSet.

type RepoTagType 

type RepoTagType struct {
	Repo RepoType
	Tag  TagType
}

RepoTagType represents a docker repository and tag.

type RepoType 

type RepoType string

RepoType represents docker repositories.

type Script 

type Script interface {
	//We expect YAML output from scripts that needs parsing of output by Banyan service
	Run(imageID ImageIDType) ([]byte, error)
	Name() string
}

Script is the common interface to run sripts inside a container

type ScriptInfo 

type ScriptInfo struct {
	// contains filtered or unexported fields
}

Script info for all types (e.g., bash, python, etc.)

func (ScriptInfo) Name 

func (sh ScriptInfo) Name() string

Name gives the name of the script

func (ScriptInfo) Run 

func (sh ScriptInfo) Run(imageID ImageIDType) (b []byte, err error)

Run handles running of a script inside an image

type TagInfo 

type TagInfo struct {
	Repo   RepoType
	TagMap map[TagType]ImageIDType
}

TagInfo records the tag-to-image mappings for a single Docker repository.

type TagType 

type TagType string

TagType represents docker repository tags.

type V1Compat 

type V1Compat struct {
	V1Compatibility string
}

type V2Manifest 

type V2Manifest struct {
	History []V1Compat
}

type V2Tag 

type V2Tag struct {
	Name string
	Tags []string
}

type Writer 

type Writer interface {
	// Write output obtained by all the scripts to the appropriate writer plugin
	// Note: outMapMap maps: ImageID -> Script -> Output
	WriteImageAllData(outMapMap map[string]map[string]interface{})

	// Append Image metadata to the appropriate writer plugin
	AppendImageMetadata(imageMetadata []ImageMetadataInfo)

	// Remoe Image metadta from the appropriate writer plugin
	RemoveImageMetadata(imageMetadata []ImageMetadataInfo)
}

This is a writer plugin interface. Currently supported plugins are: "fileWriter": writes to a file in desired format, and "banyanWriter": invokes banyan API to send data to SAAS dashboard

func NewFileWriter 

func NewFileWriter(format string, dir string) Writer

Source Files

View all Source files

Directories

Path Synopsis
cmd
collector
Package config has definitions about the overall configuration of Collector.
 Package config has definitions about the overall configuration of Collector.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.