claircore

README

ClairCore

ClairCore provides a set of go modules which handle scanning container layers for installed packages and reporting any discovered vulnerabilities.
ClairCore is designed to be embedded into a service wrapper.

For a full overview see: ClairCore Book

Usage

Two packages exist libindex and libvuln.
These packages export the methods for indexing an image's contents and matching the results of the index to vulnerabilities respectively.

libindex usage

Creating an instance

opts := &libindex.Opts{
    ConnString: "postgres://host:port",
    Migrations: true,
    // see definition for more configuration options
}
lib := libindex.New(opts)

call libindex with a populated Manifest

m := &claircore.Manifest{
    ...
}

ir, err := lib.Index(m)
if err != nil {
    log.Printf("%v", err)
}
if ir.State == "IndexError" {
    log.Printf("scan failed: %s", sr.Err)
}

libvuln usage

creating an instance

opts := &libvuln.Opts{
    ConnString: "postgres://host:port",
    Migrations: true,
    // see definition for more configuration option
}
lib := libvuln.New(opts)

call libvuln with a populated IndexReport

ir := &claircore.IndexReport{
    ...
}
vr, err := libvuln.Scan(ir)
if err != nil {
    log.Printf("%v", err)
}

Libvuln will first initialize all updaters before returning from its constructor.
Controlling how many updaters initialize in parallel is provided via the libvuln.Opts struct

To further understand how these packages work together see:
Highlevel Architecture
Vulnerability Matching

Local development and testing

The following targets start and stop a local development environment

make local-dev-up
make local-dev-down

If you modify libvuln or libindex code the following make targets will restart the services with your changes

make libindexhttp-restart
make libvulnhttp-restart

With the local development environment up the following make target runs all tests including integration

make integration

The following make target runs unit tests which do not require a database or local development environment

make unit

Documentation

Overview

Package claircore has foundational types for the claircore module.

Additional documentation can be found at http://quay.github.io/claircore/

Index

• Constants
• Variables
• type Digest
  • func NewDigest(algo string, sum []byte) (Digest, error)
  • func ParseDigest(digest string) (Digest, error)
  • func (d Digest) Algorithm() string
  • func (d Digest) Checksum() []byte
  • func (d Digest) Hash() hash.Hash
  • func (d Digest) MarshalText() ([]byte, error)
  • func (d *Digest) Scan(i interface{}) error
  • func (d Digest) String() string
  • func (d *Digest) UnmarshalText(t []byte) error
  • func (d Digest) Value() (driver.Value, error)
• type DigestError
  • func (e *DigestError) Error() string
  • func (e *DigestError) Unwrap() error
• type Distribution
• type Environment
• type IndexRecord
• type IndexReport
  • func (report *IndexReport) IndexRecords() []*IndexRecord
• type Layer
  • func (l *Layer) Fetched() bool
  • func (l *Layer) Files(paths ...string) (map[string]*bytes.Buffer, error)
  • func (l *Layer) Reader() (io.ReadCloser, error)
  • func (l *Layer) SetLocal(f string) error
• type Manifest
• type Package
• type Repository
• type Severity
• type Vulnerability
• type VulnerabilityReport

Constants

const (
	SHA256 = "sha256"
	SHA512 = "sha512"
)

Variables

var ErrNotFound = errors.New("claircore: unable to find any requested files")

ErrNotFound is returned by Layer.Files if none of the requested files are found.

Types

type Digest

type Digest struct {
	// contains filtered or unexported fields
}

Digest is a type representing the hash of some data.

It's used throughout claircore packages as an attempt to remain independent of a specific hashing algorithm.

func NewDigest

func NewDigest(algo string, sum []byte) (Digest, error)

NewDigest constructs a Digest.

func ParseDigest

func ParseDigest(digest string) (Digest, error)

ParseDigest constructs a Digest from a string, ensuring it's well-formed.

func (Digest) Algorithm

func (d Digest) Algorithm() string

Algorithm returns a string representation of the algorithm used for this digest.

func (Digest) Checksum

func (d Digest) Checksum() []byte

Checksum returns the checksum byte slice.

func (Digest) Hash

func (d Digest) Hash() hash.Hash

Hash returns an instance of the hashing algorithm used for this Digest.

func (Digest) MarshalText

func (d Digest) MarshalText() ([]byte, error)

MarshalText implements encoding.TextMarshaler.

func (*Digest) Scan

func (d *Digest) Scan(i interface{}) error

Scan implements sql.Scanner.

func (Digest) String

func (d Digest) String() string

func (*Digest) UnmarshalText

func (d *Digest) UnmarshalText(t []byte) error

UnmarshalText implements encoding.TextUnmarshaler.

func (Digest) Value

func (d Digest) Value() (driver.Value, error)

Value implements driver.Valuer.

type DigestError

type DigestError struct {
	// contains filtered or unexported fields
}

DigestError is the concrete type backing errors returned from Digest's methods.

func (*DigestError) Error

func (e *DigestError) Error() string

Error implements error.

func (*DigestError) Unwrap

func (e *DigestError) Unwrap() error

Unwrap enables errors.Unwrap.

type Distribution

type Distribution struct {
	// unique ID of this distribution. this will be created as discovered by the library
	// and used for persistence and hash map indexes.
	ID string `json:"id"`
	// A lower-case string (no spaces or other characters outside of 0–9, a–z, ".", "_" and "-") identifying the operating system, excluding any version information
	// and suitable for processing by scripts or usage in generated filenames. Example: "DID=fedora" or "DID=debian".
	DID string `json:"did"`
	// A string identifying the operating system.
	// example: "Ubuntu"
	Name string `json:"name"`
	// A string identifying the operating system version, excluding any OS name information,
	// possibly including a release code name, and suitable for presentation to the user.
	// example: "16.04.6 LTS (Xenial Xerus)"
	Version string `json:"version"`
	// A lower-case string (no spaces or other characters outside of 0–9, a–z, ".", "_" and "-") identifying the operating system release code name,
	// excluding any OS name information or release version, and suitable for processing by scripts or usage in generated filenames
	// example: "xenial"
	VersionCodeName string `json:"version_code_name"`
	// A lower-case string (mostly numeric, no spaces or other characters outside of 0–9, a–z, ".", "_" and "-")
	// identifying the operating system version, excluding any OS name information or release code name,
	// example: "16.04"
	VersionID string `json:"version_id"`
	// A string identifying the OS architecture
	// example: "x86_64"
	Arch string `json:"arch"`
	// Optional common platform enumeration identifier
	CPE string `json:"cpe"`
	// A pretty operating system name in a format suitable for presentation to the user.
	// May or may not contain a release code name or OS version of some kind, as suitable. If not set, defaults to "PRETTY_NAME="Linux"".
	// example: "PRETTY_NAME="Fedora 17 (Beefy Miracle)"".
	PrettyName string `json:"pretty_name"`
}

Distribution is the accompanying system context of a package. this information aides in CVE detection.

Distribution is modeled after the os-release file found in all linux distributions.

type Environment

type Environment struct {
	// the package database the associated package was discovered in
	PackageDB string `json:"package_db"`
	// the layer in which the associated package was introduced
	IntroducedIn Digest `json:"introduced_in"`
	// the ID of the distribution the package was discovered on
	DistributionID string `json:"distribution_id"`
	// the ID of the repository where this package was downloaded from (currently not used)
	RepositoryID string `json:"repository_id"`
}

Environment describes the surrounding environment a package was discovered in.

Environment must be accompanied by a parent structure which maps IDs to data models in order to have meaning. In our case this is IndexReport or VulnerabilityReport.

type IndexRecord

type IndexRecord struct {
	Package      *Package
	Distribution *Distribution
	Repository   *Repository
}

IndexRecord is an entry in the IndexReport.

IndexRecords provide full access to contextual package structures such as Distribution and Repository.

A list of these can be thought of as an "unpacked" IndexReport

type IndexReport

type IndexReport struct {
	// the manifest hash this IndexReport is describing
	Hash Digest `json:"manifest_hash"`
	// the current state of the index operation
	State string `json:"state"`
	// all discovered packages in this manifest key'd by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest key'd by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest key'd by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in key'd by package id
	Environments map[string][]*Environment `json:"environments"`
	// whether the index operation finished successfully
	Success bool `json:"success"`
	// an error string in the case the index did not succeed
	Err string `json:"err"`
}

IndexReport provides a database for discovered artifacts in an image.

IndexReports make heavy usage of lookup maps to associate information without repetition.

func (*IndexReport) IndexRecords

func (report *IndexReport) IndexRecords() []*IndexRecord

IndexRecords returns a list of IndexRecords derived from the IndexReport

type Layer

type Layer struct {
	// Hash is a content addressable hash uniqely identifying this layer.
	// Libindex will treat layers with this same hash as identical.
	Hash    Digest              `json:"hash"`
	URI     string              `json:"uri"`
	Headers map[string][]string `json:"headers"`
	// contains filtered or unexported fields
}

Layer is a container image filesystem layer. Layers are stacked on top of each other to comprise the final filesystem of the container image.

func (*Layer) Fetched

func (l *Layer) Fetched() bool

func (*Layer) Files

func (l *Layer) Files(paths ...string) (map[string]*bytes.Buffer, error)

Files retrieves specific files from the layer's tar archive.

An error is returned only if none of the requested files are found.

The returned map may contain more entries than the number of paths requested. All entries in the map are keyed by paths that are relative to the tar-root. For example, requesting paths of "/etc/os-release", "./etc/os-release", and "etc/os-release" will all result in any found content being stored with the key "etc/os-release".

func (*Layer) Reader

func (l *Layer) Reader() (io.ReadCloser, error)

Reader returns a ReadCloser of the layer.

It should also implement io.Seeker, and should be a tar stream.

func (*Layer) SetLocal

func (l *Layer) SetLocal(f string) error

type Manifest

type Manifest struct {
	// content addressable hash. should be able to be computed via
	// the hashes of all included layers
	Hash Digest `json:"hash"`
	// an array of filesystem layers indexed in the same order as the cooresponding image
	Layers []*Layer `json:"layers"`
}

Manifest represents a docker image. Layers array MUST be indexed in the order that image layers are stacked.

type Package

type Package struct {
	// unique ID of this package. this will be created as discovered by the library
	// and used for persistence and hash map indexes
	ID string `json:"id"`
	// the name of the package
	Name string `json:"name"`
	// the version of the package
	Version string `json:"version"`
	// type of package. currently expectations are binary or source
	Kind string `json:"kind,omitempty"`
	// if type is a binary package a source package maybe present which built this binary package.
	// must be a pointer to support recursive type:
	Source *Package `json:"source,omitempty"`
	// the file system path or prefix where this package resides
	PackageDB string `json:"-"`
	// a hint on which repository this package was downloaded from
	RepositoryHint string `json:"-"`
}

type Repository

type Repository struct {
	ID   string `json:"id"`
	Name string `json:"name"`
	Key  string `json:"key"`
	URI  string `json:"uri"`
}

Repository is a package repository

type Severity

type Severity string

const (
	Unknown    Severity = "Unknown"
	Negligible Severity = "Negligible"
	Low        Severity = "Low"
	Medium     Severity = "Medium"
	High       Severity = "High"
	Critical   Severity = "Critical"
	Defcon1    Severity = "Defcon1"
)

type Vulnerability

type Vulnerability struct {
	// unique ID of this vulnerability. this will be created as discovered by the library
	// and used for persistence and hash map indexes
	ID string `json:"id"`
	// the updater that discovered this vulnerability
	Updater string `json:"updater"`
	// the name of the vulnerability. for example if the vulnerability exists in a CVE database this
	// would the unique CVE name such as CVE-2017-11722
	Name string `json:"name"`
	// the description of the vulnerability
	Description string `json:"description"`
	// any links to more details about the vulnerability
	Links string `json:"links"`
	// the severity string retrieved from the security database
	Severity string `json:"severity"`
	// a normalized Severity type providing client guaranteed severity information
	NormalizedSeverity Severity `json:"normalized_severity"`
	// the package information associated with the vulnerability. ideally these fields can be matched
	// to packages discovered by libindex PackageScanner structs.
	Package *Package `json:"-"`
	// the distribution information associated with the vulnerability.
	Dist *Distribution `json:"-"`
	// the repository information associated with the vulnerability
	Repo *Repository `json:"-"`
	// a string specifying the package version the fix was relased in
	FixedInVersion string `json:"fixed_in_version"`
}

type VulnerabilityReport

type VulnerabilityReport struct {
	// the manifest hash this vulnerability report is describing
	Hash Digest `json:"manifest_hash"`
	// all discovered packages in this manifest keyed by package id
	Packages map[string]*Package `json:"packages"`
	// all discovered distributions in this manifest keyed by distribution id
	Distributions map[string]*Distribution `json:"distributions"`
	// all discovered repositories in this manifest keyed by repository id
	Repositories map[string]*Repository `json:"repository"`
	// a list of environment details a package was discovered in keyed by package id
	Environments map[string][]*Environment `json:"environments"`
	// all discovered vulnerabilities affecting this manifest
	Vulnerabilities map[string]*Vulnerability `json:"vulnerabilities"`
	// a lookup table associating package ids with 1 or more vulnerability ids. keyed by package id
	PackageVulnerabilities map[string][]string `json:"package_vulnerabilities"`
}

VulnerabilityReport provides a report of packages and their associated vulnerabilities.