scalibr

README

OSV-SCALIBR

OSV-SCALIBR (Software Composition Analysis Library) is an extensible library providing:

• File system scanner used to extract software inventory data (e.g. installed language packages) and detect known vulnerabilities or generate SBOMs. See the list of currently supported software inventory types.
• Container analysis functionality (e.g. layer-based extraction)
• Guided Remediation (generating upgrade patches for transitive vulnerabilities)
• And more!

This can be used as a library with a custom wrapper to perform scans on e.g. container images (only linux-based currently) or remote hosts, or via the OSV-Scanner CLI. It comes with built-in plugins for inventory extraction and vulnerability detection and it also allows users to run their custom plugins.

Prerequisites

To build OSV-SCALIBR, you'll need to have go installed. Follow https://go.dev/doc/install.

How to use

Via the OSV-Scanner CLI

If your use case is known vulnerability scanning and extraction in a CLI context, check out the OSV-Scanner usage guide.

Note: Not all OSV-SCALIBR functionality is available via OSV-Scanner yet. Check out this migration guide for more information.

Via the OSV-SCALIBR wrapper binary

1. go install github.com/google/osv-scalibr/binary/scalibr@latest
2. scalibr --result=result.textproto

See the result proto definition for details about the scan result format.

Run scalibr --help for a list of additional CLI args.

As a library:

1. Import github.com/google/osv-scalibr into your Go project
2. Create a new scalibr.ScanConfig struct, configure the extraction and detection plugins to run
3. Call scalibr.New().Scan() with the config
4. Parse the returned scalibr.ScanResults

See below for an example code snippet.

On a container image

Add the --remote-image flag to scan a remote container image. Example:

scalibr --result=result.textproto --remote-image=alpine@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5

Or the --image-tarball flag to scan a locally saved image tarball like ones produced with docker save my-image > my-image.tar. Example:

scalibr --result=result.textproto --image-tarball=my-image.tar

Note: As mentioned previously only linux-based container images are supported currently. Follow issue #953 for tracking Windows image container scanning support.

SPDX generation

OSV-SCALIBR supports generating the result of inventory extraction as an SPDX v2.3 file in json, yaml or tag-value format. Example usage:

scalibr -o spdx23-json=result.spdx.json

Some fields in the generated SPDX can be overwritten:

scalibr -spdx-document-name="Custom name" --spdx-document-namespace="Custom-namespace" --spdx-creators=Organization:Google -o spdx23-json=result.spdx.json

Running built-in plugins

With the standalone binary

The binary runs SCALIBR's "recommended" internal plugins by default. You can enable more plugins with the --plugins= flags. See the definition files for a list of all built-in plugins and their CLI flags (extractors (fs), extractors (standalone), detectors, annotators, enrichers).

With the library

A collection of all built-in plugin modules can be found in the definition files (extractors (fs), extractors (standalone), detectors, annotators, enrichers). To enable them, just import plugins/list and add the appropriate plugin names to the scan config, e.g.

import (
  "context"
  scalibr "github.com/google/osv-scalibr"
  pl "github.com/google/osv-scalibr/plugins/list"
  scalibrfs "github.com/google/osv-scalibr/fs"
)
plugins, _ := pl.FromNames([]string{"os", "cis", "vex"})
cfg := &scalibr.ScanConfig{
  ScanRoots: scalibrfs.RealFSScanRoots("/"),
  Plugins:   plugins,
}
results := scalibr.New().Scan(context.Background(), cfg)

You can also specify your scanning host's capabilities to only enable plugins whose requirements are satisfied (e.g. network access, OS-specific plugins):

import (
  ...
  "github.com/google/osv-scalibr/plugin"
)
capab := &plugin.Capabilities{
  OS:            plugin.OSLinux,
  Network:       plugin.NetworkOnline,
  DirectFS:      true,
  RunningSystem: true,
}
...
cfg := &scalibr.ScanConfig{
  ScanRoots: scalibrfs.RealFSScanRoots("/"),
  Plugins:   plugin.FilterByCapabilities(plugins, capab),
}
...

Creating + running custom plugins

Custom plugins can only be run when using OSV-SCALIBR as a library.

1. Create an implementation of the OSV-SCALIBR Extractor or Detector interface.
2. Add the newly created struct to the scan config and run the scan, e.g.

import (
  "github.com/google/osv-scalibr/plugin"
  scalibr "github.com/google/osv-scalibr"
)
cfg := &scalibr.ScanConfig{
  Root:                 "/",
  Plugins: []plugin.Plugin{&myExtractor{}},
}
results := scalibr.New().Scan(context.Background(), cfg)

A note on cross-platform

OSV-SCALIBR is compatible with Linux and has experimental support for Windows and Mac. When a new plugin is implemented for OSV-SCALIBR, we need to ensure that it will not break other platforms. Our runners will generally catch compatibility issues, but to ensure everything is easy when implementing a plugin, here are a few recommendations to keep in mind:

• Ensure you work with file paths using the filepath library. For example, avoid using /my/path but prefer filepath.Join('my', 'path') instead.
• If the plugin can only support one system (e.g. a windows-specific detector), the layout will generally be to have two versions of the file:
  • file_system.go: where system is the targeted system (e.g. file_windows.go) that contains the code specific to the target system. It must also contain the adequate go build constraint.
  • file_dummy.go: contains the code for every other system. It generally does nothing and just ensures that the code compiles on that system;
• Because of the way our internal automation works, we generally require unit tests to be defined for every platform and be filtered out dynamically if not compatible. In other words, a test should be filtered in/out using if runtime.GOOS rather than a //go:build constraint. Here is an example.

Custom logging

You can make the OSV-SCALIBR library log using your own custom logger by passing an implementation of the log.Logger interface to log.SetLogger():

import (
  customlog "path/to/custom/log"
  "github.com/google/osv-scalibr/log"
  scalibr "github.com/google/osv-scalibr"
)
cfg := &scalibr.ScanConfig{ScanRoot: "/"}
log.SetLogger(&customlog.Logger{})
results := scalibr.New().Scan(context.Background(), cfg)
log.Info(results)

Contributing

Read how to contribute to OSV-SCALIBR.

Look for any open issues or unowned Patch Reward work you'd like to contribute to.

To build and test your local changes, run make and make test. A local scalibr binary will be generated in the repo base.

Some of your code contributions might require regenerating protos. This can happen when, say, you want to contribute a new inventory type. For such cases, you'll need to install a few dependencies:

• protoc: Install the appropriate precompiled protoc binary.
  • For Mac, you can also install via HomeBrew.
• protoc-gen-go: Run go install google.golang.org/protobuf/cmd/protoc-gen-go

and then run make protos or ./build_protos.sh.

Disclaimers

OSV-SCALIBR is not an official Google product.

Documentation

Overview

Package scalibr provides an interface for running software inventory extraction and security finding detection on a machine.

Index

• func CmpPackages(a, b *extractor.Package) int
• type ScanConfig
  • func (cfg *ScanConfig) EnableRequiredPlugins() error
  • func (cfg *ScanConfig) ValidatePluginRequirements() error
• type ScanResult
• type Scanner
  • func New() *Scanner
  • func (Scanner) Scan(ctx context.Context, config *ScanConfig) (sr *ScanResult)
  • func (s Scanner) ScanContainer(ctx context.Context, img image.Image, config *ScanConfig) (sr *ScanResult, err error)

Functions

func CmpPackages

func CmpPackages(a, b *extractor.Package) int

CmpPackages is a comparison helper fun to be used for sorting Package structs.

Types

type ScanConfig

type ScanConfig struct {
	Plugins []plugin.Plugin
	// Capabilities that the scanning environment satisfies, e.g. whether there's
	// network access. Some plugins can only run if certain requirements are met.
	Capabilities *plugin.Capabilities
	// ScanRoots contain the list of root dir used by file walking during extraction.
	// All extractors and detectors will assume files are relative to these dirs.
	// Example use case: Scanning a container image or source code repo that is
	// mounted to a local dir.
	ScanRoots []*scalibrfs.ScanRoot
	// Optional: Individual file or dir paths to extract inventory from. If specified,
	// the extractors will only look at the specified files or at the contents of the
	// specified directories during the filesystem traversal.
	// Note that on real filesystems these are not relative to the ScanRoots and
	// thus need to be in sub-directories of one of the ScanRoots.
	PathsToExtract []string
	// Optional: If true, only the files in the top-level directories in PathsToExtract are
	// extracted and sub-directories are ignored.
	IgnoreSubDirs bool
	// Optional: Directories that the file system walk should ignore.
	// Note that on real filesystems these are not relative to the ScanRoots and
	// thus need to be in sub-directories of one of the ScanRoots.
	// TODO(b/279413691): Also skip local paths, e.g. "Skip all .git dirs"
	DirsToSkip []string
	// Optional: If the regex matches a directory, it will be skipped.
	SkipDirRegex *regexp.Regexp
	// Optional: If the glob matches a directory, it will be skipped.
	SkipDirGlob glob.Glob
	// Optional: Files larger than this size in bytes are skipped. If 0, no limit is applied.
	MaxFileSize int
	// Optional: Skip files declared in .gitignore files in source repos.
	UseGitignore bool
	// Optional: stats allows to enter a metric hook. If left nil, no metrics will be recorded.
	Stats stats.Collector
	// Optional: Whether to read symlinks.
	ReadSymlinks bool
	// Optional: Limit for visited inodes. If 0, no limit is applied.
	MaxInodes int
	// Optional: By default, inventories stores a path relative to the scan root. If StoreAbsolutePath
	// is set, the absolute path is stored instead.
	StoreAbsolutePath bool
	// Optional: If true, print a detailed analysis of the duration of each extractor.
	PrintDurationAnalysis bool
	// Optional: If true, fail the scan if any permission errors are encountered.
	ErrorOnFSErrors bool
}

ScanConfig stores the config settings of a scan run such as the plugins to use and the dir to consider the root of the scanned system.

func (*ScanConfig) EnableRequiredPlugins

func (cfg *ScanConfig) EnableRequiredPlugins() error

EnableRequiredPlugins adds those plugins to the config that are required by enabled plugins (such as Detectors or Enrichers) but have not been explicitly enabled.

func (*ScanConfig) ValidatePluginRequirements

func (cfg *ScanConfig) ValidatePluginRequirements() error

ValidatePluginRequirements checks that the scanning environment's capabilities satisfy the requirements of all enabled plugin.

type ScanResult

type ScanResult = result.ScanResult

ScanResult stores the results of a scan incl. scan status and inventory found. TODO: b/425645186 - Remove this alias once all callers are migrated to the result package.

type Scanner

type Scanner struct{}

Scanner is the main entry point of the scanner.

func New

func New() *Scanner

New creates a new scanner instance.

func (Scanner) Scan

func (Scanner) Scan(ctx context.Context, config *ScanConfig) (sr *ScanResult)

Scan executes the extraction/detection/annotation/etc. plugins using the provided scan config.

func (Scanner) ScanContainer

func (s Scanner) ScanContainer(ctx context.Context, img image.Image, config *ScanConfig) (sr *ScanResult, err error)

ScanContainer scans the provided container image for packages and security findings using the provided scan config. It populates the LayerDetails field of the packages with the origin layer details. Functions to create an Image from a tarball, remote name, or v1.Image are available in the artifact/image/layerscanning/image package.