licenses

README

licenses

Go library for matching license text against ScanCode's license rule corpus.

The corpus is embedded in the package. Matching needs no network access, cgo, or Python.

Install

Install the repository scanner with Homebrew:

brew tap git-pkgs/git-pkgs
brew trust --tap git-pkgs/git-pkgs
brew install licenses

Or with Go:

go install github.com/git-pkgs/licenses/cmd/licenses@latest

Add the matching library to a Go module:

go get github.com/git-pkgs/licenses

Scan a repository

licenses .
licenses /path/to/repository
licenses -json /path/to/repository > licenses.json
licenses -scope all -max-files 0 /path/to/repository

The command reports detections by file with the matching rule, expression, rule kind, score, coverage, and byte range. JSON is used when output is redirected; terminals get a text report. Skipped files and directories are named with the reason they were skipped.

Reference rules that join text across document blocks are reported as clues outside files such as LICENSE, COPYING, and NOTICE. Soft-wrapped lines, including lines with a shared source-comment prefix, remain detections. Demoted matches stay visible in the file report but do not contribute to the repository expression totals.

The default project scope skips hidden, dependency, build, cache, and test-data directories. Use -scope all for dependency-license scans. An explicit -skip list applies in either scope. Both scopes exclude .git.

Regular text files are limited to 1 MiB and 32 directory levels. Project scope also has a 10,000-file default limit. All scope requires an explicit -max-files value; use -max-files 0 for an unlimited dependency scan. Setting any limit to zero removes that guard.

The scanner accepts UTF-8, UTF-16LE or UTF-16BE with a byte-order mark, and Latin-1. Reported byte ranges refer to the original file. JSON reports use schema version 1. Schema 1 is additive: consumers must ignore unknown fields and accept file records with empty detections when clues are present. With -matched-text, matched contains decoded UTF-8 rather than the original encoded bytes.

Detection and expression records include identification. Its value is identified, partial, or SPDX's NOASSERTION. Partial expressions contain both non-placeholder and ScanCode placeholder identifiers. NOASSERTION detections confirm license-related text without naming another license identifier.

The three per-file summary counts overlap when one file contains detections in more than one identification state.

Each match reports the method that produced it. hash is a whole-file match against a rule text, exact is a rule token sequence found within the file, and spdx-id is an SPDX-License-Identifier tag line whose expression bytes are not already covered by a rule match. Tag expressions are parsed strictly with github.com/git-pkgs/spdx and reported using ScanCode license keys, so BSD-3-Clause becomes bsd-new. Valid custom LicenseRef-* values become unknown-spdx and report NOASSERTION. Malformed expressions and unknown bare identifiers do not produce an spdx-id match. Tag matches use the rule id spdx-license-identifier.

Exit status 0 means detections were found, 1 is a fatal command error, 2 means the scan was incomplete because of per-file errors or the file limit, and 3 means no conclusive detections were found.

Use the library

matcher, err := licenses.New()
if err != nil {
	return err
}

result, err := matcher.Match(ctx, text)
if err != nil {
	return err
}

for _, detection := range result.Detections {
	fmt.Println(detection.Expression)
}

Matching uses normalized whole-text hashes, exact token sequences, and SPDX-License-Identifier tag lines. It does not use fuzzy or sequence matching, so edits within a license text can prevent a match.

Corpus

The ScanCode commit is pinned in CORPUS_VERSION. Regenerate the embedded index from a clean checkout at that commit:

go run ./cmd/corpusgen \
  -scancode /path/to/scancode-toolkit \
  -version-file CORPUS_VERSION \
  -output internal/corpus/corpus.bin.gz

Conformance

The exact matcher passes 1,535 of 1,786 cases (85.95%) from ScanCode's four active data-driven detection suites. Run the suite against a ScanCode checkout at the commit in CORPUS_VERSION:

SCANCODE_TESTDATA=/path/to/scancode-toolkit/tests/licensedcode/data \
  go test . -run '^TestScanCodeConformanceExact$' -v

Known differences are recorded in the conformance baseline. CI fails if an existing result changes or a new difference appears.

Benchmarks

Run the matching benchmarks with:

GOMAXPROCS=1 go test \
  -run '^$' \
  -bench . \
  -benchmem \
  -benchtime 1s \
  -count 5 \
  .

Run the repository scan benchmark with:

go test ./cmd/licenses \
  -run '^$' \
  -bench '^BenchmarkScanRepository$' \
  -benchmem \
  -count 5

To benchmark local checkouts with the same scan defaults:

LICENSES_BENCH_REPOS=/path/to/repo1:/path/to/repo2 \
  go test ./cmd/licenses \
  -run '^$' \
  -bench '^BenchmarkScanRepositories$' \
  -benchmem \
  -count 5

Use the standard go test flags to select benchmarks or change their duration and sample count.

License

The Go code is released under the MIT License. ScanCode's license and rule data is licensed under CC-BY-4.0. See NOTICE for attribution and modification details.

Documentation

Overview

Package licenses matches byte slices against the ScanCode license rule corpus. Matching is exact after token normalization, so edits within a license can prevent a match.

Index

• Variables
• type CorpusInfo
• type Detection
• type Identification
• type Kind
• type Match
• type Matcher
  • func New(options ...Option) (*Matcher, error)
  • func (m *Matcher) Corpus() CorpusInfo
  • func (m *Matcher) Match(ctx context.Context, b []byte) (Result, error)
• type Method
• type Option
  • func WithMatchedText() Option
• type Result

Variables

var ErrTooManyMatches = errors.New("licenses: too many exact-match candidates")

ErrTooManyMatches is returned when an input produces more exact-match candidates than the matcher can safely filter.

Types

type CorpusInfo

type CorpusInfo struct {
	Version      string // ScanCode version recorded in CORPUS_VERSION.
	RuleCount    int    // Number of license texts and rules in the index.
	SourceCommit string // Full ScanCode Toolkit source commit.
}

CorpusInfo identifies the ScanCode corpus used for a result.

type Detection

type Detection struct {
	// Expression is copied exactly from the ScanCode rule.
	Expression string
	// Identification is derived from the identifiers in Expression.
	Identification Identification
	// Matches contains the rule matches that state Expression.
	Matches []Match
}

Detection groups matches that state the same license expression.

type Identification

type Identification string

Identification states whether a detected expression names concrete licenses.

const (
	// Identified means the expression contains no ScanCode placeholder
	// identifiers.
	Identified Identification = "identified"
	// Partial means the expression contains both non-placeholder and ScanCode
	// placeholder identifiers.
	Partial Identification = "partial"
	// NoAssertion uses SPDX's NOASSERTION term when the expression contains
	// only ScanCode placeholder identifiers.
	NoAssertion Identification = "NOASSERTION"
)

type Kind

type Kind string

Kind identifies the ScanCode category of a matched rule.

const (
	// KindUnknown identifies a rule without an is_license_* category.
	KindUnknown Kind = "unknown"
	// KindText identifies a full license text rule.
	KindText Kind = "text"
	// KindNotice identifies a license notice rule.
	KindNotice Kind = "notice"
	// KindTag identifies a license tag rule.
	KindTag Kind = "tag"
	// KindReference identifies a license reference rule.
	KindReference Kind = "reference"
	// KindIntro identifies a license introduction rule.
	KindIntro Kind = "intro"
	// KindClue identifies a weak license clue rule.
	KindClue Kind = "clue"
)

type Match

type Match struct {
	// RuleID is the ScanCode rule identifier.
	RuleID string
	// LicenseIDs contains identifiers copied from the rule expression.
	LicenseIDs []string
	// Kind identifies the ScanCode category of the matched rule.
	Kind Kind
	// Method identifies the exact matching stage that produced the match.
	Method Method
	// Score is the rule's 0-100 relevance, not a similarity score.
	Score float64
	// Coverage is 100 for every exact match.
	Coverage float64
	// Start is the inclusive byte offset into the input.
	Start int
	// End is the exclusive byte offset into the input.
	End int
	// Matched is a copy of input[Start:End] when WithMatchedText is set.
	// It is nil otherwise.
	Matched []byte
}

Match describes one rule match in the input.

type Matcher

type Matcher struct {
	// contains filtered or unexported fields
}

Matcher matches byte slices against an immutable embedded corpus.

func New

func New(options ...Option) (*Matcher, error)

New loads the embedded corpus. The decoded corpus is shared by every Matcher in the process.

func (*Matcher) Corpus

func (m *Matcher) Corpus() CorpusInfo

Corpus returns information about the embedded corpus used by m. It returns the zero value for a nil or uninitialized Matcher.

func (*Matcher) Match

func (m *Matcher) Match(ctx context.Context, b []byte) (Result, error)

Match finds exact normalized rule matches in b. It returns ErrTooManyMatches when the input exceeds the exact-match candidate limit; callers can identify it with errors.Is.

type Method

type Method string

Method identifies the matching stage that produced a match.

const (
	// Hash is a normalized whole-text hash match.
	Hash Method = "hash"
	// Exact is an exact token-sequence match within a larger input.
	Exact Method = "exact"
	// SpdxID is a strictly parsed SPDX-License-Identifier tag match.
	SpdxID Method = "spdx-id"
)

type Option

type Option func(*matcherOptions)

Option configures a Matcher.

func WithMatchedText

func WithMatchedText() Option

WithMatchedText includes a copy of each matched input range in Match.Matched.

type Result

type Result struct {
	Detections []Detection
	Clues      []Match
	Corpus     CorpusInfo
}

Result contains conclusive detections and weaker clue matches.