ld

README

go-license-detector

Project license detector - a command line application or a library, written in Go. It scans the given directory for license files, normalizes and hashes them and outputs all found fuzzy matches with the list of reference texts. The returned names follow SPDX standard.

Why? There are no similar projects exist which can be compiled into a native binary without dependencies and also supports the whole SPDX license database (≈400 items). The implementation is also fast and the API is easy to use.

The license texts are taken directly from license-list-data repository. The detection algorithm is not template matching; this directly implies that go-license-detector does not provide any legal guarantees. The intended area of it's usage is data mining.

Installation

go get -v -d gopkg.in/src-d/go-license-detector.v1/...

Contributions

...are welcome, see CONTRIBUTING.md and code of conduct.

License

Apache 2.0, see LICENSE.md.

Algorithm

1. Find files in the root directory which may represent a license. E.g. LICENSE or license.md.
2. If the file is Markdown or reStructuredText, render to HTML and then convert to plain text. Original HTML files are also converted.
3. Normalize the text according to SPDX recommendations.
4. Split the text into unigrams and build the weighted bag of words.
5. Calculate Weighted MinHash.
6. Apply Locality Sensitive Hashing and pick the reference licenses which are close.
7. For each of the candidate, calculate the Levenshtein distance - D. the corresponding text is the single line with each unigram represented by a single rune (character).
8. Set the similarity as 1 - D / L where L is the number of unigrams in the quieried license.

This pipeline guarantees constant time queries, though requires some initialization to preprocess the reference licenses.

If there are not license files found:

1. Look for README files.
2. If the file is Markdown or reStructuredText, render to HTML and then convert to plain text. Original HTML files are also converted.
3. Scan for words like "copyright", "license" and "released under". Take the neighborhood.
4. Run Named Entity Recognition (NER) over that surrounding context and extract the possible license name.
5. Match it against the list of license names from SPDX.

Usage

Command line:

license-detector /path/to/project
license-detector https://github.com/src-d/go-git

Library:

import "gopkg.in/src-d/go-license-detector.v1"

func main() {
	licenses, err := ld.InvestigateProjectLicenses("/path/to/project")
}

Regenerate binary data

The SPDX licenses are included into the binary. To update them, run

make bindata.go

Documentation

Index

• Variables
• func Asset(name string) ([]byte, error)
• func AssetDir(name string) ([]string, error)
• func AssetInfo(name string) (os.FileInfo, error)
• func AssetNames() []string
• func ExtractLicenseFiles(files []string, reader func(string) (string, error)) []string
• func ExtractReadmeFiles(files []string, reader func(string) (string, error)) []string
• func InvestigateFilesLicenses(fileNames []string, reader func(string) (string, error)) (map[string]float32, error)
• func InvestigateLicenseText(text string) map[string]float32
• func InvestigateLicenseTexts(texts []string) map[string]float32
• func InvestigateProjectLicenses(path string) (map[string]float32, error)
• func InvestigateReadmeText(text string) map[string]float32
• func InvestigateReadmeTexts(texts []string) map[string]float32
• func MustAsset(name string) []byte
• func NormalizeLicenseText(text string, strict bool) string
• func PreprocessHTML(htmlSource string) string
• func PreprocessMarkdown(text string) string
• func PreprocessRestructuredText(text string) string
• func RestoreAsset(dir, name string) error
• func RestoreAssets(dir, name string) error
• type LicenseDatabase
  • func (db LicenseDatabase) Length() int
  • func (db *LicenseDatabase) Load()
  • func (db *LicenseDatabase) QueryLicenseText(text string) map[string]float32
  • func (db *LicenseDatabase) QueryReadmeText(text string) map[string]float32
  • func (db LicenseDatabase) VocabularySize() int
• type WeightedMinHasher
  • func NewWeightedMinHasher(dim int, sampleSize int, seed int64) *WeightedMinHasher
  • func (wmh *WeightedMinHasher) Hash(values []float32, indices []int) []uint64

Variables

var (
	// ErrNoLicenseFound is raised if no license files were found.
	ErrNoLicenseFound = errors.New("no license file was found")
)

Functions

func Asset

func Asset(name string) ([]byte, error)

Asset loads and returns the asset for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetDir

func AssetDir(name string) ([]string, error)

AssetDir returns the file names below a certain directory embedded in the file by go-bindata. For example if you run go-bindata on data/... and data contains the following hierarchy:

data/
  foo.txt
  img/
    a.png
    b.png

then AssetDir("data") would return []string{"foo.txt", "img"} AssetDir("data/img") would return []string{"a.png", "b.png"} AssetDir("foo.txt") and AssetDir("notexist") would return an error AssetDir("") will return []string{"data"}.

func AssetInfo

func AssetInfo(name string) (os.FileInfo, error)

AssetInfo loads and returns the asset info for the given name. It returns an error if the asset could not be found or could not be loaded.

func AssetNames

func AssetNames() []string

AssetNames returns the names of the assets.

func ExtractLicenseFiles

func ExtractLicenseFiles(files []string, reader func(string) (string, error)) []string

ExtractLicenseFiles returns the list of possible license texts. The file names are matched against the template. Reader is used to to read file contents.

func ExtractReadmeFiles

func ExtractReadmeFiles(files []string, reader func(string) (string, error)) []string

ExtractReadmeFiles searches for README files. Reader is used to to read file contents.

func InvestigateFilesLicenses

func InvestigateFilesLicenses(
	fileNames []string, reader func(string) (string, error)) (map[string]float32, error)

InvestigateFilesLicenses scans the given list of file names, reads them with `reader` and detects the licenses. Each match has the confidence assigned, from 0 to 1, 1 means 100% confident.

func InvestigateLicenseText

func InvestigateLicenseText(text string) map[string]float32

InvestigateLicenseText takes the license text and returns the most probable reference licenses matched. Each match has the confidence assigned, from 0 to 1, 1 means 100% confident.

func InvestigateLicenseTexts

func InvestigateLicenseTexts(texts []string) map[string]float32

InvestigateLicenseTexts takes the list of candidate license texts and returns the most probable reference licenses matched. Each match has the confidence assigned, from 0 to 1, 1 means 100% confident.

func InvestigateProjectLicenses

func InvestigateProjectLicenses(path string) (map[string]float32, error)

InvestigateProjectLicenses returns the most probable reference licenses matched for the given file tree. Each match has the confidence assigned, from 0 to 1, 1 means 100% confident.

func InvestigateReadmeText

func InvestigateReadmeText(text string) map[string]float32

InvestigateReadmeText scans the README file for licensing information and outputs probable names found with Named Entity Recognition from NLP.

func InvestigateReadmeTexts

func InvestigateReadmeTexts(texts []string) map[string]float32

InvestigateReadmeTexts scans README files for licensing information and outputs the probable names using NER.

func MustAsset

func MustAsset(name string) []byte

MustAsset is like Asset but panics when Asset would return an error. It simplifies safe initialization of global variables.

func NormalizeLicenseText

func NormalizeLicenseText(text string, strict bool) string

NormalizeLicenseText makes a license text ready for analysis. It follows SPDX guidelines at https://spdx.org/spdx-license-list/matching-guidelines

func PreprocessHTML

func PreprocessHTML(htmlSource string) string

PreprocessHTML converts HTML to plain text. E.g. it rips all the tags.

func PreprocessMarkdown

func PreprocessMarkdown(text string) string

PreprocessMarkdown converts Markdown to plain text. It tries to revert all the decorations.

func PreprocessRestructuredText

func PreprocessRestructuredText(text string) string

PreprocessRestructuredText converts ReStructuredText to plain text. It tries to revert all the decorations.

func RestoreAsset

func RestoreAsset(dir, name string) error

RestoreAsset restores an asset under the given directory

func RestoreAssets

func RestoreAssets(dir, name string) error

RestoreAssets restores an asset under the given directory recursively

Types

type LicenseDatabase

type LicenseDatabase struct {
	Debug bool
	// contains filtered or unexported fields
}

LicenseDatabase holds the license texts, their hashes and the hashtables to query for nearest neighbors.

func (LicenseDatabase) Length

func (db LicenseDatabase) Length() int

Length returns the number of registered licenses.

func (*LicenseDatabase) Load

func (db *LicenseDatabase) Load()

Load takes the licenses from the embedded storage, normalizes, hashes them and builds the LSH hashtables.

func (*LicenseDatabase) QueryLicenseText

func (db *LicenseDatabase) QueryLicenseText(text string) map[string]float32

QueryLicenseText returns the most similar registered licenses.

func (*LicenseDatabase) QueryReadmeText

func (db *LicenseDatabase) QueryReadmeText(text string) map[string]float32

QueryReadmeText tries to detect licenses mentioned in the README.

func (LicenseDatabase) VocabularySize

func (db LicenseDatabase) VocabularySize() int

VocabularySize returns the number of unique unigrams.

type WeightedMinHasher

type WeightedMinHasher struct {
	// contains filtered or unexported fields
}

WeightedMinHasher calculates Weighted MinHash-es. https://ekzhu.github.io/datasketch/weightedminhash.html

func NewWeightedMinHasher

func NewWeightedMinHasher(dim int, sampleSize int, seed int64) *WeightedMinHasher

NewWeightedMinHasher initializes a new instance of WeightedMinHasher. `dim` is the bag size. `sampleSize` is the hash length. `seed` is the random generator seed, as Weighted MinHash is probabilistic.

func (*WeightedMinHasher) Hash

func (wmh *WeightedMinHasher) Hash(values []float32, indices []int) []uint64

Hash calculates the Weighted MinHash from the weighted bag of features. Each feature has an index and a value.