README

Service for assisted code review, that allows running custom code Analyzers on pull requests.

GitHub version Build Status Development Code Coverage Go Report Card GoDoc

WebsiteDocumentationBlogSlackTwitter

Introduction

With source{d} Lookout, we’re introducing a service for assisted code review, that allows running custom code analyzers on pull requests.

Jump to the Quickstart section to start using it!

Table of Contents

Motivation and Scope

source{d} is the company driving the Machine Learning on Code (#MLonCode) movement. Doing Machine Learning on Code consists of applying ML techniques to train models that can cluster, identify and predict useful aspects of source code and software repositories.

source{d} Lookout is the first step towards a full suite of Machine Learning on Code applications for AI-assisted coding, but you can also create your own analyzers without an ML approach.

The benefits of using source{d} Lookout are:

  • Keep your code base style/patterns consistent.
  • Language agnostic assisted code reviews.
  • Identify where to focus your attention on code reviews.
  • Automatically warn about common mistakes before human code review.

Current Status

Currently, source{d} Lookout is in development process.

Further Reading

This repository contains the code of source{d} Lookout and the project documentation, which you can also see properly rendered at https://docs.sourced.tech/lookout.

Quickstart

There are several ways to run source{d} Lookout; we recommend to use docker-compose because it's straightforward, but you can learn more about the different ways to run source{d} Lookout.

Please refer to the Configuring source{d} Lookout guide for documentation about the config.yml file, and to know how to configure source{d} Lookout to analyze your repositories, or to use your own analyzers.

There is docker-compose.yml config file for Docker Compose to start source{d} Lookout, its dependencies (bblfsh and PostgreSQL) and a dummy analyzer which will add some stats to the watched pull requests.

To do so, clone this repository or download docker-compose.yml directly.

Create the config.yml file in the same directory where docker-compose.yml is. You can use config.yml.tpl as a template. Make sure that you specify in the config.yml the repositories that will be watched by source{d} Lookout. Then run, passing a valid GitHub user/token:

$ docker-compose pull
$ GITHUB_USER=<user> GITHUB_TOKEN=<token> docker-compose up --force-recreate

Once it is running, source{d} Lookout will start posting the comments returned by dummy analyzer into the pull requests opened at GitHub in the repositories that you configured to be watched.

You can stop it by pressing ctrl+c

If you want to try source{d} Lookout with your own analyzer instead of dummy one, you must run it in advance, then set it into config.yml and then run:

$ docker-compose pull
$ GITHUB_USER=<user> GITHUB_TOKEN=<token> docker-compose up --force-recreate lookout bblfsh postgres

If you need to reset the database to a clean state, you should drop the postgres container. To do so, stop running source{d} Lookout with ctrl+c and then execute:

$ docker rm lookout_postgres_1

Available Analyzers

This is the list of the known implemented analyzers for source{d} Lookout:

Name Description Targeted files Maturity level
style-analyzer Code style analyzer development
terraform Checks if Terraform files are correctly formatted Terraform usable
gometalint Reports gometalinter results on pull requests Go testing and demo
sonarcheck Reports SonarSource checks results on pull requests using bblfsh UAST Java testing and demo
flake8 Reports flake8 results on pull requests Python testing and demo
npm-audit Reports issues with newly added dependencies using npm-audit JavaScript development
function-name analyzer Applies a translation model from function identifiers to function names. development

Create an Analyzer

If you are developing an Analyzer, or you want more info about how they work, please check the documentation about source{d} Lookout analyzers.

Contribute

Contributions are more than welcome, if you are interested please take a look at our Contributing Guidelines.

Community

source{d} has an amazing community of developers and contributors who are interested in Code As Data and/or Machine Learning on Code. Please join us! 👋

Code of Conduct

All activities under source{d} projects are governed by the source{d} code of conduct.

License

Affero GPL v3.0 or later, see LICENSE.

Documentation

Overview

Package lookout provides gRPC APIs to implement analysis servers and clients.

Services

There are two services:

1. Analyzer: analyzers process source code changes and provide analysis

results as text comments, possibly linked to specific parts of the code.

2. Data: provides simple access to source code changes.

Index

Constants

This section is empty.

Variables

View Source
var (
	// NoErrStopWatcher if a new error of this kind is returned by EventHandler
	// the Watcher.Watch function exits without error.
	NoErrStopWatcher = errors.NewKind("Stop watcher")
)

Functions

func RegisterAnalyzerServer

func RegisterAnalyzerServer(s *grpc.Server, srv AnalyzerServer)

func RegisterDataServer

func RegisterDataServer(s *grpc.Server, srv *DataServerHandler)

Types

type AnalysisStatus

type AnalysisStatus int

AnalysisStatus is the status reported to the provider to inform that we are performing an analysis, or that it has finished

const (

	// ErrorAnalysisStatus represents an error status
	ErrorAnalysisStatus AnalysisStatus
	// FailureAnalysisStatus represents a failure status
	FailureAnalysisStatus
	// PendingAnalysisStatus represents a pending status
	PendingAnalysisStatus
	// SuccessAnalysisStatus represents a success status
	SuccessAnalysisStatus
)

func (AnalysisStatus) String

func (st AnalysisStatus) String() string

type Analyzer

type Analyzer struct {
	Client AnalyzerClient
	Config AnalyzerConfig
}

Analyzer is a struct of analyzer client and config

type AnalyzerClient

type AnalyzerClient = pb.AnalyzerClient

func NewAnalyzerClient

func NewAnalyzerClient(conn *grpc.ClientConn) AnalyzerClient

type AnalyzerComments

type AnalyzerComments struct {
	Config   AnalyzerConfig
	Comments []*Comment
}

AnalyzerComments contains a group of comments and the config for the analyzer that created them

type AnalyzerCommentsGroups

type AnalyzerCommentsGroups []AnalyzerComments

AnalyzerCommentsGroups list of AnalyzerComments

func (AnalyzerCommentsGroups) Count

func (g AnalyzerCommentsGroups) Count() int

Count returns the total number of comments

func (AnalyzerCommentsGroups) Dedup

Dedup filters duplicated comments

func (AnalyzerCommentsGroups) Filter

Filter filters comments groups using CommentsFilterFn

type AnalyzerConfig

type AnalyzerConfig struct {
	Name string
	// Addr is gRPC URL.
	// can be defined only in global config, repository-scoped configuration is ignored
	Addr string
	// Disabled repository-scoped configuration can accept only true value, false value is ignored
	Disabled bool
	// Feedback is a url to be linked after each comment
	Feedback string
	// Settings any configuration for an analyzer
	Settings map[string]interface{}
}

AnalyzerConfig is a configuration of analyzer

type AnalyzerServer

type AnalyzerServer = pb.AnalyzerServer

type Change

type Change = pb.Change

type ChangeGetter

type ChangeGetter interface {
	// GetChanges returns a ChangeScanner that scans all changes according
	// to the request.
	GetChanges(context.Context, *ChangesRequest) (ChangeScanner, error)
}

ChangeGetter is used to retrieve code changes.

type ChangeScanner

type ChangeScanner interface {
	// Next advances the scanner to the next change. It returns true if a new
	// change is found, and false otherwise. After the user is done scanning,
	// Err must be called to check if all changes were consumed or there was an
	// error.
	Next() bool
	// Err returns any error found during scanning.
	Err() error
	// Change returns the current change.
	Change() *Change
	// Close closes the scanner.
	Close() error
}

ChangeScanner is a scanner for changes.

type ChangesRequest

type ChangesRequest = pb.ChangesRequest

type ClientChangeScanner

type ClientChangeScanner struct {
	// contains filtered or unexported fields
}

func (*ClientChangeScanner) Change

func (s *ClientChangeScanner) Change() *Change

func (*ClientChangeScanner) Close

func (s *ClientChangeScanner) Close() error

func (*ClientChangeScanner) Err

func (s *ClientChangeScanner) Err() error

func (*ClientChangeScanner) Next

func (s *ClientChangeScanner) Next() bool

type ClientFileScanner

type ClientFileScanner struct {
	// contains filtered or unexported fields
}

func (*ClientFileScanner) Close

func (s *ClientFileScanner) Close() error

func (*ClientFileScanner) Err

func (s *ClientFileScanner) Err() error

func (*ClientFileScanner) File

func (s *ClientFileScanner) File() *File

func (*ClientFileScanner) Next

func (s *ClientFileScanner) Next() bool

type Comment

type Comment = pb.Comment

type CommentsFilterFn

type CommentsFilterFn func(*Comment) (skip bool, err error)

CommentsFilterFn is a function that filters comments

type CommitRevision

type CommitRevision = pb.CommitRevision

CommitRevision defines a range of commits, from a base to a head

type DataClient

type DataClient struct {
	// contains filtered or unexported fields
}

func NewDataClient

func NewDataClient(cc *grpc.ClientConn) *DataClient

func (*DataClient) GetChanges

func (c *DataClient) GetChanges(ctx context.Context, in *ChangesRequest, opts ...grpc.CallOption) (
	ChangeScanner, error)

func (*DataClient) GetFiles

func (c *DataClient) GetFiles(ctx context.Context, in *FilesRequest, opts ...grpc.CallOption) (
	FileScanner, error)

type DataServerHandler

type DataServerHandler struct {
	ChangeGetter ChangeGetter
	FileGetter   FileGetter
}

func (*DataServerHandler) GetChanges

func (s *DataServerHandler) GetChanges(req *ChangesRequest,
	srv pb.Data_GetChangesServer) (err error)

func (*DataServerHandler) GetFiles

func (s *DataServerHandler) GetFiles(req *FilesRequest, srv pb.Data_GetFilesServer) (err error)

type Event

type Event interface {
	// ID returns the EventID.
	ID() EventID
	// Type returns the EventType, in order to identify the concreate type of
	// the event.
	Type() EventType
	// Revision returns a commit revision, containing the head and the base of
	// the changes.
	Revision() *CommitRevision
	// Validate returns an error if the event is malformed
	Validate() error
	// GetProvider returns the name of the provider that created this event
	GetProvider() string
	// GetOrganizationID returns the organization to which this event's repository
	// belongs to
	GetOrganizationID() string
}

Event represents a repository event.

type EventHandler

type EventHandler func(context.Context, Event) error

EventHandler is the function to be called when a new event happens.

func CachedHandler

func CachedHandler(fn EventHandler) EventHandler

CachedHandler wraps an EventHandler, keeping a cache to skip successfully processed Events

type EventID

type EventID = pb.EventID

EventID is a unique hash id for an event

type EventResponse

type EventResponse = pb.EventResponse

type EventType

type EventType = pb.EventType

EventType defines the supported event types

type File

type File = pb.File

type FileGetter

type FileGetter interface {
	// GetFiles returns a FilesScanner that scans all files according
	// to the request.
	GetFiles(context.Context, *FilesRequest) (FileScanner, error)
}

FileGetter is used to retrieve all code for a revision.

type FileScanner

type FileScanner interface {
	// Next advances the scanner to the next file. It returns true if a new
	// file is found, and false otherwise. After the user is done scanning,
	// Err must be called to check if all files were consumed or there was an
	// error.
	Next() bool
	// Err returns any error found during scanning.
	Err() error
	// File returns the current file.
	File() *File
	// Close closes the scanner.
	Close() error
}

FileScanner is a scanner for files.

type FilesRequest

type FilesRequest = pb.FilesRequest

type FnChangeScanner

type FnChangeScanner struct {
	Scanner ChangeScanner
	Fn      func(*Change) (bool, error)
	OnStart func() error
	// contains filtered or unexported fields
}

FnChangeScanner implements ChangeScanner using functions

func (*FnChangeScanner) Change

func (s *FnChangeScanner) Change() *Change

func (*FnChangeScanner) Close

func (s *FnChangeScanner) Close() error

func (*FnChangeScanner) Err

func (s *FnChangeScanner) Err() error

func (*FnChangeScanner) Next

func (s *FnChangeScanner) Next() bool

type FnFileScanner

type FnFileScanner struct {
	Scanner FileScanner
	Fn      func(*File) (bool, error)
	OnStart func() error
	// contains filtered or unexported fields
}

FnFileScanner implements FileScanner using functions

func (*FnFileScanner) Close

func (s *FnFileScanner) Close() error

func (*FnFileScanner) Err

func (s *FnFileScanner) Err() error

func (*FnFileScanner) File

func (s *FnFileScanner) File() *File

func (*FnFileScanner) Next

func (s *FnFileScanner) Next() bool

type Poster

type Poster interface {
	// Post posts comments about an event.
	// poster should make sure comments weren't posted before if safe is true
	Post(ctx context.Context, e Event, cs []AnalyzerComments, safe bool) error

	// Status sends the current analysis status to the provider
	Status(context.Context, Event, AnalysisStatus) error
}

Poster can post comments about an event.

type PushEvent

type PushEvent struct {
	pb.PushEvent
	// OrganizationID is the organization to which this event's repository
	// belongs to
	OrganizationID string
}

PushEvent represents a Push to a git repository. It wraps the pb.PushEvent adding information only relevant to lookout, and not for the analyzers.

func (*PushEvent) GetOrganizationID

func (e *PushEvent) GetOrganizationID() string

GetOrganizationID returns the organization to which this event's repository belongs to

func (*PushEvent) GetProvider

func (e *PushEvent) GetProvider() string

GetProvider returns the name of the provider that created this event

type ReferencePointer

type ReferencePointer = pb.ReferencePointer

ReferencePointer is a pointer to a git refererence in a repository

type RepositoryInfo

type RepositoryInfo = pb.RepositoryInfo

RepositoryInfo contains information about a repository

type ReviewEvent

type ReviewEvent struct {
	pb.ReviewEvent
	// OrganizationID is the organization to which this event's repository
	// belongs to
	OrganizationID string
}

ReviewEvent represents a Review (pull request in case of GitHub) being created or updated. It wraps the pb.PushEvent adding information only relevant to lookout, and not for the analyzers.

func (*ReviewEvent) GetOrganizationID

func (e *ReviewEvent) GetOrganizationID() string

GetOrganizationID returns the organization to which this event's repository belongs to

func (*ReviewEvent) GetProvider

func (e *ReviewEvent) GetProvider() string

GetProvider returns the name of the provider that created this event

type Watcher

type Watcher interface {
	// Watch for new events triggering the EventHandler for each new issue,
	// it stops until an error is returned by the EventHandler. Network errors
	// or other temporal errors are handled as non-fatal errors, just logging it.
	Watch(context.Context, EventHandler) error
}

Watcher watch for new events in given provider.

Directories

Path Synopsis
cmd
Package dummy implements an example analyzer.
Package dummy implements an example analyzer.
provider
service
bblfsh
Package bblfsh provides access to bblfsh parsing for the data service.
Package bblfsh provides access to bblfsh parsing for the data service.
git
Package git provides access to git repositories.
Package git provides access to git repositories.
models
Code generated by https://github.com/src-d/go-kallax.
Code generated by https://github.com/src-d/go-kallax.
util
cli