codeintelutils

README

⚠️ This project has been merged into github.com/sourcegraph/sourcegraph ⚠️

Codeintel Utilities

This repo contains common functionality shared between a Sourcegraph instance and src-cli.

Documentation

Index

• Constants
• Variables
• func Gzip(source io.Reader) io.Reader
• func ReadIndexerName(r io.Reader) (string, error)
• func SplitFile(filename string, maxPayloadSize int) (files []string, _ func(error) error, err error)
• func SplitReader(r io.ReaderAt, maxPayloadSize int) func() io.Reader
• func SplitReaderIntoFiles(r io.ReaderAt, maxPayloadSize int) (files []string, _ func(error) error, err error)
• func StitchFiles(filename string, makePartFilename PartFilenameFunc, decompress, compress bool) error
• func StitchFilesReader(makePartFilename PartFilenameFunc, decompress bool) (io.Reader, error)
• func UploadIndex(opts UploadIndexOpts) (int, error)
• type PartFilenameFunc
• type RequestLogger
• type UploadIndexOpts
• type UploadProgressEvent

Constants

const MaxBufferSize = 128 * 1024

MaxBufferSize is the maximum size of the metaData line in the dump. This should be large enough to be able to read the output of lsif-tsc for most cases, which will contain all glob-expanded file names in the indexing of JavaScript projects.

Data point: lodash's metaData vertex constructed by the args `*.js test/*.js --AllowJs --checkJs` is 10639 characters long.

const ProgressUpdateInterval = time.Millisecond * 100

Variables

var ErrInvalidMetaDataVertex = errors.New("invalid metaData vertex")

ErrInvalidMetaDataVertex occurs when the first line of an LSIF index is not a valid metadata vertex.

var ErrMetadataExceedsBuffer = errors.New("metaData vertex exceeds buffer")

ErrMetadataExceedsBuffer occurs when the first line of an LSIF index is too long to read.

var ErrUnauthorized = errors.New("unauthorized upload")

ErrUnauthorized occurs when the upload endpoint returns a 401 response.

Functions

func Gzip

func Gzip(source io.Reader) io.Reader

Gzip decorates a source reader by gzip compressing its contents.

func ReadIndexerName

func ReadIndexerName(r io.Reader) (string, error)

ReadIndexerName returns the name of the tool that generated the given index contents. This function reads only the first line of the file, where the metadata vertex is assumed to be in all valid dumps.

func SplitFile

func SplitFile(filename string, maxPayloadSize int) (files []string, _ func(error) error, err error)

SplitFile writes the contents of the given file into a series of temporary files, each of which are gzipped and are no larger than the given max payload size. The file names are returned in the order in which they were written. The cleanup function removes all temporary files, and wraps the error argument with any additional errors that happen during cleanup.

func SplitReader

func SplitReader(r io.ReaderAt, maxPayloadSize int) func() io.Reader

SplitReader returns a function that returns readers that return a slice of the reader at maxPayloadSize bytes long. Each reader returned will operate on the next sequential slice from the source reader.

func SplitReaderIntoFiles

func SplitReaderIntoFiles(r io.ReaderAt, maxPayloadSize int) (files []string, _ func(error) error, err error)

SplitReaderIntoFiles writes the contents of the given reader into a series of temporary files, each of which are gzipped and are no larger than the given max payload size. The file names are returned in the order in which they were written. The cleanup function removes all temporary files, and wraps the error argument with any additional errors that happen during cleanup.

func StitchFiles

func StitchFiles(filename string, makePartFilename PartFilenameFunc, decompress, compress bool) error

StitchFiles combines multiple compressed file parts into a single file. Each part on disk be concatenated into a single file. The content of each part is written to the new file sequentially. If decompress is true, then each file part is gunzipped while read. If compress is true, the new file will be gzipped. On success, the part files are removed.

func StitchFilesReader

func StitchFilesReader(makePartFilename PartFilenameFunc, decompress bool) (io.Reader, error)

StitchFilesReader combines multiple compressed file parts into a single reader. Each part on disk is concatenated into a single file. The content of each part is decompressed and written to returned reader sequentially. On success, the part files are removed.

func UploadIndex

func UploadIndex(opts UploadIndexOpts) (int, error)

UploadIndex uploads the given index file to the upload endpoint. If the upload file is large, it may be split into multiple chunks locally and uploaded over multiple requests.

Types

type PartFilenameFunc

type PartFilenameFunc func(index int) string

PartFilenameFunc constructs the name of a part file from its part index.

type RequestLogger

type RequestLogger interface {
	LogRequest(req *http.Request)
	LogResponse(req *http.Request, resp *http.Response, body []byte, elapsed time.Duration)
}

type UploadIndexOpts

type UploadIndexOpts struct {
	Endpoint             string
	Path                 string
	AccessToken          string
	AdditionalHeaders    map[string]string
	Repo                 string
	Commit               string
	Root                 string
	Indexer              string
	GitHubToken          string
	File                 string
	AssociatedIndexID    *int
	MaxPayloadSizeBytes  int
	MaxRetries           int
	RetryInterval        time.Duration
	UploadProgressEvents chan UploadProgressEvent
	Logger               RequestLogger
}

type UploadProgressEvent

type UploadProgressEvent struct {
	NumParts      int
	Part          int
	Progress      float64
	TotalProgress float64
}