Documentation

Overview

    Package archiver implements the pipeline to efficiently archive file sets to an isolated server as fast as possible.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    func PushDirectory

    func PushDirectory(a *Archiver, root, relDir string) (map[*PendingItem]PushedDirectoryItem, map[string]string, error)

      PushDirectory walks a directory at root and pushes the files in it. relDir is a relative directory to offset relative paths against the root directory. For symlink that points to another file under the same root (called *in-tree*), the symlink is preserved. Otherwise, the symlink is followed (*out-of-tree*), and is pushed as an ordinary file.

      It does not return until all the pushed items are hashed, which is done asynchornously.

      The returned files are in a map from its relative path to PushedDirectoryItem. The returned in-tree symlinks are in a map from its relative path to the relative path it points to.

      Types

      type Archiver

      type Archiver struct {
      	// contains filtered or unexported fields
      }

        Archiver is an high level interface to an isolatedclient.Client.

        Uses a 4 stages pipeline, each doing work concurrently:

        - Deduplicating similar requests or known server hot cache hits.
        - Hashing files.
        - Batched cache hit lookups on the server.
        - Uploading cache misses.
        

        func New

          New returns a thread-safe Archiver instance.

          If not nil, out will contain tty-oriented progress information.

          ctx will be used for logging.

          func (*Archiver) Close

          func (a *Archiver) Close() error

            Close waits for all pending files to be done. If an error occurred during processing, it is returned.

            func (*Archiver) Hash

            func (a *Archiver) Hash() crypto.Hash

              Hash returns the hashing algorithm used by this archiver.

              func (*Archiver) Push

              func (a *Archiver) Push(displayName string, source isolatedclient.Source, priority int64) *PendingItem

                Push schedules item upload to the isolate server. Smaller priority value means earlier processing.

                func (*Archiver) PushFile

                func (a *Archiver) PushFile(displayName, path string, priority int64) *PendingItem

                  PushFile schedules file upload to the isolate server. Smaller priority value means earlier processing.

                  func (*Archiver) Stats

                  func (a *Archiver) Stats() *Stats

                    Stats returns a copy of the statistics.

                    type BundlingChecker

                    type BundlingChecker struct {
                    	Hit, Miss CountBytes
                    	// contains filtered or unexported fields
                    }

                      BundlingChecker uses the isolatedclient.Client to check whether items are available on the server. BundlingChecker methods are safe to call concurrently.

                      func NewChecker

                      func NewChecker(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *BundlingChecker

                        NewChecker creates a new Checker with the given isolated client. maxConcurrent controls maximum number of check requests to be in-flight at once. The provided context is used to make all requests to the isolate server.

                        func (*BundlingChecker) AddItem

                        func (c *BundlingChecker) AddItem(item *Item, isolated bool, callback CheckerCallback)

                          AddItem adds the given item to the checker for testing, and invokes the provided callback asynchronously. The isolated param indicates whether the given item represents a JSON isolated manifest (as opposed to a regular file). In the case of an error, the callback may never be invoked.

                          func (*BundlingChecker) Close

                          func (c *BundlingChecker) Close() error

                            Close shuts down the checker, blocking until all pending items have been checked with the server. Close returns the first error encountered during the checking process, if any. After Close has returned, Checker is guaranteed to no longer invoke any previously-provided callback.

                            func (*BundlingChecker) Hash

                            func (c *BundlingChecker) Hash() crypto.Hash

                              Hash returns the hashing algorithm used by this checker.

                              func (*BundlingChecker) PresumeExists

                              func (c *BundlingChecker) PresumeExists(item *Item)

                                PresumeExists causes the Checker to report that item exists on the server.

                                type Checker

                                type Checker interface {
                                	Close() error
                                	AddItem(item *Item, isolated bool, callback CheckerCallback)
                                	PresumeExists(item *Item)
                                	Hash() crypto.Hash
                                }

                                  A Checker checks whether items are available on the server. It has a single implementation, *BundlingChecker. See BundlingChecker for method documentation.

                                  type CheckerCallback

                                  type CheckerCallback func(*Item, *isolatedclient.PushState)

                                    CheckerCallback is the callback used by Checker to indicate whether a file is present on the isolate server. If the item not present, the callback will be include the PushState necessary to upload it. Otherwise, the PushState will be nil.

                                    type ConcurrentUploader

                                    type ConcurrentUploader struct {
                                    	// contains filtered or unexported fields
                                    }

                                      ConcurrentUploader uses an isolatedclient.Client to upload items to the server. All methods are safe for concurrent use.

                                      func NewUploader

                                      func NewUploader(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *ConcurrentUploader

                                        NewUploader creates a new ConcurrentUploader with the given isolated client. maxConcurrent controls maximum number of uploads to be in-flight at once. The provided context is used to make all requests to the isolate server.

                                        func (*ConcurrentUploader) Close

                                        func (u *ConcurrentUploader) Close() error

                                          Close waits for any pending uploads (and associated done callbacks) to complete, and returns the first encountered error if any. Uploader cannot be used once it is closed.

                                          func (*ConcurrentUploader) Upload

                                          func (u *ConcurrentUploader) Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())

                                            Upload uploads an item from an isolated.Source. Upload does not block. If not-nil, the done func will be invoked on upload completion (both success and failure).

                                            func (*ConcurrentUploader) UploadBytes

                                            func (u *ConcurrentUploader) UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())

                                              UploadBytes uploads an item held in-memory. UploadBytes does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). The provided byte slice b must not be modified until the upload is completed. TODO(djd): Consider using Upload directly and deleting UploadBytes.

                                              func (*ConcurrentUploader) UploadFile

                                              func (u *ConcurrentUploader) UploadFile(item *Item, ps *isolatedclient.PushState, done func())

                                                UploadFile uploads a file from disk. UploadFile does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). TODO(djd): Consider using Upload directly and deleting UploadFile.

                                                type CountBytes

                                                type CountBytes struct {
                                                	// contains filtered or unexported fields
                                                }

                                                  CountBytes aggregates a count of files and the number of bytes in them.

                                                  func (*CountBytes) Bytes

                                                  func (cb *CountBytes) Bytes() int64

                                                    Bytes returns total byte count.

                                                    func (*CountBytes) Count

                                                    func (cb *CountBytes) Count() int

                                                      Count returns the file count.

                                                      type IsolatedSummary

                                                      type IsolatedSummary struct {
                                                      	// Name is the base name an isolated file with any extension stripped
                                                      	Name   string
                                                      	Digest isolated.HexDigest
                                                      }

                                                        IsolatedSummary contains an isolate name and its digest.

                                                        type Item

                                                        type Item struct {
                                                        	Path    string
                                                        	RelPath string
                                                        	Size    int64
                                                        	Mode    os.FileMode
                                                        
                                                        	Digest isolated.HexDigest
                                                        }

                                                          Item represents a file or symlink referenced by an isolate file.

                                                          type PendingItem

                                                          type PendingItem struct {
                                                          	// Immutable.
                                                          	DisplayName string // Name to use to qualify this item
                                                          	// contains filtered or unexported fields
                                                          }

                                                            PendingItem is an item being processed.

                                                            It is caried over from pipeline stage to stage to do processing on it.

                                                            func (*PendingItem) Digest

                                                            func (i *PendingItem) Digest() isolated.HexDigest

                                                              Digest returns the calculated digest once calculated, empty otherwise.

                                                              func (*PendingItem) Error

                                                              func (i *PendingItem) Error() error

                                                                Error returns any error that occurred for this item if any.

                                                                func (*PendingItem) SetErr

                                                                func (i *PendingItem) SetErr(err error)

                                                                  SetErr forcibly set an item as failed. Normally not used by callers.

                                                                  func (*PendingItem) WaitForHashed

                                                                  func (i *PendingItem) WaitForHashed()

                                                                    WaitForHashed hangs until the item hash is known.

                                                                    type PushedDirectoryItem

                                                                    type PushedDirectoryItem struct {
                                                                    	// Absolute path to the item.
                                                                    	FullPath string
                                                                    
                                                                    	// Relativie path to the item within the directory.
                                                                    	RelPath string
                                                                    
                                                                    	// FileInfo of the item.
                                                                    	Info os.FileInfo
                                                                    }

                                                                      PushedDirectoryItem represents a file within the directory being pushed.

                                                                      type Stats

                                                                      type Stats struct {
                                                                      	Hits   []units.Size  // Bytes; each item is immutable.
                                                                      	Pushed []*UploadStat // Misses; each item is immutable.
                                                                      }

                                                                        Stats is statistics from the Archiver.

                                                                        func (*Stats) PackedHits

                                                                        func (s *Stats) PackedHits() ([]byte, error)

                                                                          PackedHits returns the size of hit items in packed format.

                                                                          func (*Stats) PackedMisses

                                                                          func (s *Stats) PackedMisses() ([]byte, error)

                                                                            PackedMisses returns size of missed items in packed format.

                                                                            func (*Stats) TotalBytesHits

                                                                            func (s *Stats) TotalBytesHits() units.Size

                                                                              TotalBytesHits is the number of bytes not uploaded due to cache hits on the server.

                                                                              func (*Stats) TotalBytesPushed

                                                                              func (s *Stats) TotalBytesPushed() units.Size

                                                                                TotalBytesPushed returns the sum of bytes uploaded.

                                                                                func (*Stats) TotalHits

                                                                                func (s *Stats) TotalHits() int

                                                                                  TotalHits is the number of cache hits on the server.

                                                                                  func (*Stats) TotalMisses

                                                                                  func (s *Stats) TotalMisses() int

                                                                                    TotalMisses returns the number of cache misses on the server.

                                                                                    type TarringArchiver

                                                                                    type TarringArchiver struct {
                                                                                    	// contains filtered or unexported fields
                                                                                    }

                                                                                      TarringArchiver archives the files specified by an isolate file to the server, Small files are combining into tar archives before uploading.

                                                                                      func NewTarringArchiver

                                                                                      func NewTarringArchiver(checker Checker, uploader Uploader) *TarringArchiver

                                                                                        NewTarringArchiver constructs a TarringArchiver.

                                                                                        func (*TarringArchiver) Archive

                                                                                        func (ta *TarringArchiver) Archive(args *TarringArgs) (IsolatedSummary, error)

                                                                                          Archive uploads a single isolate.

                                                                                          type TarringArgs

                                                                                          type TarringArgs struct {
                                                                                          	Deps          []string
                                                                                          	RootDir       string
                                                                                          	IgnoredPathRe string
                                                                                          	Isolated      string
                                                                                          	Isol          *isolated.Isolated
                                                                                          }

                                                                                            TarringArgs wraps all the args for TarringArchiver.Archive().

                                                                                            type UploadStat

                                                                                            type UploadStat struct {
                                                                                            	Duration time.Duration
                                                                                            	Size     units.Size
                                                                                            	Name     string
                                                                                            }

                                                                                              UploadStat is the statistic for a single upload.

                                                                                              type UploadTracker

                                                                                              type UploadTracker struct {
                                                                                              	// contains filtered or unexported fields
                                                                                              }

                                                                                                UploadTracker uploads and keeps track of files.

                                                                                                func (*UploadTracker) Files

                                                                                                func (ut *UploadTracker) Files() map[string]isolated.File

                                                                                                  Files returns the files which have been uploaded. Note: files may not have completed uploading until the tracker's Checker and Uploader have been closed.

                                                                                                  func (*UploadTracker) Finalize

                                                                                                  func (ut *UploadTracker) Finalize(isolatedPath string) (IsolatedSummary, error)

                                                                                                    Finalize creates and uploads the isolate JSON at the isolatePath, and closes the checker and uploader. It returns the isolate name and digest. Finalize should only be called after UploadDeps.

                                                                                                    func (*UploadTracker) UploadDeps

                                                                                                    func (ut *UploadTracker) UploadDeps(parts partitionedDeps) error

                                                                                                      UploadDeps uploads all of the items in parts.

                                                                                                      type Uploader

                                                                                                      type Uploader interface {
                                                                                                      	Close() error
                                                                                                      	Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())
                                                                                                      	UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())
                                                                                                      	UploadFile(item *Item, ps *isolatedclient.PushState, done func())
                                                                                                      }

                                                                                                        Uploader uploads items to the server. It has a single implementation, *ConcurrentUploader. See ConcurrentUploader for method documentation.