Documentation

Overview

    Package tarring implements an efficient archiver that uploads the files specified by an isolate file to the server, Small files are combining into tar archives before uploading.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    This section is empty.

    Types

    type ArchiveArgs

    type ArchiveArgs struct {
    	Deps          []string
    	RootDir       string
    	IgnoredPathRe string
    	Isolated      string
    	Isol          *isolated.Isolated
    }

      ArchiveArgs wraps all the args for Archiver.Archive().

      type Archiver

      type Archiver struct {
      	// contains filtered or unexported fields
      }

        Archiver archives the files specified by an isolate file to the server, Small files are combining into tar archives before uploading.

        func NewArchiver

        func NewArchiver(checker Checker, uploader Uploader) *Archiver

          NewArchiver constructs a Archiver.

          func (*Archiver) Archive

          func (ta *Archiver) Archive(args *ArchiveArgs) (IsolatedSummary, error)

            Archive uploads a single isolate.

            type BundlingChecker

            type BundlingChecker struct {
            	Hit, Miss CountBytes
            	// contains filtered or unexported fields
            }

              BundlingChecker uses the isolatedclient.Client to check whether items are available on the server. BundlingChecker methods are safe to call concurrently.

              func NewChecker

              func NewChecker(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *BundlingChecker

                NewChecker creates a new Checker with the given isolated client. maxConcurrent controls maximum number of check requests to be in-flight at once. The provided context is used to make all requests to the isolate server.

                func (*BundlingChecker) AddItem

                func (c *BundlingChecker) AddItem(item *Item, isolated bool, callback CheckerCallback)

                  AddItem adds the given item to the checker for testing, and invokes the provided callback asynchronously. The isolated param indicates whether the given item represents a JSON isolated manifest (as opposed to a regular file). In the case of an error, the callback may never be invoked.

                  func (*BundlingChecker) Close

                  func (c *BundlingChecker) Close() error

                    Close shuts down the checker, blocking until all pending items have been checked with the server. Close returns the first error encountered during the checking process, if any. After Close has returned, Checker is guaranteed to no longer invoke any previously-provided callback.

                    func (*BundlingChecker) Hash

                    func (c *BundlingChecker) Hash() crypto.Hash

                      Hash returns the hashing algorithm used by this checker.

                      func (*BundlingChecker) PresumeExists

                      func (c *BundlingChecker) PresumeExists(item *Item)

                        PresumeExists causes the Checker to report that item exists on the server.

                        type Checker

                        type Checker interface {
                        	Close() error
                        	AddItem(item *Item, isolated bool, callback CheckerCallback)
                        	PresumeExists(item *Item)
                        	Hash() crypto.Hash
                        }

                          A Checker checks whether items are available on the server. It has a single implementation, *BundlingChecker. See BundlingChecker for method documentation.

                          type CheckerCallback

                          type CheckerCallback func(*Item, *isolatedclient.PushState)

                            CheckerCallback is the callback used by Checker to indicate whether a file is present on the isolate server. If the item not present, the callback will be include the PushState necessary to upload it. Otherwise, the PushState will be nil.

                            type ConcurrentUploader

                            type ConcurrentUploader struct {
                            	// contains filtered or unexported fields
                            }

                              ConcurrentUploader uses an isolatedclient.Client to upload items to the server. All methods are safe for concurrent use.

                              func NewUploader

                              func NewUploader(ctx context.Context, client *isolatedclient.Client, maxConcurrent int) *ConcurrentUploader

                                NewUploader creates a new ConcurrentUploader with the given isolated client. maxConcurrent controls maximum number of uploads to be in-flight at once. The provided context is used to make all requests to the isolate server.

                                func (*ConcurrentUploader) Close

                                func (u *ConcurrentUploader) Close() error

                                  Close waits for any pending uploads (and associated done callbacks) to complete, and returns the first encountered error if any. Uploader cannot be used once it is closed.

                                  func (*ConcurrentUploader) Upload

                                  func (u *ConcurrentUploader) Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())

                                    Upload uploads an item from an isolated.Source. Upload does not block. If not-nil, the done func will be invoked on upload completion (both success and failure).

                                    func (*ConcurrentUploader) UploadBytes

                                    func (u *ConcurrentUploader) UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())

                                      UploadBytes uploads an item held in-memory. UploadBytes does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). The provided byte slice b must not be modified until the upload is completed. TODO(djd): Consider using Upload directly and deleting UploadBytes.

                                      func (*ConcurrentUploader) UploadFile

                                      func (u *ConcurrentUploader) UploadFile(item *Item, ps *isolatedclient.PushState, done func())

                                        UploadFile uploads a file from disk. UploadFile does not block. If not-nil, the done func will be invoked on upload completion (both success and failure). TODO(djd): Consider using Upload directly and deleting UploadFile.

                                        type CountBytes

                                        type CountBytes struct {
                                        	// contains filtered or unexported fields
                                        }

                                          CountBytes aggregates a count of files and the number of bytes in them.

                                          func (*CountBytes) Bytes

                                          func (cb *CountBytes) Bytes() int64

                                            Bytes returns total byte count.

                                            func (*CountBytes) Count

                                            func (cb *CountBytes) Count() int

                                              Count returns the file count.

                                              type IsolatedSummary

                                              type IsolatedSummary struct {
                                              	// Name is the base name an isolated file with any extension stripped
                                              	Name   string
                                              	Digest isolated.HexDigest
                                              }

                                                IsolatedSummary contains an isolate name and its digest.

                                                type Item

                                                type Item struct {
                                                	Path    string
                                                	RelPath string
                                                	Size    int64
                                                	Mode    os.FileMode
                                                
                                                	Digest isolated.HexDigest
                                                }

                                                  Item represents a file or symlink referenced by an isolate file.

                                                  type UploadTracker

                                                  type UploadTracker struct {
                                                  	// contains filtered or unexported fields
                                                  }

                                                    UploadTracker uploads and keeps track of files.

                                                    func (*UploadTracker) Files

                                                    func (ut *UploadTracker) Files() map[string]isolated.File

                                                      Files returns the files which have been uploaded. Note: files may not have completed uploading until the tracker's Checker and Uploader have been closed.

                                                      func (*UploadTracker) Finalize

                                                      func (ut *UploadTracker) Finalize(isolatedPath string) (IsolatedSummary, error)

                                                        Finalize creates and uploads the isolate JSON at the isolatePath, and closes the checker and uploader. It returns the isolate name and digest. Finalize should only be called after UploadDeps.

                                                        func (*UploadTracker) UploadDeps

                                                        func (ut *UploadTracker) UploadDeps(parts partitionedDeps) error

                                                          UploadDeps uploads all of the items in parts.

                                                          type Uploader

                                                          type Uploader interface {
                                                          	Close() error
                                                          	Upload(name string, src isolatedclient.Source, ps *isolatedclient.PushState, done func())
                                                          	UploadBytes(name string, b []byte, ps *isolatedclient.PushState, done func())
                                                          	UploadFile(item *Item, ps *isolatedclient.PushState, done func())
                                                          }

                                                            Uploader uploads items to the server. It has a single implementation, *ConcurrentUploader. See ConcurrentUploader for method documentation.