builder

package
Version: v0.0.0-...-89ced18 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 28, 2021 License: Apache-2.0 Imports: 53 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewDefaultExecuteResponse

func NewDefaultExecuteResponse(request *remoteworker.DesiredState_Executing) *remoteexecution.ExecuteResponse

NewDefaultExecuteResponse creates an ExecuteResponse message that contains all fields that BuildExecutor should set by default.

Types

type BuildClient

type BuildClient struct {
	// contains filtered or unexported fields
}

BuildClient is a client for the Remote Worker protocol. It can send synchronization requests to a scheduler, informing it of the current state of the worker, while also obtaining requests for executing build actions.

func NewBuildClient

func NewBuildClient(scheduler remoteworker.OperationQueueClient, buildExecutor BuildExecutor, filePool filesystem.FilePool, clock clock.Clock, workerID map[string]string, instanceNamePrefix digest.InstanceName, platform *remoteexecution.Platform, sizeClass uint32) *BuildClient

NewBuildClient creates a new BuildClient instance that is set to the initial state (i.e., being idle).

func (*BuildClient) InExecutingState

func (bc *BuildClient) InExecutingState() bool

InExecutingState returns true if the worker is executing an action, or still needs to successfully synchronize against the scheduler to communicate the completion of an action.

If this function returns false, it is safe for the worker to stop synchronizing against the scheduler without causing any operations to fail.

func (*BuildClient) Run

func (bc *BuildClient) Run() error

Run a iteration of the Remote Worker client, by performing a single synchronization against the scheduler.

type BuildDirectory

type BuildDirectory interface {
	UploadableDirectory

	// Methods inherited from filesystem.Directory.
	Mkdir(name path.Component, perm os.FileMode) error
	Mknod(name path.Component, perm os.FileMode, dev int) error
	Remove(name path.Component) error
	RemoveAll(name path.Component) error

	// Identical to EnterDirectory(), except that it returns a
	// BuildDirectory object.
	EnterBuildDirectory(name path.Component) (BuildDirectory, error)

	// Installs a set of hooks into the directory to intercept I/O
	// operations. The FilePool may be used to allocate storage
	// space. The ErrorLogger may be used to report fatal I/O
	// errors. Implementations of BuildDirectory are free to let
	// this be a no-op, with the disadvantage that they cannot apply
	// resource limits or provide rich I/O error messages.
	InstallHooks(filePool re_filesystem.FilePool, errorLogger util.ErrorLogger)

	// Recursively merges the contents of a Directory stored in the
	// Content Addressable Storage into a local directory. If this
	// process is synchronous, this function can return a
	// synchronous error. If this process is lazy/asynchronous, the
	// provided ErrorLogger may be used to return an error.
	MergeDirectoryContents(ctx context.Context, errorLogger util.ErrorLogger, digest digest.Digest) error
}

BuildDirectory is a directory that may be used for the purpose of running build actions. BuildDirectory shares some operations with filesystem.Directory, but it has a couple of custom operations that implementations may use to run actions in a more efficient and manageable way.

func NewFUSEBuildDirectory

func NewFUSEBuildDirectory(directory fuse.PrepopulatedDirectory, directoryFetcher cas.DirectoryFetcher, contentAddressableStorage blobstore.BlobAccess, inodeNumberTree fuse.InodeNumberTree) BuildDirectory

NewFUSEBuildDirectory creates a BuildDirectory that is backed by a fuse.PrepopulatedDirectory. Instead of creating all files in the input root explicitly, it calls PrepopulatedDirectory.CreateChildren to add special file and directory nodes whose contents are read on demand.

func NewNaiveBuildDirectory

func NewNaiveBuildDirectory(directory filesystem.DirectoryCloser, directoryFetcher cas.DirectoryFetcher, fileFetcher cas.FileFetcher, contentAddressableStorage blobstore.BlobAccess) BuildDirectory

NewNaiveBuildDirectory creates a BuildDirectory that is backed by a simple filesystem.Directory with all of the operations implemented in a naive way. Namely, MergeDirectoryContents() recursively loads all directories from the Content Addressable Storage (CAS) and requests that all of their files are copied into the build directory.

This implementation is intended to be used in combination with regular local file systems. The downside of such file systems is that we cannot populate them on demand. All of the input files must be present before invoking the build action.

type BuildDirectoryCreator

type BuildDirectoryCreator interface {
	GetBuildDirectory(ctx context.Context, actionDigest digest.Digest, mayRunInParallel bool) (BuildDirectory, *path.Trace, error)
}

BuildDirectoryCreator is used by LocalBuildExecutor to obtain build directories in which build actions are executed.

func NewCleanBuildDirectoryCreator

func NewCleanBuildDirectoryCreator(base BuildDirectoryCreator, idleInvoker *cleaner.IdleInvoker) BuildDirectoryCreator

NewCleanBuildDirectoryCreator is an adapter for BuildDirectoryCreator that upon acquistion and release calls into a Cleaner. This Cleaner may, for example, be set up to empty out the build directory. This guarantees that build actions aren't able to see data left behind by ones that ran previously.

func NewRootBuildDirectoryCreator

func NewRootBuildDirectoryCreator(buildDirectory BuildDirectory) BuildDirectoryCreator

NewRootBuildDirectoryCreator is a BuildDirectoryCreator that repeatedly hands out a single directory present on the current system. Additional decorators are used to run builds in subdirectories, so that build actions may run in parallel.

func NewSharedBuildDirectoryCreator

func NewSharedBuildDirectoryCreator(base BuildDirectoryCreator, nextParallelActionID *atomic.Uint64) BuildDirectoryCreator

NewSharedBuildDirectoryCreator is an adapter for BuildDirectoryCreator that causes build actions to be executed inside a subdirectory within the build directory, as opposed to inside the build directory itself. The subdirectory is either named after the action digest of the build action or uses an incrementing number, based on whether collisions may occur.

This adapter can be used to add concurrency to a single worker. When executing build actions in parallel, every build action needs its own build directory.

type BuildExecutor

type BuildExecutor interface {
	Execute(ctx context.Context, filePool filesystem.FilePool, instanceName digest.InstanceName, request *remoteworker.DesiredState_Executing, executionStateUpdates chan<- *remoteworker.CurrentState_Executing) *remoteexecution.ExecuteResponse
}

BuildExecutor is the interface for the ability to run Bazel execute requests and yield an execute response.

func NewCachingBuildExecutor

func NewCachingBuildExecutor(base BuildExecutor, contentAddressableStorage, actionCache blobstore.BlobAccess, browserURL *url.URL) BuildExecutor

NewCachingBuildExecutor creates an adapter for BuildExecutor that stores action results in the Action Cache (AC) if they may be cached. If they may not be cached, they are stored in the Content Addressable Storage (CAS) instead.

In both cases, a link to bb_browser is added to the ExecuteResponse, so that the user may inspect the Action and ActionResult in detail.

func NewCostComputingBuildExecutor

func NewCostComputingBuildExecutor(base BuildExecutor, expensesPerSecond map[string]*resourceusage.MonetaryResourceUsage_Expense) BuildExecutor

NewCostComputingBuildExecutor wraps an existing BuildExecutor, adding the computed cost of the action to the prepopulated AuxiliaryMetadata field of the ActionResult. The provided expenses are represented on a per-second basis and are then multiplied by the amount of seconds that it took for a worker to complete the Action.

func NewFilePoolStatsBuildExecutor

func NewFilePoolStatsBuildExecutor(buildExecutor BuildExecutor) BuildExecutor

NewFilePoolStatsBuildExecutor creates a decorator for BuildExecutor that annotates ExecuteResponses to contain usage statistics of the FilePool. FilePools are used to allocate temporary files that are generated by the build action (e.g., output files).

func NewLocalBuildExecutor

func NewLocalBuildExecutor(contentAddressableStorage blobstore.BlobAccess, buildDirectoryCreator BuildDirectoryCreator, runner runner_pb.RunnerClient, clock clock.Clock, inputRootCharacterDevices map[path.Component]int, maximumMessageSizeBytes int, environmentVariables map[string]string) BuildExecutor

NewLocalBuildExecutor returns a BuildExecutor that executes build steps on the local system.

func NewLoggingBuildExecutor

func NewLoggingBuildExecutor(base BuildExecutor, browserURL *url.URL) BuildExecutor

NewLoggingBuildExecutor wraps an existing BuildExecutor, adding basic logging. A link to bb_browser is printed prior to executing the action. A JSON representation of the ExecuteResponse is logged after completion.

func NewMetricsBuildExecutor

func NewMetricsBuildExecutor(buildExecutor BuildExecutor) BuildExecutor

NewMetricsBuildExecutor creates a decorator for BuildExecutor that exposes the statistics stored in ExecutedActionMetadata as Prometheus metrics.

func NewStorageFlushingBuildExecutor

func NewStorageFlushingBuildExecutor(base BuildExecutor, flush StorageFlusher) BuildExecutor

NewStorageFlushingBuildExecutor is an adapter for BuildExecutor that calls a callback after every operation. The callback is typically used to flush pending writes to underlying storage, to ensure that other processes in the cluster have a consistent view of the completion of the operation.

func NewTimestampedBuildExecutor

func NewTimestampedBuildExecutor(buildExecutor BuildExecutor, clock clock.Clock, workerName string) BuildExecutor

NewTimestampedBuildExecutor creates a decorator for BuildExecutor that augments the ActionResult that is part of the ExecuteResponse with ExecutedActionMetadata. More concretely, it ensures that the ActionResult contains the name of the worker performing the build and timing information.

type InMemoryBuildQueue

type InMemoryBuildQueue struct {
	// contains filtered or unexported fields
}

InMemoryBuildQueue implements a BuildQueue that can distribute requests through the Remote Worker protocol to worker processes. All of the state of the build queue (i.e., list of queued execution requests and list of workers) is kept in memory.

func NewInMemoryBuildQueue

func NewInMemoryBuildQueue(contentAddressableStorage blobstore.BlobAccess, clock clock.Clock, uuidGenerator util.UUIDGenerator, configuration *InMemoryBuildQueueConfiguration, maximumMessageSizeBytes int, defaultPlatformHooks PlatformHooks) *InMemoryBuildQueue

NewInMemoryBuildQueue creates a new InMemoryBuildQueue that is in the initial state. It does not have any queues, workers or queued execution requests. All of these are created by sending it RPCs.

func (*InMemoryBuildQueue) AddDrain

AddDrain inserts a new drain into the list of drains currently tracked by the platform queue.

func (*InMemoryBuildQueue) Execute

Execute an action by scheduling it in the build queue. This call blocks until the action is completed.

func (*InMemoryBuildQueue) GetCapabilities

GetCapabilities returns the Remote Execution protocol capabilities that this service supports.

func (*InMemoryBuildQueue) GetOperation

GetOperation returns detailed information about a single operation identified by name.

func (*InMemoryBuildQueue) KillOperation

KillOperation requests that an operation that is currently QUEUED or EXECUTING is moved the COMPLETED stage immediately. The next time any worker associated with the operation contacts the scheduler, it is requested to stop executing the operation.

func (*InMemoryBuildQueue) ListDrains

ListDrains returns a list of all the drains that are present within a given platform queue.

func (*InMemoryBuildQueue) ListInvocations

ListInvocations returns properties of all client invocations for which one or more operations are either queued or executing within a given platform queue.

When justQueuedInvocations is false, entries for invocations are returned even if they have no queued operations; only ones that are being executed right now. Entries will be sorted by invocation ID.

When justQueuedInvocations is true, entries for invocations are returned only if they have queued operations. Entries will be sorted by priority at which operations are scheduled.

func (*InMemoryBuildQueue) ListOperations

ListOperations returns detailed information about all of the operations tracked by the InMemoryBuildQueue.

func (*InMemoryBuildQueue) ListPlatformQueues

ListPlatformQueues returns a list of all platform queues currently managed by the scheduler.

func (*InMemoryBuildQueue) ListQueuedOperations

ListQueuedOperations returns properties of all queued operations contained for a given invocation within a platform queue.

func (*InMemoryBuildQueue) ListWorkers

ListWorkers returns basic properties of all workers for a given platform queue.

func (*InMemoryBuildQueue) RegisterPredeclaredPlatformQueue

func (bq *InMemoryBuildQueue) RegisterPredeclaredPlatformQueue(instanceNamePrefix digest.InstanceName, platformMessage *remoteexecution.Platform, workerInvocationStickinessLimit time.Duration, maximumQueuedBackgroundLearningOperations int, backgroundLearningOperationPriority int32, maximumSizeClass uint32, platformHooks PlatformHooks) error

RegisterPredeclaredPlatformQueue adds a platform queue to InMemoryBuildQueue that remains present, regardless of whether workers appear.

The main purpose of this method is to create platform queues that are capable of using multiple size classes, as a maximum size class and initialsizeclass.Analyzer can be provided for specifying how operations are assigned to size classes.

func (*InMemoryBuildQueue) RemoveDrain

RemoveDrain removes a drain from the list of drains currently tracked by the platform queue.

func (*InMemoryBuildQueue) Synchronize

Synchronize the state of a worker with the scheduler. This call is used by a worker to report the completion of an operation and to request more work.

func (*InMemoryBuildQueue) TerminateWorkers

TerminateWorkers can be used to indicate that workers are going to be terminated in the nearby future. This function will block until any operations running on the workers complete, thereby allowing the workers to be terminated without interrupting operations.

func (*InMemoryBuildQueue) WaitExecution

WaitExecution attaches to an existing operation that was created by Execute(). This call can be used by the client to reattach to an operation in case of network failure.

type InMemoryBuildQueueConfiguration

type InMemoryBuildQueueConfiguration struct {
	// ExecutionUpdateInterval specifies how frequently Execute()
	// and WaitExecution() should stream updates for a task to the
	// client.
	ExecutionUpdateInterval time.Duration

	// OperationWithNoWaitersTimeout specifies how long an operation
	// may remain registered without having a single client calling
	// Execute() or WaitExecution() on it.
	OperationWithNoWaitersTimeout time.Duration

	// PlatformQueueWithNoWorkersTimeout specifies how long a
	// platform may remain registered by InMemoryBuildQueue when no
	// Synchronize() calls are received for any workers.
	PlatformQueueWithNoWorkersTimeout time.Duration

	// BusyWorkerSynchronizationInterval specifies how frequently
	// busy workers should be instructed to report their state, even
	// if no changes to their running state occurred.
	BusyWorkerSynchronizationInterval time.Duration

	// GetIdleWorkerSynchronizationInterval returns the maximum
	// amount of time a synchronization performed by a worker
	// against the scheduler may block. Once this amount of time is
	// passed, the worker is instructed to resynchronize, as a form
	// of health checking.
	//
	// Implementations may add jitter to this value to ensure
	// synchronization requests get smeared out over time.
	GetIdleWorkerSynchronizationInterval func() time.Duration

	// WorkerTaskRetryCount specifies how many times a worker may
	// redundantly request that a single task is started. By
	// limiting this, we can prevent a single task from
	// crash-looping a worker indefinitely.
	WorkerTaskRetryCount int

	// WorkerWithNoSynchronizationsTimeout specifies how long a
	// worker may remain registered by InMemoryBuildQueue when no
	// Synchronize() calls are received.
	WorkerWithNoSynchronizationsTimeout time.Duration
}

InMemoryBuildQueueConfiguration contains all the tunable settings of the InMemoryBuildQueue.

type InvocationIDExtractor

type InvocationIDExtractor interface {
	ExtractInvocationID(ctx context.Context, instanceName digest.InstanceName, action *remoteexecution.Action, requestMetadata *remoteexecution.RequestMetadata) (*anypb.Any, error)
}

InvocationIDExtractor is a helper type that is used by InMemoryBuildQueue to extract an invocation ID message out of incoming requests. The invocation ID is an identifier for a set of operations that should be scheduled collectively and fairly with respect to other sets.

var RequestMetadataInvocationIDExtractor InvocationIDExtractor = requestMetadataInvocationIDExtractor{}

RequestMetadataInvocationIDExtractor is a default implementation of InvocationIDExtractor that creates invocation ID messages that simply contain the 'tool_invocation_id' value provided by REv2 clients. When used, separate client invocations will all be scheduled fairly with respect to each other.

type OutputHierarchy

type OutputHierarchy struct {
	// contains filtered or unexported fields
}

OutputHierarchy is used by LocalBuildExecutor to track output directories and files that are expected to be generated by the build action. OutputHierarchy can be used to create parent directories of outputs prior to execution, and to upload outputs into the CAS after execution.

func NewOutputHierarchy

func NewOutputHierarchy(command *remoteexecution.Command) (*OutputHierarchy, error)

NewOutputHierarchy creates a new OutputHierarchy that uses the working directory and the output paths specified in an REv2 Command message.

func (*OutputHierarchy) CreateParentDirectories

func (oh *OutputHierarchy) CreateParentDirectories(d BuildDirectory) error

CreateParentDirectories creates parent directories of outputs. This function is called prior to executing the build action.

func (*OutputHierarchy) UploadOutputs

func (oh *OutputHierarchy) UploadOutputs(ctx context.Context, d UploadableDirectory, contentAddressableStorage blobstore.BlobAccess, digestFunction digest.Function, actionResult *remoteexecution.ActionResult) error

UploadOutputs uploads outputs of the build action into the CAS. This function is called after executing the build action.

type PlatformHooks

type PlatformHooks interface {
	InvocationIDExtractor
	initialsizeclass.Analyzer
}

PlatformHooks contains a set of methods that can be provided for any platform managed by InMemoryBuildQueue to analyze incoming execution requests. For each of these requests, it can compute what the invocation ID is and on which size class the action needs to be executed.

type StorageFlusher

type StorageFlusher func(context.Context) error

StorageFlusher is a callback that is invoked by NewStorageFlushingBuildExecutor to flush contents to storage.

type UploadableDirectory

type UploadableDirectory interface {
	// Methods inherited from filesystem.Directory.
	Close() error
	EnterUploadableDirectory(name path.Component) (UploadableDirectory, error)
	Lstat(name path.Component) (filesystem.FileInfo, error)
	ReadDir() ([]filesystem.FileInfo, error)
	Readlink(name path.Component) (string, error)

	// Upload a file into the Content Addressable Storage.
	UploadFile(ctx context.Context, name path.Component, digestFunction digest.Function) (digest.Digest, error)
}

UploadableDirectory is a directory that can be uploaded into the Content Addressable Storage. It is provided to OutputHierarchy.UploadOutputs(), which traverses it and uploads paths that were specified in the Action message.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to