Documentation
¶
Overview ¶
Package remoteexec provides proxy to remoteexec server.
TODO: goma client should use credential described in https://cloud.google.com/remote-build-execution/docs/getting-started
Index ¶
- Variables
- func ExecuteAndWait(ctx context.Context, c Client, req *rpb.ExecuteRequest, ...) (string, *rpb.ExecuteResponse, error)
- type Adapter
- type ByteStream
- type Client
- func (c Client) BatchReadBlobs(ctx context.Context, req *rpb.BatchReadBlobsRequest, opts ...grpc.CallOption) (*rpb.BatchReadBlobsResponse, error)
- func (c Client) BatchUpdateBlobs(ctx context.Context, req *rpb.BatchUpdateBlobsRequest, opts ...grpc.CallOption) (*rpb.BatchUpdateBlobsResponse, error)
- func (c Client) ByteStream() bpb.ByteStreamClient
- func (c Client) CAS() rpb.ContentAddressableStorageClient
- func (c Client) Cache() rpb.ActionCacheClient
- func (c Client) Capabilities() rpb.CapabilitiesClient
- func (c Client) Exec() rpb.ExecutionClient
- func (c Client) Execute(ctx context.Context, req *rpb.ExecuteRequest, opts ...grpc.CallOption) (rpb.Execution_ExecuteClient, error)
- func (c Client) FindMissingBlobs(ctx context.Context, req *rpb.FindMissingBlobsRequest, opts ...grpc.CallOption) (*rpb.FindMissingBlobsResponse, error)
- func (c Client) GetActionResult(ctx context.Context, req *rpb.GetActionResultRequest, opts ...grpc.CallOption) (*rpb.ActionResult, error)
- func (c Client) GetCapabilities(ctx context.Context, req *rpb.GetCapabilitiesRequest, opts ...grpc.CallOption) (*rpb.ServerCapabilities, error)
- func (c Client) GetTree(ctx context.Context, req *rpb.GetTreeRequest, opts ...grpc.CallOption) (rpb.ContentAddressableStorage_GetTreeClient, error)
- func (c Client) QueryWriteStatus(ctx context.Context, in *bpb.QueryWriteStatusRequest, opts ...grpc.CallOption) (*bpb.QueryWriteStatusResponse, error)
- func (c Client) Read(ctx context.Context, in *bpb.ReadRequest, opts ...grpc.CallOption) (bpb.ByteStream_ReadClient, error)
- func (c Client) UpdateActionResult(ctx context.Context, req *rpb.UpdateActionResultRequest, ...) (*rpb.ActionResult, error)
- func (c Client) WaitExecution(ctx context.Context, req *rpb.WaitExecutionRequest, opts ...grpc.CallOption) (rpb.Execution_WaitExecutionClient, error)
- func (c Client) Write(ctx context.Context, opts ...grpc.CallOption) (bpb.ByteStream_WriteClient, error)
- type CmdStorage
- type DigestCache
- type SpanTimeout
Constants ¶
This section is empty.
Variables ¶
var DefaultSpanTimeout = SpanTimeout{ Inventory: 1 * time.Second, InputTree: 60 * time.Second, Setup: 1 * time.Second, CheckCache: 3 * time.Second, CheckMissing: 10 * time.Second, UploadBlobs: 60 * time.Second, Execute: 0, Response: 30 * time.Second, }
DefaultSpanTimeout is default timeout.
var ( DefaultViews = []*view.View{ { Description: `Number of current running exec operations`, Measure: numRunningOperations, Aggregation: view.Sum(), }, { Description: "Number of requests per wrapper types", TagKeys: []tag.Key{ wrapperTypeKey, }, Measure: wrapperCount, Aggregation: view.Count(), }, { Measure: unknownFlagCount, TagKeys: []tag.Key{ compilerNameKey, }, Aggregation: view.Count(), }, { Description: "Size to allocate buffer for input files", TagKeys: []tag.Key{ allocStatusKey, }, Measure: inputBufferAllocSize, Aggregation: view.Sum(), }, { Description: "Time in inventory check", Measure: execInventoryTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time in input tree construction", Measure: execInputTreeTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time in setup", Measure: execSetupTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time to check cache", Measure: execCheckCacheTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time to check missing", Measure: execCheckMissingTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time to upload blobs", Measure: execUploadBlobsTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time to execute", Measure: execExecuteTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time in response", Measure: execResponseTime, Aggregation: defaultLatencyDistribution, }, { Description: "Time in RBE queue", Measure: rbeQueueTime, TagKeys: rbeTagKeys, Aggregation: defaultLatencyDistribution, }, { Description: "Time in RBE worker", Measure: rbeWorkerTime, TagKeys: rbeTagKeys, Aggregation: defaultLatencyDistribution, }, { Description: "Time in RBE input", Measure: rbeInputTime, TagKeys: rbeTagKeys, Aggregation: defaultLatencyDistribution, }, { Description: "Time in RBE exec", Measure: rbeExecTime, TagKeys: rbeTagKeys, Aggregation: defaultLatencyDistribution, }, { Description: "Time in RBE output", Measure: rbeOutputTime, TagKeys: rbeTagKeys, Aggregation: defaultLatencyDistribution, }, } )
Functions ¶
func ExecuteAndWait ¶
func ExecuteAndWait(ctx context.Context, c Client, req *rpb.ExecuteRequest, opts ...grpc.CallOption) (string, *rpb.ExecuteResponse, error)
ExecuteAndWait executes and action remotely and wait its response. it returns operation name, response and error.
Types ¶
type Adapter ¶
type Adapter struct { execpb.UnimplementedExecServiceServer // InstancePrefix is the prefix (dirname) of the full RBE instance name. // e.g. If instance name == "projects/$PROJECT/instances/default_instance", // then InstancePrefix is "projects/$PROJECT/instances" InstancePrefix string // InstanceBaseName is the name (basename) of the full RBE instance name. // If emtpy, use "default_instance". InstanceBaseName string Inventory exec.Inventory // ExecTimeout is timeout of Action in RBE. ExecTimeout time.Duration // SpanTimeout is timeout of each span in a Goma Exec request. SpanTimeout SpanTimeout // Client is remoteexec API client. Client Client InsecureClient bool // GomaFile handles output files from remoteexec's cas to goma's FileBlob. GomaFile fpb.FileServiceClient // key: goma file hash. DigestCache DigestCache // CmdStorage is a storage for command files. CmdStorage CmdStorage // Tool details put in request metadata. ToolDetails *rpb.ToolDetails // FileLookupSema specifies concurrency to look up file // contents from file-cache-server to be converted to CAS. FileLookupSema chan struct{} // CASBlobLookupSema specifies concurrency to look up file blobs in in cas.lookupBlobsInStore(), // which calls Store.Get(). CASBlobLookupSema chan struct{} // OutputFileSema specifies concurrency to download files from CAS and store in // file server in gomaOutput.toFileBlob(). OutputFileSema chan struct{} // Ratio to enable hardening. HardeningRatio float64 // Ratio to use nsjail for hardening. NsjailRatio float64 // sha256 file hash to disable hardening. DisableHardenings []string // MissingInputLimit is the maximum number of missing inputs to list in // a response. If there are more, the server randomly picks this many // inputs to respond with. 0 indicates no limit. MissingInputLimit int // contains filtered or unexported fields }
Adapter is an adapter from goma API to remoteexec API.
func (*Adapter) Exec ¶
Exec handles goma Exec requests with remoteexec backend.
- compute input tree and Action. 1.1. construct input tree from req. 1.2. construct Action message from req.
- checks the ActionCache using GetActionResult. if hit, go to 7.
- queries the ContentAddressableStorage using FindMissingBlobs
- uploads any missing blobs to the ContentAddressableStorage using bytestream.Write and BatchUpdateBlobs.
- executes the action using Execute.
- awaits completion of the action using the longrunning.Operations.
- looks a the ActionResult
- If the action is successful, uses bytestream.Read to download any outputs it does not already have; embed it in response, or will serve it by LookupFile later
- job is complete 9.1. convert ExecResp from ExecuteResponse. for small outputs, embed in resp. otherwise use FILE_META.
type ByteStream ¶
type ByteStream struct { // Adapter provides an interface to the RBE API. Adapter *Adapter // The name of the RBE instance, e.g. "projects/$PROJECT/instances/default_instance" InstanceName string }
ByteStream is a proxy that reads/writes data to/from a server, as defined in googleapis/bytestream. In the context of the exec server, it accesses a resource on the RBE server, specifically a resource under an RBE instance.
func (*ByteStream) QueryWriteStatus ¶
func (bs *ByteStream) QueryWriteStatus(ctx context.Context, req *pb.QueryWriteStatusRequest) (*pb.QueryWriteStatusResponse, error)
Write proxies bytestream QueryWriteStatus call.
func (*ByteStream) Read ¶
func (bs *ByteStream) Read(req *pb.ReadRequest, s pb.ByteStream_ReadServer) error
Read proxies bytestream Read stream.
func (*ByteStream) Write ¶
func (bs *ByteStream) Write(s pb.ByteStream_WriteServer) error
Write proxies bytestream Write stream.
type Client ¶
type Client struct { *grpc.ClientConn CallOptions []grpc.CallOption Retry rpc.Retry }
Client is a remoteexec API client to ClientConn. CallOptions will be added when calling RPC.
prcred, _ := oauth.NewApplicationDefault(ctx, "https://www.googleapis.com/auth/cloud-build-service") conn, _ := grpc.DialContext(ctx, target, grpc.WithPerRPCCredentials(prcred), grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{}))) client := &remoteexec.Client{conn}
func (Client) BatchReadBlobs ¶
func (c Client) BatchReadBlobs(ctx context.Context, req *rpb.BatchReadBlobsRequest, opts ...grpc.CallOption) (*rpb.BatchReadBlobsResponse, error)
BatchReadBlobs downloads many blobs at once.
func (Client) BatchUpdateBlobs ¶
func (c Client) BatchUpdateBlobs(ctx context.Context, req *rpb.BatchUpdateBlobsRequest, opts ...grpc.CallOption) (*rpb.BatchUpdateBlobsResponse, error)
BatchUpdateBlobs uploads many blobs at once.
func (Client) ByteStream ¶
func (c Client) ByteStream() bpb.ByteStreamClient
ByteStream returns byte stream client. https://godoc.org/google.golang.org/genproto/googleapis/bytestream#ByteStreamClient
func (Client) CAS ¶
func (c Client) CAS() rpb.ContentAddressableStorageClient
CAS returns content addressable storage client. https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L168
func (Client) Cache ¶
func (c Client) Cache() rpb.ActionCacheClient
Cache returns action cache client. https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L117
func (Client) Capabilities ¶
func (c Client) Capabilities() rpb.CapabilitiesClient
Capabilities returns capabilities client.
func (Client) Exec ¶
func (c Client) Exec() rpb.ExecutionClient
Exec returns execution client. https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L34
func (Client) Execute ¶
func (c Client) Execute(ctx context.Context, req *rpb.ExecuteRequest, opts ...grpc.CallOption) (rpb.Execution_ExecuteClient, error)
Execute executes an action remotely.
func (Client) FindMissingBlobs ¶
func (c Client) FindMissingBlobs(ctx context.Context, req *rpb.FindMissingBlobsRequest, opts ...grpc.CallOption) (*rpb.FindMissingBlobsResponse, error)
FindMissingBlobs determines if blobs are present in the CAS.
func (Client) GetActionResult ¶
func (c Client) GetActionResult(ctx context.Context, req *rpb.GetActionResultRequest, opts ...grpc.CallOption) (*rpb.ActionResult, error)
GetActionResult retrieves a cached execution result.
func (Client) GetCapabilities ¶
func (c Client) GetCapabilities(ctx context.Context, req *rpb.GetCapabilitiesRequest, opts ...grpc.CallOption) (*rpb.ServerCapabilities, error)
GetCapabilities returns the server capabilities configuration.
func (Client) GetTree ¶
func (c Client) GetTree(ctx context.Context, req *rpb.GetTreeRequest, opts ...grpc.CallOption) (rpb.ContentAddressableStorage_GetTreeClient, error)
GetTree fetches the entire directory tree rooted at a node.
func (Client) QueryWriteStatus ¶
func (c Client) QueryWriteStatus(ctx context.Context, in *bpb.QueryWriteStatusRequest, opts ...grpc.CallOption) (*bpb.QueryWriteStatusResponse, error)
QueryWriteStatus is used to find the committed_size for a resource that is being written, which can be then be used as the write_offset for the next Write call.
func (Client) Read ¶
func (c Client) Read(ctx context.Context, in *bpb.ReadRequest, opts ...grpc.CallOption) (bpb.ByteStream_ReadClient, error)
Read is used to retrieve the contents of a resource as a sequence of bytes.
func (Client) UpdateActionResult ¶
func (c Client) UpdateActionResult(ctx context.Context, req *rpb.UpdateActionResultRequest, opts ...grpc.CallOption) (*rpb.ActionResult, error)
UpdateActionResult uploads a new execution result.
func (Client) WaitExecution ¶
func (c Client) WaitExecution(ctx context.Context, req *rpb.WaitExecutionRequest, opts ...grpc.CallOption) (rpb.Execution_WaitExecutionClient, error)
WaitExecution waits for an execution operation to complete.
func (Client) Write ¶
func (c Client) Write(ctx context.Context, opts ...grpc.CallOption) (bpb.ByteStream_WriteClient, error)
Write is used to send the contents of a resource as a sequence of bytes.
type CmdStorage ¶
CmdStorage is an interface to retrieve cmd file contents.
type DigestCache ¶
DigetCache caches digest for goma file hash.
type SpanTimeout ¶ added in v0.0.17
type SpanTimeout struct { Inventory time.Duration InputTree time.Duration Setup time.Duration CheckCache time.Duration CheckMissing time.Duration UploadBlobs time.Duration Execute time.Duration Response time.Duration }
SpanTimeout specifies Timeout for exec span. 0 is no time out.
Source Files
¶
Directories
¶
Path | Synopsis |
---|---|
Package cas manages content addressable storage.
|
Package cas manages content addressable storage. |
Package datasource provides data source from local file, bytes etc.
|
Package datasource provides data source from local file, bytes etc. |
Package digest handles content digest for remote executon API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L633
|
Package digest handles content digest for remote executon API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L633 |
Package merkletree operates on a merkle tree for remote execution API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L838
|
Package merkletree operates on a merkle tree for remote execution API, https://github.com/bazelbuild/remote-apis/blob/c1c1ad2c97ed18943adb55f06657440daa60d833/build/bazel/remote/execution/v2/remote_execution.proto#L838 |