v0.0.0-...-870f2d0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Apr 19, 2024 License: BSD-3-Clause Imports: 41 Imported by: 40


Go Reference

Package buildlet contains client tools for working with a buildlet server.



Package buildlet contains client tools for working with a buildlet server.



View Source
const (
	// GomoteCreateStreamVersion is the gomote protocol version at which JSON streamed responses started.
	GomoteCreateStreamVersion = "20191119"

	// GomoteCreateMinVersion is the oldest "gomote create" protocol version that's still supported.
	GomoteCreateMinVersion = "20160922"


View Source
var ErrClosed = errors.New("buildlet: Client closed")
View Source
var ErrQuotaExceeded = errors.New("quota exceeded")

ErrQuotaExceeded matches errors.Is when VM creation fails with a quota error. Currently, it only supports GCE quota errors.

View Source
var ErrTimeout = errors.New("buildlet: timeout waiting for command to complete")

ErrTimeout is a sentinel error that represents that waiting for a command to complete has exceeded the given timeout.

View Source
var GCEGate func()

GCEGate optionally specifies a function to run before any GCE API call. It's intended to be used to bound QPS rate to GCE.

View Source
var NoKeyPair = KeyPair{}

NoKeyPair is used by the coordinator to speak http directly to buildlets, inside their firewall, without TLS.


func RegisterFlags

func RegisterFlags()

RegisterFlags registers "user" and "staging" flags that control the behavior of NewCoordinatorClientFromFlags. These are used by remote client commands like gomote.


type Client

type Client interface {
	ConnectSSH(user, authorizedPubKey string) (net.Conn, error)
	IPPort() string
	InstanceName() string
	IsBroken() bool
	Name() string
	ProxyRoundTripper() http.RoundTripper
	SetDescription(v string)
	SetDialer(dialer func(context.Context) (net.Conn, error))
	SetHTTPClient(httpClient *http.Client)
	SetInstanceName(v string)
	SetName(name string)
	SetOnHeartbeatFailure(fn func())
	Status(ctx context.Context) (Status, error)
	String() string
	URL() string

Client is an interface that represent the methods exposed by client. The fake buildlet client should be used instead of client when testing things that use the client interface. This includes a number of coordinator-internal details; users outside the coordinator should use RemoteClient.

func NewClient

func NewClient(ipPort string, kp KeyPair) Client

NewClient returns a *client that will manipulate ipPort, authenticated using the provided keypair.

This constructor returns immediately without testing the host or auth.

func StartNewVM

func StartNewVM(creds *google.Credentials, buildEnv *buildenv.Environment, instName, hostType string, opts VMOpts) (Client, error)

StartNewVM boots a new VM on GCE and returns a buildlet client configured to speak to it.

type CoordinatorClient

type CoordinatorClient struct {
	// Auth specifies how to authenticate to the coordinator.
	Auth UserPass

	// Instance optionally specifies the build coordinator to connect
	// to. If zero, the production coordinator is used.
	Instance build.CoordinatorInstance
	// contains filtered or unexported fields

A CoordinatorClient makes calls to the build coordinator.

func NewCoordinatorClientFromFlags

func NewCoordinatorClientFromFlags() (*CoordinatorClient, error)

NewCoordinatorClientFromFlags constructs a CoordinatorClient for the current user.

func (*CoordinatorClient) CreateBuildlet

func (cc *CoordinatorClient) CreateBuildlet(builderType string) (RemoteClient, error)

CreateBuildlet creates a new buildlet of the given builder type on cc.

This takes a builderType (instead of a hostType), but the returned buildlet can be used as any builder that has the same underlying buildlet type. For instance, a linux-amd64 buildlet can act as either linux-amd64 or linux-386-387.

It may expire at any time. To release it, call Client.Close.

func (*CoordinatorClient) CreateBuildletWithStatus

func (cc *CoordinatorClient) CreateBuildletWithStatus(builderType string, status func(types.BuildletWaitStatus)) (RemoteClient, error)

CreateBuildletWithStatus is like CreateBuildlet but accepts an optional status callback.

func (*CoordinatorClient) NamedBuildlet

func (cc *CoordinatorClient) NamedBuildlet(name string) (RemoteClient, error)

NamedBuildlet returns a buildlet client for the named remote buildlet. Names are not validated. Use Client.Status to check whether the client works.

func (*CoordinatorClient) RemoteBuildlets

func (cc *CoordinatorClient) RemoteBuildlets() ([]RemoteBuildlet, error)

type DirEntry

type DirEntry struct {
	// Line is of the form "drw-rw-rw\t<name>" and then if a regular file,
	// also "\t<size>\t<modtime>". in either case, without trailing newline.
	// TODO: break into parsed fields?
	Line string

DirEntry is the information about a file on a buildlet.

func (DirEntry) Digest

func (de DirEntry) Digest() string

Digest returns the SHA-1 digest of the file, such as "da39a3ee5e6b4b0d3255bfef95601890afd80709". It returns the empty string if the digest isn't included.

func (DirEntry) IsDir

func (de DirEntry) IsDir() bool

IsDir reports whether de describes a directory. That is, it tests for the os.ModeDir bit being set in de.Perm().

func (DirEntry) Name

func (de DirEntry) Name() string

Name returns the relative path to the file, such as "src/net/http/" or "src/net/http/jar.go".

func (DirEntry) Perm

func (de DirEntry) Perm() string

Perm returns the permission bits in string form, such as "-rw-r--r--" or "drwxr-xr-x".

func (DirEntry) String

func (de DirEntry) String() string

type EC2Client

type EC2Client struct {
	// contains filtered or unexported fields

EC2Client is the client used to create buildlets on EC2.

func NewEC2Client

func NewEC2Client(client *cloud.AWSClient) *EC2Client

NewEC2Client creates a new EC2Client.

func (*EC2Client) StartNewVM

func (c *EC2Client) StartNewVM(ctx context.Context, buildEnv *buildenv.Environment, hconf *dashboard.HostConfig, vmName, hostType string, opts *VMOpts) (Client, error)

StartNewVM boots a new VM on EC2, waits until the client is accepting connections on the configured port and returns a buildlet client configured communicate with it.

type ExecOpts

type ExecOpts struct {
	// Output is the output of stdout and stderr.
	// If nil, the output is discarded.
	Output io.Writer

	// Dir is the directory from which to execute the command,
	// as an absolute or relative path using the buildlet's native
	// path separator, or a slash-separated relative path.
	// If relative, it is relative to the buildlet's work directory.
	// Dir is optional. If not specified, it defaults to the directory of
	// the command, or the work directory if SystemLevel is set.
	Dir string

	// Args are the arguments to pass to the cmd given to Client.Exec.
	Args []string

	// ExtraEnv are KEY=VALUE pairs to append to the buildlet
	// process's environment.
	ExtraEnv []string

	// Path, if non-nil, specifies the PATH variable of the executed process's
	// environment. Each path in the list should use separators native to the
	// buildlet's platform, and a non-nil empty list clears the path.
	// The following expansions apply:
	//   - the string "$PATH" expands to any existing PATH element(s)
	//   - the substring "$WORKDIR" expands to buildlet's temp workdir
	// After expansion, the list is joined with an OS-specific list
	// separator and supplied to the executed process as its PATH
	// environment variable.
	Path []string

	// SystemLevel controls whether the command is expected to be found outside of
	// the buildlet's environment.
	SystemLevel bool

	// Debug, if true, instructs to the buildlet to print extra debug
	// info to the output before the command begins executing.
	Debug bool

	// OnStartExec is an optional hook that runs after the 200 OK
	// response from the buildlet, but before the output begins
	// writing to Output.
	OnStartExec func()

ExecOpts are options for a remote command invocation.

type FakeClient

type FakeClient struct {
	// contains filtered or unexported fields

FakeClient is a fake buildlet client used for testing. Not all functions are implemented.

func (*FakeClient) Close

func (fc *FakeClient) Close() error

Close is a fake client closer.

func (*FakeClient) ConnectSSH

func (fc *FakeClient) ConnectSSH(user, authorizedPubKey string) (net.Conn, error)

ConnectSSH connects to a fake SSH server.

func (*FakeClient) Exec

func (fc *FakeClient) Exec(ctx context.Context, cmd string, opts ExecOpts) (remoteErr, execErr error)

Exec fakes the execution.

func (*FakeClient) GetTar

func (fc *FakeClient) GetTar(ctx context.Context, dir string) (io.ReadCloser, error)

GetTar gives a vake tar zipped directory.

func (*FakeClient) IPPort

func (fc *FakeClient) IPPort() string

IPPort provides a fake ip and port pair.

func (*FakeClient) InstanceName

func (fc *FakeClient) InstanceName() string

InstanceName gives the fake instance name.

func (*FakeClient) IsBroken

func (fc *FakeClient) IsBroken() bool

IsBroken returns a fake broken response.

func (*FakeClient) ListDir

func (fc *FakeClient) ListDir(ctx context.Context, dir string, opts ListDirOpts, fn func(DirEntry)) error

ListDir lists a directory on a fake buildlet.

func (*FakeClient) MarkBroken

func (fc *FakeClient) MarkBroken()

MarkBroken marks the fake client as broken.

func (*FakeClient) Name

func (fc *FakeClient) Name() string

Name is the name of the fake client.

func (*FakeClient) ProxyRoundTripper

func (fc *FakeClient) ProxyRoundTripper() http.RoundTripper

ProxyRoundTripper provides a fake proxy.

func (*FakeClient) ProxyTCP

func (fc *FakeClient) ProxyTCP(port int) (io.ReadWriteCloser, error)

ProxyTCP provides a fake proxy.

func (*FakeClient) Put

func (fc *FakeClient) Put(ctx context.Context, r io.Reader, path string, mode os.FileMode) error

Put places a file on a fake buildlet.

func (*FakeClient) PutTar

func (fc *FakeClient) PutTar(ctx context.Context, r io.Reader, dir string) error

PutTar fakes putting a tar zipped file on a buildldet.

func (*FakeClient) PutTarFromURL

func (fc *FakeClient) PutTarFromURL(ctx context.Context, tarURL, dir string) error

PutTarFromURL fakes putting a tar zipped file on a builelt.

func (*FakeClient) RemoteName

func (fc *FakeClient) RemoteName() string

RemoteName gives the remote name of the fake buildlet.

func (*FakeClient) RemoveAll

func (fc *FakeClient) RemoveAll(ctx context.Context, paths ...string) error

RemoveAll deletes the provided paths, relative to the work directory for a fake buildlet.

func (*FakeClient) SetDescription

func (fc *FakeClient) SetDescription(v string)

SetDescription sets the description on a fake client.

func (*FakeClient) SetDialer

func (fc *FakeClient) SetDialer(dialer func(context.Context) (net.Conn, error))

SetDialer sets the function that creates a new connection to the fake buildlet.

func (*FakeClient) SetHTTPClient

func (fc *FakeClient) SetHTTPClient(httpClient *http.Client)

SetHTTPClient sets the HTTP client on a fake client.

func (*FakeClient) SetInstanceName

func (fc *FakeClient) SetInstanceName(v string)

SetInstanceName sets the GCE or EC2 instance name on a fake client.

func (*FakeClient) SetName

func (fc *FakeClient) SetName(name string)

SetName sets the name on a fake client.

func (*FakeClient) SetOnHeartbeatFailure

func (fc *FakeClient) SetOnHeartbeatFailure(fn func())

SetOnHeartbeatFailure sets a function to be called when heartbeats against this fake buildlet fail.

func (*FakeClient) Status

func (fc *FakeClient) Status(ctx context.Context) (Status, error)

Status provides a status on the fake client.

func (*FakeClient) String

func (fc *FakeClient) String() string

String provides a fake string representation of the client.

func (*FakeClient) URL

func (fc *FakeClient) URL() string

URL is the URL for a fake buildlet.

func (*FakeClient) WorkDir

func (fc *FakeClient) WorkDir(ctx context.Context) (string, error)

WorkDir is the working directory for the fake buildlet.

type GCEError

type GCEError struct {
	OpErrors []*compute.OperationErrorErrors

func (*GCEError) Error

func (q *GCEError) Error() string

func (*GCEError) Is

func (q *GCEError) Is(target error) bool

type GRPCCoordinatorClient

type GRPCCoordinatorClient struct {
	Client protos.GomoteServiceClient

func (*GRPCCoordinatorClient) CreateBuildlet

func (c *GRPCCoordinatorClient) CreateBuildlet(ctx context.Context, builderType string) (RemoteClient, error)

func (*GRPCCoordinatorClient) CreateBuildletWithStatus

func (c *GRPCCoordinatorClient) CreateBuildletWithStatus(ctx context.Context, builderType string, status func(types.BuildletWaitStatus)) (RemoteClient, error)

type KeyPair

type KeyPair struct {
	CertPEM string
	KeyPEM  string

KeyPair is the TLS public certificate PEM file and its associated private key PEM file that a builder will use for its HTTPS server. The zero value means no HTTPs, which is used by the coordinator for machines running within a firewall.

func NewKeyPair

func NewKeyPair() (KeyPair, error)

func (KeyPair) IsZero

func (kp KeyPair) IsZero() bool

func (KeyPair) Password

func (kp KeyPair) Password() string

Password returns the SHA1 of the KeyPEM. This is used as the HTTP Basic Auth password.

type ListDirOpts

type ListDirOpts struct {
	// Recursive controls whether the directory is listed
	// recursively.
	Recursive bool

	// Skip are the directories to skip, relative to the directory
	// passed to ListDir. Each item should contain only forward
	// slashes and not start or end in slashes.
	Skip []string

	// Digest controls whether the SHA-1 digests of regular files
	// are returned.
	Digest bool

ListDirOpts are options for Client.ListDir.

type RemoteBuildlet

type RemoteBuildlet struct {
	HostType    string // "host-linux-bullseye"
	BuilderType string // "linux-386-387"
	Name        string // "buildlet-adg-openbsd-386-2"
	Created     time.Time
	Expires     time.Time

type RemoteClient

type RemoteClient interface {
	Close() error
	Exec(ctx context.Context, cmd string, opts ExecOpts) (remoteErr, execErr error)
	GetTar(ctx context.Context, dir string) (io.ReadCloser, error)
	ListDir(ctx context.Context, dir string, opts ListDirOpts, fn func(DirEntry)) error
	Put(ctx context.Context, r io.Reader, path string, mode os.FileMode) error
	PutTar(ctx context.Context, r io.Reader, dir string) error
	PutTarFromURL(ctx context.Context, tarURL, dir string) error
	ProxyTCP(port int) (io.ReadWriteCloser, error)
	RemoteName() string
	RemoveAll(ctx context.Context, paths ...string) error
	WorkDir(ctx context.Context) (string, error)

RemoteClient is a subset of methods that can be used by a gomote client.

type Status

type Status struct {
	Version int // buildlet version, coordinator rejects value that is too old (see minBuildletVersion).

Status provides status information about the buildlet.

A coordinator can use the provided information to decide what, if anything, to do with a buildlet.

type UserPass

type UserPass struct {
	Username string // "user-$USER"
	Password string // buildlet key

type VM

type VM struct {
	// Name is the name of the GCE VM instance.
	// For example, it's of the form "mote-bradfitz-plan9-386-foo",
	// and not "plan9-386-foo".
	Name   string
	IPPort string
	TLS    KeyPair
	Type   string // buildlet type

type VMOpts

type VMOpts struct {
	// Zone is the GCE zone to create the VM in.
	// Optional; defaults to provided build environment's zone.
	Zone string

	// ProjectID is the GCE project ID (e.g. "foo-bar-123", not
	// the numeric ID).
	// Optional; defaults to provided build environment's project ID ("name").
	ProjectID string

	// TLS optionally specifies the TLS keypair to use.
	// If zero, http without auth is used.
	TLS KeyPair

	// Optional description of the VM.
	Description string

	// Optional metadata to put on the instance.
	Meta map[string]string

	// DeleteIn optionally specifies a duration at which
	// to delete the VM.
	// If zero, a short default is used (not enough for longtest builders).
	// Negative means no deletion timeout.
	DeleteIn time.Duration

	// OnInstanceRequested optionally specifies a hook to run synchronously
	// after the computeService.Instances.Insert call, but before
	// waiting for its operation to proceed.
	OnInstanceRequested func()

	// OnInstanceCreated optionally specifies a hook to run synchronously
	// after the instance operation succeeds.
	OnInstanceCreated func()

	// OnInstanceCreated optionally specifies a hook to run synchronously
	// after the computeService.Instances.Get call.
	// Only valid for GCE resources.
	OnGotInstanceInfo func(*compute.Instance)

	// OnInstanceCreated optionally specifies a hook to run synchronously
	// after the EC2 instance information is retrieved.
	// Only valid for EC2 resources.
	OnGotEC2InstanceInfo func(*cloud.Instance)

	// OnBeginBuildletProbe optionally specifies a hook to run synchronously
	// before StartNewVM tries to hit buildletURL to see if it's up yet.
	OnBeginBuildletProbe func(buildletURL string)

	// OnEndBuildletProbe optionally specifies a hook to run synchronously
	// after StartNewVM tries to hit the buildlet's URL to see if it's up.
	// The hook parameters are the return values from http.Get.
	OnEndBuildletProbe func(*http.Response, error)

	// SkipEndpointVerification does not verify that the builder is listening
	// on port 80 or 443 before creating a buildlet client.
	SkipEndpointVerification bool

	// UseIAPTunnel uses an IAP tunnel to connect to buildlets on GCP.
	UseIAPTunnel bool

	// DiskSizeGB specifies the size of the boot disk in base-2 GB. The default
	// disk size is used if unset.
	// Only valid for GCE resources.
	DiskSizeGB int64

VMOpts control how new VMs are started.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL