README

Go Reference

golang.org/x/build/buildlet

Package buildlet contains client tools for working with a buildlet server.

Documentation

Overview

    Package buildlet contains client tools for working with a buildlet server.

    Index

    Constants

    View Source
    const (
    	// GomoteCreateStreamVersion is the gomote protocol version at which JSON streamed responses started.
    	GomoteCreateStreamVersion = "20191119"
    
    	// GomoteCreateMinVersion is the oldest "gomote create" protocol version that's still supported.
    	GomoteCreateMinVersion = "20160922"
    )

    Variables

    View Source
    var (
    	// TODO(evanbrown): resource requirements should be
    	// defined per-builder in dashboard/builders.go
    	BuildletCPU      = api.MustParse("2")         // 2 Cores
    	BuildletCPULimit = api.MustParse("8")         // 8 Cores
    	BuildletMemory   = api.MustParse("4000000Ki") // 4,000,000Ki RAM
    )
    View Source
    var ErrClosed = errors.New("buildlet: Client closed")
    View Source
    var ErrTimeout = errors.New("buildlet: timeout waiting for command to complete")
    View Source
    var GCEGate func()

      GCEGate optionally specifies a function to run before any GCE API call. It's intended to be used to bound QPS rate to GCE.

      View Source
      var NoKeyPair = KeyPair{}

        NoKeyPair is used by the coordinator to speak http directly to buildlets, inside their firewall, without TLS.

        Functions

        func DestroyVM

        func DestroyVM(ts oauth2.TokenSource, proj, zone, instance string) error

          DestroyVM sends a request to delete a VM. Actual VM description is currently (2015-01-19) very slow for no good reason. This function returns once it's been requested, not when it's done.

          func RegisterFlags

          func RegisterFlags()

            RegisterFlags registers "user" and "staging" flags that control the behavior of NewCoordinatorClientFromFlags. These are used by remote client commands like gomote.

            Types

            type Client

            type Client struct {
            	// contains filtered or unexported fields
            }

              A Client interacts with a single buildlet.

              func NewClient

              func NewClient(ipPort string, kp KeyPair) *Client

                NewClient returns a *Client that will manipulate ipPort, authenticated using the provided keypair.

                This constructor returns immediately without testing the host or auth.

                func StartNewVM

                func StartNewVM(creds *google.Credentials, buildEnv *buildenv.Environment, instName, hostType string, opts VMOpts) (*Client, error)

                  StartNewVM boots a new VM on GCE and returns a buildlet client configured to speak to it.

                  func StartPod

                  func StartPod(ctx context.Context, kubeClient *kubernetes.Client, podName, hostType string, opts PodOpts) (*Client, error)

                    StartPod creates a new pod on a Kubernetes cluster and returns a buildlet client configured to speak to it.

                    func (*Client) Close

                    func (c *Client) Close() error

                      Closes destroys and closes down the buildlet, destroying all state immediately.

                      func (*Client) ConnectSSH

                      func (c *Client) ConnectSSH(user, authorizedPubKey string) (net.Conn, error)

                        ConnectSSH opens an SSH connection to the buildlet for the given username. The authorizedPubKey must be a line from an ~/.ssh/authorized_keys file and correspond to the private key to be used to communicate over the net.Conn.

                        func (*Client) DestroyVM

                        func (c *Client) DestroyVM(ts oauth2.TokenSource, proj, zone, instance string) error

                          DestroyVM shuts down the buildlet and destroys the VM instance.

                          func (*Client) Exec

                          func (c *Client) Exec(ctx context.Context, cmd string, opts ExecOpts) (remoteErr, execErr error)

                            Exec runs cmd on the buildlet.

                            Two errors are returned: one is whether the command succeeded remotely (remoteErr), and the second (execErr) is whether there were system errors preventing the command from being started or seen to completition. If execErr is non-nil, the remoteErr is meaningless.

                            If the context's deadline is exceeded, the returned execErr is ErrTimeout.

                            func (*Client) GCEInstanceName

                            func (c *Client) GCEInstanceName() string

                              GCEInstanceName gets an instance name for GCE buildlets. This value differs from the buildlet name used in the CLI and web interface. For non-GCE buildlets, this will return an empty string.

                              func (*Client) GetTar

                              func (c *Client) GetTar(ctx context.Context, dir string) (io.ReadCloser, error)

                                GetTar returns a .tar.gz stream of the given directory, relative to the buildlet's work dir. The provided dir may be empty to get everything.

                                func (*Client) IPPort

                                func (c *Client) IPPort() string

                                func (*Client) IsBroken

                                func (c *Client) IsBroken() bool

                                  IsBroken reports whether this client is broken in some way.

                                  func (*Client) ListDir

                                  func (c *Client) ListDir(ctx context.Context, dir string, opts ListDirOpts, fn func(DirEntry)) error

                                    ListDir lists the contents of a directory. The fn callback is run for each entry. The directory dir itself is not included.

                                    func (*Client) MarkBroken

                                    func (c *Client) MarkBroken()

                                      MarkBroken marks this client as broken in some way.

                                      func (*Client) Name

                                      func (c *Client) Name() string

                                        Name returns the name of this buildlet. It returns the first non-empty string from the name given to SetName, its remote buildlet name, its ip:port, or "(unnamed-buildlet)" in the case where ip:port is empty because there's a custom dialer.

                                        func (*Client) ProxyRoundTripper

                                        func (c *Client) ProxyRoundTripper() http.RoundTripper

                                          ProxyRoundTripper returns a RoundTripper that sends HTTP requests directly through to the underlying buildlet, adding auth and X-Buildlet-Proxy headers as necessary. This is really only intended for use by the coordinator.

                                          func (*Client) ProxyTCP

                                          func (c *Client) ProxyTCP(port int) (io.ReadWriteCloser, error)

                                            ProxyTCP connects to the given port on the remote buildlet. The buildlet client must currently be a gomote client (RemoteName != "") and the target type must be a VM type running on GCE. This was primarily created for RDP to Windows machines, but it might get reused for other purposes in the future.

                                            func (*Client) Put

                                            func (c *Client) Put(ctx context.Context, r io.Reader, path string, mode os.FileMode) error

                                              Put writes the provided file to path (relative to workdir) and sets mode.

                                              func (*Client) PutTar

                                              func (c *Client) PutTar(ctx context.Context, r io.Reader, dir string) error

                                                PutTar writes files to the remote buildlet, rooted at the relative directory dir. If dir is empty, they're placed at the root of the buildlet's work directory. The dir is created if necessary. The Reader must be of a tar.gz file.

                                                func (*Client) PutTarFromURL

                                                func (c *Client) PutTarFromURL(ctx context.Context, tarURL, dir string) error

                                                  PutTarFromURL tells the buildlet to download the tar.gz file from tarURL and write it to dir, a relative directory from the workdir. If dir is empty, they're placed at the root of the buildlet's work directory. The dir is created if necessary. The url must be of a tar.gz file.

                                                  func (*Client) RemoteName

                                                  func (c *Client) RemoteName() string

                                                    RemoteName returns the name of this client's buildlet on the coordinator. If this buildlet isn't a remote buildlet created via gomote, this returns the empty string.

                                                    func (*Client) RemoveAll

                                                    func (c *Client) RemoveAll(ctx context.Context, paths ...string) error

                                                      RemoveAll deletes the provided paths, relative to the work directory.

                                                      func (*Client) SetDescription

                                                      func (c *Client) SetDescription(v string)

                                                        SetDescription sets a short description of where the buildlet connection came from. This is used by the build coordinator status page, mostly for debugging.

                                                        func (*Client) SetDialer

                                                        func (c *Client) SetDialer(dialer func(context.Context) (net.Conn, error))

                                                          SetDialer sets the function that creates a new connection to the buildlet. By default, net.Dialer.DialContext is used. SetDialer has effect only when TLS isn't used.

                                                          TODO(bradfitz): this is only used for ssh connections to buildlets, which previously required the client to do its own net.Dial + upgrade request. But now that the net/http client supports read/write bodies for protocol upgrades, we could change how ssh works and delete this.

                                                          func (*Client) SetGCEInstanceName

                                                          func (c *Client) SetGCEInstanceName(v string)

                                                            SetGCEInstanceName sets an instance name for GCE buildlets. This value differs from the buildlet name used in the CLI and web interface.

                                                            func (*Client) SetHTTPClient

                                                            func (c *Client) SetHTTPClient(httpClient *http.Client)

                                                              SetHTTPClient replaces the underlying HTTP client. It should only be called before the Client is used.

                                                              func (*Client) SetName

                                                              func (c *Client) SetName(name string)

                                                              func (*Client) SetOnHeartbeatFailure

                                                              func (c *Client) SetOnHeartbeatFailure(fn func())

                                                                SetOnHeartbeatFailure sets a function to be called when heartbeats against this builder fail, or when the client is destroyed with Close. The function fn is never called more than once. SetOnHeartbeatFailure must be set before any use of the buildlet.

                                                                func (*Client) SetReleaseMode

                                                                func (c *Client) SetReleaseMode(v bool)

                                                                  SetReleaseMode sets whether this client is being used in "release mode", to prepare the final binaries to be shipped to users. All this mode does for now is disable pargzip multi-stream gzip files. See golang.org/issue/19052.

                                                                  SetReleaseMode must be set before using the client.

                                                                  func (*Client) Status

                                                                  func (c *Client) Status(ctx context.Context) (Status, error)

                                                                    Status returns an Status value describing this buildlet.

                                                                    func (*Client) String

                                                                    func (c *Client) String() string

                                                                    func (*Client) URL

                                                                    func (c *Client) URL() string

                                                                      URL returns the buildlet's URL prefix, without a trailing slash.

                                                                      func (*Client) WorkDir

                                                                      func (c *Client) WorkDir(ctx context.Context) (string, error)

                                                                        WorkDir returns the absolute path to the buildlet work directory.

                                                                        type CoordinatorClient

                                                                        type CoordinatorClient struct {
                                                                        	// Auth specifies how to authenticate to the coordinator.
                                                                        	Auth UserPass
                                                                        
                                                                        	// Instance optionally specifies the build coordinator to connect
                                                                        	// to. If zero, the production coordinator is used.
                                                                        	Instance build.CoordinatorInstance
                                                                        	// contains filtered or unexported fields
                                                                        }

                                                                          A CoordinatorClient makes calls to the build coordinator.

                                                                          func NewCoordinatorClientFromFlags

                                                                          func NewCoordinatorClientFromFlags() (*CoordinatorClient, error)

                                                                            NewCoordinatorClientFromFlags constructs a CoordinatorClient for the current user.

                                                                            func (*CoordinatorClient) CreateBuildlet

                                                                            func (cc *CoordinatorClient) CreateBuildlet(builderType string) (*Client, error)

                                                                              CreateBuildlet creates a new buildlet of the given builder type on cc.

                                                                              This takes a builderType (instead of a hostType), but the returned buildlet can be used as any builder that has the same underlying buildlet type. For instance, a linux-amd64 buildlet can act as either linux-amd64 or linux-386-387.

                                                                              It may expire at any time. To release it, call Client.Close.

                                                                              func (*CoordinatorClient) CreateBuildletWithStatus

                                                                              func (cc *CoordinatorClient) CreateBuildletWithStatus(builderType string, status func(types.BuildletWaitStatus)) (*Client, error)

                                                                                CreateBuildletWithStatus is like CreateBuildlet but accepts an optional status callback.

                                                                                func (*CoordinatorClient) NamedBuildlet

                                                                                func (cc *CoordinatorClient) NamedBuildlet(name string) (*Client, error)

                                                                                  NamedBuildlet returns a buildlet client for the named remote buildlet. Names are not validated. Use Client.Status to check whether the client works.

                                                                                  func (*CoordinatorClient) RemoteBuildlets

                                                                                  func (cc *CoordinatorClient) RemoteBuildlets() ([]RemoteBuildlet, error)

                                                                                  type DirEntry

                                                                                  type DirEntry struct {
                                                                                  	// contains filtered or unexported fields
                                                                                  }

                                                                                    DirEntry is the information about a file on a buildlet.

                                                                                    func (DirEntry) Digest

                                                                                    func (de DirEntry) Digest() string

                                                                                      Digest returns the SHA-1 digest of the file, such as "da39a3ee5e6b4b0d3255bfef95601890afd80709". It returns the empty string if the digest isn't included.

                                                                                      func (DirEntry) IsDir

                                                                                      func (de DirEntry) IsDir() bool

                                                                                        IsDir reports whether de describes a directory. That is, it tests for the os.ModeDir bit being set in de.Perm().

                                                                                        func (DirEntry) Name

                                                                                        func (de DirEntry) Name() string

                                                                                          Name returns the relative path to the file, such as "src/net/http/" or "src/net/http/jar.go".

                                                                                          func (DirEntry) Perm

                                                                                          func (de DirEntry) Perm() string

                                                                                            Perm returns the permission bits in string form, such as "-rw-r--r--" or "drwxr-xr-x".

                                                                                            func (DirEntry) String

                                                                                            func (de DirEntry) String() string

                                                                                            type EC2Client

                                                                                            type EC2Client struct {
                                                                                            	// contains filtered or unexported fields
                                                                                            }

                                                                                              EC2Client is the client used to create buildlets on EC2.

                                                                                              func NewEC2Client

                                                                                              func NewEC2Client(client *cloud.AWSClient) *EC2Client

                                                                                                NewEC2Client creates a new EC2Client.

                                                                                                func (*EC2Client) StartNewVM

                                                                                                func (c *EC2Client) StartNewVM(ctx context.Context, buildEnv *buildenv.Environment, hconf *dashboard.HostConfig, vmName, hostType string, opts *VMOpts) (*Client, error)

                                                                                                  StartNewVM boots a new VM on EC2, waits until the client is accepting connections on the configured port and returns a buildlet client configured communicate with it.

                                                                                                  type ExecOpts

                                                                                                  type ExecOpts struct {
                                                                                                  	// Output is the output of stdout and stderr.
                                                                                                  	// If nil, the output is discarded.
                                                                                                  	Output io.Writer
                                                                                                  
                                                                                                  	// Dir is the directory from which to execute the command.
                                                                                                  	// It is optional. If not specified, it defaults to the directory of
                                                                                                  	// the command, or the work directory if SystemLevel is set.
                                                                                                  	Dir string
                                                                                                  
                                                                                                  	// Args are the arguments to pass to the cmd given to Client.Exec.
                                                                                                  	Args []string
                                                                                                  
                                                                                                  	// ExtraEnv are KEY=VALUE pairs to append to the buildlet
                                                                                                  	// process's environment.
                                                                                                  	ExtraEnv []string
                                                                                                  
                                                                                                  	// Path, if non-nil, specifies the PATH variable of the executed
                                                                                                  	// process's environment. A non-nil empty list clears the path.
                                                                                                  	// The following expansions apply:
                                                                                                  	//   - the string "$PATH" expands to any existing PATH element(s)
                                                                                                  	//   - the substring "$WORKDIR" expands to buildlet's temp workdir
                                                                                                  	// After expansions, the list is joined with an OS-specific list
                                                                                                  	// separator and supplied to the executed process as its PATH
                                                                                                  	// environment variable.
                                                                                                  	Path []string
                                                                                                  
                                                                                                  	// SystemLevel controls whether the command is run outside of
                                                                                                  	// the buildlet's environment.
                                                                                                  	SystemLevel bool
                                                                                                  
                                                                                                  	// Debug, if true, instructs to the buildlet to print extra debug
                                                                                                  	// info to the output before the command begins executing.
                                                                                                  	Debug bool
                                                                                                  
                                                                                                  	// OnStartExec is an optional hook that runs after the 200 OK
                                                                                                  	// response from the buildlet, but before the output begins
                                                                                                  	// writing to Output.
                                                                                                  	OnStartExec func()
                                                                                                  }

                                                                                                    ExecOpts are options for a remote command invocation.

                                                                                                    type KeyPair

                                                                                                    type KeyPair struct {
                                                                                                    	CertPEM string
                                                                                                    	KeyPEM  string
                                                                                                    }

                                                                                                      KeyPair is the TLS public certificate PEM file and its associated private key PEM file that a builder will use for its HTTPS server. The zero value means no HTTPs, which is used by the coordinator for machines running within a firewall.

                                                                                                      func NewKeyPair

                                                                                                      func NewKeyPair() (KeyPair, error)

                                                                                                      func (KeyPair) IsZero

                                                                                                      func (kp KeyPair) IsZero() bool

                                                                                                      func (KeyPair) Password

                                                                                                      func (kp KeyPair) Password() string

                                                                                                        Password returns the SHA1 of the KeyPEM. This is used as the HTTP Basic Auth password.

                                                                                                        type ListDirOpts

                                                                                                        type ListDirOpts struct {
                                                                                                        	// Recursive controls whether the directory is listed
                                                                                                        	// recursively.
                                                                                                        	Recursive bool
                                                                                                        
                                                                                                        	// Skip are the directories to skip, relative to the directory
                                                                                                        	// passed to ListDir. Each item should contain only forward
                                                                                                        	// slashes and not start or end in slashes.
                                                                                                        	Skip []string
                                                                                                        
                                                                                                        	// Digest controls whether the SHA-1 digests of regular files
                                                                                                        	// are returned.
                                                                                                        	Digest bool
                                                                                                        }

                                                                                                          ListDirOpts are options for Client.ListDir.

                                                                                                          type PodOpts

                                                                                                          type PodOpts struct {
                                                                                                          	// ProjectID is the GCE project ID. Required.
                                                                                                          	ProjectID string
                                                                                                          
                                                                                                          	// ImageRegistry specifies the Docker registry Kubernetes
                                                                                                          	// will use to create the pod. Required.
                                                                                                          	ImageRegistry string
                                                                                                          
                                                                                                          	// TLS optionally specifies the TLS keypair to use.
                                                                                                          	// If zero, http without auth is used.
                                                                                                          	TLS KeyPair
                                                                                                          
                                                                                                          	// Description optionally describes the pod.
                                                                                                          	Description string
                                                                                                          
                                                                                                          	// Labels optionally specify key=value strings that Kubernetes
                                                                                                          	// can use to filter and group pods.
                                                                                                          	Labels map[string]string
                                                                                                          
                                                                                                          	// DeleteIn optionally specifies a duration at which
                                                                                                          	// to delete the pod.
                                                                                                          	DeleteIn time.Duration
                                                                                                          
                                                                                                          	// OnPodCreating optionally specifies a hook to run synchronously
                                                                                                          	// after the pod create request has been made, but before the create
                                                                                                          	// has succeeded.
                                                                                                          	OnPodCreating func()
                                                                                                          
                                                                                                          	// OnPodCreated optionally specifies a hook to run synchronously
                                                                                                          	// after the pod create request succeeds.
                                                                                                          	OnPodCreated func()
                                                                                                          
                                                                                                          	// OnGotPodInfo optionally specifies a hook to run synchronously
                                                                                                          	// after the pod Get call.
                                                                                                          	OnGotPodInfo func()
                                                                                                          }

                                                                                                            PodOpts control how new pods are started.

                                                                                                            type RemoteBuildlet

                                                                                                            type RemoteBuildlet struct {
                                                                                                            	HostType    string // "host-linux-jessie"
                                                                                                            	BuilderType string // "linux-386-387"
                                                                                                            	Name        string // "buildlet-adg-openbsd-386-2"
                                                                                                            	Created     time.Time
                                                                                                            	Expires     time.Time
                                                                                                            }

                                                                                                            type Status

                                                                                                            type Status struct {
                                                                                                            	Version int // buildlet version, coordinator rejects any value less than 1.
                                                                                                            }

                                                                                                              Status provides status information about the buildlet.

                                                                                                              A coordinator can use the provided information to decide what, if anything, to do with a buildlet.

                                                                                                              type UserPass

                                                                                                              type UserPass struct {
                                                                                                              	Username string // "user-$USER"
                                                                                                              	Password string // buildlet key
                                                                                                              }

                                                                                                              type VM

                                                                                                              type VM struct {
                                                                                                              	// Name is the name of the GCE VM instance.
                                                                                                              	// For example, it's of the form "mote-bradfitz-plan9-386-foo",
                                                                                                              	// and not "plan9-386-foo".
                                                                                                              	Name   string
                                                                                                              	IPPort string
                                                                                                              	TLS    KeyPair
                                                                                                              	Type   string // buildlet type
                                                                                                              }

                                                                                                              func ListVMs

                                                                                                              func ListVMs(ts oauth2.TokenSource, proj, zone string) ([]VM, error)

                                                                                                                ListVMs lists all VMs.

                                                                                                                type VMOpts

                                                                                                                type VMOpts struct {
                                                                                                                	// Zone is the GCE zone to create the VM in.
                                                                                                                	// Optional; defaults to provided build environment's zone.
                                                                                                                	Zone string
                                                                                                                
                                                                                                                	// ProjectID is the GCE project ID (e.g. "foo-bar-123", not
                                                                                                                	// the numeric ID).
                                                                                                                	// Optional; defaults to provided build environment's project ID ("name").
                                                                                                                	ProjectID string
                                                                                                                
                                                                                                                	// TLS optionally specifies the TLS keypair to use.
                                                                                                                	// If zero, http without auth is used.
                                                                                                                	TLS KeyPair
                                                                                                                
                                                                                                                	// Optional description of the VM.
                                                                                                                	Description string
                                                                                                                
                                                                                                                	// Optional metadata to put on the instance.
                                                                                                                	Meta map[string]string
                                                                                                                
                                                                                                                	// DeleteIn optionally specifies a duration at which
                                                                                                                	// to delete the VM.
                                                                                                                	// If zero, a reasonable default is used.
                                                                                                                	// Negative means no deletion timeout.
                                                                                                                	DeleteIn time.Duration
                                                                                                                
                                                                                                                	// OnInstanceRequested optionally specifies a hook to run synchronously
                                                                                                                	// after the computeService.Instances.Insert call, but before
                                                                                                                	// waiting for its operation to proceed.
                                                                                                                	OnInstanceRequested func()
                                                                                                                
                                                                                                                	// OnInstanceCreated optionally specifies a hook to run synchronously
                                                                                                                	// after the instance operation succeeds.
                                                                                                                	OnInstanceCreated func()
                                                                                                                
                                                                                                                	// OnInstanceCreated optionally specifies a hook to run synchronously
                                                                                                                	// after the computeService.Instances.Get call.
                                                                                                                	// Only valid for GCE resources.
                                                                                                                	OnGotInstanceInfo func(*compute.Instance)
                                                                                                                
                                                                                                                	// OnInstanceCreated optionally specifies a hook to run synchronously
                                                                                                                	// after the EC2 instance information is retrieved.
                                                                                                                	// Only valid for EC2 resources.
                                                                                                                	OnGotEC2InstanceInfo func(*cloud.Instance)
                                                                                                                
                                                                                                                	// OnBeginBuildletProbe optionally specifies a hook to run synchronously
                                                                                                                	// before StartNewVM tries to hit buildletURL to see if it's up yet.
                                                                                                                	OnBeginBuildletProbe func(buildletURL string)
                                                                                                                
                                                                                                                	// OnEndBuildletProbe optionally specifies a hook to run synchronously
                                                                                                                	// after StartNewVM tries to hit the buildlet's URL to see if it's up.
                                                                                                                	// The hook parameters are the return values from http.Get.
                                                                                                                	OnEndBuildletProbe func(*http.Response, error)
                                                                                                                
                                                                                                                	// SkipEndpointVerification does not verify that the builder is listening
                                                                                                                	// on port 80 or 443 before creating a buildlet client.
                                                                                                                	SkipEndpointVerification bool
                                                                                                                }

                                                                                                                  VMOpts control how new VMs are started.