dexec

package module
v0.1.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 21, 2023 License: Apache-2.0 Imports: 21 Imported by: 0

README

dexec GoDoc

dexec is a small Go library allowing you to run processes inside Docker containers as if you are running them locally using os/exec package. Read documentation at godoc.org or see examples.

Using dexec, you can do stream processing, off-load computationally expensive parts of your program to a remote fleet of Docker engines. Its interface is strikingly similar to os/exec.

Examples

Check out the following examples:

A Short Example

It takes only a 4-line code change to convert a piece of code using os/exec to use dexec to start running your stuff inside containers.

Here is a minimal Go program that runs echo in a container:

package main

import (
	"log"

	"github.com/ahmetalpbalkan/dexec"
	"github.com/fsouza/go-dockerclient"
)

func main(){
	cl, _ := docker.NewClient("unix:///var/run/docker.sock")
	d := dexec.Docker{cl}

	m, _ := dexec.ByCreatingContainer(docker.CreateContainerOptions{
	Config: &docker.Config{Image: "busybox"}})

	cmd := d.Command(m, "echo", `I am running inside a container!`)
	b, err := cmd.Output()
	if err != nil { log.Fatal(err) }
	log.Printf("%s", b)
}

Output: I am running inside a container!

Use Cases

This library is intended for providing an execution model that looks and feels like os/exec package.

You might want to use this library when:

  • You want to execute a process, but run it in a container with extra security and finer control over resource usage with Docker –and change your code minimally.

  • You want to execute a piece of work on a remote machine (or even better, a pool of machines or a cluster) through Docker. Especially useful to distribute computationally expensive workloads.

For such cases, this library abstracts out the details of executing the process in a container and gives you a cleaner interface you are already familiar with.

Check out more examples →

Read more →

Analytics

Documentation

Overview

Package dexec provides an interface similar to os/exec to run external commands inside containers.

Please read documentation carefully about semantic differences between os/exec and dexec.

Use Case

This utility is intended to provide an execution environment without changing the existing code a lot to execute a command on a remote machine (or a pool of machines) running Docker engine or locally to limit resource usage and have extra security.

Dependencies

The package needs the following dependencies to work: 

go get github.com/fsouza/go-dockerclient

Known issues

- You may receive empty stdout/stderr from commands if the executed command does not end with a trailing new line or has a different flushing behavior.

- StdinPipe/StdoutPipe/StderrPipe should be used with goroutines (one that executes Cmd.Wait() and another one that writes to/reads from the pipe. Otherwise, the code may hang 10% of the time due to some timing issue.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RandomString added in v0.1.2 

func RandomString(n int) string

Types

type Cmd 

type Cmd interface {
	// StdoutPipe returns a pipe that will be connected to the command's standard output when
	// the command starts.
	//
	// Wait will close the pipe after seeing the command exit or in error conditions.
	StdoutPipe() (io.ReadCloser, error)
	// StderrPipe returns a pipe that will be connected to the command's standard error when
	// the command starts.
	//
	// Wait will close the pipe after seeing the command exit or in error conditions.
	StderrPipe() (io.ReadCloser, error)
	// SetStderr sets the stderr writer
	SetStderr(writer io.Writer)
	// StdinPipe returns a pipe that will be connected to the command's standard input
	// when the command starts.
	//
	// Different than os/exec.StdinPipe, returned io.WriteCloser should be closed by user.
	StdinPipe() (io.WriteCloser, error)
	// GetPID will return a unique identifier for the running command. The meaning of the identifier
	// may be implementation specific
	GetPID() string
	// Start starts the specified command but does not wait for it to complete.
	Start() error
	// Wait waits for the command to exit. It must have been started by Start.
	//
	// If the container exits with a non-zero exit code, the error is of type
	// *ExitError. Other error types may be returned for I/O problems and such.
	//
	// Different than os/exec.Wait, this method will not release any resources
	// associated with Cmd (such as file handles).
	Wait() error
	// Kill will stop a running command
	Kill() error
	// Run starts the specified command and waits for it to complete.
	//
	// If the command runs successfully and copying streams are done as expected,
	// the error is nil.
	//
	// If the container exits with a non-zero exit code, the error is of type
	// *ExitError. Other error types may be returned for I/O problems and such.
	Run() error
	// Output runs the command and returns its standard output.
	//
	// If the container exits with a non-zero exit code, the error is of type
	// *ExitError. Other error types may be returned for I/O problems and such.
	//
	// If c.Stderr was nil, Output populates ExitError.Stderr.
	Output() ([]byte, error)
	// CombinedOutput runs the command and returns its combined standard output and
	// standard error.
	//
	// Docker API does not have strong guarantees over ordering of messages. For instance:
	//     >&1 echo out; >&2 echo err
	// may result in "out\nerr\n" as well as "err\nout\n" from this method.
	CombinedOutput() ([]byte, error)
	// SetDir sets the working directory for the command
	SetDir(dir string)
	// Cleanup cleans up any resources that were created for the command
	Cleanup() error
}

func Command added in v0.1.2 

func Command(client interface{}, config Config) Cmd

type CommandDetails added in v0.1.2 

type CommandDetails struct {
	ExecutorId      int64
	ChainExecutorId int64
	ResultId        int64
}

type Config added in v0.1.2 

type Config struct {
	ContainerConfig ContainerConfig
	NetworkConfig   NetworkConfig
	TaskConfig      TaskConfig
	CommandDetails  CommandDetails
	Logger          *logrus.Entry
	NewRelic        *newrelic.Application
	Namespace       string
}

type ContainerClient added in v0.1.2 

type ContainerClient interface {
	Docker | Containerd
}

type ContainerConfig added in v0.1.2 

type ContainerConfig struct {
	Image  string
	User   string
	Env    []string
	Mounts []Mount
}

type Containerd added in v0.1.2 

type Containerd struct {
	ContainerdClient
	Namespace string
}

func (Containerd) Command added in v0.1.2 

func (c Containerd) Command(method Execution[Containerd], name string, arg ...string) *ContainerdCmd

type ContainerdClient added in v0.1.5 

type ContainerdClient interface {
	WithLease(context.Context, ...leases.Opt) (context.Context, func(ctx context.Context) error, error)
	IsServing(context.Context) (bool, error)
	LoadContainer(context.Context, string) (containerd.Container, error)
	Containers(context.Context, ...string) ([]containerd.Container, error)
	Reconnect() error
}

ContainerdClient is the interface spec of all the calls we use from *containerd.Client . Having the interface makes it easier to mock the client in tests

type ContainerdCmd added in v0.1.2 

type ContainerdCmd struct {
	*GenericCmd[Containerd]
}

type CreateTaskOptions added in v0.1.2 

type CreateTaskOptions struct {
	Image          string
	Mounts         []specs.Mount
	User           string
	Env            []string
	CommandTimeout time.Duration
	WorkingDir     string
	CommandDetails CommandDetails
}

type Docker 

type Docker struct {
	*docker.Client
}

Docker contains connection to Docker API. Use github.com/fsouza/go-dockerclient to initialize *docker.Client.

func (Docker) Command 

func (d Docker) Command(method Execution[Docker], name string, arg ...string) *DockerCmd

Command returns the Cmd struct to execute the named program with given arguments using specified execution method.

For each new Cmd, you should create a new instance for "method" argument.

type DockerCmd added in v0.1.2 

type DockerCmd struct {
	*GenericCmd[Docker]
}

DockerCmd represents an external command being prepared or run.

A DockerCmd cannot be reused after calling its Run, Output or CombinedOutput methods.

type Execution 

type Execution[T ContainerClient] interface {
	// contains filtered or unexported methods
}

Execution determines how the command is going to be executed. Currently the only method is ByCreatingContainer.

func ByCreatingContainer 

func ByCreatingContainer(opts docker.CreateContainerOptions) (Execution[Docker], error)

ByCreatingContainer is the execution strategy where a new container with specified options is created to execute the command.

The container will be created and started with Cmd.Start and will be deleted before Cmd.Wait returns.

func ByCreatingTask added in v0.1.2 

func ByCreatingTask(opts CreateTaskOptions, logger *logrus.Entry) (Execution[Containerd], error)

type ExitError 

type ExitError struct {
	// ExitCode holds the non-zero exit code of the container
	ExitCode int

	// Stderr holds the standard error output from the command
	// if it *Cmd executed through Output() and Cmd.Stderr was not
	// set.
	Stderr []byte
}

ExitError reports an unsuccessful exit by a command.

func (*ExitError) Error 

func (e *ExitError) Error() string

type GenericCmd added in v0.1.2 

type GenericCmd[T ContainerClient] struct {
	// Path is the path or name of the command in the container.
	Path string

	// Arguments to the command in the container, excluding the command
	// name as the first argument.
	Args []string

	// Env is environment variables to the command. If Env is nil, Run will use
	// Env specified on Method or pre-built container image.
	Env []string

	// Dir specifies the working directory of the command. If Dir is the empty
	// string, Run uses Dir specified on Method or pre-built container image.
	Dir string

	// Stdin specifies the process's standard input.
	// If Stdin is nil, the process reads from the null device (os.DevNull).
	//
	// Run will not close the underlying handle if the Reader is an *os.File
	// differently than os/exec.
	Stdin io.Reader

	// Stdout and Stderr specify the process's standard output and error.
	// If either is nil, they will be redirected to the null device (os.DevNull).
	//
	// Run will not close the underlying handles if they are *os.File differently
	// than os/exec.
	Stdout io.Writer
	Stderr io.Writer

	Method Execution[T]

	NewRelic *newrelic.Application
	// contains filtered or unexported fields
}

func (*GenericCmd[T]) Cleanup added in v0.1.2 

func (g *GenericCmd[T]) Cleanup() error

Cleanup cleans up any resources that were created for the command

func (*GenericCmd[T]) CombinedOutput added in v0.1.2 

func (g *GenericCmd[T]) CombinedOutput() ([]byte, error)

CombinedOutput runs the command and returns its combined standard output and standard error.

Docker API does not have strong guarantees over ordering of messages. For instance: 

>&1 echo out; >&2 echo err

may result in "out\nerr\n" as well as "err\nout\n" from this method.

func (*GenericCmd[T]) GetPID added in v0.1.2 

func (g *GenericCmd[T]) GetPID() string

GetPID will return the container ID of the Cmd's running container. This is useful for when we need to cleanup the process before completion or store its container ID

func (*GenericCmd[T]) Kill added in v0.1.2 

func (g *GenericCmd[T]) Kill() error

Kill will stop a running container

func (*GenericCmd[T]) Output added in v0.1.2 

func (g *GenericCmd[T]) Output() ([]byte, error)

Output runs the command and returns its standard output.

If the container exits with a non-zero exit code, the error is of type *ExitError. Other error types may be returned for I/O problems and such.

If c.Stderr was nil, Output populates ExitError.Stderr.

func (*GenericCmd[T]) Run added in v0.1.2 

func (g *GenericCmd[T]) Run() error

Run starts the specified command and waits for it to complete.

If the command runs successfully and copying streams are done as expected, the error is nil.

If the container exits with a non-zero exit code, the error is of type *ExitError. Other error types may be returned for I/O problems and such.

func (*GenericCmd[T]) SetDir added in v0.1.2 

func (g *GenericCmd[T]) SetDir(dir string)

SetDir sets the working directory for the command

func (*GenericCmd[T]) SetStderr added in v0.1.2 

func (g *GenericCmd[T]) SetStderr(writer io.Writer)

SetStderr sets the stderr writer

func (*GenericCmd[T]) Start added in v0.1.2 

func (g *GenericCmd[T]) Start() error

Start starts the specified command but does not wait for it to complete.

func (*GenericCmd[T]) StderrPipe added in v0.1.2 

func (g *GenericCmd[T]) StderrPipe() (io.ReadCloser, error)

StderrPipe returns a pipe that will be connected to the command's standard error when the command starts.

Wait will close the pipe after seeing the command exit or in error conditions.

func (*GenericCmd[T]) StdinPipe added in v0.1.2 

func (g *GenericCmd[T]) StdinPipe() (io.WriteCloser, error)

StdinPipe returns a pipe that will be connected to the command's standard input when the command starts.

Different than os/exec.StdinPipe, returned io.WriteCloser should be closed by user.

func (*GenericCmd[T]) StdoutPipe added in v0.1.2 

func (g *GenericCmd[T]) StdoutPipe() (io.ReadCloser, error)

StdoutPipe returns a pipe that will be connected to the command's standard output when the command starts.

Wait will close the pipe after seeing the command exit or in error conditions.

func (*GenericCmd[T]) Wait added in v0.1.2 

func (g *GenericCmd[T]) Wait() error

Wait waits for the command to exit. It must have been started by Start.

If the container exits with a non-zero exit code, the error is of type *ExitError. Other error types may be returned for I/O problems and such.

Different than os/exec.Wait, this method will not release any resources associated with Cmd (such as file handles).

type Mount added in v0.1.2 

type Mount struct {
	Type        string
	Source      string
	Destination string
	Options     []string
}

type NetworkConfig added in v0.1.2 

type NetworkConfig struct {
	DNS        []string
	DNSSearch  []string
	DNSOptions []string
}

type Stats added in v0.1.5 

type Stats struct {
	Running          int
	Created          int
	Stopped          int
	Paused           int
	Pausing          int
	Unknown          int
	DeadlineExceeded int
	Errors           int
}

func GetStats added in v0.1.5 

func GetStats(client interface{}) (Stats, error)

type TaskConfig added in v0.1.2 

type TaskConfig struct {
	Executable string
	Args       []string
	Timeout    time.Duration
	WorkingDir string
}

Source Files

View all Source files

Directories

Path Synopsis
examples
100-hello
200-stdin-stdout
300-pipes
400-exit-code
500-video-processing
600-parallel-compute

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.