prox

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 11, 2020 License: BSD-2-Clause Imports: 25 Imported by: 0

README

Prox Go GitHub release License GoDoc

Notes: this is a pre release. There is no promise regarding API stability or breaking changes

Prox is a process runner for Procfile-based applications inspired by foreman. With prox you can run several processes defined in a Procfile concurrently within a single terminal. All process outputs are merged but prefixed with their corresponding process names. One of the major use cases for this is the local development of an application that consist of multiple processes (e.g. microservices and storage backends). With a process runner you can easily start this "stack" of applications and inspect its output in a single shell while interacting with the application.

You may ask why not just use docker for local development since it provides similar functionality to run multiple processes, especially when using docker-compose. The reason is ease of development and a fast development cycle also for small code changes. It just takes longer than necessary to recompile a binary and additionally build the docker image. Also the extra file system and process isolation that are one of dockers many benefits in a production environment can become quite a nuisance during local development.

Features

Prox primary use case is as a development tool to run your entire application stack on your local machine. Apart from running all components, Proxs primary goal is to help you understand what the application is doing and sometimes help to debug why a component has crashed.

Error reporting

Like other process managers, prox will stop the entire stack when one of the managed processes has crashed. This way the system fails fast and it is the developers task to understand and fix the problem. This usually entails searching through the log output for the first fatal error which caused the system to go down. Prox helps with this by reporting the name and exit code of the process that was the root cause for the stack shutdown.

Log parsing

Today it is good practice for applications to emit structured log output so it can be parsed and used easily. Prox detects if a process encodes its logs as JSON and can use this information to reformat and color the output. By default this is used to highlight error messages but the user can specify custom formatting as well.

In the future log parsing can also be used during error reporting to print the last error message of the component which crashed the stack.

Prox Server

Another thing that distinguishes Prox from other foreman clones is that when prox starts the application, it will also listen on a unix socket for requests. This makes it possible to interact with a running Prox instance in another terminal, for instance to tail the logs of a subset of processes. This can be useful when working with many processes where the merged output of all applications can be rather spammy and is hard to be read by humans.

The current version of the prox server only implements tailing but you can take a look at the IDEAS.md file for other functionality that might be implemented later on.

Proxfile

Advanced users can use a slightly more complicated Proxfile which serves as an opt-in alternative to the Procfile but with more features (see usage below).

Installation

All you need to run prox is a single binary. You can either use a prebuilt binary or build prox from source.

Prebuilt binaries

Download the binary from the releases page and extract the prox binary somewhere in your path.

Building from source

If you want to compile prox from source you need a working installation of Go version 1.9 or greater.

You can either install prox directly via go get or use the make install target. The preferred way is the installation via make since this compiles version information into the prox binary. You can inspected the version of your prox binary via prox version. This is helpful when reporting issues and debugging but it is otherwise of no use.

go get -v github.com/fgrosse/prox/cmd/prox
cd $GOPATH/src/github.com/fgrosse/prox
make install

Usage

You always need a Procfile or Proxfile which defines all processes that you want to run.

Simple Procfile usage
$ cat Procfile
# You can use comments, empty lines are ignored as well
worker: my-worker -v /etc/foo # prox uses your $PATH and passes arguments and flags as expected

foo-service: CONFIG_DIR=$PWD/config foo-serve # You can set and use environment variables per job
bar-service: bar-serve # Additionally all processes inherit your shells environment
baz-service: baz-serve # If there is a .env file it will be used to set variables for all processes

Optionally you can create a .env file which must contain a new-line delimited list of key=value pairs which specify additional environment variables that are exported to all processes defined within the Procfile.

$ cat .env
NAMESPACE=production
FOO_URL=file://home/fgrosse/src/github.com/foo/bar

# Again you may use empty lines and comments starting with '#'
ETCD_ENDPOINT=localhost:2379
LOG=*:debug,xxx:info,cache:error,db:info

# You can also use environment variables that you have defined earlier or that
# are defined in the shell that started prox.
PATH=/etc/foo/$NAMESPACE/baz:$PATH

# Spaces are allowed in values without any extra quoting
GREETING=hello world

Then change into the directory which contains your Procfile and .env and start prox.

$ prox
echo1    │ I am a process
echo2    │ Hello World
redis    │ 14773:C 03 Oct 21:17:26.487 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis    │ 14773:C 03 Oct 21:17:26.487 # Redis version=4.0.1, bits=64, commit=00000000, modified=0, pid=14773, just started
…

In order to follow the logs of a specific process open another terminal.

prox tail redis
redis    │ 15249:M 03 Oct 21:21:13.044 # Server initialized
redis    │ 15249:M 03 Oct 21:21:13.045 * DB loaded from disk: 0.000 seconds
redis    │ 15249:M 03 Oct 21:21:13.045 * Ready to accept connections
…

For a detailed description of all prox commands and flags refer to the output of prox help.

Advanced Proxfile usage

Instead of using a standard Procfile and .env file you can combine both in a Proxfile. Additionally this gives you access to more features such as custom coloring of structured log output.

$ cat Proxfile
version: 1 # The Proxfile file format is versioned

# Internally the Proxfile is parsed as YAML.
# You can use comments, empty lines are ignored as well.

processes:
  redis: redis-server # Like the Procfile you specify processes as "name: shell script"

  foo-service:
    script: foo-service --debug -a -b 42
    env:
      - "CONFIG_DIR=$PWD/config foo-serve"

  echo:
    script: "echo $LISTEN_ADDR"
    env:
      - "LISTEN_ADDR=localhost:1232"

  example-3:
    script: my-app
    tags:
      errors:
        color: red
        condition:
          field: level
          value: "/error|fatal/i"

Similar Projects

  • foreman: the original process runner by David Dollar
  • forego: a 1-1 port of foreman to Go
  • goreman: another clone of foreman with some undocumented RPC functionality via TCP ports (Go)
  • honcho: a Python port of foreman
  • spm: Simple Process Manager with client/server communication via unix sockets (Go)
  • overmind: a process manager for Procfile-based applications that relies on tmux sessions
  • and more …

Dependencies

Prox uses go mod as dependency management tool. All vendored dependencies are specified in the go.mod file and their licenses are copied into LICENSE-THIRD-PARTY. Prox itself mainly relies on the Go standard library, zap for logging, cobra/viper for the CLI and pkg/errors for error wrapping as well as hashicorp/go-multierror.

License

Prox is licensed under the BSD 2-clause License. Please see the LICENSE file for details. The individual licenses of the vendored dependencies can be found in the LICENSE-THIRD-PARTY file.

Contributing

Contributions are always welcome (use pull requests). Before you start working on a bigger feature its always best to discuss ideas in a new github issue. For each pull request make sure that you covered your changes and additions with unit tests.

Please keep in mind that I might not always be able to respond immediately but I usually try to react within a week or two ☺.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NewLogger 

func NewLogger(w io.Writer, debug bool) *zap.Logger

NewLogger creates a *zap.Logger that emits its log messages through the given io.Writer. If debug is true the log level will also include debug and info messages. Otherwise only warning and errors will be logged.

func TestNewServerAndClient 

func TestNewServerAndClient(t TestReporter, w io.Writer) (server *Server, client *Client, executor *TestExecutor, done func())

func Validate added in v0.3.0 

func Validate(pp []Process) error

Validate checks if all given processes are valid and no process name is used multiple times. If an error is returned it will be a multierror.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

A Client connects to a Server via a unix socket to provide access to a running prox server.

func NewClient 

func NewClient(socketPath string, debug bool) (*Client, error)

NewClient creates a new prox Client and immediately connects it to a prox Server via a unix socket. It is the callers responsibility to eventually close the client to release the underlying socket connection.

func (*Client) Close 

func (c *Client) Close() error

Close closes the socket connection to the prox server.

func (*Client) List 

func (c *Client) List(ctx context.Context, output io.Writer) error

List fetches a list of running processes from the server and prints it via the given output.

func (*Client) Tail 

func (c *Client) Tail(ctx context.Context, processNames []string, output io.Writer) error

Tail requests and "follows" the logs for a set of processes from a server and prints them to the output. This function blocks until the context is done or the connection to the server is closed by either side.

type Environment 

type Environment map[string]string

Environment is a set of key value pairs that are used to set environment variables for processes.

func NewEnv 

func NewEnv(values []string) Environment

NewEnv creates a new Environment and immediately sets all given key=value pairs.

func SystemEnv 

func SystemEnv() Environment

SystemEnv creates a new Environment from the operating systems environment variables.

func (Environment) Expand 

func (e Environment) Expand(input string) string

Expand replaces ${var} or $var in the input string with the corresponding values of e. If the variable is not found in e then an empty string is returned.

func (Environment) Get added in v0.3.0 

func (e Environment) Get(key, defaultValue string) string

Get retrieves the key from the Environment or returns the given default value if that key was not set

func (Environment) List 

func (e Environment) List() []string

List returns all variables of e as a list of key=value pairs.

func (Environment) ParseEnvFile 

func (e Environment) ParseEnvFile(r io.Reader) error

ParseEnvFile reads environment variables that should be set on all processes from the ".env" file.

The format of the ".env" file is expected to be a newline separated list of key=value pairs which represent the environment variables that should be used by all started processes. Trimmed lines which are empty or start with a "#" are ignored and can be used to add comments.

All values are expanded using the Environment. Additionally values in the env file can use other values which have been defined in earlier lines above. If a value refers to an unknown variable then it is replaced with the empty string.

func (Environment) Set 

func (e Environment) Set(s string)

Set splits the input string at the first "=" character (if any) and sets the resulting key and value on e.

func (Environment) SetAll 

func (e Environment) SetAll(vars []string)

SetAll assigns a list of key=value pairs on e.

type Executor 

type Executor struct {
	// contains filtered or unexported fields
}

An Executor manages a set of processes. It is responsible for running them concurrently and waits until they have finished or an error occurs.

func NewExecutor 

func NewExecutor(debug bool) *Executor

NewExecutor creates a new Executor. The debug flag controls whether debug logging should be activated. If debug is false then only warnings and errors will be logged.

func (*Executor) DisableColoredOutput 

func (e *Executor) DisableColoredOutput()

DisableColoredOutput disables colored prefixes in the output.

func (*Executor) Info 

func (e *Executor) Info(processName string) ProcessInfo

Info returns information about a running process. If there is no such process running process a ProcessInfo with a PID of -1 is returned.

func (*Executor) Run 

func (e *Executor) Run(ctx context.Context, processes []Process) error

Run starts all processes and blocks until all processes have finished or the context is done (e.g. canceled). If a process crashes or the context is canceled early, all running processes receive an interrupt signal.

type Process 

type Process struct {
	Name   string
	Script string
	Env    Environment
	Output StructuredOutput // optional
}

Process holds all information about a process that is executed by prox.

func ParseProcFile 

func ParseProcFile(reader io.Reader, env Environment) ([]Process, error)

ParseProcFile parses a Procfile from the given reader and returns the corresponding set of Processes, each configured with the given Environment.

A Procfile defines one process per line. TODO: write more about the format TODO: comments and empty lines

func ParseProxFile 

func ParseProxFile(reader io.Reader, env Environment) ([]Process, error)

func (Process) CommandLine 

func (p Process) CommandLine() ([]string, error)

CommandLine returns the shell command line that would be executed when the given Process is started.

func (Process) Validate added in v0.3.0 

func (p Process) Validate() error

Validate checks that the Process configuration is complete and without errors. If an error is returned it will be a multierror.

type ProcessInfo 

type ProcessInfo struct {
	Name   string
	PID    int
	Uptime time.Duration
}

ProcessInfo contains information about a running process.

type Proxfile 

type Proxfile struct {
	Version   string
	Processes map[string]ProxfileProcess
}

type ProxfileProcess 

type ProxfileProcess struct {
	Script string
	Env    []string

	Format string // e.g. json
	Fields struct {
		Message string
		Level   string
	}
	Tags map[string]struct {
		Color     string
		Condition struct {
			Field string
			Value string
		}
	}
}

func (*ProxfileProcess) UnmarshalYAML 

func (p *ProxfileProcess) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements the gopkg.in/yaml.v2.Unmarshaler interface.

type Server 

type Server struct {
	*Executor
	// contains filtered or unexported fields
}

A Server wraps an Executor to expose its functionality via a unix socket.

func NewExecutorServer 

func NewExecutorServer(socketPath string, debug bool) *Server

NewExecutorServer creates a new Server. This function does not start the Executor nor does it listen on the unix socket just yet. To start the Server and Executor the Server.Run(…) function must be used.

func (*Server) Close 

func (s *Server) Close() error

Close closes the Servers listener.

func (*Server) Run 

func (s *Server) Run(ctx context.Context, pp []Process) error

Run opens a unix socket using the path that was passed via NewExecutor and then starts the Executor. It is the callers responsibility to eventually call Server.Close() in order to close the unix socket connect.

type StructuredOutput added in v0.2.0 

type StructuredOutput struct {
	Format       string // e.g. "json", the zero value makes prox auto-detect the format
	MessageField string
	LevelField   string

	TaggingRules []TaggingRule
	TagColors    map[string]string
}

StructuredOutput contains all configuration to setup advanced functionality for structured logs.

func DefaultStructuredOutput added in v0.2.0 

func DefaultStructuredOutput(env Environment) StructuredOutput

DefaultStructuredOutput returns the default configuration for processes that do not specify structured log output specifically.

type TaggingRule 

type TaggingRule struct {
	Field string
	Value string // either a concrete string or a regex like so: "/error|fatal/i"
	Tag   string
}

A TaggingRule may be applied to a structured log message to tag it. These tags can then later be used to change the log lines appearance or to modify its content.

type TestExecutor 

type TestExecutor struct {
	*Executor

	Error error
	// contains filtered or unexported fields
}

func TestNewExecutor 

func TestNewExecutor(w io.Writer) *TestExecutor

func (*TestExecutor) IsDone 

func (e *TestExecutor) IsDone() bool

func (*TestExecutor) Run 

func (e *TestExecutor) Run(processes ...*TestProcess)

func (*TestExecutor) Stop 

func (e *TestExecutor) Stop()

type TestProcess 

type TestProcess struct {
	PID    int
	Uptime time.Duration
	// contains filtered or unexported fields
}

func (*TestProcess) Fail 

func (p *TestProcess) Fail()

func (*TestProcess) Finish 

func (p *TestProcess) Finish()

func (*TestProcess) FinishInterrupt 

func (p *TestProcess) FinishInterrupt()

func (*TestProcess) HasBeenInterrupted 

func (p *TestProcess) HasBeenInterrupted() bool

func (*TestProcess) HasBeenStarted 

func (p *TestProcess) HasBeenStarted() bool

func (*TestProcess) Info 

func (p *TestProcess) Info() ProcessInfo

func (*TestProcess) Name 

func (p *TestProcess) Name() string

func (*TestProcess) Run 

func (p *TestProcess) Run(ctx context.Context) error

func (*TestProcess) ShouldBlockOnInterrupt 

func (p *TestProcess) ShouldBlockOnInterrupt()

func (*TestProcess) ShouldSay 

func (p *TestProcess) ShouldSay(t TestReporter, msg string)

func (*TestProcess) String 

func (p *TestProcess) String() string

type TestReporter 

type TestReporter interface {
	Log(args ...interface{})
	Fatal(args ...interface{})
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
prox

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.