kog

README

Kog - process workflow engine (PWE) executor

A cross-platform CLI to execute a PWE job.

Execute Jobs

Kog supports executing a job either via docker directly or as a kubernetes job

• --executor docker default.
  • Set in DOCKER_HOST environment variable to a valid docker host, for example tcp://localhost:2375
• --executor k8s
  • This uses the KUBECONFIG env var and the current context to determine which cluster and namespace to use

Workflows

There are few workflows checked in under data folder. The simplest workflow to execute is data/test/success.yaml, which executes echo command in alpine container.

kog run -f https://gitlab.com/f5-pwe/kog/raw/master/data/test/success.yaml
kog run -f /local/path/to/workflow.json

The --file|-f flag works for both remote and local files encoded in either json or yaml

Workflow Action Container Specification

1. Containers MUST have an entrypoint that reads the KOG_CONTEXT env var as a base64 encoded json serialized Context
2. Containers MUST print to stdout a prefixed line that contains the json serialized ActionResult
   • eg KOG:{"result":"success","rc":0,"message":"auto success","context":{"action_name":"auto-success","correlation_id":"btzb6nwmekwj","run_id":"btzb6nwmengl","step_name":"start","task_name":"success test"}
   • the context from the ActionResult is passed into the next step in the workflow
3. IF you support the docker executor type your Docker file MUST have an ENV KOG_ENV_CONTEXT=true
   • Example Dockerfile
   • This is just to tell the docker executor to not use stdin to pass the context, this is how it used to be done and I did not want to break backwards compatibility

Context

Every kog job and step has a Context that it is executed within. This context is passed to each workflow step, which can then use this context to configure details about how this step is configured. Each step returns a context that is used in the next step, so it is common to put results from this step into the context before returning it so the next step can use those details

• --context /path/to/context.json

The --context flag works for both remote and local files encoded in either json or yaml

Environment Variables

There are three was to pass env vars to a workflow step

1. --env VARIABLE=something
2. --env-file /path/to/data.env this works for both remote and local files in the dotenv file format
3. --import-current-env you SHOULD always use this with --ignore-env to exclude env vars you do NOT want passed to your workflow step

On docker the env variables are all passed in via --env and is thus inherently insecure, do NOT pass passwords, tokens, etc this way unless you are 100% running on a local docker daemon and are 100% sure the box is secure

On kubernetes all the env variables are saves as a kubernetes Secret then loaded into the job via the normal kubernetes methods. This is only remotely more secure then the docker method, so still do not pass passwords or tokens this way.

Volume Mounts

The docker executor supports mounting volumes into each step action, the symantics are identical to the docker --volume flag

Kubernetes does not yet support volume mounting. If you are interested in this feature feel free to do a MR

Logging

By default, Kog logs to console in logfmt format. It is possible to switch format to JSON by settings --log-formatter flag to json. The log settings will also be propagated to all spawned containers using passed context.

Remote Logging

To enable log aggregation, it is useful to set --remote-logger flag to syslog. You can also specify the remote logger address --remote-logger-addr 10.0.0.15:514 which is useful if you are passing your logs to logstash or some other log aggregation tool Enabling remote logger does not disable local logs, but adds a handler to ship logs to a remote location.

Status Notification

To enable step result notifications, set --notifier flag to either console, or https://url.example.com/kog/notifications This will enable sending a notification on every step start to keep track of a job progress. You can specify multiple notifiers for each run

HTTP Notifiers

passing a url to the notifier flag will enable kog to make an http POST request to that URL with the json serialized pwe.Job object as the body

Return Codes

By default the kog run command will use the job status to map to return codes, use --use-job-rc=false to always return 0 (except for issues with the kog command itself)

• 127 - error with the kog command itself
• Job Status Return Codes

Upgrading and Switching to specific versions

As of v1.4.0 kog supports the upgrade and switch-to commands

• kog upgrade to upgrade to the latest version
• kog switch-to --version #.#.# to switch to a specific version
• kog switch-to --list to list the latest 10 versions and show which version you are on.

$ kog switch-to --list
> 1.5.0
1.4.5
1.4.4
1.4.3
1.4.2
1.4.1
1.4.0

Dev Environment Setup

1. Install Go language on your system. You can use built-in package manager system and look for go or golang. For more advanced instruction, see official documentation.
2. Create the folder for the project, this is important since Go expects packages to be under

mkdir -p ~/projects/f5-pwe
cd ~/projects//f5-pwe

1. Clone this project

git clone https://gitlab.com/f5-pwe/kog

1. Init the dependencies

cd kog
make init

1. Compile and install kog into current directory

make build

1. Verify you can run kog

./bin/kog-$GOOS-$GOARCH run -f https://gitlab.com/f5-pwe/kog/raw/master/data/test/success.yaml

Documentation

Overview

Copyright 2018 F5 Networks

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2018 F5 Networks

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• func DecodeResults(result []byte, logger log.Logger) (actions.ActionResult, error)
• func DecodeStepResults(ctx variables.Context, logger log.Logger) (results []actions.ActionResult, err error)
• func ExecuteStep(executor Executor, step workflow.WorkflowStep, job *pwe.Job, ...) actions.ActionResult
• func FileOrURLReader(taskFilename string, logger log.Logger) (reader io.Reader, err error)
• func NewStepResult(result actions.ActionStatus, message string) actions.ActionResult
• func NextStep(step workflow.WorkflowStep, code actions.ActionStatus) string
• func ParseContext(data []byte) (ctx variables.Context, err error)
• func ParseJob(data []byte) (job *pwe.Job, err error)
• type Executor
  • func NewDockerExecutor(logger log.Logger) (e Executor, err error)
  • func Newk8sExecutor(logger log.Logger) (e Executor, err error)
• type Job
• type JobResult
  • func ExecuteJob(executor Executor, job *pwe.Job, logger log.Logger, notifierHooks *NotifyHooks, ...) (retval JobResult)
• type NotifyHook
  • func NewConsoleHook(logger log.Logger) NotifyHook
  • func NewHTTPNotifyHook(baseURL string, logger log.Logger) NotifyHook
• type NotifyHooks
  • func (h *NotifyHooks) AddNotification(hook NotifyHook)
  • func (h NotifyHooks) FireNotification(job *pwe.Job) error

Functions

func DecodeResults

func DecodeResults(result []byte, logger log.Logger) (actions.ActionResult, error)

func DecodeStepResults

func DecodeStepResults(ctx variables.Context, logger log.Logger) (results []actions.ActionResult, err error)

func ExecuteStep

func ExecuteStep(executor Executor, step workflow.WorkflowStep, job *pwe.Job, ctx variables.Context,
	logger log.Logger, notifierHooks *NotifyHooks, extraEnv map[string]string, volumes []string) actions.ActionResult

func FileOrURLReader

func FileOrURLReader(taskFilename string, logger log.Logger) (reader io.Reader, err error)

func NewStepResult

func NewStepResult(result actions.ActionStatus, message string) actions.ActionResult

func NextStep

func NextStep(step workflow.WorkflowStep, code actions.ActionStatus) string

func ParseContext

func ParseContext(data []byte) (ctx variables.Context, err error)

func ParseJob

func ParseJob(data []byte) (job *pwe.Job, err error)

Types

type Executor

type Executor interface {
	Prepare(job *pwe.Job) error
	CreateJob(inputData, correlationId, jobName, stepName, runId string, action actions.Action, labels map[string]string, volumes []string) error
	GetJob() Job
	AttachIO() error
	SendInput(inputData string) error
	ProcessOutput(resultText *string)
	Wait(timeout time.Duration) (int, error)
	Cleanup() error
}

func NewDockerExecutor

func NewDockerExecutor(logger log.Logger) (e Executor, err error)

func Newk8sExecutor

func Newk8sExecutor(logger log.Logger) (e Executor, err error)

type Job

type Job interface {
	ID() string
	Input(input io.WriteCloser) io.WriteCloser
	Output(output io.ReadCloser) io.ReadCloser
}

type JobResult

type JobResult struct {
	ID            string        `json:"correlation_id" yaml:"correlation_id"`
	Name          string        `json:"task_name" yaml:"task_name"`
	Status        pwe.JobStatus `json:"status" yaml:"status"`
	Message       string        `json:"message" yaml:"message"`
	NotifyStatus  string        `json:"notify_status" yaml:"notify_status"`
	NotifyMessage string        `json:"notify_message" yaml:"notify_message"`
}

func ExecuteJob

func ExecuteJob(executor Executor, job *pwe.Job, logger log.Logger, notifierHooks *NotifyHooks, extraEnv map[string]string, volumes []string) (retval JobResult)

type NotifyHook

type NotifyHook interface {
	FireNotification(job *pwe.Job) error
	Logger() log.Logger
}

func NewConsoleHook

func NewConsoleHook(logger log.Logger) NotifyHook

func NewHTTPNotifyHook

func NewHTTPNotifyHook(baseURL string, logger log.Logger) NotifyHook

type NotifyHooks

type NotifyHooks []NotifyHook

func (*NotifyHooks) AddNotification

func (h *NotifyHooks) AddNotification(hook NotifyHook)

func (NotifyHooks) FireNotification

func (h NotifyHooks) FireNotification(job *pwe.Job) error