yaml

README

YAML CLI processor

A CLI tool for querying and transforming YAML data: Grep matching objects, join YAML documents, get/add/edit/delete YAML nodes matching given selector, loop over objects and/or data arrays etc.

[input.yml] => [query or transformations] => [output.yml]

Note: The input YAML data might contain multiple YAML documents separated by ---.

• One-liner commands
  • yaml get selector
  • yaml print selector
  • yaml set "selector: value"
  • yaml default "selector: value"
  • yaml delete selector
  • yaml cat file1.yml file2.yml fileN.yml
  • yaml count
  • yaml doc $index
  • yaml len arraySelector
• Grep YAML documents (objects)
  • yaml grep "selector: value" ...
    • Grep k8s deployment object by name
  • yaml grep -v "selector: value" ...
    • Grep all k8s objects that don't create any Pods
    • Print first container's image of linkerd2 deployment objects
• Useful Kubernetes examples
  • Push all non-pod objects to k8s
  • Rollout k8s deployments from desired-state files sequentially
• Known issues
• Transformation files
  • yaml apply file1.yt file2.yt fileN.yt
    • Examples of transformation YAML files
• Feedback
• License

One-liner commands

yaml get selector

Get value of a node matching the given selector.

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml get spec.containers[0].image
nats-streaming:0.10.0

yaml print selector

Print full node matching the given selector.

$ kubectl get pod/nats-8576dfb67-vg6v7 -o yaml | yaml print kind apiVersion metadata.annotations
kind: Pod
apiVersion: v1
metadata.annotations:
  kubernetes.io/psp: eks.privileged

yaml set "selector: value"

Add/overwrite field's value.

$ cat input.yml | yaml set "metadata.labels.environment: staging" > output.yml

yaml default "selector: value"

Set field's value, if no such value exists yet.

$ cat input.yml | yaml default "metadata.labels.environment: staging" > output.yml

yaml delete selector

Delete specific field.

$ cat input.yml | yaml delete "metadata.labels.environment" > output.yml

yaml cat file1.yml file2.yml fileN.yml

Join multiple YAML files into a single file with multiple documents separated by ---.

$ yaml cat k8s-apps/*.yml > output.yml

yaml count

Print number of YAML documents within the input YAML stream.

input.yml

document: this is doc 1
---
document: this is doc 2

$ yaml count input.yml
2

yaml doc $index

Print Nth (index=0..N-1) YAML document from the input YAML stream.

input.yml

document: this is doc 1
---
document: this is doc 2

$ yaml doc 1
document: this is doc 2

yaml len arraySelector

Print number of items within an array matching the given selector.

Useful for ranging over arrays.

pods=$(kubectl get pods -o yaml)
count=$(echo "$pods" | yaml len items)
for ((i=0; i < $count; i++)); do
    echo "$pods" | yaml get items[$i].status.phase
done

Grep YAML documents (objects)

Grep documents/objects matching all of the given selector: value pairs.

If a provided value is an array (ie. selector: [first, second]), the matching value must match at least one of the provided values (logical "OR").

yaml grep "selector: value" ...

Grep k8s deployment object by name

$ cat desired-state.yml | yaml grep "kind: Deployment" "metadata.name: linkerd"

yaml grep -v "selector: value" ...

Inverse grep.

Grep all k8s objects that don't create any Pods

$ cat desired-state.yml | yaml grep -v "kind: [Deployment, Pod, Job, ReplicaSet, ReplicationController]"

Print first container's image of linkerd2 deployment objects

$ cat linkerd.yml | yaml grep "kind: Deployment" | yaml get "spec.template.spec.containers[0].image"
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/web:stable-2.4.0
prom/prometheus:v2.10.0
gcr.io/linkerd-io/grafana:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0
gcr.io/linkerd-io/controller:stable-2.4.0

Useful Kubernetes examples

Push all non-pod objects to k8s

$ cat desired-state.yml | yaml grep -v "kind: [Deployment, Pod, Job]"

Rollout k8s deployments from desired-state files sequentially

$ for file in *.yml; do
    out=$(cat $file | yaml grep "kind: Deployment" | kubectl apply -f -)

    for deploy in $(echo "$out" | cut -d' ' -f1); do
        kubectl rollout status --timeout 180s $deploy || {
            kubectl rollout undo $deploy
            exit 1
        }
    done
  done

Known issues

1. Merging complex nodes doesn't work well

set:
    metadata:
        we:
            cant:
                merge:
                    complex: objects
                    such:
                        as: this
                        properly:
                            just: yet

We'll want to fix this later. For now, use explicit paths to the final nodes:

set:
    metadata.we.cant.merge.complex: objects
    metadata.we.cant.merge.such.as: this
    metadata.we.cant.merge.such.properly.just: yet

1. Wildcard array[*] matching doesn't work yet

match:
    spec.template.spec.containers[*].name: prometheus

1. Selectors with . dots in the selector path, ie.

metadata:
    annotations:
        linkerd.io/inject: enabled

Since the match selectors are separated with . dots, we'll have to figure out how to support these selector keys with inner . dots.

We might wanna support

delete:
    - metadata.annotations."linkerd.io/inject"
set:
    metadata.annotations."rbac.authorization.kubernetes.io/autoupdate": true

Transformation files

All of the above examples, and more, can be described in YAML transformation file syntax. Multiple such transformations can be applied at once.

yaml apply file1.yt file2.yt fileN.yt

Apply multiple YAML "transformations", see the .yt file syntax below.

$ yaml cat k8s-apps/*.yml | yaml apply staging.yt enable-linkerd.yt > staging/desired-state.yml

staging.yt:

match:
    # all YAML objects
set:
    metadata.labels.environment: staging
---
match:
    kind: Deployment
    metadata.name: api
set:
    metadata.labels.first: updated-label
    spec.replicas: 3

enable-linkerd.yt:

match:
  kind: [Deployment, Pod]
default:
  metadata.annotations:
    linkerd.io/inject: enabled

Changes applied to the original object:

 apiVersion: apps/v1
 kind: Deployment
 metadata:
     name: api
     labels:
-        first: label
+        first: updated-label
         second: label
+        environment: staging
+    annotations:
+        linkerd.io/inject: enabled
 spec:
-    replicas: 1
+    replicas: 3
 ...

Examples of transformation YAML files

match:
    kind: Deployment
set:
    spec.template.spec.containers[*]:
        imagePullPolicy: IfNotPresent

match:
    kind: Deployment
set:
    spec.template.spec.nodeSelector:
        worker: generic

match:
    kind: Deployment
    metadata.name: api
set:
    spec.replicas: 3

match:
    kind: Ingress
set:
    metadata:
        annotations:
            kubernetes.io/ingress.class: nginx

match:
    kind: Deployment
delete: spec.replicas

match:
    kind: Deployment
    spec.template.spec.containers[0].image: nats-streaming
set:
    spec.template.spec.containers[0].image: nats-streaming:0.15.1

Feedback

Any feedback welcome! Please open issues and feature requests..

License

Licensed under the MIT License.

Documentation

Index

• func Bytes(in *Node) []byte
• func Delete(doc *yaml.Node, path string) error
• func Get(doc *yaml.Node, path string) (*yaml.Node, error)
• func Set(doc *yaml.Node, path string, value *yaml.Node) error
• func SetDefault(doc *yaml.Node, path string, value *yaml.Node) error
• func Write(w io.Writer, in *Node) error
• type Node
  • func Parse(in []byte) (*Node, error)
• type Transformation
  • func Transformations(r io.Reader) ([]*Transformation, error)
  • func (t *Transformation) Apply(doc *yaml.Node) error
  • func (t *Transformation) ApplyDefault(doc *yaml.Node) error
  • func (t *Transformation) ApplyDeletes(doc *yaml.Node) error
  • func (t *Transformation) ApplySet(doc *yaml.Node) error
  • func (t *Transformation) MustMatchAll(doc *yaml.Node) (bool, error)

Functions

func Bytes

func Bytes(in *Node) []byte

func Delete

func Delete(doc *yaml.Node, path string) error

func Get

func Get(doc *yaml.Node, path string) (*yaml.Node, error)

func Set

func Set(doc *yaml.Node, path string, value *yaml.Node) error

func SetDefault

func SetDefault(doc *yaml.Node, path string, value *yaml.Node) error

func Write

func Write(w io.Writer, in *Node) error

Types

type Node

type Node = yaml.Node

func Parse

func Parse(in []byte) (*Node, error)

type Transformation

type Transformation struct {
	Matches  map[string]yaml.Node `yaml:"match"`
	Sets     yaml.Node            `yaml:"set"`
	Defaults yaml.Node            `yaml:"default"`
	Deletes  []string             `yaml:"delete"`
}

func Transformations

func Transformations(r io.Reader) ([]*Transformation, error)

func (*Transformation) Apply

func (t *Transformation) Apply(doc *yaml.Node) error

func (*Transformation) ApplyDefault

func (t *Transformation) ApplyDefault(doc *yaml.Node) error

func (*Transformation) ApplyDeletes

func (t *Transformation) ApplyDeletes(doc *yaml.Node) error

func (*Transformation) ApplySet

func (t *Transformation) ApplySet(doc *yaml.Node) error

func (*Transformation) MustMatchAll

func (t *Transformation) MustMatchAll(doc *yaml.Node) (bool, error)