task

README

oci-build task

A fork of vito/oci-build-task with support for authenticating against ECR using an IAM role. See * params section for variables to set to enable ECR support.

A Concourse task for building OCI images. Currently uses buildkit for building.

A stretch goal of this is to support running without privileged: true, though it currently still requires it.

• usage
  • image_resource
  • params
  • inputs
  • outputs
  • caches
  • run
• migrating from the docker-image resource
• differences from builder task
• example

usage

The task implementation is available as an image on Docker Hub at vito/oci-build-task. (This image is built from Dockerfile using the oci-build task itself.)

This task implementation started as a spike to explore patterns around reusable tasks to hopefully lead to a proper RFC. Until that RFC is written and implemented, configuration is still done by way of providing your own task config as follows:

image_resource

First, your task needs to point to the oci-build-task image:

image_resource:
  type: registry-image
  source:
    repository: vito/oci-build-task

params

Next, any of the following optional parameters may be specified:

• $CONTEXT (default .): the path to the directory to provide as the context for the build.

• $DOCKERFILE (default $CONTEXT/Dockerfile): the path to the Dockerfile to build.

• $BUILD_ARG_*: params prefixed with BUILD_ARG_ will be provided as build args. For example BUILD_ARG_foo=bar, will set the foo build arg as bar.

• $BUILD_ARGS_FILE (default empty): path to a file containing build args in the form foo=bar, one per line. Empty lines are skipped.

  Example file contents:

  EMAIL=me@yopmail.com
  HOW_MANY_THINGS=1
  DO_THING=false
  

• $TARGET (default empty): a target build stage to build.

• $TARGET_FILE (default empty): path to a file containing the name of the target build stage to build.

• $UNPACK_ROOTFS (default false): unpack the image as Concourse's image format (rootfs/, metadata.json) for use with the image task step option.

• $LOGIN (default empty): if set to ECR the task will attempt to gain ECR credentials from AWS (using IAM role). The ECR credentials will be written to ~/.docker/config.json within the config and configured to the default account repository (eg. https://<account-number>.dkr.ecr.<region>.amazonaws.com). This behaviour can be changed using the $REGISTRIES and $REGION flags. If set to PASS the task will use the $PASS field to populate the docker auth.

• $PASS (default empty): if used with $LOGIN set to PASS this should include the docker username/password in the form of username:pass - for example passed in as ((username)):((password)). These will be base64 encoded and used in the ./docker/config.json file.

• $REG_URL (default "https://index.docker.io/v1/"): if used with $LOGIN set to PASS this specifies the URL to be used for docker login.

• $REGISTRIES (default empty): a list of account IDs to use when creating ECR credentials. Optional use in conjunction with $ECR var.

• $REGION (default eu-west-1): the region to use when creating ECR credentials.

Note: this is the main pain point with reusable tasks - env vars are kind of an awkward way to configure a task. Once the RFC lands these will turn into a JSON structure similar to configuring params on a resource, and task params will become env instead.

inputs

There are no required inputs - your task should just list each artifact it needs as an input. Typically this is in close correlation with $CONTEXT:

params:
  CONTEXT: my-image

inputs:
- name: my-image

Should your build be dependent on multiple inputs, you may want to leave $CONTEXT as its default (.) and set an explicit path to the $DOCKERFILE:

params:
  DOCKERFILE: my-repo/Dockerfile

inputs:
- name: my-repo
- name: some-dependency

It might also make sense to place one input under another, like so:

params:
  CONTEXT: my-repo

inputs:
- name: my-repo
- name: some-dependency
  path: my-repo/some-dependency

Or, to fully rely on the default behavior and use path to wire up the context accordingly, you could set your primary context as path: . and set up any additional inputs underneath:

inputs:
- name: my-repo
  path: .
- name: some-dependency

outputs

A single output named image may be configured:

outputs:
- name: image

The output will contain the following files:

• image.tar: the OCI image tarball. This tarball can be uploaded to a registry using the Registry Image resource.

• digest: the digest of the OCI config. This file can be used to tag the image after it has been loaded with docker load, like so:

  docker load -i image/image.tar
  docker tag $(cat image/digest) my-name
  

If $UNPACK_ROOTFS is configured, the following additional entries will be created:

• rootfs/*: the unpacked contents of the image's filesystem.

• metadata.json: a JSON file containing the image's env and user configuration.

This is a Concourse-specific format to support using the newly built image for a subsequent task by pointing the task step's image option to the output, like so:

plan:
- task: build-image
  output_mapping: {image: my-built-image}
- task: use-image
  image: my-built-image

(The output_mapping here is just for clarity; alternatively you could just set image: image.)

Note: at some point Concourse will likely standardize on OCI instead.

caches

Caching can be enabled by caching the cache path on the task:

caches:
- path: cache

run

Your task should run the build executable:

run:
  path: build

migrating from the docker-image resource

The docker-image resource was previously used for building and pushing a Docker image to a registry in one fell swoop.

The oci-build task, in contrast, only supports building images - it does not support pushing or even tagging the image. It can be used to build an image and use it for a subsequent task image without pushing it to a registry, by configuring $UNPACK_ROOTFS.

In order to push the newly built image, you can use a resource like the registry-image resource like so:

resources:
- get: my-image-src
  type: git
  source:
    uri: https://github.com/...

- name: my-image
  type: registry-image
  source:
    repository: my-user/my-repo

jobs:
- name: build-and-push
  plan:
  # fetch repository source (containing Dockerfile)
  - get: my-image-src

  # build using `oci-build` task
  #
  # note: this task config could be pushed into `my-image-src` and loaded using
  # `file:` instead
  - task: build
    privileged: true
    config:
      platform: linux

      image_resource:
        type: registry-image
        source:
          repository: vito/oci-build-task

      inputs:
      - name: my-image-src
        path: .

      outputs:
      - name: image

      run:
        path: build

  # push using `registry-image` resource
  - put: my-image
    params: {image: image/image.tar}

differences from builder task

The builder task was a stepping stone that led to the oci-build task. It is now deprecated. The transition should be relatively smooth, with the following differences:

• The oci-build task does not support configuring $REPOSITORY or $TAG.
  • for running the image with docker, a digest file is provided which can be tagged with docker tag
  • for pushing the image, the repository and tag are configured in the registry-image resource
• The oci-build task has a more efficient caching implementation. By using buildkit directly we can make use of its local cache exporter/importer, which doesn't require a separate translation step for saving into the task cache.
• This task is written in Go instead of Bash, and has tests!

example

This repo contains an example.yml, which builds the image for the task itself:

fly -t dev execute -c example.yml -o image=. -p
docker load -i image.tar

Documentation

Index

• func GetLogin()
• type Buildkitd
  • func SpawnBuildkitd() (*Buildkitd, error)
  • func (buildkitd *Buildkitd) Cleanup() error
• type Config
• type ImageMetadata
• type Request
• type Response
  • func Build(buildkitd *Buildkitd, outputsDir string, req Request) (Response, error)

Functions

func GetLogin

func GetLogin()

GetLogin Creates Docker config.json via IAM role

Types

type Buildkitd

type Buildkitd struct {
	Addr string
	// contains filtered or unexported fields
}

func SpawnBuildkitd

func SpawnBuildkitd() (*Buildkitd, error)

func (*Buildkitd) Cleanup

func (buildkitd *Buildkitd) Cleanup() error

type Config

type Config struct {
	Debug bool `json:"debug" envconfig:"optional"`

	ContextDir     string `json:"context"              envconfig:"CONTEXT,optional"`
	DockerfilePath string `json:"dockerfile,omitempty" envconfig:"DOCKERFILE,optional"`

	Target     string `json:"target"      envconfig:"optional"`
	TargetFile string `json:"target_file" envconfig:"optional"`

	BuildArgs     []string `json:"build_args"      envconfig:"optional"`
	BuildArgsFile string   `json:"build_args_file" envconfig:"optional"`

	// Unpack the OCI image into Concourse's rootfs/ + metadata.json image scheme.
	//
	// Theoretically this would go away if/when we standardize on OCI.
	UnpackRootfs bool `json:"unpack_rootfs" envconfig:"optional"`
}

Config contains the configuration for the task.

In the future, when Concourse supports a 'reusable task' interface, this will be provided as a JSON request on `stdin`.

For now, and for backwards-compatibility, we will also support taking values from task params (i.e. env), hence the use of `envconfig:`.

type ImageMetadata

type ImageMetadata struct {
	Env  []string `json:"env"`
	User string   `json:"user"`
}

ImageMetadata is the schema written to manifest.json when producing the legacy Concourse image format (rootfs/..., metadata.json).

type Request

type Request struct {
	ResponsePath string `json:"response_path"`
	Config       Config `json:"config"`
}

Request is the request payload sent from Concourse to execute the task.

This is currently not really exercised by Concourse; it's a mock-up of what a future 'reusable tasks' design may look like.

type Response

type Response struct {
	Outputs []string `json:"outputs"`
}

Response is sent back to Concourse by writing this structure to the `response_path` specified in the request.

This is also a mock-up. Right now it communicates the available outputs, which may be useful to assist pipeline authors in knowing what artifacts are available after a task excutes.

In the future, pipeline authors may list which outputs they would like to propagate to the rest of the build plan, by specifying `outputs` or `output_mapping` like so:

task: build
outputs: [image]

task: build
output_mapping: {image: my-image}

Outputs may also be 'cached', meaning their previous value will be present for subsequent runs of the task:

task: build
outputs: [image]
caches: [cache]

func Build

func Build(buildkitd *Buildkitd, outputsDir string, req Request) (Response, error)