pipeline-builder

README

Pipeline Builder

The Pipeline Builder is a collection of tools related to GitHub Actions and other GitHub-related concerns for Paketo-style buildpacks. This collection includes GitHub actions, a CLI that configures pipelines for repositories, and a collection of descriptors for existing repositories.

• Pipeline Builder
  • Octo Pipelines
    • Getting Started
    • Common Workflows
      • Dependendabot
      • Labels
      • Pipeline Updater
      • Release Drafter
    • Descriptor
      • github (REQUIRED)
      • codeowners
      • package
      • builder
      • docker_credentials
      • http_credentials
      • dependencies
      • test
      • path
      • offline_packages
      • republish_images
      • actions
  • Actions
    • Adoptium Dependency
    • Alibaba Dragonwell Dependency
    • Amazon Corretto Dependency
    • AppDynamics Dependency
    • Aqua Security Dependency
    • Azul Zulu Dependency
    • Bellsoft Liberica Dependency
    • CA APM Dependency
    • CF Java Index Dependency
    • Clojure Tools Dependency
    • Compute Artifact Dependencies
    • Draft Release
    • Foojay Dependency
    • GCS Dependency
    • GitHub Release Dependency
    • Google Stackdriver Profiler Dependency
    • GraalVM Dependency
    • Gradle Dependency
    • IBM Semeru Dependency
    • JProfiler Dependency
    • JRebel Dependency
    • Leiningen Dependency
    • Liberty Dependency
    • Maven Dependency
    • New Relic Dependency
    • NPM Dependency
    • OverOps Dependency
    • Paketo Deps Dependency
    • Aternity Dependency
    • Rustup Init Dependency
    • Skywalking Dependency
    • Spring Generations
    • Tomcat Dependency
    • Tomee Dependency
    • YourKit Dependency
  • License

Octo Pipelines

Octo Pipelines is a CLI that generates a collection of GitHub Workflows and other Github-related artifacts in a repository. The pipeline generator takes a simplified descriptor as input and transforms it into more complicated workflows and artifacts.

Getting Started

The command line can either be built or run directly using Go.

$: go run github.com/paketo-buildpacks/pipeline-builder/cmd/octo --descriptor <DESCRIPTOR>

The input is a YAML-based descriptor, examples of which can be found in .github/pipeline-descriptor.yml in other repositories.

Common Workflows

Regardless of the contents of the descriptor the pipeline builder creates a minimal set of workflows and artifacts.

Dependendabot

• Example dependabot.yml

A Dependendabot configuration file is added to all repositories. It will always add an update for github-actions, and if the repository contains a go.mod file, it will also add an update for gomod.

Labels

• Example Labels
• Example labels.yml
• Example synchronize-labels.yml
• Example minimal-labels.yml

In order to facilitate the automated creation of draft releases and notes, a set of semver-scope labels (semver:major, semver:minor, and semver:patch) and type labels (type:bug, type:dependency-upgrade, type:documentation, type:enhancement, type:question, type:task) are added to the repository. In addition, a workflow that enforces exactly one of each label is attached to every PR.

Pipeline Updater

• Example update-pipeline.yml

A pipeline-updating workflow is added to the repository to ensure that workflows are kept up to date as the pipeline-builder evolves.

Release Drafter

• Example Release Notes
• Example release-drafter.yml
• Example update-draft-release.yml

Draft release notes are created on every commit to main. These notes take into account every PR since the previous release in order to create a division of the types of changes that were made and the semver scope of the change to work out the next release number.

Descriptor

The descriptor is a YAML document with a number of top-level keys that correspond to new workflows, modified workflows, or artifacts. All top-level keys except github are optional.

github (REQUIRED)

github:
  username: ${{ secrets.JAVA_GITHUB_USERNAME }}
  token:    ${{ secrets.JAVA_GITHUB_TOKEN }}
  mappers:
  - '|tanzu-buildpacks|paketo-buildpacks|'
  - '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'

github is an object the describes the GitHub identity that the pipeline should use in the many places that GitHub API access is required. The token must be granted the repo, write:packages, and workflow scopes.

The mappers list is optional and is used by the Draft Release workflow. Mappers are used to control how the action looks up metadata in Github for a buildpack. See the action notes for more information.

codeowners

codeowners:
- path:  "*"
  owner: "@paketo-buildpacks/java-buildpacks"

• Example CODEOWNERS

codeowners is a list of objects test describe a path and owner. Each of these objects corresponds to a line in CODEOWNERS file.

package

package:
  repository:           ""
  repositories:         ["index.docker.io/paketobuildpacks/adoptium","gcr.io/paketo-buildpacks/adoptium"]
  include_dependencies: false
  register:             true
  registry_token:       ${{ secrets.JAVA_REGISTRY_TOKEN }}
  source_path:          subdir/adoptium
  platform:
    os: linux

• Example create-package.yml
• Example test.yml

package is an object that describes the repositories a buildpackage should be published to as well as whether to include the buildpackage's dependencies when creating it (false by default). If defined, a create-package workflow is created that creates and publishes a new package when a release is published as well as adds a create-package job to the tests workflow that is run on each PR and each commit. It will also add additional content to the draft release notes about the contents of the build package and will update the digest of the buildpackage in the published release notes. If register is true, after the package is created, it is registered with the Buildpack Registry Index.

repository is the primary repository. It is the repository registered with the Buildpack Registry Index. It defaults to the first entry in repositories so if a value is specified, this will take preference over repositories[0].

repositories is the list of repositories that the image should be published to. The list is an array of strings. If repository is not set then the first registry address will be used for repository, which is the address that's added to the Buildpack Registry Index. When multiple repositories are set, the image will be packaged once and then copied to subsequent repository addresses specified for efficiency.

If you have a single repository to which you want to publish, using repository is preferred but it is the same as setting repositories to a single repository. If you have multiple repositories, you may set both repository and repositories but that's only necessary if you need a repository registered with the Buildpack Registry Index that's not the first in the repositories list.

source_path is the optional path to the buildpack's directory relative to the repository's root. Defaults to the repository root.

platform describes what platform the created package should be built for. os can be set to linux or windows (linux by default).

builder

builder:
  repository: gcr.io/projectriff/builder

• Example create-builder.yml
• Example test.yml

builder is an object that describes the repository a builder should be published to. If defined, a create-builder workflow is created that's creates and publishes a new builder when a release is published as well as adds a create-builder job to the tests workflow that is run on each PR and each commit. It will also add additional content to the draft release notes about the contents of the build package and will update the digest of the builder in the published release notes. Finally it will add update workflows for the lifecycle and builder image elements of the builder.toml file.

docker_credentials

docker_credentials:
- registry: gcr.io
  username: _json_key
  password: ${{ secrets.JAVA_GCLOUD_SERVICE_ACCOUNT_KEY }}

docker_credentials is a list of objects that describes a registry, username, and password for authenticating against a Docker registry. In every job that registry access is required, a Docker Login Action is created and conditionalized so that it will only run on pushes to main and on PRs that come from the same repository (to protect the secrets).

http_credentials

http_credentials:
- host:     download.aquasec.com
  username: ${{ secrets.AQUA_SECURITY_USERNAME }}
  password: ${{ secrets.AQUA_SECURITY_PASSWORD }}

http_credentials is a list of objects that describe a host, username, and password for authenticating requests against an HTTP endpoint. In every job that HTTP access is required a .netrc file is written and conditionalized so that it will only be written on pushes to main and on PRs that come from the same repository (to protect the secrets).

dependencies

dependencies:
- name:            JRE 11
  id:              jre
  version_pattern: "11\\.[\\d]+\\.[\\d]+"
  purl_pattern:    "version-11\\.[\\d]+\\.[\\d]+"
  cpe_pattern:     "patch[\\d]+"
  uses:            docker://ghcr.io/paketo-buildpacks/actions/adoptium-dependency:main
  with:
    implementation: hotspot
    type:           jre
    version:        "[11,12)"

• Example update-*.yml

dependencies is a list of objects that define how dependencies are detected and updated by describing an optional name (defaults to id), id that matches a buildpack.toml defined dependency id, an optional version_pattern that defines which dependency with a given id to update, optional purl_pattern and cpe_pattern values which allow you to provide alternative patterns for updating the CPEs and PURL (they default to the version pattern), a uses to define which GitHub Action to use to find the next version, and a with used to configure the GitHub action to use to find the next version. If defined, each object will create an update workflow that is responsible for detecting a new version, updating buildpack.toml and opening a PR to include the change in the repository, if appropriate.

test

test:
  steps:
  - name: Install richgo
    run: |
      #!/usr/bin/env bash

      set -euo pipefail

      echo "Installing richgo ${RICHGO_VERSION}"

      mkdir -p "${HOME}"/bin
      echo "${HOME}/bin" >> "${GITHUB_PATH}"

      curl \
        --location \
        --show-error \
        --silent \
        "https://github.com/kyoh86/richgo/releases/download/v${RICHGO_VERSION}/richgo_${RICHGO_VERSION}_linux_amd64.tar.gz" \
      | tar -C "${HOME}"/bin -xz richgo
  - name: Run Tests
    run: |
      #!/usr/bin/env bash

      set -euo pipefail

      richgo test ./...
    env:
      RICHGO_FORCE_COLOR: "1"

test is an object that allows you to override the default test configuration by specifying a list of steps as defined by the GitHub Actions reference documentation. The default value is described in the example above.

path

path: ..

path is a pointer to a Git repository, on your local filesystem, where files will be created. If unspecified, the default is ...

offline_packages

offline_packages:
- source: paketo-buildpacks/adoptium
  target: gcr.io/tanzu-buildpacks/adoptium
  tag_prefix: my-buildpack/
  source_path: subdir/adoptium
  platform:
    os: linux

offline_packages is a list of objects that describe a source GitHub repository and a target Docker registry location. If defined, each object will create a create-package workflow that is responsible for detecting a new online buildpackage release and creating a matching offline buildpackage release and publishing it.

source_path is the optional path to the buildpack's directory relative to the repository's root. Defaults to the repository root.

tag_prefix is the optional prefix to filter for when detecting the buildpack's version via tags. Defaults to empty string.

platform describes what platform the created package should be built for. os can be set to linux or windows (linux by default).

republish_images

republish_images:
- source: gcr.io/paketo-buildpacks/bellsoft-liberica
  target: gcr.io/tanzu-buildpacks/bellsoft-liberica-lite
  id: tanzu-buildpacks/bellsoft-liberica-lite
  target_repo: paketo-buildpacks/bellsoft-liberica
  tag_prefix: java/

republish_images is a list of objects that describe a source Docker registry location, a target Docker registry location, and a new buildpack id. If defined, each object will create a republish-image workflow that is responsible for detecting new source images, modifying the buildpack id of the image to the new id, and publishing a new target image with the modified buildpack id.

The target_repo setting can be used to point the workflow to a different source control repository for the target image. It's assumed to be the current repository, but for cases where you have a utility repo which is republishing multiple images you need to point the workflow to are repository from which it can read the git tags and extract the target version. The example above, points it to the source image's code repository so it'll end up with a source and target version that are in sync.

tag_prefix is the optional prefix to filter for when detecting the buildpack's version via tags. Defaults to empty string.

actions

actions:
- source: adoptium-dependency
  target: ghcr.io/paketo-buildpacks/actions/adoptium-dependency

• Example create-action-*.yml

actions is a list of objects that describe a source directory within this repository's actions/ directory, and a target Docker registry location. If defined, each object will create a create-action workflow that is responsible for building and publishing new versions of the action.

Actions

This repository contains the source code and build pipelines for a collection of GitHub actions that are used by buildpacks. Most of these actions are used to find the latest version of given dependencies.

All the of the dependency actions also accept a pre_release configuration option (true by default) of whether to return pre-release versions.

Adoptium Dependency

The Adoptium Dependency queries the Adoptium API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/adoptium-dependency:main
with:
  implementation: hotspot
  type:           jre
  version:        "[11,12)"

Alibaba Dragonwell Dependency

The Alibaba Dependency watches Alibaba repositories for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/alibaba-dragonwell-dependency:main
with:
  glob:       Alibaba_Dragonwell_[\d\.]+_x64_linux.tar.gz$
  repository: dragonwell<jdk-version>
  token:      ${{ secrets.JAVA_GITHUB_TOKEN }}

Amazon Corretto Dependency

The Amazon Corretto Dependency watches Amazon Corretto repositories for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/amazon-corretto-dependency:main
with:
  glob:       amazon-corretto-11\.[\d.-]+-linux-x64.tar.gz
  repository: corretto-11

AppDynamics Dependency

The AppDynamics Dependency queries the AppDynamics API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/appdynamics-dependency:main
with:
  type: sun-jvm

Aqua Security Dependency

The Aqua Security Dependency queries the Aqua Security API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/aqua-security-dependency:main
with:
  username: ${{ secrets.AQUA_SECURITY_USERNAME }}
  password: ${{ secrets.AQUA_SECURITY_PASSWORD }}

Azul Zulu Dependency

The Azul Zulu Dependency queries the Azul Zulu API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/azul-zulu-dependency:main
with:
  type:    headfull
  version: "11"

Bellsoft Liberica Dependency

The Bellsoft Liberica Dependency queries the Bellsoft Liberica API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/bellsoft-liberica-dependency:main
with:
  type:    jre
  version: "11"

CA APM Dependency

The CA APM Dependency watches the CA APM Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/ca-apm-dependency:main
with:
  type: java

CF Java Index Dependency

The CF Java Index Dependency querys a CloudFoundry Java buildpack v2 index.yml file for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/cf-java-index-dependency:main
with:
  repository_root: <repo-root>

Clojure Tools Dependency

The Clojure Tools Dependency watches Clojure Tools repositories for new versions. It then filters based on the stable.properties file in their brew tap repo, allowing it to pick the most recent stable version.

uses: docker://ghcr.io/paketo-buildpacks/actions/clojure-tools-dependency:main
with:
  token:      ${{ secrets.JAVA_GITHUB_TOKEN }}

Compute Artifact Dependencies

The Compute Artifact Dependencies takes a buildpack image and produces the list of included dependencies. It's used for generating release notes. If the image is a composite buildpack, then it will recursively load all of the referenced buildpacks, pulling metadata directly from Github, and give a compete, sorted and deduplicated list of dependencies.

A Github token is strongly recommended. This is for fetching information from Github, and without it you might get rate limited. It is required for private repositories.

The package and version entries are required. They are required to indiciate the specific buildpack that will be described.

A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of gcr.io/foo/bar:1.2.3 but the Github project is at github.com/org/foo-bar then you can add a mapper of |foo\/bar|org/foo-bar| to have compute-artifact-dependencies try both. You can specify as many mappers as you want, with each mapper generating a new URI for compute-artifact-dependencies to try. It will stop when it hits the first URL that successfully pulls back the buildpack.toml file.

The output is artifact-reference-description, which can be referenced by other jobs.

uses: docker://ghcr.io/paketo-buildpacks/actions/compute-artifact-description:main
with:
  github_token: ${{ secrets.JAVA_GITHUB_TOKEN }}
  package:      "gcr.io/paketo-buildpacks/java"
  version:      "1.2.3"
  mapper_1:     '|tanzu-buildpacks|paketo-buildpacks|'
  mapper_2:     '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'

Draft Release

The draft-release action is used to augment release notes generated by the release-drafter action. The release-drafter pulls in notes based on PR changes. The draft-release action is what augments those with all of the buildpack specific information. For example, dependencies included or order groups.

The draft-release action needs to be able to look up and remotely fetch buildpack metadata. It does this by obtaining the buildpack.toml file for any dependent buildpacks from Github. This requires mapping the image URI to the Github project URI, which is not always one-to-one. As such, the action supports mappers.

A mapper will add additional permutations of Github project URIs to try. For example, if you have an image URI of gcr.io/foo/bar:1.2.3 but the Github project is at github.com/org/foo-bar then you can add a mapper of |foo\/bar|org/foo-bar| to have draft-release try both. You can specify as many mappers as you want, with each mapper generating a new URI for draft-release to try. It will stop when it hits the first URL that successfully pulls back the buildpack.toml file.

uses: docker://ghcr.io/paketo-buildpacks/actions/draft-release:main
with:
  github_token:     ${{ secrets.JAVA_GITHUB_TOKEN }}
  mapper_1:         '|tanzu-buildpacks|paketo-buildpacks|'
  mapper_2:         '|tanzu-buildpacks\/|pivotal-cf/tanzu-|'
  release_body:     ${{ steps.release-drafter.outputs.body }}
  release_id:       ${{ steps.release-drafter.outputs.id }}
  release_name:     ${{ steps.release-drafter.outputs.name }}
  release_tag_name: ${{ steps.release-drafter.outputs.tag_name }}

Foojay Dependency

The Foojay Dependency queries the Foojay API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/foojay-dependency:main
with:
  distro:  microsoft
  type:    jdk
  version: "11"

GCS Dependency

The GCS Dependency watches GCS Buckets for new versions. The first capture in glob is used as the version.

uses: docker://ghcr.io/paketo-buildpacks/actions/gcs-dependency:main
with:
  bucket: projectriff
  glob:   command-function-invoker/command-function-invoker-linux-amd64-([\d\.]+).tgz

GitHub Release Dependency

The GitHub Release Dependency watches GitHub Releases for new versions. The first capture in glob is used as the version.

uses: docker://ghcr.io/paketo-buildpacks/actions/github-release-dependency:main
with:
  glob:       sapmachine-jre-.+_linux-x64_bin.tar.gz
  owner:      SAP
  repository: SapMachine
  tag_filter: sapmachine-(11.*)
  token:      ${{ secrets.GITHUB_TOKEN }}

Google Stackdriver Profiler Dependency

The Google Stackdriver Profiler Dependency watches the Google Cloud Profiler bucket for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/google-stackdriver-profiler-dependency:main

GraalVM Dependency

The GraalVM Dependency watches the GraalVM Repository for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/graalvm-dependency:main
with:
  glob:    graalvm-ce-java11-linux-amd64-.+.tar.gz
  token:   ${{ secrets.GITHUB_TOKEN }}
  version: 11

Gradle Dependency

The Gradle Dependency queries the Gradle API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/gradle-dependency:main

IBM Semeru Dependency

The IBM Semeru Dependency queries the Gradle API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/ibm-semeru-dependency:main
with:
  glob: ibm-semeru-open-jdk_x64_linux_.+_openj9-.+.tar.gz
  owner: ibmruntimes
  repository: semeru8-binaries
  tag_filter: jdk(.*8.*)
  token: ${{ secrets.JAVA_GITHUB_TOKEN }}

JProfiler Dependency

The JProfiler Dependency watches the JProfiler Changelog for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/jprofiler-dependency:main

JRebel Dependency

The JRebel Dependency watches the JRebel Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/jrebel-dependency:main

Leiningen Dependency

The Leiningen Dependency watches the Leiningen Repository for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/leiningen-dependency:main
with:
  token: ${{ secrets.GITHUB_TOKEN }}

Liberty Dependency

The Liberty Dependency watches a Maven Repository for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/liberty-dependency:main
with:
  uri:         https://repo1.maven.org/maven2
  group_id:    org.apache.maven
  artifact_id: apache-maven
  classifier:  bin
  packaging:   tar.gz

Maven Dependency

The Maven Dependency queries a Maven Repository for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/maven-dependency:main
with:
  uri:         https://repo1.maven.org/maven2
  group_id:    org.apache.maven
  artifact_id: apache-maven
  classifier:  bin
  packaging:   tar.gz

New Relic Dependency

The New Relic Dependency watches the New Relic Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/new-relic-dependency:main
with:
  type: php

NPM Dependency

The NPM Dependency queries the NPM API for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/npm-dependency:main
with:
  package: "@google-cloud/debug-agent"

OverOps Dependency

The OverOps Dependency watches the OverOps Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/overops-dependency:main

Paketo Deps Dependency

The Paketo Deps Dependency queries the Paketo Deps Server for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/paketo-deps-dependency:main
with:
  name: rust

Aternity Dependency

The Aternity Dependency watches the Aternity Bucket for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/aternity-dependency:main

Rustup Init Dependency

The Rustup Init Dependency queries the Rustup Github Project for new versions. The target specifies the target triple to download.

uses: docker://ghcr.io/paketo-buildpacks/actions/rustup-init-dependency:main
with:
  target: x86_64-unknown-linux-musl
  token: ${{ secrets.PAKETO_BOT_GITHUB_TOKEN }}

Skywalking Dependency

The Skywalking Dependency watches the Apache Skywalking Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/skywalking-dependency:main

Spring Generations

The Spring Generations queries the Spring Generations API for the generation lifecycle of Spring projects.

uses: docker://ghcr.io/paketo-buildpacks/actions/spring-generations:main
with:
  uri: https://spring.io/api

Tomcat Dependency

The Tomcat Dependency watches the Apache Tomcat Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/tomcat-dependency:main
with:
  uri: https://downloads.apache.org/tomcat/tomcat-9

Tomee Dependency

The Tomee Dependency watches the Apache Tomee Download Page for new versions. Available distributions are microprofile, webprofile, plus or plume.

uses: docker://ghcr.io/paketo-buildpacks/actions/tomee-dependency:main
with:
  uri: https://downloads.apache.org/tomee/
  dist: webprofile
  major: 9

YourKit Dependency

The YourKit Dependency watches the YourKit Download Page for new versions.

uses: docker://ghcr.io/paketo-buildpacks/actions/yourkit-dependency:main

License

This library is released under version 2.0 of the Apache License.