bitrise-step-pull-intermediate-files

README

Pull Pipeline intermediate files

The Step downloads Pipeline intermediate files to a local folder.

Description

The Step downloads Pipeline intermediate files to a local folder. These intermediate files are build artifacts generated by Workflows in a Pipeline intended to be shared with subsequent Workflows.

Make sure to add this Step after you have uploaded the intermediate files. You can upload the intermediate files using the Deploy to Bitrise.io Step's Files to share between pipeline stages input. The directories you specify will be archived and uploaded as a single file.

When uploading the Pipeline intermediate files, you must assign environment variable keys to them in the Files to share between pipeline stages input. After downloading the files, the environment variable key will point to the file's local path.

When downloading an artifact, that was a directory originally, they are extracted, and the specified environment variable will point to the directory's local path.

By default, all artifacts generated by any workflow of the pipeline are downloaded. This can be limited by setting the Artifact source input.

Please note that this step is designed to be executed on the CI only.

Configuring the Step

To configure the Step:

1. Specify which workflows' artifacts to download in the Artifact source input. By default, all workflows artifacts will be downloaded.

NOTE: You can list multiple artifacts by separating them using a comma. For example: {stage1}.{workflow1},{stage2}.{workflow2}

1. (Optional) Set the Enable verbose logging input to true if you want to log additional information for debugging purposes.

🧩 Get started

Add this step directly to your workflow in the Bitrise Workflow Editor.

You can also run this step directly with Bitrise CLI.

Examples

Basic step config

steps:
- pull-intermediate-files@1:
    inputs:
    - verbose: "true"
    - artifact_sources: stage-1\..*

Use the artifact_sources input variable to limit the downloads to a set of stages or workflows:

• stage1.workflow1 - Gets the artifacts from the stage1's workflow1.
• stage1\..* - Gets all artifacts from the stage1's workflows.
• .*\.workflow1 - Gets workflow1s' artifacts from the previous stages.
• .* - Gets every generated artifacts from the previous stages.

Wildcard based artifact pull

During a pipeline, workflows receive the finished stages and workflows object. Developers can find it on a build VM's environment variable: BITRISEIO_FINISHED_STAGES.

Let's suppose that we get the following JSON object about the previously finished stages and workflows.

[
  {
    "id": "083aa861-55b1-4132-ba70-0dfcd48fe929",
    "name": "stage-1",
    "workflows": [
      {
        "external_id": "73d33fb5-35c6-495f-bd80-015ae681db33",
        "finished_at": "2021-12-07T14:04:45Z",
        "id": "b1c6f0a1-06e7-4f63-a172-ac541a467d71",
        "name": "placeholder",
        "started_at": "2021-12-07T14:04:27Z",
        "status": "succeeded"
      },
      {
        "external_id": "39404bee-52ba-4ca2-8508-91489e7f6afa",
        "finished_at": "2021-12-07T14:05:07Z",
        "id": "f3bda7bb-37be-409f-9291-b377717cba60",
        "name": "textfile_generator",
        "started_at": "2021-12-07T14:04:48Z",
        "status": "succeeded"
      }
    ]
  },
  {
    "id": "4919fe0e-877a-45ca-ab25-7da2ddf54bce",
    "name": "stage-2",
    "workflows": [
      {
        "external_id": "ed0da0cf-66cc-4109-b23f-8a156d61b0c3",
        "finished_at": "2021-12-07T14:06:41Z",
        "id": "f572ca4e-2f06-40f1-a4cf-c208af15ff28",
        "name": "deployer",
        "started_at": "2021-12-07T14:06:13Z",
        "status": "succeeded"
      },
      {
        "external_id": "05130ce4-825b-4ca1-a9be-4f54413e5dcd",
        "finished_at": "2021-12-07T14:07:04Z",
        "id": "861fd1be-48b1-4a6b-ae4c-ee5449eaa6b6",
        "name": "textfile_generator",
        "started_at": "2021-12-07T14:06:45Z",
        "status": "succeeded"
      }
    ]
  }
]

As the key names in the object are self-describing, we will not cover those names except the external_id. The external_id is the build's slug in the PipelineService context.

Let's see the following use-cases, the use cases first part is the demand, the second is the artifact_sources config:

• As a developer, I would like to get the build artifact(s) of the stage-1's placeholder's workflow: stage-1.placeholder.

• As a developer, I would like to get the build artifact(s) of the stage-2's deployer's workflow and the stage-1's placeholder's workflow: stage-1.placeholder,stage-2.deployer. The two expressions are separated by a comma.

• As a developer, I would like to retrieve already generated artifacts: .*. As the example shows, developers can use regex.

• As a developer, I would like to retrieve the generated artifacts from the stage-2 stage: stage-2\..*.

• As a developer, I would like to get the textfile_generator workflow artifacts: .*\.textfile_generator

And so on. The syntax is: {stage-name}.{workflow-name}. Do not forget to escape the special characters when using a regex pattern.

⚙️ Configuration

Inputs

Key	Description	Flags	Default
artifact_sources	A comma (,) separated list of the Stage and Workflow paths ({stage}.{workflow}), used to specify which workflows' artifacts to download. The input uses a {stage}.{workflow} syntax. The dot character (.) is the delimiter between the Stage and the Workflow. You can use regular expressions. Do not forget to escape the special characters. Examples: - stage1.workflow1 - Gets the artifacts from the stage1's workflow1. - stage1\..* - Gets all artifacts from the stage1's workflows. - .*\.workflow1 - Gets workflow1s' artifacts from the previous stages. - .* - Gets every generated artifacts from the previous stages.	required	.*
verbose	Enable logging additional information for debugging	required	false
app_slug	The slug that uniquely identifies your app on bitrise.io. It’s part of the app URL, too.	required	$BITRISE_APP_SLUG
finished_stage	This is a JSON representation of the finished stages for which the step can download build artifacts.	required	$BITRISEIO_FINISHED_STAGES
bitrise_api_base_url	The base URL of the Bitrise API used to process the download requests.	required	https://api.bitrise.io
bitrise_api_access_token	The OAuth access token that authorizes to call the Bitrise API.	sensitive	$BITRISEIO_ARTIFACT_PULL_TOKEN

Outputs

There are no outputs defined in this step

🙋 Contributing

We welcome pull requests and issues against this repository.

For pull requests, work on your changes in a forked repository and use the Bitrise CLI to run step tests locally.

Note: this step's end-to-end tests (defined in e2e/bitrise.yml) are working with secrets which are intentionally not stored in this repo. External contributors won't be able to run those tests. Don't worry, if you open a PR with your contribution, we will help with running tests and make sure that they pass.

Learn more about developing steps:

• Create your own step
• Testing your Step