read_write_pubsub

Documentation

Overview

read_write_pubsub is a pipeline example using the fhirio connector to read FHIR resources from GCS, write them to a GCP FHIR store, and, if a PubSub topic is provided, read the written resources from the FHIR store and log them based on the PubSub notifications about store updates.

Pre-requisites: 1. NDJSON-encoded FHIR resources stored in GCS. 2. Dataflow Runner enabled: https://cloud.google.com/dataflow/docs/quickstarts. 3. A Google Cloud FHIR store. Optionally, PubSub notifications set up on the store. (see: https://cloud.google.com/healthcare-api/docs/concepts/pubsub).

Running this pipeline requires providing a fully qualified GCS address (potentially containing wildcards) to where your FHIR resources are stored, a path to the FHIR store where the resources should be written to, and, optionally, the PubSub topic name your FHIR store is sending notifications to, in addition to the usual flags for the Dataflow runner.

An example command for executing this pipeline on GCP is as follows:

export PROJECT="$(gcloud config get-value project)"
export TEMP_LOCATION="gs://MY-BUCKET/temp"
export STAGING_LOCATION="gs://MY-BUCKET/staging"
export REGION="us-central1"
export SOURCE_GCS_LOCATION="gs://MY_BUCKET/path/to/resources/**"
export FHIR_STORE_PATH="MY_FHIR_STORE_PATH"
export PUBSUB_TOPIC="MY_FHIR_STORE_TOPIC"
cd ./sdks/go
go run ./examples/fhirio/read_write_pubsub/read_write_pubsub.go \
  --runner=dataflow \
  --temp_location=$TEMP_LOCATION \
  --staging_location=$STAGING_LOCATION \
  --project=$PROJECT \
  --region=$REGION \
  --worker_harness_container_image=apache/beam_go_sdk:latest \
  --sourceGcsLocation=$SOURCE_GCS_LOCATION \
  --fhirStore=$FHIR_STORE_PATH \
  --pubsubTopic=$PUBSUB_TOPIC