rdss-archivematica-channel-adapter

README

RDSS Archivematica Channel Adapter

• Introduction
• Usage
  • Consumer
  • Publisher
• Configuration
  • Environment variables
  • Configuration file
• Reusability
• Profiling
• Diagram

Introduction

THIS IS A PROTOTYPE!

This repository holds the source code of the channel adapter that connects Archivematica to RDSS's messaging API.

The adapter is written in Go as a standalone application that runs next to Archivematica. Its main role is to abstract the complexities and specifics of the underlying queuing system from its users.

Usage

We're producing a single binary executable file: rdss-archivematica-channel-adapter with two subcommands: consumer and publisher. You can build the Docker image locally running the following command:

$ docker build -t rdss-archivematica-channel-adapter .

The consumer is the component that brings Archivematica functionality to RDSS. It consumes the messages that come from the stream and convert them into Archivematica-specific calls. You can start it running:

$ docker run rdss-archivematica-channel-adapter consumer

On the other hand, the publisher is the component that brings RDSS functionality to Archivematica. It is implemented as a gRPC server which and it encapsulates the asynchronous nature of the messaging interaction, exposing regular synchronous methods to the application logic or the client. You can start it running:

$ docker run rdss-archivematica-channel-adapter publisher

Certain configuration parameter are required though. See the configuration section in this document for more details.

Configuration

The adapter is not configurable via command-line flags. You can choose between environment variables and the configuration file, having the former method precedence over the latter.

Environment variables

The following is a list of supported environment variables. They need to be prefixed with the string RDSS_ARCHIVEMATICA_ADAPTER_, e.g. RDSS_ARCHIVEMATICA_ADAPTER_LOGGING.LEVEL=DEBUG. Notice that the dot is used to separate nested attributes.

➖ LOGGING.LEVEL

Default: "INFO"

All severity levels defined by RFC 5424 are supported.

➖ AMCLIENT.URL

Default: ""

Archivematica API - URL, e.g.: "https://my.archivematica.internal:9000.

➖ AMCLIENT.USER

Default: ""

Archivematica API - Username, e.g.: "demo".

➖ AMCLIENT.KEY

Default: ""

Archivematica API - Key, e.g.: "eid3Aitheijoo1ohce2pho4eiDei0lah".

➖ S3.FORCE_PATH_STYLE

Default: false

When set to true, the bucket name is always left in the request URL and never moved to the host as a sub-domain.

➖ S3.INSECURE_SKIP_VERIFY

Default: false

When set to true, the client will skip the TLS verification process.

➖ S3.ENDPOINT

Default: ""

When set to a non-empty string, it's used as the AWS service endpoint, e.g.: "https://127.0.0.1:4567".

➖ S3.ACCESS_KEY

Default: ""

When set to a non-empty string, it's combined with S3.SECRET_KEY to set up the static credential object.

➖ S3.SECRET_KEY

Default: ""

When set to a non-empty string, it's combined with S3.ACCESS_KEY to set up the static credential object.

➖ S3.REGION

Default: ""

AWS Region. If empty, the AWS SDK will throw an error. E.g.: eu-west-2.

➖ CONSUMER.ARCHIVEMATICA_SHARED_DIR

Default: "/var/archivematica/sharedDirectory"

Location of the Archivematica Shared Directory.

➖ CONSUMER.BACKEND

Default: "builtin"

The default ("builtin") is a simple non-persisted hash in memory. A better alternative that can be shared by multiple consumers is the dynamodb backend which uses a single DynamoDB table.

➖ CONSUMER.DYNAMODB_TLS

Default: true

When set to true, the client uses the TLS protocol to communicate securely with the server. If you are using a clone of DynamoDB like dynalite you may prefer to have it disabled.

➖ CONSUMER.DYNAMODB_TABLE

Default: ""

The DynamoDB table name which must conform the naming rules.

➖ CONSUMER.DYNAMODB_ENDPOINT

Default: ""

When set to a non-empty string, it's used as the AWS service endpoint, e.g.: "http://127.0.0.1:9999".

➖ CONSUMER.DYNAMODB_REGION

Default: ""

AWS Region. If empty, the AWS SDK will throw an error. E.g.: eu-west-2.

➖ PUBLISHER.LISTEN

Default: "0.0.0.0:8000"

Address of the gRPC server found in the publisher.

➖ PUBLISHER.TLS

Default: false

When set to true, the gRPC server is built with the TLS integration enabled which is helpful if you want your clients to authenticate the server and encrypt all the data exchanged between the clients and the server.

➖ PUBLISHER.TLS_CERT_FILE

Default: ""

When PUBLISHER.TLS is set to true, this is used to describe the location of the public key of the X509 key pair. The file must contain PEM encoded data and it may contain intermediate certificates following the leaf certificate to form a certificate chain.

➖ PUBLISHER.TLS_KEY_FILE

Default: ""

When PUBLISHER.TLS is set to true, this is used to describe the location of the private key of the X509 key pair. The file must contain PEM encoded data.

➖ BROKER.BACKEND

Default: "kinesis"

The name of the backend used as the RDSS broker. "kinesis" is the backend currently supported in addition to "backendmock"`, developed for testing purposes only.

➖ BROKER.VALIDATION

Default: true

The adapter implements a message validator that uses the canonical RDSS JSON Schema documents. The validator can be configured in the following ways:

• When set to true, the adapter will reject invalid messages and the validation issues will be logged with DEBUG level.
• When set to warnings, the adapter will not reject invalid messages but the validation issues will be logged with DEBUG level.
• When set to false, message validation will not be performed.

➖ BROKER.QUEUES.MAIN

Default: "main"

Name of the main message queue.

➖ BROKER.QUEUES.INVALID

Default: "invalid"

Name of the invalid message queue.

➖ BROKER.QUEUES.ERROR

Default: "error"

Name of the error message queue.

➖ BROKER.REPOSITORY.BACKEND

Default: "builtin"

The Local Data Repository backend. The default ("builtin") is a simple non-persisted hash in memory. A better alternative that can be shared by multiple consumers is the dynamodb backend which uses a single DynamoDB table.

➖ BROKER.REPOSITORY.DYNAMODB_TLS

Default: true

When set to true, the client uses the TLS protocol to communicate securely with the server. If you are using a clone of DynamoDB like dynalite you may prefer to have it disabled.

➖ BROKER.REPOSITORY.DYNAMODB_TABLE

Default: ""

The DynamoDB table name which must conform the naming rules.

➖ BROKER.REPOSITORY.DYNAMODB_ENDPOINT

Default: ""

When set to a non-empty string, it's used as the AWS service endpoint, e.g.: "http://127.0.0.1:9999".

➖ BROKER.REPOSITORY.DYNAMODB_REGION

Default: ""

AWS Region. If empty, the AWS SDK will throw an error. E.g.: eu-west-2.

➖ BROKER.KINESIS.APP_NAME

Default: "rdss_am"

This is used to prefix the names of the DynamoDB tables used to share state across multiple consumers. If you are setting up more than one consumer remember to define the same value in all of them.

➖ BROKER.KINESIS.REGION

Default: ""

AWS Region. If empty, the AWS SDK will throw an error. E.g.: eu-west-2.

➖ BROKER.KINESIS.TLS

Default: true

When set to true, the client uses the TLS protocol to communicate securely with the server. If you are using a clone of DynamoDB like dynalite you may prefer to have it disabled.

➖ BROKER.KINESIS.ENDPOINT

Default: ""

When set to a non-empty string, it's used as the AWS service endpoint, e.g.: "http://127.0.0.1:4567".

➖ BROKER.KINESIS.TLS_DYNAMODB

Default: true

When set to true, the client uses the TLS protocol to communicate securely with the server. If you are using a clone of DynamoDB like dynalite you may prefer to have it disabled.

➖ BROKER.KINESIS.ENDPOINT_DYNAMODB

Default: ""

When set to a non-empty string, it's used as the AWS service endpoint, e.g.: "http://127.0.0.1:9999".

Configuration file

The adapter will try to read config in TOML format from $HOME/.rdss-archivematica-channel-adapter.toml and /etc/archivematica/rdss-archivematica-channel-adapter.toml. Alternatively, you can pass a different location with the global --config string flag.

Notice how the environment variables in the previous section map to the nested configuration sections in the configuration file, e.g.:

[logging]
level = "INFO"

[amclient]
url = "http://dadada"
user = "demo"
key = "12345"

Code reusability

A few Go packages found in this repository are agnostic to Archivematica and could be used by other vendors:

• github.com/JiscRDSS/rdss-archivematica-channel-adapter/amclient is an Archivematica HTTP API client.
• github.com/JiscRDSS/rdss-archivematica-channel-adapter/broker is a RDSS client conforming to the RDSS messaging API. Both consumer and publisher packages in this repository are use cases. If you want to know more, there is a comprehensive example in broker_test.go.
• github.com/JiscRDSS/rdss-archivematica-channel-adapter/s3 is a small S3 wrapper used to download files.

Profiling

Both consumer and publisher serve Go runtime profiling data in the format expected by the pprof visualization tool - they listen on 0.0.0.0:6060. Learn how to use the tool.

Diagram