dp-import-reporter

README

dp-import-reporter

Getting started

Service is authenticated against zebedee, one can run dp-auth-api-stub to mimic service identity check in zebedee.

Run make debug

Kafka scripts

Scripts for updating and debugging Kafka can be found here(dp-data-tools)

Configuration

Environment variable	Default	Description
BIND_ADDR	:22200	The port to bind the application healhcheck endpoint to
KAFKA_ADDR	localhost:9092	The addresses of the kafka brokers (CSV)
KAFKA_VERSION	1.0.2	The version of Kafka
KAFKA_SEC_PROTO	unset	if set to TLS, kafka connections will use TLS ([ref-1])
KAFKA_SEC_CLIENT_KEY	unset	PEM for the client key ([ref-1])
KAFKA_SEC_CLIENT_CERT	unset	PEM for the client certificate ([ref-1])
KAFKA_SEC_CA_CERTS	unset	CA cert chain for the server cert ([ref-1])
KAFKA_SEC_SKIP_VERIFY	false	ignores server certificate issues if true ([ref-1])
KAFKA_OFFSET_OLDEST	true	start consuming kafka topics at oldest message (if false, starts at newest)
CONSUMER_GROUP	dp-event-reporter	The kafka consumer group
CONSUMER_TOPIC	report-events	The kafka consumer topic
DATASET_API_URL	http://localhost:22000	The URL of the dataset API
DATASET_API_AUTH_TOKEN	D0108EA-825D-411C-9B1D-41EF7727F465	The Auth token for the Dataset API
CACHE_SIZE	100 * 1024 * 1024	The size of the in memory cache
CACHE_EXPIRY	60	The time to live (in seconds) of the cache
GRACEFUL_SHUTDOWN_TIMEOUT	5s	The shutdown timeout in seconds (time.Duration)
SERVICE_AUTH_TOKEN	AB0A5CFA-3C55-4FA8-AACC-F98039BED0AC	The service authorization token
ZEBEDEE_URL	http://localhost:8082	The host name for Zebedee

[ref-1]: https://github.com/ONSdigital/dp-kafka/tree/main/examples#tls kafka TLS examples documentation

Contributing

See CONTRIBUTING for details.

License

Copyright © 2016-2021, Office for National Statistics

Released under MIT license, see LICENSE for details.