- Build instructions
Harp is for Harpocrates (Ancient Greek: Ἁρποκράτης) the god of silence, secrets and confidentiality in the Hellenistic religion. - Wikipedia
This tool was initially developed while I was at Elastic, to be able to continue to maintain Harp without the upstream dependency, I decided to do a hard-fork of the Elastic repository.
I'm going to introduce breaking changes from the Elastic original version.
Harp is a tool set to operate secret management by contract. The objective is
to reduce the value centric management by handling secret data in a
reproducible way by providing a technical stack to describe how your value is
managed by contracts and pipelines. Furthermore, we know that
naming thing is hard,
as a consequence a secret could be
consistently associated to a
secret identifier used as a key to refer to the secret value. Finally, the secret
can hold additional metadata (ownership, rotation period, leak severity, etc.)
which can be consumed during the pipeline executions.
These key/value associations (path ⇒ value) form a
Bundle stored in an
immutable file named a
Container acts as a pivot format to
allow Harp commands to communicate and create data management pipelines.
In addition to that, it provides a
template engine used to generate various
confidence values (password, passphrase, crypto keys, etc.) and allow more
sophisticated rendering operations (configuration files, etc.).
Finally, it provides a
SDK to allow developers to integrate
in their products, and/or extend the Harp pipeline features by creating new
- Secret management is in essence a collection of processes that must be auditable, executable and reproducible for infosec and operation requirements;
- Secret provisioning must be designed with secret rotation as a day one task, due to the fact that secret data must be rotated periodically to keep its secret property;
Developersshould negotiate secret value for the secret consumer they are currently developing, by the contract based on a path (reference to the secret) and a value specification (for code contract) without the knowledge of the final deployed value;
Secret Operatorsuse different set of tools to achieve secret management operation which increases the error/secret exposure probability due to tool count involved in the process (incompatibility, changes, etc.);
- Without a defined secret naming convention, the secret storage becomes difficult to handle in time (naming is hard) and secret naming could not be helped to get a consistent, reliable and flexible secret tree;
- Secret storage backend can use various implementations in different environments and should be provisioned consistently;
- When you use
Terraformfor secret management, you have the cleartext value stored in the state. To protect the state you have to deploy a complex infrastructure. To simplify this we use harp for secret provisioning and use the secret reference in the Terraform topology.
- You want to have a
single secret valueand you are asking yourself
how to generate a strong password- Harp has a template engine with secret value generation functions to allow you to generate such values.
- You have
thousands secretsto handle to deploy your platform/customers
on multiple cloud providerswith
different secret storages- Harp will help you to define consistent secret provisioning bundles and pipelines.
- You need a
ephemeral secret storageto
bootstrapyour long term cloud secret storage - Harp will help you to create secret containers that can be consumed on deployment.
- You want to
migrate massivelyyour secrets from one secret storage to another - Harp provides you a secret container to store these secrets while they are going to be distributed in other secret storage implementations.
- You have to
alter/modifiya secret (rotation/deprecation/renewal) - Harp provides you a
storage agnostic operation set, so that you can define a specification to describe how your secret operation is going to be applied offline on the secret container.
How does it work?
Like a Data pipeline but for secret
harp allows you to handle secrets using deterministic pipelines expressed
using an atomic series of CLI operations applied to a commonly shared container
immutable and standalone file system used to store secret collection (Bundle)
generated from a template engine via user specification, or external secret
value coming from files or external secret storage.
These pipelines use the immutable container file system as a data exchange
protocol and could be extended for new input, intermediary operation or output
via plugins created with the
Each applied transformation creates a container with transformed data inside. This will enforce container reproducibility by eliminating cumulative side effects applied to the same container.
The container handles for you the confidentiality and integrity protection applied to the secret collection stored inside and manipulated by copy during the pipeline execution.
What can I do?
Harp provides :
- A methodology to design your secret management;
- Secret naming convention (CSO);
- A defined common language and complete processes to achieve secret management operations;
- A SDK to create your own tools to orchestrate your secret management pipelines;
- A container manipulation library exposed as
- A secret bundle specification to store and manipulate secrets exposed as
on-steroidtemplate engine exposed as
- A path name validation library exposed as
- A container manipulation library exposed as
- A CLI for secret management implementation
- CI/CD integration;
- Based on human-readable definitions (YAML);
- In order to create auditable and reproducible pipelines.
- An extensible tool which can be enhanced via plugins.
And allows :
- Bundle level operations
- Create a bundle from scratch / template / JSON (more via plugins);
- Generate a complete bundle using a YAML Descriptor (
BundleTemplate) to describe secret and their usages;
- Read value stored in the K/V virtual file system;
- Update the K/V virtual file system;
- Reproducible patch applied on immutable container (copy-on-write);
- Import / Export to Vault.
- Immutable container level operations
- Seal / Unseal a container for integrity and confidentiality property conservation to enforce at-rest encryption (aes256-gcm96 or chacha20-poly1305);
- Multiple identities sealing algorithm;
Is it used internally at zntrio? - Yes. It is used to generate bootstrap secrets used to bootstrap the new region infrastructure components. #ChickenEggProblem
Harp is only supporting
Vault? - No, it has been published with only vault support built-in, but it supports many other secret storage implementations via plugins.
What's the difference with
Vault? - HashiCorp Vault is an encrypted highly available K/V store with advanced authorization engine, it doesn't handle secret provisioning for you. You can't ask Vault to generate secrets for your application and store them using a defined logic. Harp is filling this requirement.
harp artifacts and source code is released under Apache 2.0 Software License.
Download a release or build from source.
$ git clone email@example.com:zntrio/harp.git $ export HARP_REPOSITORY=$(pwd)/harp
Setup dev environment
With nix flake
nix on your system, if not already installed.
$ sudo install -d -m755 -o $(id -u) -g $(id -g) /nix $ curl -L https://nixos.org/nix/install | sh
More information? - https://nixos.wiki/wiki/Nix_Installation_Guide
$ cd $HARP_REPOSITORY $ nix develop
Non-nix managed environment
Check your go version
Only last 2 minor versions of a major are supported.
Harp is compiled with :
$ go version go version go1.19.4 linux/amd64
Simple go version manager - https://github.com/stefanmaric/g
Mage is an alternative to Make where language used is Go. You can install it using 2 different methods.From source
# Install mage git clone https://github.com/magefile/mage cd mage go run bootstrap.go
# Go to tools submodule cd $HARP_REPOSITORY/tools # Resolve dependencies go mod tidy go mod vendor # Pull tools sources, compile them and install executable in tools/bin mage
❯ mage -l Targets: api:generate protobuf objects from proto definitions. build* harp executable. code:format source code and process imports. code:generate SDK code (mocks, tests, etc.) code:licenser apply copyright banner to source code. code:lint code using golangci-lint. compile harp code to create an executable. docker:harp build harp docker image docker:tools prepares docker images with go toolchain and project tools. homebrew generates homebrew formula from compiled artifacts. release harp version and cross-compile code to produce all artifacts. releaser:harp releases harp artifacts using docker pipeline. test:cli Test harp application. test:unit Test harp application. * default target
You can find more Harp feature extensions - https://github.com/zntrio/harp-plugins
Here is the list of external projects used as inspiration :
Package pkg exposes public Harp SDK.
|Package pkg exposes public Harp SDK.|
Package bundle exposes Bundle object manipulation functions.
|Package bundle exposes Bundle object manipulation functions.|
Package container exposes Secret Container operations.
|Package container exposes Secret Container operations.|
Package mock is a generated GoMock package.
|Package mock is a generated GoMock package.|
Package bech32 is a modified version of the reference implementation of BIP173.
|Package bech32 is a modified version of the reference implementation of BIP173.|
Package tlsconfig provides primitives to retrieve secure-enough TLS configurations for both clients and servers.
|Package tlsconfig provides primitives to retrieve secure-enough TLS configurations for both clients and servers.|
Package logical is a generated GoMock package.
|Package logical is a generated GoMock package.|