vaultsync

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 2, 2026 License: MIT Imports: 13 Imported by: 0

README 

= vaultsync
:doctype: article
:description: Sync secrets between HashiCorp Vault and local YAML files
:toc: macro
:toclevels: 3
ifdef::env-github[]
:tip-caption: :rocket:
:!showtitle:
:icons: font
endif::[]

++++
<p align="center">
  <img width="275" height="275" src="docs/vaultsync.png">
</p>
<p align="center">
    <h1 align="center">vaultsync</h1>


    <h3 align="center">sync secrets between HashiCorp Vault and local YAML files</h3>
        <p align="center">
      <img height=20 src="https://img.shields.io/badge/Go-%2300ADD8.svg?style=for-the-badge&logo=go&logoColor=white">
  <img height=20 src="https://img.shields.io/badge/HashiCorp_Vault-FFEC6E?style=for-the-badge&logo=vault&logoColor=black">
  <img height=20 src="https://img.shields.io/badge/License-MIT-green?style=for-the-badge">
  <a href="https://github.com/kriipke/vaultsync/actions/workflows/ci.yml"><img height=20 src="https://github.com/kriipke/vaultsync/actions/workflows/ci.yml/badge.svg" alt="CI status"></a>

</p>
<br/>
<p align="center">
  <img width="700" src="docs/vhs/full-cycle.gif" alt="vaultsync demo: list, pull, edit, and push secrets">
</p>
++++

A CLI tool for syncing secrets between HashiCorp Vault and local YAML files, with support for HCP (HashiCorp Cloud Platform) Vault. The core sync logic is also importable as a Go library (see <<using-as-a-library,Using as a Library>>).

toc::[]

== Features

* *List secrets* from Vault KVv2 engines
* *Pull secrets* to local YAML files with directory structure mirroring
* *Push secrets* back to Vault from YAML files
* *Bulk pull/push* via the importable Go library, driven by a config file at `~/.config/vaultsync/config.yaml`
* *Dry-run mode* with diff output (enhanced with delta if available)
* *HCP Vault support* with enterprise namespaces
* *Automatic diff tool detection* (delta, diff-so-fancy)

== Installation

=== Pre-built Binaries

Download the latest release for your platform from the https://github.com/kriipke/vaultsync/releases[releases page].

==== Linux/macOS

[source,bash]
----
# Download and extract (set VERSION to the latest release tag and match your platform)
VERSION=v0.1.8
curl -L "https://github.com/kriipke/vaultsync/releases/download/${VERSION}/vaultsync-${VERSION}-linux-amd64.tar.gz" | tar xz

# Make executable and move to PATH
chmod +x vaultsync
sudo mv vaultsync /usr/local/bin/
----

==== Windows

Download the `.zip` file and extract `vaultsync.exe` to a directory in your PATH.

=== Build from Source

[source,bash]
----
git clone https://github.com/kriipke/vaultsync.git
cd vaultsync
go build -o vaultsync ./cmd/vaultsync
----

== Usage

=== Environment Variables

[source,bash]
----
export VAULT_ADDR="https://your-vault.example.com:8200"
export VAULT_TOKEN="your-vault-token"
----

For HCP Vault:

[source,bash]
----
export VAULT_ADDR="https://myvault.hashicorp.cloud:8200"
export VAULT_TOKEN="your-hcp-token"
----

=== Commands

==== List Secrets

[source,bash]
----
vaultsync [--kv-engine=name] list <namespace> [path]

# Examples
vaultsync list my-namespace                    # list all secrets in default 'kv' engine
vaultsync list my-namespace app                # list secrets under 'app' path
vaultsync --kv-engine=secrets list my-namespace app  # use 'secrets' engine instead of 'kv'
----

==== Pull Secrets to Files

[source,bash]
----
vaultsync [--kv-engine=name] pull <namespace> [path] [output-dir]

# Examples
vaultsync pull my-namespace                     # pull all from 'kv' to ./secrets/
vaultsync pull my-namespace app                 # pull 'app' path to ./secrets/app/
vaultsync pull my-namespace app ./secrets      # pull 'app' path to ./secrets/app/
vaultsync --kv-engine=secrets pull my-namespace app  # use 'secrets' engine
----

==== Push Secrets from Files

[source,bash]
----
vaultsync [--kv-engine=name] push <namespace> [path] [input-dir] [--dry-run]

# Examples
vaultsync push my-namespace --dry-run           # dry-run all from ./secrets/
vaultsync push my-namespace                     # push all from ./secrets/
vaultsync push my-namespace app --dry-run       # dry-run 'app' path from ./secrets/app/
vaultsync push my-namespace app ./secrets      # push 'app' from ./secrets/app/
----

==== Bulk Pull and Push from Config

Bulk, config-driven sync is available programmatically through the Go library
(`LoadVaultSyncConfig` together with `RunPullAll`/`RunPushAll`), not as a CLI
subcommand — see <<using-as-a-library,Using as a Library>> for a runnable
example. The config format below describes what the library reads. Config-driven
syncs write files directly into each configured `local_path` and do not add a
`.yaml` suffix.

VaultSync reads its config from a single fixed location:

[source,bash]
----
~/.config/vaultsync/config.yaml
----

A sample config lives at `config.example.yaml` in the repository root. It is illustrative only and is never read by the tool — copy it into place and edit it:

[source,bash]
----
mkdir -p ~/.config/vaultsync
cp config.example.yaml ~/.config/vaultsync/config.yaml
----

Supported YAML formats:

[source,yaml]
----
syncs:
  - namespace: team-a
    vault_path: app/database
    local_path: /absolute/path/to/team-a-secrets
  - namespace: team-b
    vault_path: shared/config
    local_path: /absolute/path/to/team-b-secrets
----

[source,yaml]
----
- namespace: team-a
  vault_path: app/database
  local_path: /absolute/path/to/team-a-secrets
- namespace: team-b
  vault_path: shared/config
  local_path: /absolute/path/to/team-b-secrets
----

`vault_path` is relative to the selected KV engine. `local_path` must be absolute unless `root_dir` is set, in which case relative values are resolved under `<root_dir>/secrets`.

You can also define a top-level `root_dir` and use relative `local_path` values. In that mode, VaultSync resolves each sync target under `<root_dir>/secrets`:

[source,yaml]
----
root_dir: ~/vaultsync-demo
syncs:
  - namespace: team-a
    vault_path: app/database
    local_path: dev
  - namespace: team-b
    vault_path: shared/config
    local_path: qa
----

The example above writes to `~/vaultsync-demo/secrets/dev` and `~/vaultsync-demo/secrets/qa`.

For config-driven syncs, the configured `local_path` is the direct root for that Vault path. For example, if `vault_path` is `kubernetes/dev/example-app` and `local_path` resolves to `~/vaultsync-demo/secrets/dev`, then `ls ~/vaultsync-demo/secrets/dev` will show the secret files immediately instead of another nested `kubernetes/dev/example-app` directory tree.

=== Workflow Example

[source,bash]
----
# 1. Pull secrets from Vault
vaultsync pull my-namespace

# 2. Edit files in ./secrets/
# Files are organized like: ./secrets/app/database.yaml

# 3. Preview changes with enhanced diff
vaultsync push my-namespace --dry-run | delta

# 4. Push changes back to Vault
vaultsync push my-namespace
----

== Directory Structure

Direct `pull` and `push` commands mirror the requested Vault path and use `.yaml` files:

*Vault Path -> File Path (default 'kv' engine)*

* `kv/app/database` -> `./secrets/app/database.yaml`
* `kv/shared/config` -> `./secrets/shared/config.yaml`

*For specific subpaths:*

* Pull from `app` -> Files in `./secrets/app/`
* Push to `app` -> Reads files from `./secrets/app/`

*Custom KV engines:*

* `--kv-engine=secrets` with path `app/db` -> `./secrets/app/db.yaml`

Config-driven bulk sync (via the library) uses each configured local directory as the direct root and writes extensionless files:

* `vault_path: kubernetes/dev/example-app` with `local_path: ~/vaultsync-demo/secrets/dev` -> `~/vaultsync-demo/secrets/dev/database`
* Nested Vault secrets below that path still create subdirectories only for the secret path segments below the configured base.

== Enhanced Diff Output

The tool automatically detects and uses enhanced diff tools if available:

. *delta* - Side-by-side diffs with syntax highlighting
. *diff-so-fancy* - Enhanced unified diffs

Install any of these tools to get improved `--dry-run` output:

[source,bash]
----
# Install delta
cargo install git-delta

# Or via package managers
brew install git-delta          # macOS
sudo apt install git-delta      # Ubuntu
----

== File Format

Secrets are stored as YAML content with the secret keys as top-level properties. Direct CLI syncs use `.yaml` files; config-driven syncs use extensionless filenames.

[source,yaml]
----
# ./secrets/app/database.yaml or ~/vaultsync-demo/secrets/dev/database
host: db.example.com
port: 5432
username: myapp
password: secret123
----

[#using-as-a-library]
== Using as a Library

Beyond the CLI, the repository root is `package vaultsync`, so other Go applications can import and reuse the Vault sync logic directly.

[source,go]
----
package main

import (
	"log"

	"github.com/kriipke/vaultsync"
)

func main() {
	client, err := vaultsync.NewVaultClientFromEnv("my-namespace")
	if err != nil {
		log.Fatal(err)
	}

	ref := vaultsync.NewSecretRef("kv", "app")

	if err := client.PullSecretsToFilesAt(ref, "./secrets"); err != nil {
		log.Fatal(err)
	}
}
----

Useful exported entry points:

* `vaultsync.NewVaultClient(address, token, namespace)`
* `vaultsync.NewVaultClientFromEnv(namespace)`
* `vaultsync.NewSecretRef(kvEngine, path)`
* `(*vaultsync.VaultClient).ListSecretsAt(...)`
* `(*vaultsync.VaultClient).GetSecretAt(...)`
* `(*vaultsync.VaultClient).PutSecretAt(...)`
* `(*vaultsync.VaultClient).PullSecretsToFilesAt(...)`
* `(*vaultsync.VaultClient).PushSecretsFromFilesAt(...)`
* `vaultsync.LoadVaultSyncConfig()`
* `vaultsync.RunPullAll(...)` / `vaultsync.RunPushAll(...)` — bulk config-driven sync

=== Config-Driven Bulk Sync

Bulk pull/push is exposed through the library rather than the CLI. Load the
config from `~/.config/vaultsync/config.yaml` (the format is documented under
<<_bulk_pull_and_push_from_config,Usage>>) and run every configured target in a
single call:

[source,go]
----
cfg, err := vaultsync.LoadVaultSyncConfig()
if err != nil {
	log.Fatal(err)
}

// Pull every configured target into its local_path.
if err := vaultsync.RunPullAll(cfg, "kv", vaultsync.NewVaultClientFromEnv); err != nil {
	log.Fatal(err)
}

// Preview pushes for every configured target (dry-run).
if err := vaultsync.RunPushAll(cfg, "kv", true, vaultsync.NewVaultClientFromEnv); err != nil {
	log.Fatal(err)
}
----

== Contributing

. Fork the repository
. Create a feature branch
. Make your changes
. Add tests if applicable
. Submit a pull request

== License

MIT License - see link:LICENSE[LICENSE] file for details.

Documentation

Index

Constants

View Source 
const DefaultKVEngine = "kv"

DefaultKVEngine is the KV v2 secrets engine used for config-driven syncs when no engine is specified.

Variables

View Source 
var ErrSecretNotFound = errors.New("vault secret not found")

Functions

func DefaultConfigPath 

func DefaultConfigPath() (string, error)

func RunPullAll 

func RunPullAll(cfg *VaultSyncConfig, kvEngine string, newClient ClientFactory) error

RunPullAll pulls every sync target defined in cfg into its configured local_path. Each target's local_path is treated as the direct output root, so secret files are written there without an extension (matching the documented pull-all behavior). Failures are aggregated so a single unreachable target does not abort syncing of the others.

func RunPushAll 

func RunPushAll(cfg *VaultSyncConfig, kvEngine string, dryRun bool, newClient ClientFactory) error

RunPushAll pushes secret files from each sync target's local_path back to its configured Vault path. When dryRun is true, changes are previewed instead of written. As with RunPullAll, per-target failures are aggregated rather than fatal.

Types

type ClientFactory 

type ClientFactory func(namespace string) (*VaultClient, error)

ClientFactory constructs a VaultClient scoped to the given namespace. The default factory, NewVaultClientFromEnv, reads VAULT_ADDR/VAULT_TOKEN from the environment; tests inject their own to drive a mock transport.

type HTTPError 

type HTTPError struct {
	StatusCode int
	Body       string
}

func (*HTTPError) Error 

func (e *HTTPError) Error() string

type SecretRef 

type SecretRef struct {
	Engine string
	Path   string
}

func NewSecretRef 

func NewSecretRef(engine, path string) SecretRef

func (SecretRef) MetadataPath 

func (r SecretRef) MetadataPath() string

type SyncTarget 

type SyncTarget struct {
	Namespace string `yaml:"namespace"`
	VaultPath string `yaml:"vault_path"`
	LocalPath string `yaml:"local_path"`
}

type VaultClient 

type VaultClient struct {
	Address   string
	Token     string
	Namespace string

	Output    io.Writer
	ErrOutput io.Writer
	// contains filtered or unexported fields
}

func NewVaultClient 

func NewVaultClient(address, token, namespace string) *VaultClient

func NewVaultClientFromEnv 

func NewVaultClientFromEnv(namespace string) (*VaultClient, error)

func (*VaultClient) GetSecretAt 

func (v *VaultClient) GetSecretAt(ref SecretRef) (map[string]interface{}, error)

func (*VaultClient) ListSecretsAt 

func (v *VaultClient) ListSecretsAt(ref SecretRef) ([]string, error)

func (*VaultClient) PullSecretsRecursivelyAt 

func (v *VaultClient) PullSecretsRecursivelyAt(ref SecretRef) (map[string]map[string]interface{}, error)

func (*VaultClient) PullSecretsToFilesAt 

func (v *VaultClient) PullSecretsToFilesAt(ref SecretRef, outputDir string) error

func (*VaultClient) PullSecretsToFilesDirectAt 

func (v *VaultClient) PullSecretsToFilesDirectAt(ref SecretRef, outputDir string) error

func (*VaultClient) PushSecretsFromFilesAt 

func (v *VaultClient) PushSecretsFromFilesAt(inputDir string, ref SecretRef, dryRun bool) error

func (*VaultClient) PushSecretsFromFilesDirectAt 

func (v *VaultClient) PushSecretsFromFilesDirectAt(inputDir string, ref SecretRef, dryRun bool) error

func (*VaultClient) PutSecretAt 

func (v *VaultClient) PutSecretAt(ref SecretRef, secretData map[string]interface{}) error

type VaultListResponse 

type VaultListResponse struct {
	Data struct {
		Keys []string `json:"keys"`
	} `json:"data"`
}

type VaultSecretResponse 

type VaultSecretResponse struct {
	Data struct {
		Data     map[string]interface{} `json:"data"`
		Metadata struct {
			Version int `json:"version"`
		} `json:"metadata"`
	} `json:"data"`
}

type VaultSyncConfig 

type VaultSyncConfig struct {
	RootDir string       `yaml:"root_dir"`
	Syncs   []SyncTarget `yaml:"syncs"`
}

func LoadVaultSyncConfig 

func LoadVaultSyncConfig() (*VaultSyncConfig, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
vaultsync command
Command vaultsync is the CLI entrypoint for syncing secrets between HashiCorp Vault and local YAML files.
 Command vaultsync is the CLI entrypoint for syncing secrets between HashiCorp Vault and local YAML files.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.