cosmovisor

package module
v1.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 11, 2022 License: Apache-2.0 Imports: 20 Imported by: 1

README

Cosmovisor

cosmovisor is a small process manager for Cosmos SDK application binaries that monitors the governance module for incoming chain upgrade proposals. If it sees a proposal that gets approved, cosmovisor can automatically download the new binary, stop the current binary, switch from the old binary to the new one, and finally restart the node with the new binary.

Design

Cosmovisor is designed to be used as a wrapper for a Cosmos SDK app:

  • it will pass arguments to the associated app (configured by DAEMON_NAME env variable). Running cosmovisor run arg1 arg2 .... will run app arg1 arg2 ...;
  • it will manage an app by restarting and upgrading if needed;
  • it is configured using environment variables, not positional arguments.

Note: If new versions of the application are not set up to run in-place store migrations, migrations will need to be run manually before restarting cosmovisor with the new binary. For this reason, we recommend applications adopt in-place store migrations.

Note: If validators would like to enable the auto-download option (which we don't recommend), and they are currently running an application using Cosmos SDK v0.42, they will need to use Cosmovisor v0.1. Later versions of Cosmovisor do not support Cosmos SDK v0.44.3 or earlier if the auto-download option is enabled.

Contributing

Cosmovisor is part of the Cosmos SDK monorepo, but it's a separate module with it's own release schedule.

Release branches have the following format release/cosmovisor/vA.B.x, where A and B are a number (e.g. release/cosmovisor/v1.3.x). Releases are tagged using the following format: cosmovisor/vA.B.C.

Setup

Installation

You can download Cosmovisor from the GitHub releases.

To install the latest version of cosmovisor, run the following command:

go install github.com/cosmos/cosmos-sdk/cosmovisor/cmd/cosmovisor@latest

To install a previous version, you can specify the version. IMPORTANT: Chains that use Cosmos-SDK v0.44.3 or earlier (eg v0.44.2) and want to use auto-download feature MUST use cosmovisor v0.1.0

go install github.com/cosmos/cosmos-sdk/cosmovisor/cmd/cosmovisor@v0.1.0

Run cosmovisor version to check the cosmovisor version.

You can also install from source by pulling the cosmos-sdk repository and switching to the correct version and building as follows:

git clone git@github.com:cosmos/cosmos-sdk
cd cosmos-sdk
git checkout cosmovisor/vx.x.x
make cosmovisor

This will build cosmovisor in /cosmovisor directory. Afterwards you may want to put it into your machine's PATH like as follows:

cp cosmovisor/cosmovisor ~/go/bin/cosmovisor

Note: If you are using go v1.15 or earlier, you will need to use go get, and you may want to run the command outside a project directory.

Command Line Arguments And Environment Variables

The first argument passed to cosmovisor is the action for cosmovisor to take. Options are:

  • help, --help, or -h - Output cosmovisor help information and check your cosmovisor configuration.
  • run - Run the configured binary using the rest of the provided arguments.
  • version - Output the cosmovisor version and also run the binary with the version argument.

All arguments passed to cosmovisor run will be passed to the application binary (as a subprocess). cosmovisor will return /dev/stdout and /dev/stderr of the subprocess as its own. For this reason, cosmovisor run cannot accept any command-line arguments other than those available to the application binary.

*Note: Use of cosmovisor without one of the action arguments is deprecated. For backwards compatibility, if the first argument is not an action argument, run is assumed. However, this fallback might be removed in future versions, so it is recommended that you always provide run.

cosmovisor reads its configuration from environment variables:

  • DAEMON_HOME is the location where the cosmovisor/ directory is kept that contains the genesis binary, the upgrade binaries, and any additional auxiliary files associated with each binary (e.g. $HOME/.gaiad, $HOME/.regend, $HOME/.simd, etc.).
  • DAEMON_NAME is the name of the binary itself (e.g. gaiad, regend, simd, etc.).
  • DAEMON_ALLOW_DOWNLOAD_BINARIES (optional), if set to true, will enable auto-downloading of new binaries (for security reasons, this is intended for full nodes rather than validators). By default, cosmovisor will not auto-download new binaries.
  • DAEMON_RESTART_AFTER_UPGRADE (optional, default = true), if true, restarts the subprocess with the same command-line arguments and flags (but with the new binary) after a successful upgrade. Otherwise (false), cosmovisor stops running after an upgrade and requires the system administrator to manually restart it. Note restart is only after the upgrade and does not auto-restart the subprocess after an error occurs.
  • DAEMON_RESTART_DELAY (optional, default none), allow a node operator to define a delay between the node halt (for upgrade) and backup by the specified time. The value must be a duration (e.g. 1s).
  • DAEMON_POLL_INTERVAL (optional, default 300 milliseconds), is the interval length for polling the upgrade plan file. The value must be a duration (e.g. 1s).
  • DAEMON_DATA_BACKUP_DIR option to set a custom backup directory. If not set, DAEMON_HOME is used.
  • UNSAFE_SKIP_BACKUP (defaults to false), if set to true, upgrades directly without performing a backup. Otherwise (false, default) backs up the data before trying the upgrade. The default value of false is useful and recommended in case of failures and when a backup needed to rollback. We recommend using the default backup option UNSAFE_SKIP_BACKUP=false.
  • DAEMON_PREUPGRADE_MAX_RETRIES (defaults to 0). The maximum number of times to call pre-upgrade in the application after exit status of 31. After the maximum number of retries, cosmovisor fails the upgrade.
Folder Layout

$DAEMON_HOME/cosmovisor is expected to belong completely to cosmovisor and the subprocesses that are controlled by it. The folder content is organized as follows:

.
├── current -> genesis or upgrades/<name>
├── genesis
│   └── bin
│       └── $DAEMON_NAME
└── upgrades
    └── <name>
        ├── bin
        │   └── $DAEMON_NAME
        └── upgrade-info.json

The cosmovisor/ directory incudes a subdirectory for each version of the application (i.e. genesis or upgrades/<name>). Within each subdirectory is the application binary (i.e. bin/$DAEMON_NAME) and any additional auxiliary files associated with each binary. current is a symbolic link to the currently active directory (i.e. genesis or upgrades/<name>). The name variable in upgrades/<name> is the lowercased URI-encoded name of the upgrade as specified in the upgrade module plan. Note that the upgrade name path are normalized to be lowercased: for instance, MyUpgrade is normalized to myupgrade, and its path is upgrades/myupgrade.

Please note that $DAEMON_HOME/cosmovisor only stores the application binaries. The cosmovisor binary itself can be stored in any typical location (e.g. /usr/local/bin). The application will continue to store its data in the default data directory (e.g. $HOME/.gaiad) or the data directory specified with the --home flag. $DAEMON_HOME is independent of the data directory and can be set to any location. If you set $DAEMON_HOME to the same directory as the data directory, you will end up with a configuation like the following:

.gaiad
├── config
├── data
└── cosmovisor

Usage

The system administrator is responsible for:

  • installing the cosmovisor binary
  • configuring the host's init system (e.g. systemd, launchd, etc.)
  • appropriately setting the environmental variables
  • creating the <DAEMON_HOME>/cosmovisor directory
  • creating the <DAEMON_HOME>/cosmovisor/genesis/bin folder
  • creating the <DAEMON_HOME>/cosmovisor/upgrades/<name>/bin folders
  • placing the different versions of the <DAEMON_NAME> executable in the appropriate bin folders.

cosmovisor will set the current link to point to genesis at first start (i.e. when no current link exists) and then handle switching binaries at the correct points in time so that the system administrator can prepare days in advance and relax at upgrade time.

In order to support downloadable binaries, a tarball for each upgrade binary will need to be packaged up and made available through a canonical URL. Additionally, a tarball that includes the genesis binary and all available upgrade binaries can be packaged up and made available so that all the necessary binaries required to sync a fullnode from start can be easily downloaded.

The DAEMON specific code and operations (e.g. tendermint config, the application db, syncing blocks, etc.) all work as expected. The application binaries' directives such as command-line flags and environment variables also work as expected.

Initialization

The cosmovisor init <path to executable> command creates the folder structure required for using cosmovisor.

It does the following:

  • creates the <DAEMON_HOME>/cosmovisor folder if it doesn't yet exist
  • creates the <DAEMON_HOME>/cosmovisor/genesis/bin folder if it doesn't yet exist
  • copies the provided executable file to <DAEMON_HOME>/cosmovisor/genesis/bin/<DAEMON_NAME>
  • creates the current link, pointing to the genesis folder

It uses the DAEMON_HOME and DAEMON_NAME environment variables for folder location and executable name.

The cosmovisor init command is specifically for initializing cosmovisor, and should not be confused with a chain's init command (e.g. cosmovisor run init).

Detecting Upgrades

cosmovisor is polling the $DAEMON_HOME/data/upgrade-info.json file for new upgrade instructions. The file is created by the x/upgrade module in BeginBlocker when an upgrade is detected and the blockchain reaches the upgrade height. The following heuristic is applied to detect the upgrade:

  • When starting, cosmovisor doesn't know much about currently running upgrade, except the binary which is current/bin/. It tries to read the current/update-info.json file to get information about the current upgrade name.
  • If neither cosmovisor/current/upgrade-info.json nor data/upgrade-info.json exist, then cosmovisor will wait for data/upgrade-info.json file to trigger an upgrade.
  • If cosmovisor/current/upgrade-info.json doesn't exist but data/upgrade-info.json exists, then cosmovisor assumes that whatever is in data/upgrade-info.json is a valid upgrade request. In this case cosmovisor tries immediately to make an upgrade according to the name attribute in data/upgrade-info.json.
  • Otherwise, cosmovisor waits for changes in upgrade-info.json. As soon as a new upgrade name is recorded in the file, cosmovisor will trigger an upgrade mechanism.

When the upgrade mechanism is triggered, cosmovisor will:

  1. if DAEMON_ALLOW_DOWNLOAD_BINARIES is enabled, start by auto-downloading a new binary into cosmovisor/<name>/bin (where <name> is the upgrade-info.json:name attribute);
  2. update the current symbolic link to point to the new directory and save data/upgrade-info.json to cosmovisor/current/upgrade-info.json.
Auto-Download

Generally, cosmovisor requires that the system administrator place all relevant binaries on disk before the upgrade happens. However, for people who don't need such control and want an automated setup (maybe they are syncing a non-validating fullnode and want to do little maintenance), there is another option.

NOTE: we don't recommend using auto-download because it doesn't verify in advance if a binary is available. If there will be any issue with downloading a binary, the cosmovisor will stop and won't restart an App (which could lead to a chain halt).

If DAEMON_ALLOW_DOWNLOAD_BINARIES is set to true, and no local binary can be found when an upgrade is triggered, cosmovisor will attempt to download and install the binary itself based on the instructions in the info attribute in the data/upgrade-info.json file. The files is constructed by the x/upgrade module and contains data from the upgrade Plan object. The Plan has an info field that is expected to have one of the following two valid formats to specify a download:

  1. Store an os/architecture -> binary URI map in the upgrade plan info field as JSON under the "binaries" key. For example:

    {
  "binaries": {
    "linux/amd64":"https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f"
  }
}

    You can include multiple binaries at once to ensure more than one environment will receive the correct binaries:

    {
  "binaries": {
    "linux/amd64":"https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f",
    "linux/arm64":"https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f",
    "darwin/amd64":"https://example.com/gaia.zip?checksum=sha256:aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f"
    }
}

    When submitting this as a proposal ensure there are no spaces. An example command using gaiad could look like:

    > gaiad tx gov submit-proposal software-upgrade Vega \
--title Vega \
--deposit 100uatom \
--upgrade-height 7368420 \
--upgrade-info '{"binaries":{"linux/amd64":"https://github.com/cosmos/gaia/releases/download/v6.0.0-rc1/gaiad-v6.0.0-rc1-linux-amd64","linux/arm64":"https://github.com/cosmos/gaia/releases/download/v6.0.0-rc1/gaiad-v6.0.0-rc1-linux-arm64","darwin/amd64":"https://github.com/cosmos/gaia/releases/download/v6.0.0-rc1/gaiad-v6.0.0-rc1-darwin-amd64"}}' \
--description "upgrade to Vega" \
--gas 400000 \
--from user \
--chain-id test \
--home test/val2 \
--node tcp://localhost:36657 \
--yes

  2. Store a link to a file that contains all information in the above format (e.g. if you want to specify lots of binaries, changelog info, etc. without filling up the blockchain). For example:

    https://example.com/testnet-1001-info.json?checksum=sha256:deaaa99fda9407c4dbe1d04bd49bab0cc3c1dd76fa392cd55a9425be074af01e

When cosmovisor is triggered to download the new binary, cosmovisor will parse the "binaries" field, download the new binary with go-getter, and unpack the new binary in the upgrades/<name> folder so that it can be run as if it was installed manually.

Note that for this mechanism to provide strong security guarantees, all URLs should include a SHA 256/512 checksum. This ensures that no false binary is run, even if someone hacks the server or hijacks the DNS. go-getter will always ensure the downloaded file matches the checksum if it is provided. go-getter will also handle unpacking archives into directories (in this case the download link should point to a zip file of all data in the bin directory).

To properly create a sha256 checksum on linux, you can use the sha256sum utility. For example:

sha256sum ./testdata/repo/zip_directory/autod.zip

The result will look something like the following: 29139e1381b8177aec909fab9a75d11381cab5adf7d3af0c05ff1c9c117743a7.

You can also use sha512sum if you would prefer to use longer hashes, or md5sum if you would prefer to use broken hashes. Whichever you choose, make sure to set the hash algorithm properly in the checksum argument to the URL.

Example: SimApp Upgrade

The following instructions provide a demonstration of cosmovisor using the simulation application (simapp) shipped with the Cosmos SDK's source code. The following commands are to be run from within the cosmos-sdk repository.

Chain Setup

Let's create a new chain using the v0.44 version of simapp (the Cosmos SDK demo app):

git checkout v0.44.6
make build

Clean ~/.simapp (never do this in a production environment):

./build/simd unsafe-reset-all

Set up app config:

./build/simd config chain-id test
./build/simd config keyring-backend test
./build/simd config broadcast-mode block

Initialize the node and overwrite any previous genesis file (never do this in a production environment):

./build/simd init test --chain-id test --overwrite

Set the minimum gas price to 0stake in ~/.simapp/config/app.toml:

minimum-gas-prices = "0stake"

For the sake of this demonstration, amend voting_period in genesis.json to a reduced time of 20 seconds (20s):

cat <<< $(jq '.app_state.gov.voting_params.voting_period = "20s"' $HOME/.simapp/config/genesis.json) > $HOME/.simapp/config/genesis.json

Create a validator, and setup genesis transaction:

./build/simd keys add validator
./build/simd add-genesis-account validator 1000000000stake --keyring-backend test
./build/simd gentx validator 1000000stake --chain-id test
./build/simd collect-gentxs
Prepare Cosmovisor and Start the Chain

Set the required environment variables:

export DAEMON_NAME=simd
export DAEMON_HOME=$HOME/.simapp

Set the optional environment variable to trigger an automatic app restart:

export DAEMON_RESTART_AFTER_UPGRADE=true

Create the folder for the genesis binary and copy the simd binary:

mkdir -p $DAEMON_HOME/cosmovisor/genesis/bin
cp ./build/simd $DAEMON_HOME/cosmovisor/genesis/bin

Now you can run cosmovisor with simapp v0.44:

cosmovisor run start
Update App

Update app to the latest version (e.g. v0.45).

Next, we can add a migration - which is defined using x/upgrade upgrade plan (you may refer to a past version if you are using an older Cosmos SDK release). In a migration we can do any deterministic state change.

Build the new version simd binary:

make build

Create the folder for the upgrade binary and copy the simd binary:

mkdir -p $DAEMON_HOME/cosmovisor/upgrades/test1/bin
cp ./build/simd $DAEMON_HOME/cosmovisor/upgrades/test1/bin

Open a new terminal window and submit an upgrade proposal along with a deposit and a vote (these commands must be run within 20 seconds of each other): Note, when using a v0.46+ chain, replace submit-proposal by submit-legacy-proposal.

./build/simd tx gov submit-proposal software-upgrade test1 --title upgrade --description upgrade --upgrade-height 200 --from validator --yes
./build/simd tx gov deposit 1 10000000stake --from validator --yes
./build/simd tx gov vote 1 yes --from validator --yes

The upgrade will occur automatically at height 200. Note: you may need to change the upgrade height in the snippet above if your test play takes more time.

Documentation

Index

Constants

View Source 
const (
	EnvHome                 = "DAEMON_HOME"
	EnvName                 = "DAEMON_NAME"
	EnvDownloadBin          = "DAEMON_ALLOW_DOWNLOAD_BINARIES"
	EnvRestartUpgrade       = "DAEMON_RESTART_AFTER_UPGRADE"
	EnvRestartDelay         = "DAEMON_RESTART_DELAY"
	EnvSkipBackup           = "UNSAFE_SKIP_BACKUP"
	EnvDataBackupPath       = "DAEMON_DATA_BACKUP_DIR"
	EnvInterval             = "DAEMON_POLL_INTERVAL"
	EnvPreupgradeMaxRetries = "DAEMON_PREUPGRADE_MAX_RETRIES"
)

environment variable names

Variables

View Source 
var LoggerKey struct{}

Functions

func DownloadBinary 

func DownloadBinary(cfg *Config, info upgradetypes.Plan) error

DownloadBinary will grab the binary and place it in the proper directory

func EnsureBinary 

func EnsureBinary(path string) error

EnsureBinary ensures the file exists and is executable, or returns an error

func GetDownloadURL 

func GetDownloadURL(info upgradetypes.Plan) (string, error)

GetDownloadURL will check if there is an arch-dependent binary specified in Info

func IsSkipUpgradeHeight added in v1.0.0 

func IsSkipUpgradeHeight(args []string, upgradeInfo upgradetypes.Plan) bool

IsSkipUpgradeHeight checks if pre-upgrade script must be run. If the height in the upgrade plan matches any of the heights provided in --unsafe-skip-upgrades, the script is not run.

func LogConfigOrError added in v1.1.0 

func LogConfigOrError(logger *zerolog.Logger, cfg *Config, err error)

LogConfigOrError logs either the config details or the error.

func MarkExecutable 

func MarkExecutable(path string) error

MarkExecutable will try to set the executable bits if not already set Fails if file doesn't exist or we cannot set those bits

func NewLogger added in v1.2.0 

func NewLogger() *zerolog.Logger

func OSArch 

func OSArch() string

func UpgradeBinary added in v1.2.0 

func UpgradeBinary(logger *zerolog.Logger, cfg *Config, info upgradetypes.Plan) error

UpgradeBinary will be called after the log message has been parsed and the process has terminated. We can now make any changes to the underlying directory without interference and leave it in a state, so we can make a proper restart

func UpgradeSkipHeights added in v1.0.0 

func UpgradeSkipHeights(args []string) []int

UpgradeSkipHeights gets all the heights provided when simd start --unsafe-skip-upgrades <height1> <optional_height_2> ... <optional_height_N>

Types

type Config 

type Config struct {
	Home                  string
	Name                  string
	AllowDownloadBinaries bool
	RestartAfterUpgrade   bool
	RestartDelay          time.Duration
	PollInterval          time.Duration
	UnsafeSkipBackup      bool
	DataBackupPath        string
	PreupgradeMaxRetries  int
	// contains filtered or unexported fields
}

Config is the information passed in to control the daemon

func GetConfigFromEnv 

func GetConfigFromEnv() (*Config, error)

GetConfigFromEnv will read the environmental variables into a config and then validate it is reasonable

func (*Config) BaseUpgradeDir added in v1.0.0 

func (cfg *Config) BaseUpgradeDir() string

BaseUpgradeDir is the directory containing the named upgrade directories.

func (*Config) CurrentBin 

func (cfg *Config) CurrentBin() (string, error)

CurrentBin is the path to the currently selected binary (genesis if no link is set) This will resolve the symlink to the underlying directory to make it easier to debug

func (Config) DetailString added in v1.0.0 

func (cfg Config) DetailString() string

DetailString returns a multi-line string with details about this config.

func (*Config) GenesisBin 

func (cfg *Config) GenesisBin() string

GenesisBin is the path to the genesis binary - must be in place to start manager

func (*Config) Root 

func (cfg *Config) Root() string

Root returns the root directory where all info lives

func (*Config) SetCurrentUpgrade 

func (cfg *Config) SetCurrentUpgrade(u upgradetypes.Plan) (rerr error)

SetCurrentUpgrade sets the named upgrade to be the current link, returns error if this binary doesn't exist

func (*Config) SymLinkToGenesis 

func (cfg *Config) SymLinkToGenesis() (string, error)

SymLinkToGenesis creates a symbolic link from "./current" to the genesis directory.

func (*Config) UpgradeBin 

func (cfg *Config) UpgradeBin(upgradeName string) string

UpgradeBin is the path to the binary for the named upgrade

func (*Config) UpgradeDir 

func (cfg *Config) UpgradeDir(upgradeName string) string

UpgradeDir is the directory named upgrade

func (*Config) UpgradeInfo added in v1.0.0 

func (cfg *Config) UpgradeInfo() (upgradetypes.Plan, error)

func (*Config) UpgradeInfoFilePath added in v1.0.0 

func (cfg *Config) UpgradeInfoFilePath() string

UpgradeInfoFilePath is the expected upgrade-info filename created by `x/upgrade/keeper`.

func (*Config) WaitRestartDelay added in v1.2.0 

func (cfg *Config) WaitRestartDelay()

WaitRestartDelay will block and wait until the RestartDelay has elapsed.

type Launcher added in v1.0.0 

type Launcher struct {
	// contains filtered or unexported fields
}

func NewLauncher added in v1.0.0 

func NewLauncher(logger *zerolog.Logger, cfg *Config) (Launcher, error)

func (Launcher) Run added in v1.0.0 

func (l Launcher) Run(args []string, stdout, stderr io.Writer) (bool, error)

Run launches the app in a subprocess and returns when the subprocess (app) exits (either when it dies, or *after* a successful upgrade.) and upgrade finished. Returns true if the upgrade request was detected and the upgrade process started.

func (Launcher) WaitForUpgradeOrExit added in v1.0.0 

func (l Launcher) WaitForUpgradeOrExit(cmd *exec.Cmd) (bool, error)

WaitForUpgradeOrExit checks upgrade plan file created by the app. When it returns, the process (app) is finished.

It returns (true, nil) if an upgrade should be initiated (and we killed the process) It returns (false, err) if the process died by itself, or there was an issue reading the upgrade-info file. It returns (false, nil) if the process exited normally without triggering an upgrade. This is very unlikely to happened with "start" but may happened with short-lived commands like `gaiad export ...`

type UpgradeConfig 

type UpgradeConfig struct {
	Binaries map[string]string `json:"binaries"`
}

UpgradeConfig is expected format for the info field to allow auto-download

Source Files

View all Source files

Directories

Path Synopsis
cmd
cosmovisor

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.