remove_ger

README

remove-ger

Diagnose and recover from invalid Global Exit Root (GER) injection on L2.

Overview

What it does: The remove-ger CLI validates a GER on L1 and L2, finds claims on L2 that used that GER, and classifies each claim. After an optional interactive confirmation, it runs the recovery steps (freeze bridge, remove GER, unset/set claims or force-emit corrected events as needed, restore bridge).

When to use it: Use after you have detected an invalid GER—for example via aggsender or l2gersync error logs. For how to detect invalid GERs and manual recovery context, see the Remove GER runbook. This tool automates the procedures described there.

Building

From the repository root:

• Single binary:
  go build -o remove_ger ./tools/remove_ger/cmd
• All tools (Makefile):
  make build-tools — builds the remove_ger tool (and other tools) into $(GOBIN)/remove_ger.

Config file

The tool uses the same config file(s) as the main aggkit binary: standard aggkit-config.toml format, loaded via the --cfg / -c flag. The only change versus the main config is that you must add a [RemoveGER] section.

Additional section: [RemoveGER]

Field	Type	Description
BridgeServiceURL	string	Bridge service REST API base URL (required). Used for querying claims and bridges. The tool runs a health check at startup and will fail if the service is unreachable.
SovereignAdminKey	section	Signing key with sovereign admin privileges (activate/deactivate emergency state, remove GER, unset/set claims, force-emit claim events). Supports local keystore, AWS KMS, and GCP KMS. See sub-fields below.

SovereignAdminKey sub-fields (depends on Method):

Field	Type	Description
Method	string	Signing method: "local" (keystore file), "AWS" (AWS KMS), "GCP" (GCP KMS).
Path	string	Path to the keystore file ("local" only).
Password	string	Password to decrypt the keystore ("local" only).

Example config addition

Append the following to your existing aggkit-config.toml (adjust paths and URL to your environment):

[RemoveGER]
BridgeServiceURL = "http://localhost:8080"
SovereignAdminKey = { Method = "local", Path = "/path/to/sovereign_admin_keystore.json", Password = "your-keystore-password" }

Usage examples

Interactive (diagnose, then confirm before recovery):

./remove_ger --cfg aggkit-config.toml --ger 0x0123...64_hex_chars

Non-interactive (for automation):

./remove_ger --cfg aggkit-config.toml --ger 0x0123...64_hex_chars --yes

You can pass multiple config files; later files override earlier ones (e.g. --cfg base.toml --cfg overrides.toml).

CLI flags

Flag	Short	Required	Description
--cfg	-c	Yes	Configuration file(s), same format as aggkit-config.toml.
--ger	—	Yes	Invalid GER hash to diagnose and remove (hex, 0x-prefixed, 32 bytes / 64 hex chars).
--yes	—	No	Skip interactive confirmation and run recovery immediately.
--force	—	No	Continue even if the GER exists on L1 (still diagnose and remove).

Scenarios

The tool classifies the situation and runs the matching recovery flow:

• No claims — The GER exists on L2 but no claims use it.
  Steps: Freeze bridge → remove GER → restore bridge.

• Category A (under-collateralization) — Claim(s) reference a bridge that does not exist on L1 or has different content.
  Steps: Freeze bridge → remove GER → unset those claims → restore bridge.

• Category B.1 (GER mismatch, same index) — Claim(s) have correct bridge data but wrong GER.
  Steps: Freeze bridge → remove GER → force-emit corrected claim event(s) → restore bridge.

• Category B.2 (GER and index mismatch) — Claim(s) have wrong GER and wrong index; the correct bridge exists on L1 at a different deposit count.
  Steps: Freeze bridge → remove GER → unset wrong claims → set correct claims (correct global indexes) → force-emit corrected claim events → restore bridge.

Troubleshooting

• Wrong private key / keystore: Ensure SovereignAdminKey points to a key that has sovereign admin roles on the L2 contracts (emergency bridge pause/unpause, GER removal, unset/set claims, force emit). If recovery transactions fail with auth errors, verify the key’s roles on the L2 bridge and GER manager contracts.

• Bridge service not reachable: If BridgeServiceURL is set, the tool runs a health check at startup. Connection or HTTP errors will cause an immediate exit. Check the URL, network access, and that the bridge service is running.

• GER actually exists on L1: By default the tool exits when the GER is found on L1 (treated as valid). Use --force only when you intentionally want to diagnose and remove a GER that exists on L1 (e.g. operational override).

Documentation

Index

• func ExecuteRecovery(ctx context.Context, cfg *Config, env *Env, diagnosis *DiagnosisResult) error
• func GetClaimsByGER(ctx context.Context, bridgeService *client.Client, networkID uint32, ...) ([]*bridgesync.Claim, error)
• func PrintDiagnosis(result *DiagnosisResult)
• func Run(c *cli.Context) error
• type BridgeData
• type ClaimDiagnosis
• type Config
  • func LoadConfig(c *cli.Context) (*Config, error)
• type DiagnosisResult
  • func Diagnose(ctx context.Context, env *Env, gerHash common.Hash, force bool) (*DiagnosisResult, error)
• type Env
  • func SetupEnv(ctx context.Context, cfg *Config) (*Env, error)
  • func (e *Env) Close() error
• type GERExistsOnL1Error
  • func (e GERExistsOnL1Error) Error() string
• type RemoveGERConfig
• type Scenario

Functions

func ExecuteRecovery

func ExecuteRecovery(ctx context.Context, cfg *Config, env *Env, diagnosis *DiagnosisResult) error

ExecuteRecovery runs the recovery flow for the given diagnosis. All steps execute on L2. On any error, returns immediately; the bridge may remain in emergency state for manual intervention.

func GetClaimsByGER

func GetClaimsByGER(
	ctx context.Context, bridgeService *client.Client, networkID uint32, gerHash common.Hash,
) ([]*bridgesync.Claim, error)

GetClaimsByGER queries the bridge service for DetailedClaimEvent claims that used the given GER. networkID specifies which network to query (0 for L1, L2 network ID otherwise). Exported so E2E tests can use the same query for wait and assertion as the tool.

func PrintDiagnosis

func PrintDiagnosis(result *DiagnosisResult)

PrintDiagnosis prints a human-readable diagnosis summary and recovery plan to stdout.

func Run

func Run(c *cli.Context) error

Run is the main entry point for the remove-ger CLI.

Types

type BridgeData

type BridgeData struct {
	LeafType           uint8
	OriginNetwork      uint32
	OriginAddress      common.Address
	DestinationNetwork uint32
	DestinationAddress common.Address
	Amount             *big.Int
	Metadata           []byte
	DepositCount       uint32
}

BridgeData holds L1 bridge fields needed for comparison and for CorrectBridge (B.1/B.2).

type ClaimDiagnosis

type ClaimDiagnosis struct {
	GlobalIndex   *big.Int
	DepositCount  uint32
	OriginNetwork uint32
	Category      Scenario
	CorrectBridge *BridgeData // nil for Category A
}

ClaimDiagnosis holds the classification for a single claim.

type Config

type Config struct {
	// L1NetworkConfig contains the L1 RPC URL and contract addresses.
	L1NetworkConfig ethermanconfig.L1NetworkConfig `mapstructure:"L1NetworkConfig"`

	// Common contains shared settings such as the L2 RPC URL.
	Common ethermanconfig.CommonConfig `mapstructure:"Common"`

	// BridgeL2Sync contains the L2 bridge contract address used to initialize the binding.
	BridgeL2Sync bridgesync.Config `mapstructure:"BridgeL2Sync"`

	// L2GERSync contains the L2/L1 GER contract addresses.
	L2GERSync l2gersync.Config `mapstructure:"L2GERSync"`

	RemoveGER RemoveGERConfig `mapstructure:"RemoveGER"`
}

Config holds the subset of aggkit configuration fields needed by the remove-GER tool, plus tool-specific settings in the RemoveGER section.

func LoadConfig

func LoadConfig(c *cli.Context) (*Config, error)

LoadConfig reads the TOML config file(s) specified by --cfg and unmarshals the fields required by the remove-GER tool. Uses the same template rendering pipeline as the main aggkit binary so that template variables (e.g. L1URL → L1NetworkConfig.RPC.URL) are resolved correctly.

type DiagnosisResult

type DiagnosisResult struct {
	InvalidGER     common.Hash
	GERExistsOnL1  bool
	GERExistsOnL2  bool
	GERTimestampL2 *big.Int
	Claims         []ClaimDiagnosis
	Scenario       Scenario
}

DiagnosisResult holds the result of the diagnosis phase.

func Diagnose

func Diagnose(ctx context.Context, env *Env, gerHash common.Hash, force bool) (*DiagnosisResult, error)

Diagnose runs the diagnosis phase: validate GER on L1/L2, find claims by GER, classify each claim. If GER exists on L1 and force is false, returns GERExistsOnL1Error.

type Env

type Env struct {
	// RPC clients
	L1 *ethclient.Client
	L2 *ethclient.Client

	// Bridge service REST client (required)
	BridgeService *client.Client

	// L2NetworkID is the network ID of the L2 network served by the bridge service.
	L2NetworkID uint32

	// L1 contract bindings
	L1GERManager *agglayerger.Agglayerger

	// L2 contract bindings
	L2Bridge     *agglayerbridgel2.Agglayerbridgel2
	L2GERManager *agglayergerl2.Agglayergerl2
}

Env holds all connections and contract bindings needed by the remove-ger tool. Pass it to diagnosis and recovery methods in later chunks.

func SetupEnv

func SetupEnv(ctx context.Context, cfg *Config) (*Env, error)

SetupEnv dials L1/L2, initializes contract bindings and bridge service client. BridgeServiceURL in cfg.RemoveGER is required. Exported for use by E2E tests that invoke the tool programmatically.

func (*Env) Close

func (e *Env) Close() error

Close closes all RPC connections. BridgeService has no Close.

type GERExistsOnL1Error

type GERExistsOnL1Error struct {
	GER common.Hash
}

GERExistsOnL1Error is returned when the GER exists on L1 (not invalid) and --force was not set.

func (GERExistsOnL1Error) Error

func (e GERExistsOnL1Error) Error() string

type RemoveGERConfig

type RemoveGERConfig struct {
	// SovereignAdminKey is the signing key with privileges to:
	// - activateEmergencyState / deactivateEmergencyState on the L2 bridge
	// - removeGlobalExitRoots on the L2 GER manager
	// - unsetMultipleClaims / setMultipleClaims on the L2 bridge
	// - forceEmitDetailedClaimEvent on the L2 bridge
	// Supports local keystore, AWS KMS, and GCP KMS via signertypes.SignerConfig.
	SovereignAdminKey signertypes.SignerConfig `mapstructure:"SovereignAdminKey"`

	// BridgeServiceURL is the URL of the aggkit bridge service REST API (required).
	// Used for querying claims, bridges, and proofs.
	BridgeServiceURL string `mapstructure:"BridgeServiceURL"`

	// L2NetworkID is the network ID of the L2 network served by the bridge service.
	// Required for querying L2 claims via the bridge service.
	L2NetworkID uint32 `mapstructure:"L2NetworkID"`
}

RemoveGERConfig contains configuration specific to the remove-GER tool.

type Scenario

type Scenario string

Scenario is the overall or per-claim classification from the runbook.

const (
	ScenarioNoClaims   Scenario = "no_claims"
	ScenarioCategoryA  Scenario = "category_a"
	ScenarioCategoryB1 Scenario = "category_b1"
	ScenarioCategoryB2 Scenario = "category_b2"
)