network-collector

README

Network Collector

Network Collector is a Go-based tool designed for flexible and efficient data collection from various network devices. It supports multiple protocols and drivers, allowing you to connect to and collect data from a wide range of network devices.

Features

• SSH: Collect data using SSH for devices running various operating systems like Cisco NX-OS, Juniper Junos, etc.
• HTTP: Fetch data using HTTP from devices with REST APIs, such as Arista EOS.
• Netconf: Use Netconf to interact with devices supporting the Netconf protocol.
• gNMI: Collect data using the gNMI protocol.
• RESTCONF: Fetch data using RESTCONF from devices supporting this protocol.

Installation

Download a platform archive from the GitHub releases page, extract it, and run:

./network-collector --version

To build from source instead:

1. Clone the repository:

   git clone https://github.com/gwoodwa1/network-collector.git
   cd network-collector
   

2. Install dependencies:

   go mod tidy
   

3. Build the CLI examples:

   go build -o network-collector ./cmd/network-collector
   go build -o arista-http ./cmd/arista-http
   go build -o netconf-client ./cmd/netconf-client
   go build -o gnmi-client ./cmd/gnmi-client
   go build -o restconf-client ./cmd/restconf-client
   

4. Provide credentials:

   Set credentials using environment variables:

   export NET_USER=<username>
   export NET_PASSWORD=<your password>
   

   Alternatively, pass --creds_input to any of the built commands to be prompted for a username and password. Interactive input overrides any credentials already set in the environment, and password entry is hidden:

   ./network-collector --creds_input
   

Package usage

Import the SSH module into your Go program:

import "github.com/gwoodwa1/network-collector/pkg/drivers/ssh"

Import the Arista HTTP driver:

import "github.com/gwoodwa1/network-collector/pkg/drivers/aristahttp"

Import the NETCONF driver:

import "github.com/gwoodwa1/network-collector/pkg/drivers/netconf"

Import the gNMI driver:

import "github.com/gwoodwa1/network-collector/pkg/drivers/gnmi"

Import the RESTCONF driver:

import "github.com/gwoodwa1/network-collector/pkg/drivers/restconf"

Example usage for SSH:

client := ssh.NewClient()
if err := client.Connect("192.168.1.10", "admin", "password", "cisco_nxos"); err != nil {
    log.Fatal(err)
}
output, err := client.Execute("show version")
if err != nil {
    log.Fatal(err)
}
fmt.Println(output)
if err := client.Close(); err != nil {
    log.Fatal(err)
}

Separate command examples

• cmd/network-collector: SSH example using pkg/drivers/ssh
• cmd/arista-http: HTTP example using pkg/drivers/aristahttp
• cmd/netconf-client: NETCONF example using pkg/drivers/netconf
• cmd/gnmi-client: gNMI example using pkg/drivers/gnmi
• cmd/restconf-client: RESTCONF example using pkg/drivers/restconf

CLI validation and output

The cmd/network-collector SSH example supports validation configured in config.yaml and these CLI flags:

• --json: emit consolidated machine-readable JSON for all validation results (suppresses raw command output)
• --fail-on-fail: exit with non-zero status if any validation returns fail or error, or if a device/step cannot run successfully
• --creds_input: securely prompt for credentials instead of using NET_USER and NET_PASSWORD

fail-on-fail can also be configured with fail_on_fail: true in config.yaml or the FAIL_ON_FAIL=true environment variable. The CLI flag takes precedence when provided.

Example: run validations and emit only JSON

./network-collector --json

Example: run validations and exit non-zero if any check fails

./network-collector --fail-on-fail

Configuration

The configuration is done through a config.yaml file Here’s an example of the config.yaml:

fail_on_fail: false

restconf:
  - hostname: device-eos-02
    ip: 192.168.15.7
    port: 3333
    skip_tls: true
    method: GET
    endpoint: data/openconfig-interfaces:interfaces/interface

gnmi:
  - hostname: device-eos-01
    ip: 192.168.16.10:6030
    skip_tls: true
    path: /interfaces/interface/subinterfaces/subinterface/state/description

ssh:
  - hostname: device-nxos-01
    ip: 192.168.16.1
    type: cisco_nxos
    cmd: show ip route
  - hostname: device-qfx-01
    ip: 192.168.16.1
    type: juniper_junos
    cmd: show route

http:
  - hostname: device-eos-08
    ip: 192.168.16.8
    type: arista_eos
    cmd: show version
    skip_tls: true
  - hostname: device-eos-03
    ip: 192.168.16.9
    type: arista_eos
    cmd: show ip route
    skip_tls: true

netconf:
  - hostname: device-eos-05
    ip: 192.168.16.7
    type: arista_eos
    rpc: |
      <get>
        <filter type="subtree">
          <interfaces>
            <interface>
            </interface>
          </interfaces>
        </filter>
      </get>
  - hostname: device-eos
    ip: 192.168.15.8
    type: arista_eos
    rpc: |
      <get>
        <filter type="subtree">
          <interfaces>
            <interface>
            </interface>
          </interfaces>
        </filter>
      </get>

SSH step-based commands with retry

The cmd/network-collector example supports ssh.steps, which lets you run multiple commands over the same SSH connection for a single device. Each step can include validation and optional retry behavior.

Inventory-based SSH targets

You can keep connection details in inventory.yaml and reference them from config.yaml, which avoids repeating hostnames, IP addresses, and device types across playbook entries.

Example inventory.yaml:

hosts:
  - name: router-01
    ip: 192.0.2.10
    type: cisco_ios
  - name: router-02
    ip: 192.0.2.11
    type: cisco_ios

groups:
  ios:
    hosts:
      - router-01
      - router-02

Example config.yaml:

inventory_file: inventory.yaml
parsers_file: parsers.yaml

ssh:
  - host: router-01
    cmd: show version

  - group: ios
    steps:
      - name: show-version
        cmd: show version

Use host for one inventory host, hosts for a list of inventory hosts, group for one inventory group, or groups for multiple groups. Inventory hosts can define name, hostname, ip or address, type, timeout, and operation_timeout. Values in config.yaml override inventory values, so you can set a common type or timeout at the playbook entry if needed. Existing single-node entries with inline hostname, ip, and type continue to work without an inventory file.

Staged and concurrent SSH execution

Use the top-level execution settings to run inventory devices concurrently while staggering when each device starts. This is useful for long-running software upgrades where devices should overlap without all beginning at once.

inventory_file: inventory.yaml

execution:
  max_parallel: 3
  start_interval_seconds: 120
  canary_count: 1
  failure_threshold: 2

ssh:
  - group: ios_upgrade
    steps:
      - name: perform-upgrade
        cmd: install replace harddisk:/8000-x64.iso
        return_to_prompt: false
      - name: wait-for-router
        ssh_probe:
          port: 22
          interval_seconds: 30
          max_attempts: 40
          post_wait_seconds: 120

• max_parallel is the maximum number of active SSH device runs. 0 or an omitted value preserves the default of one device at a time.
• start_interval_seconds is the minimum delay between device starts. The first device starts immediately, and a free concurrency slot does not bypass the delay.
• canary_count runs that many devices from the resolved inventory order as a separate first stage. All canaries must succeed before remaining devices are started.
• failure_threshold stops launching new devices after that many device runs have failed. Devices already running are allowed to finish. 0 disables the threshold.

For the example above, the canary runs first. If it succeeds, another device may start immediately when the canary took longer than two minutes; subsequent starts remain two minutes apart, with no more than three active devices. A failed canary stops the main stage regardless of failure_threshold.

Validation failures, connection errors, step errors, and session setup or shutdown errors count as device failures. Results are aggregated in inventory order even though devices may complete in a different order. Each device continues to write its own session log; live terminal output from simultaneously active devices may be interleaved.

Separate SSH entries targeting the same hostname/IP remain serialized and share their registered variables. This preserves playbooks that capture a value in one entry and consume it in a later entry.

Regex parser modules

You can define reusable regex parser modules in parsers.yaml, reference them from a step with parser, and validate the generated JSON with the existing gjson extractor.

Example parsers.yaml:

parsers:
  xr_install_active_summary:
    type: regex
    fields:
      active_packages:
        pattern: '(?m)^\s+(disk0:\S+)'
        group: 1
        repeated: true
      profile:
        pattern: '(?m)^(.+ Profile):'
        group: 1

Example config.yaml:

parsers_file: parsers.yaml

ssh:
  - host: xr-router-1
    steps:
      - name: parse-active-packages
        cmd: show install active summary
        parser: xr_install_active_summary
        validations:
          - extractor: gjson
            json_path: active_packages.#
            condition: gte
            expected: 1
            expected_type: int
          - extractor: gjson
            json_path: profile
            condition: contains
            expected: Default
            expected_type: string

Parser fields support pattern, optional capture group (defaults to the first capture group when present), repeated: true for arrays, and type: int for numeric coercion. Parsed JSON is written to the session log before validation.

Offline parser fixture tests

Parser and validation changes can be tested without logging into a network device. Add captured command output as plain text under cmd/network-collector/testdata/cli/, then add a case to cmd/network-collector/testdata/parser-fixtures.yaml.

Fixture cases can define parser and validations directly, or reference a parser-bearing step from cmd/network-collector/testdata/offline-config.yaml:

cases:
  - name: xr-show-alarms-brief-system-active
    input: cli/xr_show_alarms_brief_system_active.txt
    config_ref:
      ssh_index: 0
      step: parse-active-alarms

Run the offline parser suite with:

go test ./cmd/network-collector -run TestParserFixtures

The fixture runner loads parsers.yaml, parses the CLI text file, applies the referenced config step's validation or validations, and fails the test if parsing or validation does not pass.

For baseline/final comparisons, provide both baseline_input and input. The baseline output is parsed, registered into the variable named by the baseline config step's register, and then the final output is parsed and validated.

cases:
  - name: xr-active-alarms-unchanged
    baseline_input: cli/xr_show_alarms_before.txt
    input: cli/xr_show_alarms_after_unchanged.txt
    baseline_config_ref:
      ssh_index: 0
      step: capture-baseline-alarms
    config_ref:
      ssh_index: 0
      step: compare-final-alarms

  - name: xr-active-alarms-changed
    baseline_input: cli/xr_show_alarms_before.txt
    input: cli/xr_show_alarms_after_changed.txt
    baseline_config_ref:
      ssh_index: 0
      step: capture-baseline-alarms
    config_ref:
      ssh_index: 0
      step: compare-final-alarms
    expect_pass: false

Use expect_pass: false for negative fixtures that should detect a changed parser result.

Example:

name_playbook: Software Upgrade on Cisco IOS
inventory_file: inventory.yaml
parsers_file: parsers.yaml

ssh:
  - host: device-ios-03
    timeout: 20
    operation_timeout: 120
    steps:
      - name: show-version
        message: checking currently running image
        cmd: show version
        validation:
          extractor: regex
          pattern: "System image file is \"(.+)\""
          condition: contains
          expected: flash
          expected_type: string

      - name: capture-install-id
        cmd: 'show install active | include "Install ID"'
        validation:
          extractor: regex
          pattern: 'Install ID:\s+(\\d+)'
          condition: eq
          expected: 14
          expected_type: int
        register: install_id

      - name: wait-for-install-state
        wait_seconds: 30

      - name: confirm-reload
        cmd: yes
        return_to_prompt: false

      - name: wait-for-reboot
        wait_seconds: 600
        ssh_probe:
          port: 22
          interval_seconds: 30
          max_attempts: 40
          timeout_seconds: 5
          post_wait_seconds: 120

      - name: show-install-by-id
        cmd: 'show install active {{install_id}}'
        retry:
          until_pass: true
          interval_seconds: 60
          max_attempts: 5
        validation:
          extractor: regex
          pattern: 'Package ID:\s+{{install_id}}'
          condition: contains
          expected: '{{install_id}}'
          expected_type: string

The retry step keeps rerunning the command until validation passes, with the configured interval and attempt limit.

Validation steps can also run conditional actions after the final validation result:

      - name: check-current-image
        cmd: show version
        validation:
          extractor: regex
          pattern: "System image file is \"(.+)\""
          condition: contains
          expected: iosxe-17.09.04
          expected_type: string
        on_pass:
          action: exit
          message: target image is already active; stopping this device
        on_fail:
          message: target image is not active; running upgrade path
          steps:
            - name: collect-install-state
              cmd: show install summary
              validation:
                extractor: regex
                pattern: 'State:\s+(\S+)'
                condition: eq
                expected: READY
                expected_type: string
            - name: perform-upgrade
              cmd: install add file flash:iosxe-17.09.04.bin activate commit

Use on_pass or on_fail on a step with validation. Supported actions are exit/stop to stop the remaining steps for the current device without failing, fail to stop and mark the run failed, cmd to run another SSH command, steps to run a nested list of normal SSH steps, and none/noop to take no control-flow action. If an action block contains only message, the collector logs the message and continues. If it contains only cmd, action: cmd is implied. If it contains only steps, action: steps is implied. Nested steps support the same fields as top-level steps, including validation, retry, register, ssh_probe, return_to_prompt, and their own on_pass / on_fail actions. Action message and cmd values support registered variables such as {{install_id}}.

Each SSH device run is recorded under session_logs/ using the hostname and start timestamp in the filename. Set top-level name_playbook to include a playbook title in the ASCII banner at the start of each session log.

Use operation_timeout on an SSH device to increase the scrapligo operation timeout for long-running commands. The value is seconds; for example, operation_timeout: 120 gives commands up to two minutes to return to the prompt.

Use wait_seconds on a step to pause while keeping the SSH connection open. A wait-only step does not require cmd; if both wait_seconds and cmd are set, the collector waits first and then runs the command.

Use message on a step to write an operator note to stdout and the session log. Message-only steps are allowed, and messages can use registered variables such as {{install_id}}. Action messages under on_pass and on_fail are logged the same way.

Use ssh_probe after software upgrades or reloads. The collector closes the stale SSH session, probes the configured TCP port until it responds, waits post_wait_seconds after the first successful probe, reconnects SSH, and then continues with the following steps. This helps cover the gap where port 22 is accepting connections but the device is still booting.

Use return_to_prompt: false for commands that intentionally reboot or disconnect the device before a normal CLI prompt can return, such as a yes confirmation. Timeout/error from that command is treated as expected, the stale SSH client is closed, and the next wait/probe step can handle reconnecting. The collector also accepts no as a compatibility alias.

You can also register a value from a step using register: <name> and reuse it in later steps with {{<name>}} in cmd, pattern, json_path, or string expected values.

Validation semantics

Validation steps in config.yaml support extractors and typed comparisons. Use extractor: regex for CLI text output and extractor: gjson for JSON payloads, including parser output. Use validation for one rule, or validations for multiple rules. When multiple validations are configured, all rules must pass for the step to trigger on_pass; any failed or errored rule triggers on_fail.

• pattern / json_path: the extraction pattern or JSON path. Regex extraction uses the first capture group when present; otherwise it uses the whole regex match.
• condition: eq, neq, contains, not_contains, matches, gt, gte, lt, or lte.
• expected: the expected value to compare against
• expected_type (optional): string, int, or length — when provided the extracted value will be coerced to that type before comparison. This prevents accidental string vs numeric mismatches (for example, the string "100" is distinct from the integer 100 when expected_type is int). With length, regex values use string length and GJSON arrays/objects use item count.

Example equality checks added in config.yaml:

• String equality (exact match): condition: eq, expected: "RUNNING", expected_type: string
• Integer equality: condition: eq, expected: 100, expected_type: int
• Regex match against an extracted value: condition: matches, expected: '^\d+\.\d+\.\d+$'
• Length check: condition: gte, expected: 3, expected_type: length