CATMonitor

module

v0.3.5 Latest Latest Go to latest Published: Aug 25, 2026 License: Apache-2.0

README ¶

CATMonitor

Computing Availability Tools Monitor — 服务器运行指标采集、健康度评估与 Prometheus 导出守护进程

CATMonitor 是 CAT (Computing Availability Tools) 系列软件之一，用于采集服务器各部件（CPU、内存、硬盘、GPU、NPU、网卡、机箱等）的运行指标，基于采集结果评估服务器整体健康度，并以 Prometheus 格式导出供长期存储与告警。

版本信息

项目	说明
版本号	v0.3.5
发布时间	2026-08-25
平台支持	Linux (x86_64), Windows (x86_64)
许可证	Apache-2.0（见 LICENSE）

功能特性

多部件采集：CPU / 内存 / 硬盘 / GPU / NPU / 网卡 / 机箱共 7 个部件，216 个指标（含 NPU 掉卡检测 card_drop/error_code/进程信息 process_info、Disk 累计 raw counters、Network rx/tx_bytes_total，详见指标清单）
健康度评估：基于采集指标自动计算 0-100 健康分，自动检测 GPU/NPU 切换权重方案
可靠性压测：Linux 上显式运行 STREAM / HPL / HPCG / Ascend NPU Burn；CLI 与本机 Web 共享作业、报告和互斥锁，结果不直接计入健康总分
Snapshot 统一生产：daemon 作为唯一 snapshot 生产者，产出 per-component snapshot_<comp>.json + 全局 snapshot.json（health/collectors/intervals/system_specs）；只读特性（web/dfee）消费快照而不再各自采集，避免重复跑硬件
Web 仪表盘：独立 catmonitor-web 二进制，只读消费 daemon 产出的 snapshot，可视化单机健康度与各部件指标，默认端口 19322
能效监控（dfee）：独立 catmonitor-dfee 二进制，只读消费 snapshot 渲染能效指标实时图表 SPA（卡片拖拽缩放、多选下拉筛选、模块折叠），默认端口 19323；内置 Prometheus exporter（:9333/metrics）将 snapshot 映射为 node_*/dsmi_*/ipmi_*/static_* 格式，无需额外进程
Prometheus 导出（exporter）：daemon 内置 /metrics 端点（:19320），一次采集同时落盘 JSONL + 缓存导出，零额外进程
容器化部署：docker/ 提供 NPU（Debian/glibc 两步构建 + driver/nnae 挂载）与 generic（alpine 多阶段）两种镜像，build.sh 自动检测 driver，docker-compose.yml 一键编排 daemon + web + dfee 三服务，详见容器化文档
指标采集目录：configs/metrics.yaml 统一管控采哪些指标、优先级、默认是否采集，模块可覆盖
Feature-scoped 采集：features 配置列表声明各特性所需指标，internal/metrics 以 SetFeatureScope 建立白名单（各 feature metrics.yaml 的并集）；非空时只采白名单内且 priority ≥ min_priority 的指标，AnyWanted 跳过产出全 out-of-scope 的子方法，避免空跑硬件；空则用默认目录全集。同时 ComponentIntervals 取各 feature metrics.yaml 声明 interval 的最小值派生每组件采集 cadence（C_comp），覆盖 collectors.<name>.interval，使 feature 所需刷新节奏成为采集节奏
采集粒度控制：collection.min_priority 配置（low/medium/high）按优先级阈值预过滤采集，采集器经 AnyWanted DI 在执行前跳过无需采集的指标组，降低开销
故障订阅推送（faultsub）：opt-in 特性，对采集到的 NPU 指标做故障判定（卡掉线/健康状态/错误码/HBM UCE/RoCE 链路等），经 HTTP Webhook 向已订阅的外部故障管理者推送 FaultEvent，并提供订阅注册/快照/事件回补 REST API（:19321）。零新依赖（net/http），默认关闭
落后节点 KPI 输出（stragglerout）：opt-in 特性，作为 daemon 的 Storage 插件，把每次采集到的 NPU KPI 指标（温度/功耗/AICore 频率与利用率/HBM 利用率/带宽/RoCE 错误等）按"每时刻×每卡"聚合追加写为日级 JSONL，供 straggler 慢节点检测器消费，替代其自带 kpi_collect.sh。默认关闭
来源层架构：internal/source/（15 包）抽象数据获取与解析，采集器不直接读文件/执行命令，无硬件时优雅降级
跨平台：Linux / Windows 双平台，构建标签隔离平台代码
易扩展：新增部件采集器只需实现统一接口并注册，核心代码零修改

各特性功能规格见 SPEC.md，各特性的设计与规格见对应 features/<feature>/*_SPEC.md。

技术栈

项目	选型
语言	Go 1.23.4+（以 `go.mod` 为准）
平台	Linux / Windows
输出	本地文件 (JSONL) + Prometheus 文本 (`/metrics`)
配置	YAML
外部依赖	Go 仅 `gopkg.in/yaml.v3`；GPU 经 `nvidia-smi`，NPU 采集经 `dcmi`(CGo, `-tags dcmi`)/`npu-smi`/`hccn_tool`；可选 NPU 压测源码按 Mulan PSL v2 固定在 `third_party/ascend_npu_burn`，管理员另行准备匹配的 CANN/torch_npu 环境或基础镜像
Web/导出	Go 标准库 `net/http` + `//go:embed` 内嵌前端，无构建步骤

快速开始

# 编译（daemon + web + dfee 三个二进制）
go version             # 必须为 Go 1.23.4 或更高版本
make all               # 或分别 make build / make web / make dfee

# 节点存在多个 Go 版本时，显式指定已安装的新工具链
make all GO=/opt/catmonitor/toolchains/go1.25.1/bin/go

# 配置（Linux）：开启 snapshot 生产以供 web/dfee 只读消费
cp configs/catmonitor.yaml /etc/catmonitor/catmonitor.yaml
#   在 /etc/catmonitor/catmonitor.yaml 中设：
#     snapshot.enabled: true
#     snapshot.dir: /var/lib/catmonitor/snapshot
#     features: [web, dfee]   # 按特性所需指标做 scope 白名单采集（可选）

# 启动守护进程（采集 + 健康度 + Prometheus :19320 + snapshot 生产）
catmonitor daemon

# 启动只读消费者（消费 daemon 产出的 snapshot，不自行采集）
catmonitor-web -addr :19322 -snapshot-dir /var/lib/catmonitor/snapshot
catmonitor-dfee -addr :19323 -snapshot-dir /var/lib/catmonitor/snapshot

# 单次采集 / 健康检查 / 采集器列表
catmonitor collect -o table
catmonitor health -o table
catmonitor list

STREAM/HPL/HPCG 不随仓库分发二进制。Linux 管理员可用 scripts/stress/build_cpu_benchmarks.sh 从任意源码位置构建并生成可追溯 manifest；参数、安装和验收步骤见 stress 指南。 Ascend NPU Burn 源码随仓库固定，标准镜像构建无需 --source，但管理员必须先准备并加载与目标节点匹配、且含 CANN toolkit/devlib、torch_npu 和 TBE 的基础镜像。构建器显式初始化 CANN 环境并在 wheel 前做 HAL/import 预检，最终镜像提供 upstream topology 所需的 pciutils/lspci，支持正常联网、临时代理和兼容 RPM/DEB 依赖闭包离线注入；镜像构建不需要宿主机 driver mount 或 NPU 设备，真正 driver/device 验证留在固定容器和 describe npu_burn 阶段。镜像完成后可用 scripts/stress/create_npu_burn_container.sh 动态映射节点全部 /dev/davinciN 并创建管理员维护的固定容器；CATMonitor 会交叉检查容器 /dev/davinciN 与 upstream lspci PCI topology，管理员必须显式选择验证后的 NPU Burn logical ID，不使用 npu-smi Phy-ID，all 仅用于整节点独占压测。

完成 stress 资产部署、节点脚本适配并在主配置中显式启用后，再执行：

catmonitor stress doctor -o table
catmonitor stress -o table

完整安装、配置、命令、Web 仪表盘、dfee 能效监控、Prometheus 接入与示例见使用手册。

健康度评分

场景	CPU	Memory	Disk	GPU/NPU	合计
无 GPU/NPU	30	40	30	—	100
有 GPU/NPU	10	20	10	60	100

得分	等级
90-100	Excellent
75-89	Good
60-74	Warning
0-59	Critical

扣分规则与阈值见 features/health/HEALTH_SPEC.md。

文档

文档	说明
使用手册	构建、安装、配置、命令、Web/dfee/exporter 用法与示例
SPEC.md	功能规格说明书（不含技术细节）
DESIGN.md	架构与模块设计
docs/CATMonitor_indi_list.md	采集指标清单（216 项）
docs/test_report.md	测试报告（无 NPU/GPU 系统测试）
docker/README.md	容器化部署（NPU/generic 镜像 + compose 编排）
features/health/HEALTH_SPEC.md	健康度评估规格
features/stress/README.md	可靠性压测入口、规格、设计与部署验收文档
features/web/Web_SPEC.md	Web 仪表盘规格
features/dfee/dfee_SPEC.md	能效监控模块规格
features/exporter/exporter_SPEC.md	Prometheus 导出模块规格
features/faultsub/faultsub_SPEC.md	故障订阅推送模块规格
features/stragglerout/stragglerout_SPEC.md	落后节点 KPI 输出模块规格

项目结构

CATMonitor/
├── cmd/catmonitor/          # 守护进程入口（daemon/collect/health/stress/list/version）
├── internal/
│   ├── collector/           # 采集核心：Collector 接口 + Registry + Scheduler
│   ├── collectors/          # 7 个部件采集器（cpu/memory/disk/gpu/npu/network/chassis）
│   ├── source/              # 来源层（15 包：proc/sys/ipmi/lscpu/mce/dmesg/dmidecode/statfs/smartctl + dcmi/npu_smi/hccn_tool/nvidia_smi + lspci）
│   ├── metrics/             # 指标采集目录（MetricSpec/Catalog/Filter + SetFeatureScope 白名单）
│   ├── config/ platform/ storage/   # 配置 / 平台适配 / 数据存储(JSONL)
├── features/                # 特性层
│   ├── health/              #   健康度评估
│   ├── stress/              #   STREAM/HPL/HPCG/Ascend NPU Burn 可靠性压测
│   ├── snapshot/            #   Snapshot 统一生产（PerCompWriter + GlobalWriter + 原子写/只读读）
│   ├── web/                 #   Web 仪表盘（catmonitor-web，只读消费 snapshot）
│   ├── dfee/                #   能效监控（catmonitor-dfee 独立二进制，package main，只读消费 snapshot + 内置 Prometheus exporter :9333）
│   ├── exporter/            #   Prometheus 导出（CachingStorage + /metrics）
│   ├── faultsub/            #   故障订阅推送（FaultStorage + HTTP Webhook + REST）
│   └── stragglerout/        #   落后节点 KPI 文件输出（StragglerStorage + KPIWriter）
├── configs/                 # 默认配置（catmonitor.yaml + metrics.yaml）
├── docker/                  # 容器化（Dockerfile.npu/generic + build.sh + compose + README）
├── docs/                    # 文档（指标清单 / 使用手册 / 测试报告）
├── tests/ scripts/          # 测试框架与数据 / 安装脚本
└── Makefile                # make all/build/web/dfee + DCMI 头自动探测

完整目录与模块设计见 DESIGN.md。

Directories ¶

Path	Synopsis
cmd
catmonitor command
features
dfee command
exporter
faultsub Package faultsub implements CATMonitor's fault subscription & push mechanism.	Package faultsub implements CATMonitor's fault subscription & push mechanism.
health
snapshot
stragglerout Package stragglerout writes a straggler-dedicated KPI time-series file so the straggler slow-node detector can consume NPU resource metrics collected by CATMonitor without running its own kpi_collect.sh.	Package stragglerout writes a straggler-dedicated KPI time-series file so the straggler slow-node detector can consume NPU resource metrics collected by CATMonitor without running its own kpi_collect.sh.
stress Package stress runs explicitly requested, high-load benchmark jobs.	Package stress runs explicitly requested, high-load benchmark jobs.
stress/cli Package cli implements the catmonitor stress command adapter.	Package cli implements the catmonitor stress command adapter.
stress/cmd/cpu-runner command
stress/cmd/cpu-runner-client command
stress/runnerapi Package runnerapi implements the private Unix-socket protocol used by the optional CATMonitor CPU stress runner.	Package runnerapi implements the private Unix-socket protocol used by the optional CATMonitor CPU stress runner.
web command
internal
collector
collectors/chassis Package chassis provides a collector for server-level environmental metrics (system power, inlet/outlet temperature, chassis fan speed) sourced from ipmitool SDR.	Package chassis provides a collector for server-level environmental metrics (system power, inlet/outlet temperature, chassis fan speed) sourced from ipmitool SDR.
collectors/cpu
collectors/disk
collectors/gpu
collectors/memory
collectors/network
collectors/npu
config
metrics Package metrics provides the metric collection catalog: the set of metrics each collector can emit (with metadata), a default catalog file on disk, and optional per-module override files.	Package metrics provides the metric collection catalog: the set of metrics each collector can emit (with metadata), a default catalog file on disk, and optional per-module override files.
platform
source
source/dcmi Package dcmi provides a data source that calls Huawei Ascend NPU's DCMI (Device Control Management Interface) C library libdcmi.so via CGo.	Package dcmi provides a data source that calls Huawei Ascend NPU's DCMI (Device Control Management Interface) C library libdcmi.so via CGo.
source/dmesg Package dmesg provides a data source that returns the kernel ring buffer text via the external `dmesg` command.	Package dmesg provides a data source that returns the kernel ring buffer text via the external `dmesg` command.
source/dmidecode Package dmidecode provides a data source that runs the external `dmidecode --type 17` command and parses its SMBIOS Memory Device entries into typed structs describing installed DIMMs.	Package dmidecode provides a data source that runs the external `dmidecode --type 17` command and parses its SMBIOS Memory Device entries into typed structs describing installed DIMMs.
source/hccn_tool Package hccn_tool provides a data source that wraps the `hccn_tool` command for NPU network diagnostics (bandwidth, RoCE speed/link).	Package hccn_tool provides a data source that wraps the `hccn_tool` command for NPU network diagnostics (bandwidth, RoCE speed/link).
source/ipmi Package ipmi provides a data source that queries an external `ipmitool` command for sensor readings (SDR) and instantaneous power (DCMI).	Package ipmi provides a data source that queries an external `ipmitool` command for sensor readings (SDR) and instantaneous power (DCMI).
source/lscpu Package lscpu provides a data source that runs the external `lscpu` command and parses its output into a CPU topology struct.	Package lscpu provides a data source that runs the external `lscpu` command and parses its output into a CPU topology struct.
source/lspci Package lspci provides a data source that runs the external `lspci` command and caches PCI device descriptions.	Package lspci provides a data source that runs the external `lspci` command and caches PCI device descriptions.
source/mce Package mce provides a data source that counts Machine Check Exception (MCE) events from the kernel log, distinguishing Corrected (CE) and Uncorrected (UCE/UE) errors per CPU socket.	Package mce provides a data source that counts Machine Check Exception (MCE) events from the kernel log, distinguishing Corrected (CE) and Uncorrected (UCE/UE) errors per CPU socket.
source/npu_smi Package npu_smi provides a data source that wraps the `npu-smi info -t` subcommands (topology, HCCS bandwidth).	Package npu_smi provides a data source that wraps the `npu-smi info -t` subcommands (topology, HCCS bandwidth).
source/nvidia_smi Package nvidia_smi provides a data source that wraps the `nvidia-smi` command for NVIDIA GPU metrics.	Package nvidia_smi provides a data source that wraps the `nvidia-smi` command for NVIDIA GPU metrics.
source/proc Package proc provides a data source that reads and parses the Linux /proc virtual filesystem.	Package proc provides a data source that reads and parses the Linux /proc virtual filesystem.
source/smartctl Package smartctl provides a data source that wraps the external `smartctl` command (from smartmontools) to query disk SMART health.	Package smartctl provides a data source that wraps the external `smartctl` command (from smartmontools) to query disk SMART health.
source/statfs Package statfs provides a data source that wraps the statfs(2) system call to report filesystem block usage for a given mount path.	Package statfs provides a data source that wraps the statfs(2) system call to report filesystem block usage for a given mount path.
source/sys Package sys provides a data source that reads and parses the Linux /sys virtual filesystem.	Package sys provides a data source that reads and parses the Linux /sys virtual filesystem.
storage
version

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL