InfiniBand Cluster Monitor

InfiniBand subnet health: port state matrix, link error counters, SM election, LID/GID mapping, IPoIB routing and UFM API integration with bandwidth saturation detection.

NVIDIA Mellanox

UFM

Open in Generator Open in Platform

You're viewing a preview

The full script, parameters, and execution console are available with a free account.

Already have access? Sign in →

Capabilities

IB port state matrix
Symbol/link error counters
Subnet Manager election check
LID/GID mapping table
IPoIB reachability
UFM REST integration
Bandwidth saturation alerts

Required inputs

Parameters the script accepts. Defaults shown; some are vendor- or context-gated.

Parameter	Type	Default	Notes
Symbol error threshold symbol_err_threshold	number	1000	—
Saturation alert (%) sat_pct	number	90	—
Use UFM REST API use_ufm	bool	true	—

Hint 1 — Name your vendors and OS versions

Mention the exact platforms you run (NVIDIA Mellanox, UFM) so the generated ai/gpu dc logic uses the right CLI/NETCONF syntax.

Hint 2 — State your real thresholds

Provide concrete values for Symbol error threshold, Saturation alert (%), Use UFM REST API instead of the defaults — they shape what counts as a fault.

Hint 3 — Prioritize the checks you need

This tool can iB port state matrix and symbol/link error counters. Ask for the subset relevant to your incident to keep output focused.

Hint 4 — Describe your inventory format

Tell the assistant whether your device list is CSV, YAML, or JSON and which columns it has, so parsing matches your data.

Sample inventory schema

Authoritative shape of the device/policy data this script consumes.

No bundled inventory sample. The script accepts standard device lists (CSV/YAML) with hostname, mgmt IP, vendor, and credentials.

Expected output

Reference terminal output the script should produce — used as a stylistic and structural target.

terminal

[2026-06-12] INFO  Querying UFM + ibdiagnet across IB subnet (512 ports)...

GUID              LID   STATE   WIDTH  SPEED   SYM-ERR  STATUS
0x248a07...c1     0x12  ACTIVE  4x     NDR     0        OK
0x248a07...d4     0x19  DOWN    -      -       0        LINK-DOWN ←
0x248a07...e7     0x22  ACTIVE  4x     NDR     4821     ERRORS ←

[CRITICAL] LID 0x19: link DOWN — GPU node gpu-h100-07 isolated
[WARNING]  LID 0x22: 4821 symbol errors — check cable/transceiver
SM: master=ufm-sm-01 (priority 15) · standby healthy