GPU 裸金属服务器运维

GPU 集群的稳定性从物理层开始。裸金属服务器是整个 AI 基础设施的”地基”——BMC 远程管理、固件一致性、功耗散热、PCI 拓扑、硬件诊断，每一层的问题都会向上传导为”训练任务莫名 OOM""NCCL 超时""GPU 掉卡”等上层故障。

概述

裸金属服务器运维覆盖服务器的完整生命周期：上架 → 固件配置 → OS 部署 → 运行监控 → 硬件诊断 → 维护更换 → 退役下架。与虚拟化环境不同，裸金属直接操作物理硬件，没有 Hypervisor 隔离层，因此对硬件状态、固件版本、物理拓扑的掌控要求更高。

flowchart TB
    subgraph 裸金属生命周期
        A[物理上架<br/>布线/电源] --> B[固件配置<br/>BIOS/BMC/NIC/GPU VBIOS]
        B --> C[OS 部署<br/>PXE/IPXE/MAAS]
        C --> D[驱动安装<br/>GPU/CUDA/NIC/OFED]
        D --> E[运行监控<br/>BMC传感器/DCGM/IPMI]
        E --> F{硬件诊断}
        F -->|正常| E
        F -->|异常| G[故障隔离<br/>隔离节点/更换部件]
        G --> H[修复验证]
        H --> E
        E --> I[退役下架<br/>数据擦除/固件重置]
    end

一、BMC 远程管理

1.1 BMC 是什么

BMC（Baseboard Management Controller）是服务器主板上的独立管理芯片，独立于主 OS 运行，提供带外（Out-of-Band）管理能力。即使服务器宕机、OS 崩溃，BMC 仍可通过独立网口远程访问。

厂商	BMC 品牌	常见型号	访问方式
Dell	iDRAC	iDRAC9 (Lifecycle Controller)	Web / SSH / IPMI / Redfish API
HPE	iLO	iLO 5 / iLO 6	Web / SSH / IPMI / RESTful API
浪潮	BMC	不定	Web / IPMI / Redfish
超聚变	iBMC	不定	Web / IPMI / Redfish
Supermicro	BMC / IPMI	AST2500 / AST2600	Web / IPMI / Redfish
Lenovo	XCC	XClarity Controller	Web / IPMI / Redfish

1.2 IPMI 命令速查

# ====== 基本信息 ======
# 查看传感器数据（温度/风扇/电压/电源）
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> sensor list

# 查看传感器精简列表
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> sdr

# 查看系统事件日志 (SEL)
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> sel list

# 查看系统信息
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> fru

# ====== 电源管理 ======
# 远程开机
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> chassis power on

# 远程关机（硬关机）
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> chassis power off

# 软关机（通过 ACPI）
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> chassis power soft

# 重启（硬重启）
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> chassis power cycle

# 查看电源状态
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> chassis power status

# ====== 远程控制台 / 虚拟介质 ======
# 通过 SOL (Serial Over LAN) 远程查看控制台
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> sol activate

# 退出 SOL
# 按 ~. 退出

# ====== SEL 管理 ======
# 清除 SEL
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> sel clear

# 查看 SEL 事件详情
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> sel elist

# ====== MCU / 固件信息 ======
# 查看 BMC 固件版本
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> mc info

# 查看 BIOS 版本
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> fru | grep -i bios

1.3 Redfish API（现代管理接口）

Redfish 是 DMTF 标准的 RESTful API，逐步取代 IPMI 成为带外管理的主流方式。

# 查看系统信息
curl -s -k -u <user>:<pass> \
  https://<BMC_IP>/redfish/v1/Systems/1 | python3 -m json.tool

# 远程开机
curl -s -k -u <user>:<pass> \
  -X POST https://<BMC_IP>/redfish/v1/Systems/1/Actions/ComputerSystem.Reset \
  -H "Content-Type: application/json" \
  -d '{"ResetType": "On"}'

# 查看 SEL（事件日志）
curl -s -k -u <user>:<pass> \
  https://<BMC_IP>/redfish/v1/Managers/1/LogServices/SEL/Entries | python3 -m json.tool

# 虚拟介质挂载 ISO（以 iDRAC 为例）
curl -s -k -u <user>:<pass> \
  -X POST https://<BMC_IP>/redfish/v1/Managers/1/VirtualMedia/CD/Actions/VirtualMedia.InsertMedia \
  -H "Content-Type: application/json" \
  -d '{"Image": "https://nfs-server/os-install.iso", "Inserted": true}'

1.4 BMC 批量管理

"""BMC 批量管理器 —— 统一管理集群中所有裸金属服务器"""

from dataclasses import dataclass, field
from typing import Optional
import subprocess
import json
import concurrent.futures


@dataclass
class BMCHost:
    hostname: str
    bmc_ip: str
    bmc_user: str
    bmc_password: str
    vendor: str = ""          # dell / hpe / supermicro
    model: str = ""
    rack: str = ""
    u_position: str = ""
    os_ip: str = ""           # 业务网络 IP
    status: str = "unknown"   # online / offline / maintenance


@dataclass
class SensorReading:
    name: str
    value: float
    unit: str
    status: str          # ok / warning / critical
    lower_threshold: float = 0
    upper_threshold: float = 0


class BMCManager:
    """BMC 批量管理器"""

    def __init__(self, hosts: list[BMCHost]):
        self.hosts = {h.hostname: h for h in hosts}

    def get_power_status_all(self) -> dict[str, str]:
        """批量获取电源状态"""
        results = {}
        with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:
            futures = {
                executor.submit(self._get_power_status, host): name
                for name, host in self.hosts.items()
            }
            for future in concurrent.futures.as_completed(futures):
                name = futures[future]
                try:
                    results[name] = future.result(timeout=10)
                except Exception as e:
                    results[name] = f"error: {e}"
        return results

    def get_sensors(self, hostname: str) -> list[SensorReading]:
        """获取传感器数据"""
        host = self.hosts.get(hostname)
        if not host:
            return []

        cmd = [
            "ipmitool", "-I", "lanplus",
            "-H", host.bmc_ip,
            "-U", host.bmc_user,
            "-P", host.bmc_password,
            "sensor", "list",
        ]
        try:
            result = subprocess.run(
                cmd, capture_output=True, text=True, timeout=15
            )
            return self._parse_sensors(result.stdout)
        except subprocess.TimeoutExpired:
            return []
        except Exception:
            return []

    def get_sel_events(self, hostname: str) -> list[dict]:
        """获取系统事件日志"""
        host = self.hosts.get(hostname)
        if not host:
            return []
        cmd = [
            "ipmitool", "-I", "lanplus",
            "-H", host.bmc_ip,
            "-U", host.bmc_user,
            "-P", host.bmc_password,
            "sel", "elist",
        ]
        try:
            result = subprocess.run(
                cmd, capture_output=True, text=True, timeout=15
            )
            return self._parse_sel(result.stdout)
        except Exception:
            return []

    def power_on(self, hostname: str) -> bool:
        return self._power_action(hostname, "on")

    def power_off(self, hostname: str) -> bool:
        return self._power_action(hostname, "off")

    def power_cycle(self, hostname: str) -> bool:
        return self._power_action(hostname, "cycle")

    def _power_action(self, hostname: str, action: str) -> bool:
        host = self.hosts.get(hostname)
        if not host:
            return False
        cmd = [
            "ipmitool", "-I", "lanplus",
            "-H", host.bmc_ip,
            "-U", host.bmc_user,
            "-P", host.bmc_password,
            "chassis", "power", action,
        ]
        try:
            result = subprocess.run(cmd, capture_output=True, timeout=15)
            return result.returncode == 0
        except Exception:
            return False

    def _get_power_status(self, host: BMCHost) -> str:
        cmd = [
            "ipmitool", "-I", "lanplus",
            "-H", host.bmc_ip,
            "-U", host.bmc_user,
            "-P", host.bmc_password,
            "chassis", "power", "status",
        ]
        try:
            result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
            return result.stdout.strip()
        except Exception as e:
            return f"error: {e}"

    @staticmethod
    def _parse_sensors(output: str) -> list[SensorReading]:
        sensors = []
        for line in output.strip().split("\n"):
            parts = [p.strip() for p in line.split("|")]
            if len(parts) >= 4:
                name = parts[0]
                try:
                    value = float(parts[1]) if parts[1] not in ("na", "") else 0
                except ValueError:
                    value = 0
                unit = parts[2]
                status = parts[3]
                sensors.append(SensorReading(
                    name=name, value=value, unit=unit, status=status
                ))
        return sensors

    @staticmethod
    def _parse_sel(output: str) -> list[dict]:
        events = []
        for line in output.strip().split("\n"):
            parts = [p.strip() for p in line.split("|")]
            if len(parts) >= 4:
                events.append({
                    "id": parts[0],
                    "timestamp": parts[1],
                    "sensor": parts[2],
                    "event": parts[3],
                })
        return events


# 使用示例
if __name__ == "__main__":
    hosts = [
        BMCHost("gpu-node-01", "10.0.10.1", "admin", "calvin",
                vendor="dell", model="R760xa", rack="R01", u_position="U18-U21"),
        BMCHost("gpu-node-02", "10.0.10.2", "admin", "calvin",
                vendor="dell", model="R760xa", rack="R01", u_position="U22-U25"),
    ]
    mgr = BMCManager(hosts)
    # 批量获取电源状态
    power_status = mgr.get_power_status_all()
    for name, status in power_status.items():
        print(f"{name}: {status}")
    # 获取传感器数据
    sensors = mgr.get_sensors("gpu-node-01")
    for s in sensors:
        if s.status != "ok":
            print(f"  [!] {s.name}: {s.value}{s.unit} ({s.status})")

二、BIOS/UEFI 配置

2.1 GPU 服务器关键 BIOS 设置

GPU 服务器的 BIOS 配置直接影响 GPU 性能和稳定性，以下是关键项：

BIOS 设置项	推荐值	说明
Above 4G Decoding	Enabled	必须！GPU 需要大于 4GB 的 MMIO 空间
SR-IOV	Enabled	虚拟化场景需要（MIG/vGPU）
VT-d / IOMMU	Enabled	PCI Passthrough 需要
NUMA Nodes per Socket	NPS4 (AMD) / Node Interleave=Disabled	保持 NUMA 拓扑，不要 Interleave
Memory Mapped I/O	≥ 64GB	8卡 GPU 服务器 MMIO 空间需求
PCIe Link Speed	Gen5 / Auto	确保不降速
Power Profile	Performance	关闭节能模式
C-States	C0/C1 only	深度省电会导致 GPU 通信延迟抖动
Hyper-Threading	Enabled	多数训练场景受益
Secure Boot	Disabled（或配置自定义密钥）	GPU 驱动签名兼容性

2.2 BIOS 批量配置

# Dell iDRAC: 通过 racadm 批量配置 BIOS
racadm -r <BMC_IP> -u <user> -p <pass> set BIOS.MemSettings.Above4GDecoding Enabled
racadm -r <BMC_IP> -u <user> -p <pass> set BIOS.IntegratedDevices.SRIOVEnable Enabled
racadm -r <BMC_IP> -u <user> -p <pass> set BIOS.ProcSettings.VirtualizationTechnology Enabled
racadm -r <BMC_IP> -u <user> -p <pass> set BIOS.ProcSettings.C1E Disabled
racadm -r <BMC_IP> -u <user> -p <pass> set BIOS.SysProfileSettings.SysProfile Performance

# 配置变更后需要重启生效
racadm -r <BMC_IP> -u <user> -p <pass> jobqueue create BIOS.Setup.1-1
# 重启服务器
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> chassis power cycle

"""BIOS 配置批量校验"""

from dataclasses import dataclass


@dataclass
class BIOSConfig:
    hostname: str
    above_4g_decoding: bool = False
    sriov: bool = False
    vt_d: bool = False
    numa_nodes_per_socket: int = 1
    mmio_size: str = ""
    c_states_disabled: bool = False
    power_profile: str = ""
    secure_boot: bool = False


class BIOSValidator:
    """BIOS 配置校验器"""

    REQUIRED_SETTINGS = {
        "above_4g_decoding": True,
        "sriov": True,
        "vt_d": True,
        "c_states_disabled": True,
        "power_profile": "Performance",
    }

    def validate(self, config: BIOSConfig) -> list[str]:
        """校验 BIOS 配置，返回问题列表"""
        issues = []
        if not config.above_4g_decoding:
            issues.append("[严重] Above 4G Decoding 未启用 —— GPU 无法正常工作")
        if not config.sriov:
            issues.append("[警告] SR-IOV 未启用 —— 影响 MIG/vGPU 功能")
        if not config.vt_d:
            issues.append("[警告] VT-d/IOMMU 未启用 —— 影响 PCI Passthrough")
        if not config.c_states_disabled:
            issues.append("[建议] C-States 未禁用 —— 可能导致 NCCL 通信延迟抖动")
        if config.power_profile.lower() != "performance":
            issues.append("[建议] Power Profile 非 Performance —— 影响训练性能")
        if config.secure_boot:
            issues.append("[注意] Secure Boot 已启用 —— 需确认 GPU 驱动签名兼容")
        return issues

三、固件管理

3.1 固件类型与版本矩阵

GPU 裸金属服务器涉及多个固件层，版本一致性是集群稳定性的基础。

固件类型	管理工具	更新方式	频率
BMC 固件	iDRAC/iLO Web	在线/虚拟介质	季度
BIOS	iDRAC/iLO	通过 BMC	季度
NIC 固件	mlxup (Mellanox) / ethtool -e	在线 OS 内	季度
HBA/NVMe 固件	厂商工具	在线	半年
GPU VBIOS	nvidia-flashrom	在线 OS 内	跟随驱动
NVLink Switch 固件	nvswitch CLI	在线	半年
PSU 固件	BMC	通过 BMC	年度
CPLD 固件	BMC	通过 BMC	年度

3.2 GPU VBIOS 与驱动版本矩阵

# 查看当前 GPU VBIOS 版本
nvidia-smi -q | grep -i "vbios"

# 查看当前驱动版本
nvidia-smi --query-gpu=driver_version --format=csv,noheader

# 查看 CUDA 版本
nvcc --version
cat /usr/local/cuda/version.json | grep version

# 查看固件兼容性
nvidia-smi -q | grep -A2 "GPU Firmware"

版本兼容性矩阵（示例）：

GPU 驱动	CUDA	VBIOS (H100)	推荐状态
535.129.03	12.2	96.00.49.00	✅ 生产验证
535.161.07	12.2	96.00.49.00	✅ 生产验证
545.23.06	12.3	96.00.52.00	⚠️ 测试中
550.40.07	12.4	96.00.55.00	⚠️ 新功能

关键原则：GPU 驱动、CUDA、VBIOS 三者版本必须兼容。集群内所有节点版本必须一致。升级时先在测试节点验证，再灰度推广。

3.3 固件批量更新脚本

"""固件批量更新管理器"""

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class FirmwareType(Enum):
    BMC = "bmc"
    BIOS = "bios"
    NIC = "nic"
    GPU_VBIOS = "gpu_vbios"
    NVSWITCH = "nvswitch"
    PSU = "psu"
    CPLD = "cpld"


class UpdateStatus(Enum):
    PENDING = "pending"
    DOWNLOADING = "downloading"
    UPLOADING = "uploading"
    INSTALLING = "installing"
    REBOOTING = "rebooting"
    SUCCESS = "success"
    FAILED = "failed"
    ROLLBACK = "rollback"


@dataclass
class FirmwareVersion:
    component: FirmwareType
    current_version: str
    target_version: str
    compatible_driver: str = ""
    release_notes: str = ""


@dataclass
class UpdateTask:
    hostname: str
    component: FirmwareType
    current_version: str
    target_version: str
    status: UpdateStatus = UpdateStatus.PENDING
    error: str = ""
    started_at: str = ""
    completed_at: str = ""


class FirmwareUpdateManager:
    """固件批量更新管理器"""

    def __init__(self, firmware_dir: str):
        self.firmware_dir = firmware_dir
        self.tasks: list[UpdateTask] = []

    def plan_update(self, hosts: list[str],
                    component: FirmwareType,
                    target_version: str,
                    current_versions: dict[str, str]) -> list[UpdateTask]:
        """规划更新任务，只更新版本不一致的节点"""
        for host in hosts:
            current = current_versions.get(host, "")
            if current != target_version:
                self.tasks.append(UpdateTask(
                    hostname=host,
                    component=component,
                    current_version=current,
                    target_version=target_version,
                ))
        return [t for t in self.tasks if t.status == UpdateStatus.PENDING]

    def execute_update(self, task: UpdateTask) -> bool:
        """执行单个更新任务"""
        task.status = UpdateStatus.INSTALLING
        # 实际执行固件更新命令
        # Dell iDRAC: racadm firmware update
        # HPE iLO: ilorest firmware upload
        # GPU: nvidia-flashrom
        # NIC: mlxup
        # 返回 True/False
        return True

    def batch_update(self, max_concurrent: int = 1) -> dict:
        """批量更新（串行或限并发）"""
        results = {"success": 0, "failed": 0, "skipped": 0}
        for task in self.tasks:
            if task.status != UpdateStatus.PENDING:
                results["skipped"] += 1
                continue
            success = self.execute_update(task)
            if success:
                task.status = UpdateStatus.SUCCESS
                results["success"] += 1
            else:
                task.status = UpdateStatus.FAILED
                results["failed"] += 1
        return results

四、功耗与散热管理

4.1 功耗监控

GPU 服务器是”电老虎”，单台 8 卡 H100 服务器满载功耗可达 10-12kW。

# GPU 功耗实时监控
nvidia-smi --query-gpu=index,name,power.draw,power.limit,temperature.gpu,clocks.sm,clocks.mem \
  --format=csv -l 1

# 查看功耗限制
nvidia-smi -q -d POWER

# 设置功耗限制（W）
sudo nvidia-smi -pl 700    # 限制到 700W

# GPU 持久化模式（减少驱动初始化开销）
sudo nvidia-smi -pm 1

# 查看节点总功耗（通过 BMC）
ipmitool -I lanplus -H <BMC_IP> -U <user> -P <pass> dcmi power reading

4.2 散热与热节流

"""GPU 热管理监控器"""

from dataclasses import dataclass, field
from enum import Enum
import subprocess
import json


class ThermalState(Enum):
    NORMAL = "normal"           # < 70°C
    WARM = "warm"              # 70-80°C
    HOT = "hot"                # 80-85°C (热节流阈值附近)
    CRITICAL = "critical"       # > 85°C (强制降频)
    EMERGENCY = "emergency"     # > 90°C (自动关机保护)


@dataclass
class GPUThermalStatus:
    gpu_index: int
    temperature: float           # GPU 温度 °C
    power_draw: float            # 当前功耗 W
    power_limit: float           # 功耗限制 W
    clock_sm: int                # SM 时钟 MHz
    clock_mem: int               # 显存时钟 MHz
    clock_throttle_reasons: str  # 降频原因
    fan_speed: int = 0           # 风扇转速 % (如适用)
    state: ThermalState = ThermalState.NORMAL


class GPUThermalMonitor:
    """GPU 热管理监控器"""

    THRESHOLDS = {
        "warm": 70,
        "hot": 80,
        "throttle": 83,      # H100 默认热节流阈值
        "critical": 85,
        "emergency": 90,
    }

    def get_status(self) -> list[GPUThermalStatus]:
        """获取所有 GPU 的热状态"""
        cmd = [
            "nvidia-smi",
            "--query-gpu=index,temperature.gpu,power.draw,power.limit,"
            "clocks.sm,clocks.mem,clocks_throttle_reasons.active,pstate",
            "--format=csv,noheader,nounits",
        ]
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
        statuses = []
        for line in result.stdout.strip().split("\n"):
            parts = [p.strip() for p in line.split(",")]
            if len(parts) >= 7:
                temp = float(parts[1])
                statuses.append(GPUThermalStatus(
                    gpu_index=int(parts[0]),
                    temperature=temp,
                    power_draw=float(parts[2]),
                    power_limit=float(parts[3]),
                    clock_sm=int(parts[4]),
                    clock_mem=int(parts[5]),
                    clock_throttle_reasons=parts[6],
                    state=self._classify_temp(temp),
                ))
        return statuses

    def check_thermal_issues(self) -> list[dict]:
        """检查热管理问题"""
        statuses = self.get_status()
        issues = []
        for s in statuses:
            if s.state == ThermalState.EMERGENCY:
                issues.append({
                    "severity": "critical",
                    "gpu": s.gpu_index,
                    "message": f"GPU {s.gpu_index} 温度 {s.temperature}°C "
                               f"超过紧急阈值 {self.THRESHOLDS['emergency']}°C",
                    "action": "立即降低负载或关闭节点",
                })
            elif s.state == ThermalState.CRITICAL:
                issues.append({
                    "severity": "critical",
                    "gpu": s.gpu_index,
                    "message": f"GPU {s.gpu_index} 温度 {s.temperature}°C "
                               f"触发强制降频",
                    "action": "检查散热系统和风扇",
                })
            elif s.state == ThermalState.HOT:
                issues.append({
                    "severity": "warning",
                    "gpu": s.gpu_index,
                    "message": f"GPU {s.gpu_index} 温度 {s.temperature}°C "
                               f"接近热节流阈值",
                    "action": "降低功耗限制或改善散热",
                })
            # 检查是否已热节流
            if "Thermal" in s.clock_throttle_reasons:
                issues.append({
                    "severity": "warning",
                    "gpu": s.gpu_index,
                    "message": f"GPU {s.gpu_index} 正在热节流降频",
                    "action": "散热不足，检查风扇/液冷",
                })
        return issues

    def _classify_temp(self, temp: float) -> ThermalState:
        if temp >= self.THRESHOLDS["emergency"]:
            return ThermalState.EMERGENCY
        if temp >= self.THRESHOLDS["critical"]:
            return ThermalState.CRITICAL
        if temp >= self.THRESHOLDS["throttle"]:
            return ThermalState.HOT
        if temp >= self.THRESHOLDS["warm"]:
            return ThermalState.WARM
        return ThermalState.NORMAL

4.3 数据中心散热与功耗规划

GPU 型号	单卡 TDP	8卡服务器 TDP	建议机柜功率	散热方式
A100 80GB SXM	400W	~4.2kW	8-10kW	风冷/液冷
H100 80GB SXM5	700W	~10kW	15-20kW	液冷（推荐）
H200 141GB SXM	700W	~10kW	15-20kW	液冷
B200 192GB SXM	1000W	~14kW	25-30kW	液冷（必须）
L40S 350W	350W	~3.5kW	6-8kW	风冷

关键：H100/B200 级别 GPU 服务器满载功耗远超传统风冷机柜的 8kW 限制，必须规划液冷（冷板式/浸没式）和高密度供电（双路 30A/三相）。

五、PCI 拓扑与 GPU 枚举

5.1 查看 PCI 拓扑

# 查看 GPU 的 PCI 拓扑（关键！）
nvidia-smi topo -m

# 示例输出（8卡 H100 服务器）：
#         GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7  NIC0    NIC1    NIC2    NIC3
# GPU0     X    NV12  NV12  NV12  NV12  NV12  NV12  NV12  PIX     NODE    SYS     SYS
# GPU1    NV12   X    NV12  NV12  NV12  NV12  NV12  NV12  PIX     NODE    SYS     SYS
# ...
# 解读：
#   NV12 = 12条 NVLink 连接
#   PIX = 同一 PCIe Switch 下
#   NODE = 同一 NUMA Node 但不同 PCIe Switch
#   SYS = 跨 CPU Socket（跨 NUMA Node），延迟最高

# 查看 PCIe 链路信息
nvidia-smi -q | grep -A5 "PCI"

# 查看详细 PCIe 拓扑
lspci -tv | grep -i nvidia

# 查看 NUMA 亲和性
numactl --hardware
cat /sys/bus/pci/devices/<pci_address>/numa_node

# 查看 GPU 与 NIC 的 NUMA 亲和性
# 确保训练任务的 GPU 和 NIC 在同一 NUMA Node
nvidia-smi topo -m | grep -E "NIC|NIC"

5.2 GPU PCI 地址映射

"""GPU PCI 地址与设备号映射管理"""

from dataclasses import dataclass
import subprocess
import re


@dataclass
class GPUDeviceInfo:
    gpu_index: int           # nvidia-smi 中的 GPU 编号
    pci_address: str         # PCI 地址 (0000:01:00.0)
    numa_node: int           # NUMA 节点
    pcie_link_gen: int       # PCIe 代数
    pcie_link_width: str     # PCIe 带宽 (x16)
    nvlinks: int             # NVLink 连接数
    uuid: str                # GPU UUID


class GPUDeviceMapper:
    """GPU 设备映射管理器"""

    def get_all_gpus(self) -> list[GPUDeviceInfo]:
        """获取所有 GPU 的设备信息"""
        cmd = [
            "nvidia-smi",
            "--query-gpu=index,pci.bus_id,pci.link.gen.current,"
            "pci.link.width.current,uuid",
            "--format=csv,noheader",
        ]
        result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
        gpus = []
        for line in result.stdout.strip().split("\n"):
            parts = [p.strip() for p in line.split(",")]
            if len(parts) >= 5:
                pci_addr = self._normalize_pci_addr(parts[1])
                numa = self._get_numa_node(pci_addr)
                gpus.append(GPUDeviceInfo(
                    gpu_index=int(parts[0]),
                    pci_address=pci_addr,
                    numa_node=numa,
                    pcie_link_gen=int(parts[2]),
                    pcie_link_width=parts[3],
                    nvlinks=0,  # 需要从 topo -m 获取
                    uuid=parts[4],
                ))
        return gpus

    def check_pcie_health(self, gpus: list[GPUDeviceInfo]) -> list[str]:
        """检查 PCIe 链路健康度"""
        issues = []
        for gpu in gpus:
            if gpu.pcie_link_gen < 5:
                issues.append(
                    f"GPU {gpu.gpu_index}: PCIe Gen{gpu.pcie_link_gen} "
                    f"(期望 Gen5) —— 可能降速"
                )
            if "x16" not in gpu.pcie_link_width:
                issues.append(
                    f"GPU {gpu.gpu_index}: PCIe {gpu.pcie_link_width} "
                    f"(期望 x16) —— 带宽不足"
                )
        return issues

    def get_numa_affinity_map(self, gpus: list[GPUDeviceInfo]) -> dict[int, list[int]]:
        """获取 NUMA → GPU 映射"""
        numa_map: dict[int, list[int]] = {}
        for gpu in gpus:
            numa_map.setdefault(gpu.numa_node, []).append(gpu.gpu_index)
        return numa_map

    @staticmethod
    def _normalize_pci_addr(bus_id: str) -> str:
        """将 nvidia-smi 的 bus_id 转为标准 PCI 地址"""
        # 00000000:01:00.0 → 0000:01:00.0
        match = re.match(r'([0-9a-f]+):([0-9a-f]+):([0-9a-f]+)\.([0-9a-f]+)',
                         bus_id, re.I)
        if match:
            return f"0000:{match.group(2)}:{match.group(3)}.{match.group(4)}"
        return bus_id

    @staticmethod
    def _get_numa_node(pci_addr: str) -> int:
        """读取 GPU 的 NUMA 节点"""
        try:
            with open(f"/sys/bus/pci/devices/{pci_addr}/numa_node") as f:
                return int(f.read().strip())
        except FileNotFoundError:
            return -1

六、硬件诊断与故障隔离

6.1 常见硬件故障类型

故障类型	表现	诊断工具	处理方式
GPU ECC 错误	训练任务出错/崩溃	nvidia-smi -q -d ECC	记录/隔离/更换
GPU Xid 错误	GPU 掉卡/卡死	dmesg + nvidia-smi	见 GPU Xid 错误排查手册
GPU 掉卡	nvidia-smi 看不到 GPU	lspci | grep NVIDIA	重新插拔/更换
NVLink 降级	训练速度下降	nvidia-smi nvlink -s	检查物理连接
NIC 故障	NCCL 超时	ibstat / ethtool	更换网卡/线缆
内存故障	内核 oops / 训练 OOM	dmidecode / mcelog	更换内存条
PSU 故障	BMC 告警 / 随机关机	IPMI sensor	更换电源
风扇故障	温度告警	IPMI sensor	更换风扇
CPU 故障	随机 crash	mcelog / dmesg	更换 CPU
PCIe 故障	GPU/NIC 间歇性消失	lspci -vvv	检查插槽/riser

6.2 硬件健康检查脚本

"""裸金属服务器硬件健康检查"""

from dataclasses import dataclass, field
from enum import Enum
import subprocess


class HealthStatus(Enum):
    HEALTHY = "healthy"
    WARNING = "warning"
    CRITICAL = "critical"
    OFFLINE = "offline"


@dataclass
class HealthCheckResult:
    hostname: str
    status: HealthStatus
    checks: dict[str, dict] = field(default_factory=dict)
    summary: str = ""

    @property
    def is_healthy(self) -> bool:
        return self.status == HealthStatus.HEALTHY


class HardwareHealthChecker:
    """硬件健康检查器"""

    def check_all(self, hostname: str) -> HealthCheckResult:
        """执行全套硬件检查"""
        result = HealthCheckResult(hostname=hostname, status=HealthStatus.HEALTHY)

        # 1. GPU 健康
        result.checks["gpu"] = self._check_gpu()

        # 2. 内存健康
        result.checks["memory"] = self._check_memory()

        # 3. 磁盘健康
        result.checks["disk"] = self._check_disk()

        # 4. 网卡健康
        result.checks["nic"] = self._check_nic()

        # 5. BMC 传感器
        result.checks["bmc_sensors"] = self._check_bmc_sensors()

        # 汇总
        critical = sum(1 for c in result.checks.values()
                       if c.get("status") == "critical")
        warning = sum(1 for c in result.checks.values()
                      if c.get("status") == "warning")

        if critical > 0:
            result.status = HealthStatus.CRITICAL
            result.summary = f"{critical} 项严重问题, {warning} 项警告"
        elif warning > 0:
            result.status = HealthStatus.WARNING
            result.summary = f"{warning} 项警告"
        else:
            result.summary = "所有检查通过"

        return result

    def _check_gpu(self) -> dict:
        """GPU 健康检查"""
        issues = []
        try:
            # 检查 GPU 数量
            result = subprocess.run(
                ["nvidia-smi", "-L"], capture_output=True, text=True, timeout=10
            )
            gpu_count = len(result.stdout.strip().split("\n"))
            if gpu_count < 8:  # 假设 8 卡服务器
                issues.append(f"GPU 数量异常: {gpu_count}/8")

            # 检查 ECC 错误
            result = subprocess.run(
                ["nvidia-smi", "-q", "-d", "ECC"],
                capture_output=True, text=True, timeout=10
            )
            if "Uncorrectable" in result.stdout:
                if int(result.stdout.split("Uncorrectable")[1].split(":")[1].strip()) > 0:
                    issues.append("检测到不可纠正 ECC 错误")

            # 检查掉卡
            result = subprocess.run(
                ["nvidia-smi", "--query-gpu=index", "--format=csv,noheader"],
                capture_output=True, text=True, timeout=10
            )
            gpu_indices = [int(x.strip()) for x in result.stdout.strip().split("\n")]
            for i in range(8):
                if i not in gpu_indices:
                    issues.append(f"GPU {i} 掉卡")
        except Exception as e:
            return {"status": "critical", "issues": [f"GPU检查失败: {e}"]}

        return {
            "status": "critical" if issues else "healthy",
            "issues": issues,
        }

    def _check_memory(self) -> dict:
        """内存检查"""
        issues = []
        try:
            # 检查 dmesg 中的内存错误
            result = subprocess.run(
                ["dmesg", "--time-format=iso"],
                capture_output=True, text=True, timeout=10
            )
            if "EDAC" in result.stdout or "Machine Check" in result.stdout:
                issues.append("dmesg 中检测到内存错误 (EDAC/MCE)")
        except Exception:
            pass
        return {"status": "critical" if issues else "healthy", "issues": issues}

    def _check_disk(self) -> dict:
        """磁盘 SMART 检查"""
        issues = []
        try:
            result = subprocess.run(
                ["smartctl", "--scan"], capture_output=True, text=True, timeout=10
            )
            for line in result.stdout.strip().split("\n"):
                if "/dev/" in line:
                    dev = line.split()[0]
                    r = subprocess.run(
                        ["smartctl", "-H", dev],
                        capture_output=True, text=True, timeout=10
                    )
                    if "PASSED" not in r.stdout:
                        issues.append(f"磁盘 {dev} SMART 检查未通过")
        except Exception:
            pass
        return {"status": "warning" if issues else "healthy", "issues": issues}

    def _check_nic(self) -> dict:
        """网卡链路检查"""
        issues = []
        try:
            result = subprocess.run(
                ["ip", "-o", "link", "show"],
                capture_output=True, text=True, timeout=10
            )
            for line in result.stdout.strip().split("\n"):
                if "NO-CARRIER" in line and "eth" in line:
                    iface = line.split(":")[1].strip().split("@")[0]
                    issues.append(f"网卡 {iface} 链路断开")
        except Exception:
            pass
        return {"status": "warning" if issues else "healthy", "issues": issues}

    def _check_bmc_sensors(self) -> dict:
        """BMC 传感器检查"""
        issues = []
        try:
            result = subprocess.run(
                ["ipmitool", "sdr"],
                capture_output=True, text=True, timeout=10
            )
            for line in result.stdout.strip().split("\n"):
                parts = [p.strip() for p in line.split("|")]
                if len(parts) >= 4 and parts[3] != "ok" and parts[3] != "na":
                    issues.append(f"传感器异常: {parts[0]} = {parts[1]}{parts[2]} ({parts[3]})")
        except Exception:
            pass
        return {"status": "critical" if issues else "healthy", "issues": issues}

6.3 故障隔离策略

flowchart TB
    A[检测到硬件故障] --> B{故障严重度}
    B -->|GPU ECC/Xid| C[标记 GPU 为 unhealthy]
    B -->|GPU 掉卡| D[标记节点为 NotReady]
    B -->|PSU/风扇| E[BMC 告警<br/>评估是否紧急]
    B -->|NIC 故障| F[标记 NIC 为 down<br/>NCCL 回退到 Socket]

    C --> G[cordon + drain 节点]
    D --> G
    E -->|紧急| G
    F --> H[在调度层面排除该 NIC]

    G --> I[人工确认故障]
    I --> J[更换硬件部件]
    J --> K[硬件健康检查]
    K -->|通过| L[uncordon 节点<br/>重新上线]
    K -->|未通过| J

# K8s 层面隔离故障节点
kubectl cordon gpu-node-03         # 禁止调度
kubectl drain gpu-node-03 --ignore-daemonsets --delete-emptydir-data  # 驱逐工作负载

# 在节点层面标记 GPU 为不可用（GPU Operator + Node Feature Discovery）
kubectl label node gpu-node-03 nvidia.com/gpu.count=0
kubectl taint nodes gpu-node-03 hardware-issue=true:NoSchedule

# 检查修复后重新上线
kubectl uncordon gpu-node-03
kubectl label node gpu-node-03 nvidia.com/gpu.count=8
kubectl taint nodes gpu-node-03 hardware-issue=true:NoSchedule-

七、裸金属 OS 部署

7.1 PXE 网络启动

已有笔记 集群自动化部署方案 覆盖了 PXE 服务端配置，这里补充 GPU 裸金属特有的 OS 调优。

7.2 GPU 服务器 OS 内核调优

# /etc/sysctl.conf —— GPU 服务器专用调优

# 1. 内存管理
vm.swappiness = 1                    # 几乎禁用 swap，避免 GPU 训练数据被 swap
vm.overcommit_memory = 1             # 允许内存超额分配（训练框架常需要）
vm.max_map_count = 262144            # 大页表映射
vm.dirty_ratio = 10                  # 降低脏页比例，缩短 fsync 延迟
vm.dirty_background_ratio = 5

# 2. 网络调优（RDMA/RoCE 场景）
net.core.rmem_max = 2147483647       # 最大接收缓冲区 2GB
net.core.wmem_max = 2147483647       # 最大发送缓冲区 2GB
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_rmem = 4096 87380 2147483647
net.ipv4.tcp_wmem = 4096 65536 2147483647

# 3. 文件描述符
fs.file-max = 2097152
fs.nr_open = 2097152

# 4. hugepages（GPU 训练大内存映射）
# 预留 1000 个 1GB 大页
vm.nr_hugepages = 0                  # 使用透明大页替代
echo always > /sys/kernel/mm/transparent_hugepage/enabled

# 5. IRQ 亲和性
# 让 GPU 和 NIC 的中断分配到正确的 NUMA Node

# /etc/security/limits.conf —— 提高资源限制
* soft nofile 1048576
* hard nofile 1048576
* soft memlock unlimited
* hard memlock unlimited
* soft stack unlimited
* hard stack unlimited

# 内核启动参数 (GRUB)
# /etc/default/grub
GRUB_CMDLINE_LINUX="... intel_iommu=on iommu=pt \
  pci=realloc hugepagesz=1G hugepages=0 \
  transparent_hugepage=always \
  numa_balancing=disable \
  processor.max_cstate=1 \
  idle=poll \
  pcie_aspm=off"

# 说明：
# - intel_iommu=on iommu=pt: 启用 IOMMU（PCI Passthrough）
# - numa_balancing=disable: 关闭自动 NUMA 负载均衡（训练任务自行绑定）
# - processor.max_cstate=1: 限制 C-State（降低 GPU 通信延迟抖动）
# - idle=poll: CPU 空闲时轮询（极致延迟，代价是功耗）
# - pcie_aspm=off: 关闭 PCIe 节能（保持链路全速）

# 更新 GRUB
grub2-mkconfig -o /boot/grub2/grub.cfg

八、裸金属运维 Checklist

日常巡检 Checklist

检查项	命令	频率	告警阈值
GPU 数量	nvidia-smi -L | wc -l	每小时	!= 预期数量
GPU ECC 错误	nvidia-smi -q -d ECC	每5分钟	uncorrectable > 0
GPU 温度	nvidia-smi --query-gpu=temperature.gpu	每30秒	> 83°C
GPU 功耗	nvidia-smi --query-gpu=power.draw	每30秒	> TDP
NVLink 状态	nvidia-smi nvlink -s	每5分钟	任何 link down
BMC 传感器	ipmitool sdr	每5分钟	非 ok 状态
SEL 事件	ipmitool sel elist	每小时	新增 critical 事件
PSU 状态	ipmitool sdr | grep -i psu	每5分钟	非冗余/故障
NIC 链路	ip -o link show	每30秒	NO-CARRIER
磁盘 SMART	smartctl -H /dev/sd*	每天	FAILED
dmesg 错误	dmesg | grep -i error	每5分钟	新增硬件错误
节点功耗	ipmitool dcmi power reading	每5分钟	> 额定功率 90%

上架前 Checklist

• 物理安装：GPU 卡、内存条、网卡、电源模块
• 线缆连接：电源线（双路）、网线/光模块、管理线
• BMC 网络连通性：能 ping 通 BMC IP
• BIOS 配置：Above 4G / SR-IOV / VT-d / NUMA / C-States
• BMC 固件版本：符合基线
• BIOS 固件版本：符合基线
• GPU 固件版本：符合基线
• OS 安装：PXE 启动 + 自动化安装
• 内核参数：sysctl + GRUB 调优
• GPU 驱动安装：版本一致性
• nvidia-smi 验证：8卡可见，PCIe Gen5 x16
• NVLink 验证：所有 NVLink 活跃
• 网络验证：RDMA/RoCE 连通性测试
• 散热验证：满载温度 < 83°C
• 功耗验证：满载功耗 < 额定 90%
• 健康检查脚本全部通过

关联知识

• GPU 集群运维知识总览 — 本篇是总览的裸金属层补充
• 集群自动化部署方案 — PXE/MAAS 裸金属 OS 部署流程
• GPU 驱动与固件管理 — GPU 驱动和 VBIOS 管理细节
• GPU 服务器硬件选型指南 — 硬件选型参考
• NVIDIA GPU 架构演进 — GPU 架构背景知识
• NVLink 与 NVSwitch 拓扑详解 — NVLink 拓扑与 PCI 拓扑关联
• GPU Xid 错误排查手册 — GPU Xid 错误诊断
• NCCL 通信故障诊断指南 — NCCL 通信故障与硬件关联
• DCGM 监控体系详解 — GPU 监控指标体系
• RDMA 与 InfiniBand 详解 — 网络硬件与固件
• SRE 稳定性工程总览 — 裸金属层是稳定性工程的基础
• 高可用架构设计总览 — 硬件冗余设计

参考资源

• Dell iDRAC9 User’s Guide
• HPE iLO 5 User Guide
• NVIDIA DataCenter GPU Driver Documentation
• IPMI 2.0 Specification
• DMTF Redfish API Specification
• NVIDIA GPU Best Practices for Data Centers

学习时间

约 8-10 小时（含 BMC 实操 + 硬件诊断实践）

状态

• 理解 BMC 远程管理体系（IPMI/Redfish）
• 掌握 GPU 服务器关键 BIOS 配置
• 理解固件类型与版本管理
• 掌握功耗与散热管理
• 能查看和分析 PCI 拓扑
• 能执行硬件健康检查
• 掌握故障隔离策略
• 掌握 GPU 服务器 OS 内核调优
• 实际操作 BMC 批量管理
• 完成一次完整的硬件诊断流程