文章

集群自动化部署方案

#gpu#automation#deployment#ansible#bare-metal

集群自动化部署方案

GPU 集群的自动化部署涵盖从裸金属上架到可运行训练任务的全流程。本指南覆盖 OS 安装、驱动部署、K8s 初始化、GPU Operator 安装和配置管理。

概述

GPU 集群部署比通用 K8s 集群多出 GPU 驱动、CUDA、容器运行时、Device Plugin、DCGM 等组件。自动化程度直接影响运维效率和集群可扩展性。

📖 核心流程（已掌握）

裸金属上架
  → PXE/iDRAC OS 安装
    → Ansible 基础配置（网络/存储/安全）
      → GPU 驱动 + CUDA 安装
        → Container Runtime (nvidia-docker) 安装
          → K8s Cluster (kubeadm/k3s) 初始化
            → GPU Operator 安装
              → 监控采集 (dcgm-exporter + node-exporter)
                → 接入调度器 (Volcano)
                  → 健康检查 → 上线

📖 1. PXE/MAAS 裸金属 OS 部署（已掌握）

1.1 PXE 服务端配置

# 安装所需服务
yum install -y dhcp-server tftp-server syslinux httpd

# TFTP 目录结构
mkdir -p /var/lib/tftpboot/{pxelinux.cfg,ubuntu22,rocky9}
cp /usr/share/syslinux/{pxelinux.0,menu.c32,ldlinux.c32} /var/lib/tftpboot/

# DHCP 配置 /etc/dhcp/dhcpd.conf
subnet 10.0.0.0 netmask 255.255.0.0 {
    range 10.0.100.1 10.0.100.254;
    option routers 10.0.0.1;
    option domain-name-servers 10.0.0.10;
    filename "pxelinux.0";
    next-server 10.0.0.5;
}

1.2 Kickstart 示例（Rocky 9）

# /var/lib/tftpboot/pxelinux.cfg/default
DEFAULT menu.c32
PROMPT 0
TIMEOUT 50

LABEL rocky9-gpu
    MENU LABEL Rocky 9 GPU Node
    KERNEL rocky9/vmlinuz
    APPEND initrd=rocky9/initrd.img inst.ks=http://10.0.0.5/ks/rocky9-gpu.ks ip=dhcp
# kickstart /var/www/html/ks/rocky9-gpu.ks
url --url="http://10.0.0.5/rocky9/"
lang en_US.UTF-8
keyboard us
timezone Asia/Shanghai
rootpw --iscrypted $6$xxx
bootloader --location=mbr --append="rd.driver.blacklist=nouveau modprobe.blacklist=nouveau"
zerombr
clearpart --all --initlabel
autopart --type=lvm
network --bootproto=dhcp --hostname=gpu%02d --device=eno1
firewall --disabled
selinux --disabled
services --enabled=sshd,chronyd
reboot

%packages --ignoremissing
@core
@base
chrony
vim
wget
curl
nfs-utils
pciutils
%end

%post --log=/root/ks-post.log
# 禁用 nouveau 驱动
cat > /etc/modprobe.d/blacklist-nouveau.conf << EOF
blacklist nouveau
options nouveau modeset=0
EOF
dracut --force
%end

1.3 MAAS 快速部署（Ubuntu）

# MAAS 服务器安装
sudo snap install maas
sudo maas init region+rack --database-uri maas-test-db://

# 添加节点：节点侧 PXE 启动即可自动发现
# 设置节点电源管理 (IPMI)
maas $PROFILE machine update $SYSTEM_ID \
  power_type=ipmi \
  power_parameters_power_address=10.0.1.100 \
  power_parameters_power_user=admin \
  power_parameters_power_pass=password

# 批量部署 Ubuntu 22.04
maas $PROFILE machine deploy $SYSTEM_ID \
  distro_series=focal

📖 2. Ansible 自动化编排（已掌握）

2.1 Inventory 结构

# inventories/gpu-cluster/hosts.yml
all:
  children:
    control:
      hosts:
        mgmt01: { ansible_host: 10.0.0.10 }
    gpu_nodes:
      hosts:
        gpu01: { ansible_host: 10.0.1.1, gpu_model: "A100-80G", gpu_count: 8 }
        gpu02: { ansible_host: 10.0.1.2, gpu_model: "A100-80G", gpu_count: 8 }
        gpu03: { ansible_host: 10.0.1.3, gpu_model: "H100",     gpu_count: 8 }
    inference_nodes:
      hosts:
        inf01: { ansible_host: 10.0.2.1, gpu_model: "L40S", gpu_count: 4 }
    vars:
      cluster_name: "prod-gpu-cluster"
      k8s_version: "1.29.6"
      cuda_version: "12.4"
      nvidia_driver_version: "550.90.07"

2.2 核心 Playbook 结构

# site.yml — 总入口
- name: Deploy GPU Cluster
  hosts: all
  gather_facts: true
  roles:
    - common            # SSH、NTP、内核参数
    - nvidia-driver     # 驱动 + CUDA 运行时
    - containerd        # 容器运行时 + nvidia-container-toolkit
    - k8s               # kubeadm/k3s 初始化
    - gpu-operator      # NVIDIA GPU Operator Helm 部署
    - monitoring        # DCGM Exporter + Node Exporter + Prometheus

2.3 nvidia-driver Role

# roles/nvidia-driver/tasks/main.yml
- name: Blacklist nouveau
  copy:
    dest: /etc/modprobe.d/blacklist-nouveau.conf
    content: |
      blacklist nouveau
      options nouveau modeset=0

- name: Install NVIDIA driver
  shell: |
    yum-config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
    dnf install -y --setopt=obsoletes=0 \
      nvidia-driver-{{ nvidia_driver_version }} \
      cuda-toolkit-{{ cuda_version.split('.')[:2] | join('-') }}
  args:
    creates: /usr/bin/nvidia-smi

- name: Enable persistence mode
  copy:
    dest: /etc/systemd/system/nvidia-persistenced.service
    content: |
      [Unit]
      Description=NVIDIA Persistence Daemon
      [Service]
      ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced
      [Install]
      WantedBy=multi-user.target
  notify: restart nvidia-persistenced

- name: Set GPU performance defaults
  shell: |
    nvidia-smi -pm 1
    nvidia-smi -ac {{ gpu_mem_clock }},{{ gpu_core_clock }}
    nvidia-smi -e 0    # 禁用 ECC（训练场景可关闭，推荐推理场景开启）

2.4 containerd Role（含 nvidia-container-toolkit）

# roles/containerd/tasks/main.yml
- name: Install containerd
  get_url:
    url: "https://github.com/containerd/containerd/releases/download/v{{ containerd_version }}/containerd-{{ containerd_version }}-linux-amd64.tar.gz"
    dest: /tmp/containerd.tar.gz
  register: dl

- name: Install nvidia-container-toolkit
  shell: |
    curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
    curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | \
      sed 's#https://#https://#g' | tee /etc/yum.repos.d/nvidia-container-toolkit.repo
    dnf install -y nvidia-container-toolkit

- name: Configure containerd for NVIDIA runtime
  shell: nvidia-ctk runtime configure --runtime=containerd
  notify: restart containerd

# /etc/containerd/config.toml 关键配置
# /etc/containerd/config.toml — GPU 节点关键部分
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

2.5 k8s Role（kubeadm 初始化）

# roles/k8s/tasks/init-control.yml
- name: Initialize k8s control plane
  shell: |
    kubeadm init \
      --pod-network-cidr=10.244.0.0/16 \
      --service-cidr=10.96.0.0/12 \
      --apiserver-advertise-address={{ ansible_default_ipv4.address }} \
      --kubernetes-version=v{{ k8s_version }} \
      --upload-certs
  register: kubeadm_init
  when: "'control' in group_names"

- name: Install Calico CNI
  shell: |
    kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
    kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml
  when: "'control' in group_names"

- name: Join worker nodes
  shell: "{{ hostvars['mgmt01'].kubeadm_join_command }}"
  when: "'gpu_nodes' in group_names or 'inference_nodes' in group_names"

📖 3. GPU Operator Helm 部署（已掌握）

3.1 生产级 Helm Values

# gpu-operator-values.yaml
operator:
  defaultRuntime: containerd
  useNvidiaDriverCRD: true

driver:
  enabled: false          # 驱动由 Ansible 管理，Operator 不再安装
  # enabled: true         # 若让 Operator 管理驱动则开启
  # version: "550.90.07"
  # repo: nvcr.io/nvidia

toolkit:
  enabled: false           # 由 Ansible 预装
  # version: "1.14.6-ubuntu20.04"

devicePlugin:
  enabled: true
  version: "v0.15.0"
  config:
    name: device-plugin-config
    default: "time-slicing"
  args: ["--mig-strategy=mixed",
         "--pass-device-specs=true",
         "--fail-on-init-error=true"]

migManager:
  enabled: true
  config:
    name: mig-config
    default: "default-mig-parted-config"

dcgmExporter:
  enabled: true
  env:
    - name: DCGM_EXPORTER_COLLECTORS
      value: "dmon,name=pod"
  serviceMonitor:
    enabled: true
    interval: 15s

gfd:
  enabled: true    # GPU Feature Discovery

validator:
  enabled: true    # GPU Operator 自检

toolkit:
  enabled: false   # 已有 nvidia-container-toolkit

sandboxDevicePlugin:
  enabled: false

3.2 Helm 安装命令

# 添加 NVIDIA Helm repo
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# 安装 GPU Operator
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version v24.9.0 \
  --values gpu-operator-values.yaml \
  --wait \
  --timeout 10m

# 验证安装
kubectl get pods -n gpu-operator
kubectl get node -o json | jq '.items[].status.allocatable | with_entries(select(.key | startswith("nvidia")))'

3.3 Time-Slicing ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  time-slicing: |
    version: v1
    sharing:
      timeSlicing:
        renameByDefault: false
        failRequestsGreaterThanOne: false
        resources:
          - name: nvidia.com/gpu
            replicas: 4           # 每卡 4 个时间片
          - name: nvidia.com/mig-1g.10gb
            replicas: 2

📖 4. 自动化健康检查脚本（已掌握）

#!/bin/bash
# health-check.sh — GPU 节点上线前健康检查
set -euo pipefail

NODE=$1
LOG_FILE="/var/log/gpu-health-${NODE}-$(date +%Y%m%d-%H%M%S).log"

echo "=== GPU Health Check for ${NODE} ===" | tee -a "$LOG_FILE"

# 1. nvidia-smi 基础检测
echo "[1/6] nvidia-smi check..." | tee -a "$LOG_FILE"
ssh "$NODE" 'nvidia-smi -q -d TEMPERATURE,POWER,MEMORY | grep -E "GPU Current Temp|Power Draw|Total"' | tee -a "$LOG_FILE"

# ECC 错误检查
ssh "$NODE" 'nvidia-smi -q -d ECC | grep -A2 "Volatile"' | tee -a "$LOG_FILE"

# 2. NVLink 状态
echo "[2/6] NVLink check..." | tee -a "$LOG_FILE"
ssh "$NODE" 'nvidia-smi nvlink -s' | grep -c "active" | tee -a "$LOG_FILE"

# 3. CUDA sample bandwidthTest
echo "[3/6] CUDA bandwidth test..." | tee -a "$LOG_FILE"
ssh "$NODE" 'docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 \
  /usr/local/cuda/extras/demo_suite/bandwidthTest 2>/dev/null' | tee -a "$LOG_FILE"

# 4. NCCL 多卡通信测试（all_reduce 带宽）
echo "[4/6] NCCL all_reduce test..." | tee -a "$LOG_FILE"
ssh "$NODE" "docker run --rm --gpus all --network host \
  -e NCCL_DEBUG=INFO \
  nvcr.io/nvidia/pytorch:24.06-py3 \
  bash -c 'git clone -b v2.20.5 https://github.com/NVIDIA/nccl-tests.git && \
    cd nccl-tests && make MPI=0 CUDA_HOME=/usr/local/cuda && \
    ./build/all_reduce_perf -b 8 -e 512M -f 2 -g 8'" | tee -a "$LOG_FILE"

# 5. 网络带宽测试（iperf3 或 nccl 跨节点）
echo "[5/6] Network bandwidth test..." | tee -a "$LOG_FILE"
# 从目标节点到已有节点做 ib_write_bw 或 nccl 跨节点测试
ssh "$NODE" "ib_write_bw -d mlx5_0 --report_gbits 10.0.1.1 2>/dev/null || \
  iperf3 -c 10.0.1.1 -P 8" | tee -a "$LOG_FILE"

# 6. 存储性能测试
echo "[6/6] Storage benchmark..." | tee -a "$LOG_FILE"
ssh "$NODE" "fio --name=randrw --ioengine=libaio --rw=randrw --bs=4k --direct=1 \
  --size=4G --numjobs=16 --runtime=60 --group_reporting \
  --filename=/mnt/shared-storage/fio-test" | grep "iops\|bw" | tee -a "$LOG_FILE"

# 结果汇总
echo "=== Health check completed ===" | tee -a "$LOG_FILE"
grep -i "error\|fail\|ERR" "$LOG_FILE" && echo "❌ ISSUES FOUND!" || echo "✅ ALL CHECKS PASSED"

📖 5. 集群扩容流程（已掌握）

# ===== Step 1: 节点发现与信息录入 =====
# 记录节点信息到 inventory
cat >> inventories/gpu-cluster/hosts.yml << EOF
    gpu10: { ansible_host: 10.0.1.10, gpu_model: "H100", gpu_count: 8 }
EOF

# ===== Step 2: 批量运行 Ansible Playbook =====
ansible-playbook -i inventories/gpu-cluster/hosts.yml site.yml \
  --limit gpu10 \
  --extra-vars "nvidia_driver_version=550.90.07 cuda_version=12.4"

# ===== Step 3: 加入 K8s 集群 =====
# 控制节点生成 join command
ssh mgmt01 "kubeadm token create --print-join-command" > /tmp/join-cmd.sh

# 在目标节点执行
ssh gpu10 "bash -s" < /tmp/join-cmd.sh

# ===== Step 4: 打标签和污点 =====
kubectl label node gpu10 \
  nvidia.com/gpu.product=NVIDIA-H100-PCIe \
  nvidia.com/gpu.count=8 \
  node-role.kubernetes.io/gpu-worker=true

kubectl taint node gpu10 nvidia.com/gpu=true:NoSchedule

# ===== Step 5: 等待 GPU Operator 就绪 =====
kubectl wait --for=condition=Ready pod \
  -l app=nvidia-device-plugin-daemonset \
  -n gpu-operator \
  --field-selector spec.nodeName=gpu10 \
  --timeout=300s

# ===== Step 6: 运行健康检查 =====
./health-check.sh gpu10

# ===== Step 7: 验证调度能力 =====
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-test-gpu10
spec:
  nodeName: gpu10
  containers:
  - name: test
    image: nvidia/cuda:12.4.0-base-ubuntu22.04
    command: ["nvidia-smi"]
  resources:
    limits:
      nvidia.com/gpu: 1
EOF

kubectl logs gpu-test-gpu10 | grep "NVIDIA-SMI"
kubectl delete pod gpu-test-gpu10

📖 6. Day-2 运维（已掌握）

6.1 证书轮换

# K8s 证书检查
kubeadm certs check-expiration

# 自动续期（1 年内有效）
kubeadm certs renew all

# 手动续期特定证书
kubeadm certs renew apiserver
kubeadm certs renew apiserver-etcd-client
kubeadm certs renew etcd-server
kubeadm certs renew etcd-peer

# 重启组件使新证书生效
crictl ps | grep -E "kube-apiserver|kube-controller|kube-scheduler|etcd" | awk '{print $1}' | \
  xargs -I {} crictl stop {}

# 分发新 admin.conf
cp /etc/kubernetes/admin.conf ~/.kube/config

6.2 ETCD 备份与恢复

# 定期备份（建议加入 cron）
#!/bin/bash
BACKUP_DIR="/var/backups/etcd/$(date +%Y%m%d-%H%M%S)"
mkdir -p "$BACKUP_DIR"

ETCDCTL_API=3 etcdctl snapshot save "$BACKUP_DIR/etcd-snapshot.db" \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key

etcdctl snapshot status "$BACKUP_DIR/etcd-snapshot.db" --write-out=table

# 恢复流程
etcdctl snapshot restore /var/backups/etcd/etcd-snapshot.db \
  --data-dir=/var/lib/etcd-restore \
  --name=mgmt01 \
  --initial-cluster=mgmt01=https://10.0.0.10:2380 \
  --initial-advertise-peer-urls=https://10.0.0.10:2380

6.3 GPU 节点 Drain/Replace

# Step 1: 驱逐 GPU 节点
kubectl drain gpu03 --ignore-daemonsets --delete-emptydir-data --force

# Step 2: 从集群移除
kubectl delete node gpu03

# Step 3: 清理 GPU Operator（在目标节点）
ssh gpu03 "kubeadm reset -f"
ssh gpu03 "rm -rf /etc/cni /var/lib/kubelet /var/lib/etcd"

# Step 4: 硬件维护 / 更换 GPU
# ... 物理操作 ...

# Step 5: 重新加入集群（走扩容流程）
ansible-playbook -i inventories/gpu-cluster/hosts.yml site.yml --limit gpu03

# Step 6: 取消隔离
kubectl uncordon gpu03

6.4 GPU Operator 版本升级

# 先升级一个节点验证（canary）
kubectl patch daemonset -n gpu-operator nvidia-device-plugin-daemonset \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"nvidia.com/gpu.canary":"true"}}}}}}'

kubectl label node gpu03 nvidia.com/gpu.canary=true --overwrite

helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version v24.12.0 \
  --values gpu-operator-values.yaml \
  --wait

# 验证 canary 节点
nvidia-smi  # 在 canary Pod 中测试

# 全量升级
kubectl label node gpu03 nvidia.com/gpu.canary-
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --version v24.12.0 \
  --values gpu-operator-values.yaml

关联知识

学习时间

阶段时间备注
骨架创建2026-06-30框架搭建
实战补充2026-06-30PXE/Ansible/Helm 完整脚本

状态标记

📖 已掌握 — PXE/Kickstart OS 部署、Ansible 编排、GPU Operator Helm 部署、健康检查、扩容流程、Day-2 运维 📝 待补充 — GPU Operator 离线安装方案、大规模集群分批灰度策略、SR-IOV 网络集成