Ansible 配置管理实战

概述

Ansible 是 Red Hat 开发的开源配置管理工具，通过 SSH/WinRM 协议远程管理服务器，无需在目标节点安装 Agent。它用 YAML 编写 Playbook，描述系统的”期望状态”，自动实现幂等性。

一句话：Terraform 管”资源有没有”（创建 VPC、EC2），Ansible 管”资源配置对不对”（装包、改配置、启服务）。

Ansible 架构

graph TB
    subgraph "控制节点 (Control Node)"
        Playbook["Playbook<br/>YAML 剧本"]
        Inventory["Inventory<br/>主机清单"]
        Modules["Modules<br/>执行单元"]
        Roles["Roles<br/>可复用角色"]
    end

    subgraph "被管节点 (Managed Nodes)"
        N1["GPU 服务器 01<br/>10.0.1.10"]
        N2["GPU 服务器 02<br/>10.0.1.11"]
        N3["GPU 服务器 03<br/>10.0.1.12"]
        N4["存储服务器<br/>10.0.2.10"]
    end

    Playbook --> Modules
    Inventory --> Modules
    Roles --> Playbook
    Modules -->|"SSH (port 22)"| N1
    Modules -->|"SSH"| N2
    Modules -->|"SSH"| N3
    Modules -->|"SSH"| N4

    style Modules fill:#e8f5e9,stroke:#4caf50

核心概念

概念	说明	类比
Inventory	被管主机清单（INI/YAML）	通讯录
Playbook	YAML 格式的配置剧本	剧本
Task	最小执行单元（调用 Module）	一句台词
Module	执行具体操作（yum/apt/copy/service…）	演员技能
Role	一组 Task + 变量 + 模板的集合，可复用	一幕戏
Handler	只在 Task 通知时触发的操作（如重启服务）	幕后道具
Variable	变量（host_vars / group_vars / extra_vars）	剧本参数
Fact	自动收集的目标主机信息（OS、IP、CPU…）	角色资料

Inventory 管理

静态 Inventory

# inventory/production.ini
[gpu_servers]
gpu-01 ansible_host=10.0.1.10 ansible_user=root
gpu-02 ansible_host=10.0.1.11 ansible_user=root
gpu-03 ansible_host=10.0.1.12 ansible_user=root

[storage_servers]
storage-01 ansible_host=10.0.2.10 ansible_user=ubuntu

[gpu_servers:vars]
gpu_type=a100
nvidia_driver_version=550
cuda_version=12.4

[storage_servers:vars]
nfs_enabled=true

# 元组：包含所有服务器
[all:children]
gpu_servers
storage_servers

[all:vars]
ansible_python_interpreter=/usr/bin/python3

动态 Inventory

云环境中主机动态变化，用动态 Inventory 自动发现：

# inventory/aws_ec2.yml — AWS 动态 Inventory
plugin: aws_ec2
regions:
  - ap-southeast-1
  - us-west-2

# 按标签分组
keyed_groups:
  - key: tags.Role
    prefix: role
  - key: tags.Environment
    prefix: env
  - key: tags.GPU
    prefix: gpu

# 过滤条件
filters:
  tag:Environment: production
  instance-state-name: running

# 主机变量
compose:
  ansible_host: public_ip_address
  ansible_user: "'ec2-user'"

# inventory/gcp_compute.yml — GCP 动态 Inventory
plugin: google.cloud.gcp_compute
projects:
  - my-project-id
zones:
  - asia-southeast1-a
  - asia-southeast1-b
filters:
  - status = RUNNING
  - labels.environment = production
keyed_groups:
  - key: labels.gpu_type
    prefix: gpu
hostnames:
  - name
compose:
  ansible_host: network_interfaces[0].access_configs[0].natIP

# 使用动态 Inventory
ansible-inventory -i inventory/aws_ec2.yml --graph
ansible-playbook -i inventory/aws_ec2.yml site.yml --limit role_gpu

Playbook 核心语法

基本结构

---
# site.yml — 主入口 Playbook
- name: 配置 GPU 服务器集群
  hosts: gpu_servers
  become: yes                    # sudo 提权
  gather_facts: yes              # 收集主机信息

  vars:
    nvidia_driver_version: "550"
    cuda_version: "12.4"
    docker_version: "25.0"

  pre_tasks:
    - name: 确保系统包索引更新
      apt:
        update_cache: yes
      when: ansible_os_family == "Debian"

    - name: 检查 GPU 是否存在
      shell: nvidia-smi --query-gpu=name --format=csv,noheader
      register: gpu_check
      changed_when: false
      failed_when: false

    - name: 显示 GPU 信息
      debug:
        msg: "检测到 GPU: {{ gpu_check.stdout_lines }}"
      when: gpu_check.rc == 0

  roles:
    - role: nvidia_driver
      when: gpu_check.rc == 0
    - role: cuda_toolkit
      when: gpu_check.rc == 0
    - role: docker
    - role: nvidia_container_toolkit
      when: gpu_check.rc == 0
    - role: k8s_worker

  post_tasks:
    - name: 验证 NVIDIA 驱动
      command: nvidia-smi
      register: nvidia_smi
      changed_when: false

    - name: 显示 GPU 状态
      debug:
        var: nvidia_smi.stdout_lines

常用 Module 速查

Module	用途	示例
apt / yum	包管理	apt: name=nginx state=present
copy	拷贝文件	copy: src=app.conf dest=/etc/nginx/
template	Jinja2 模板渲染	template: src=nginx.conf.j2 dest=/etc/nginx/nginx.conf
service / systemd	服务管理	systemd: name=nginx state=restarted enabled=yes
file	文件/目录管理	file: path=/data state=directory mode=0755
user / group	用户管理	user: name=app shell=/bin/bash
git	Git 操作	git: repo=... dest=/opt/app version=main
command / shell	执行命令	shell: nvidia-smi register: result
lineinfile	单行编辑	lineinfile: path=/etc/hosts line='...'
blockinfile	多行块编辑	blockinfile: path=/etc/sysctl.conf ...
mount	挂载管理	mount: src=/dev/sdb path=/data fstype=ext4 state=mounted
cron	定时任务	cron: name="backup" hour=2 job="/opt/backup.sh"
uri	HTTP 请求	uri: url=http://localhost/health status=200
debug	调试输出	debug: var=result.stdout
set_fact	设置变量	set_fact: gpu_count={{ gpu_check.stdout_lines | length }}

变量与 Jinja2 模板

# group_vars/gpu_servers.yml
gpu_config:
  driver_version: "550"
  cuda_version: "12.4"
  persistence_mode: true
  power_limit: 350          # Watts
  auto_boost: 0

nvidia_container_runtime:
  repo_url: "https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
  packages:
    - nvidia-container-toolkit
  docker_runtime: "nvidia"

k8s_node_labels:
  gpu: "true"
  gpu-type: "a100"
  nvidia.com/gpu.present: "true"

{# templates/nvidia-persistence.service.j2 #}
[Unit]
Description=NVIDIA Persistence Daemon
After=network.target

[Service]
Type=forking
ExecStart=/usr/bin/nvidia-persistenced --persistence-mode -u root
ExecStop=/usr/bin/nvidia-persistenced --persistence-mode -u root --no-persistence-mode
Restart=always

[Install]
WantedBy=multi-user.target

{# templates/daemon.json.j2 — Docker 配置 with NVIDIA runtime #}
{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia",
  "data-root": "{{ docker_data_root | default('/var/lib/docker') }}",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m",
    "max-file": "3"
  },
  "registry-mirrors": [
    "https://mirror.example.com"
  ]
}

Handler — 事件触发的操作

- name: 配置 Docker daemon
  template:
    src: daemon.json.j2
    dest: /etc/docker/daemon.json
  notify: restart docker          # 通知 handler

  handlers:
    - name: restart docker
      systemd:
        name: docker
        state: restarted
        daemon_reload: yes

Handler 只在 Task 状态为 changed 时触发，且在 Play 结束时统一执行。同一个 handler 被通知多次只执行一次。

条件与循环

# 条件判断
- name: Ubuntu 系统安装 GPU 驱动
  apt:
    deb: "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/nvidia-driver-{{ nvidia_driver_version }}_open_{{ cuda_version }}.deb"
  when:
    - ansible_os_family == "Debian"
    - gpu_check.rc == 0

# 循环
- name: 创建数据目录
  file:
    path: "/data/{{ item }}"
    state: directory
    owner: "{{ app_user }}"
    mode: "0755"
  loop:
    - logs
    - cache
    - tmp
    - models

# 循环 + 字典
- name: 配置 sysctl 参数
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
    reload: yes
  loop:
    - { key: "net.core.somaxconn", value: "65535" }
    - { key: "net.ipv4.tcp_max_syn_backlog", value: "65535" }
    - { key: "vm.max_map_count", value: "262144" }
    - { key: "fs.file-max", value: "2097152" }

Role 开发与复用

Role 目录结构

roles/
└── nvidia_driver/
    ├── defaults/             # 默认变量（优先级最低，可被覆盖）
    │   └── main.yml
    ├── vars/                 # 角色变量（优先级高于 defaults）
    │   └── main.yml
    ├── tasks/                # 主任务
    │   └── main.yml
    ├── handlers/             # Handler
    │   └── main.yml
    ├── templates/            # Jinja2 模板
    │   ├── nvidia-persistence.service.j2
    │   └── nvidia-modprobe.conf.j2
    ├── files/                # 静态文件
    │   └── blacklist-nouveau.conf
    └── meta/                 # 元数据（依赖、作者）
        └── main.yml

实战 Role：NVIDIA GPU 驱动安装

# roles/nvidia_driver/defaults/main.yml
---
nvidia_driver_version: "550"
cuda_version: "12.4"
nvidia_persistence_mode: true
nvidia_power_limit: 350          # Watts, 0 = 不修改
nvidia_auto_boost: 0
nouveau_blacklist: true

# roles/nvidia_driver/tasks/main.yml
---
- name: 屏蔽 Nouveau 开源驱动
  copy:
    src: blacklist-nouveau.conf
    dest: /etc/modprobe.d/blacklist-nouveau.conf
  when: nouveau_blacklist
  notify: update initramfs

- name: 安装内核头文件和编译工具
  apt:
    name:
      - linux-headers-{{ ansible_kernel }}
      - build-essential
      - dkms
    state: present
  when: ansible_os_family == "Debian"

- name: 添加 NVIDIA 官方 APT 源
  apt:
    deb: "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu{{ ansible_distribution_version | replace('.', '') }}/{{ ansible_architecture }}/cuda-keyring_1.1-1_all.deb"
    state: present
  when: ansible_os_family == "Debian"

- name: 安装 NVIDIA 驱动
  apt:
    name: "nvidia-driver-{{ nvidia_driver_version }}"
    state: present
    update_cache: yes
  register: driver_install
  when: ansible_os_family == "Debian"

- name: 安装 CUDA Toolkit
  apt:
    name: "cuda-toolkit-{{ cuda_version }}"
    state: present
  when: ansible_os_family == "Debian"

- name: 加载 NVIDIA 内核模块
  modprobe:
    name: nvidia
    state: present
  when: driver_install.changed

- name: 启用 NVIDIA 持久化模式
  command: "nvidia-smi -pm {{ 1 if nvidia_persistence_mode else 0 }}"
  changed_when: false
  when: nvidia_persistence_mode

- name: 设置 GPU 功耗限制
  command: "nvidia-smi -pl {{ nvidia_power_limit }}"
  when: nvidia_power_limit | int > 0
  changed_when: false

- name: 配置 GPU 自动 Boost
  command: "nvidia-smi -ac {{ item }}"
  loop: "{{ gpu_application_clocks | default([]) }}"
  changed_when: false
  when: nvidia_auto_boost | int == 0

- name: 安装 nvidia-persistenced
  template:
    src: nvidia-persistence.service.j2
    dest: /etc/systemd/system/nvidia-persistenced.service
  notify: enable persistenced

# roles/nvidia_driver/handlers/main.yml
---
- name: update initramfs
  command: update-initramfs -u

- name: enable persistenced
  systemd:
    name: nvidia-persistenced
    state: started
    enabled: yes
    daemon_reload: yes

实战 Role：Docker + NVIDIA Container Toolkit

# roles/docker/tasks/main.yml
---
- name: 卸载旧版本 Docker
  apt:
    name: "{{ item }}"
    state: absent
  loop:
    - docker
    - docker-engine
    - docker.io
    - containerd
    - runc

- name: 安装 Docker 依赖
  apt:
    name:
      - ca-certificates
      - curl
      - gnupg
      - lsb-release
    state: present
    update_cache: yes

- name: 添加 Docker 官方 GPG Key
  apt_key:
    url: "https://download.docker.com/linux/{{ ansible_distribution | lower }}/gpg"
    state: present

- name: 添加 Docker APT 源
  apt_repository:
    repo: "deb [arch=amd64] https://download.docker.com/linux/{{ ansible_distribution | lower }} {{ ansible_distribution_release }} stable"
    state: present
    update_cache: yes

- name: 安装 Docker
  apt:
    name: "docker-ce{{ '='+docker_version+'*' if docker_version else '' }}"
    state: present
  register: docker_install

- name: 配置 Docker daemon
  template:
    src: daemon.json.j2
    dest: /etc/docker/daemon.json
  notify: restart docker

- name: 配置 Docker 日志轮转
  copy:
    src: docker-logrotate
    dest: /etc/logrotate.d/docker
  when: docker_logrotate | default(true)

- name: 启动并设置开机自启
  systemd:
    name: docker
    state: started
    enabled: yes

# NVIDIA Container Toolkit
- name: 添加 NVIDIA Container Toolkit 源
  apt_key:
    url: "https://nvidia.github.io/libnvidia-container/gpgkey"
    state: present
  when: "'gpu' in group_names"

- name: 添加 NVIDIA Container Toolkit APT 源
  apt_repository:
    repo: "deb https://nvidia.github.io/libnvidia-container/stable/deb/$(ARCH) /"
    state: present
    update_cache: yes
  when: "'gpu' in group_names"

- name: 安装 NVIDIA Container Toolkit
  apt:
    name: nvidia-container-toolkit
    state: present
  when: "'gpu' in group_names"
  notify: restart docker

- name: 配置 Docker 使用 NVIDIA Runtime
  command: nvidia-ctk runtime configure --runtime=docker
  when: "'gpu' in group_names"
  notify: restart docker

实战 Role：K8s Worker 节点准备

# roles/k8s_worker/tasks/main.yml
---
- name: 禁用 Swap
  swapoff:
    name: swap
    state: absent

- name: 注释 fstab 中的 swap
  replace:
    path: /etc/fstab
    regexp: '^([^#].*swap.*)$'
    replace: '#\1'

- name: 加载内核模块
  modprobe:
    name: "{{ item }}"
    state: present
  loop:
    - overlay
    - br_netfilter

- name: 持久化内核模块加载
  copy:
    dest: /etc/modules-load.d/k8s.conf
    content: |
      overlay
      br_netfilter

- name: 配置 sysctl for K8s
  sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
    reload: yes
  loop:
    - { key: "net.bridge.bridge-nf-call-iptables", value: "1" }
    - { key: "net.bridge.bridge-nf-call-ip6tables", value: "1" }
    - { key: "net.ipv4.ip_forward", value: "1" }

- name: 添加 Kubernetes APT 源
  apt_key:
    url: "https://pkgs.k8s.io/core:/stable:/v{{ k8s_version }}/deb/Release.key"
    state: present
  when: ansible_os_family == "Debian"

- name: 添加 Kubernetes APT 源
  apt_repository:
    repo: "deb https://pkgs.k8s.io/core:/stable:/v{{ k8s_version }}/deb/ /"
    state: present
    update_cache: yes

- name: 安装 containerd
  apt:
    name: containerd
    state: present

- name: 配置 containerd
  shell: |
    containerd config default > /etc/containerd/config.toml
    sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
  args:
    creates: /etc/containerd/config.toml
  notify: restart containerd

- name: 安装 Kubernetes 组件
  apt:
    name:
      - kubelet
      - kubeadm
      - kubectl
    state: present

- name: 标记 Kubernetes 包不自动升级
  dpkg_selections:
    name: "{{ item }}"
    selection: hold
  loop:
    - kubelet
    - kubeadm
    - kubectl

- name: 设置节点标签和 taint
  template:
    src: kubelet-extra-args.service.j2
    dest: /etc/systemd/system/kubelet.service.d/11-extra-args.conf
  notify: restart kubelet

Ansible 与 Terraform 协作模式

模式一：Terraform 创建资源 → Ansible 配置

Terraform Plan:
  1. 创建 VPC / 子网 / 安全组
  2. 创建 EC2 / GPU 实例
  3. 输出实例 IP → 写入 Ansible Inventory

Ansible Playbook:
  4. SSH 到实例 → 安装 NVIDIA 驱动 / Docker / K8s

# Terraform 输出动态 Inventory
resource "local_file" "ansible_inventory" {
  content = templatefile("inventory.tmpl", {
    gpu_servers = aws_instance.gpu[*].public_ip
    region      = var.region
  })
  filename = "${path.module}/../ansible/inventory/terraform_hosts.ini"
}

{# inventory.tmpl #}
[gpu_servers]
%{ for ip in gpu_servers ~}
gpu-${index(gpu_servers, ip) + 1} ansible_host=${ip} ansible_user=ubuntu
%{ endfor ~}

[gpu_servers:vars]
region=${region}

# ansible/site.yml — 使用 Terraform 生成的 Inventory
- name: 配置 GPU 服务器
  hosts: gpu_servers
  become: yes
  roles:
    - nvidia_driver
    - docker
    - nvidia_container_toolkit
    - k8s_worker

模式二：Terraform user_data + Ansible Playbook

# Terraform 用 user_data 触发 Ansible Pull 模式
resource "aws_instance" "gpu" {
  user_data = templatefile("user_data.sh.tpl", {
    playbook_repo = "https://github.com/org/ansible-playbooks.git"
    playbook_name = "gpu-server.yml"
  })
}

#!/bin/bash
# user_data.sh.tpl — 实例启动时自动拉取并执行 Playbook
apt-get update && apt-get install -y ansible git
git clone ${playbook_repo} /opt/ansible
cd /opt/ansible
ansible-playbook ${playbook_name} --connection=local -i localhost,

模式三：Ansible 管理已有基础设施

# 适合裸金属服务器 — 不经过 Terraform，直接 Ansible 管理
- name: 全量配置 GPU 集群
  import_playbook: site.yml
  vars:
    target: gpu_servers

Ansible AWX / Tower 自动化

AWX 架构

graph TB
    subgraph "AWX 架构"
        Web["Web UI<br/>可视化管理"]
        API["REST API<br/>程序化接口"]
        DB["PostgreSQL<br/>存储任务/凭据"]
        Redis["Redis<br/>任务队列"]
        Exec["Execution Environment<br/>容器化执行"]
    end

    Git["Git 仓库<br/>Playbook 源码"]
    Cloud["云平台<br/>AWS/GCP/裸金属"]

    Web --> API
    API --> DB
    API --> Redis
    Redis --> Exec
    Exec -->|"SSH"| Cloud
    Git -->|"拉取"| Exec

    style Exec fill:#e8f5e9,stroke:#4caf50

AWX 核心概念

概念	说明
Project	Git 仓库连接（存储 Playbook）
Inventory	主机清单（可动态同步云资源）
Credential	SSH/云 API 凭据（加密存储）
Job Template	Playbook + Inventory + Credential 的组合
Workflow	多个 Job Template 的编排（DAG）
Survey	运行时输入表单（变量参数化）
Execution Environment	容器化的执行环境（自定义 Python 依赖）

AWX Job Template 示例

# 通过 API 创建 Job Template
---
name: "GPU 服务器初始化"
description: "安装 NVIDIA 驱动、CUDA、Docker、K8s"
organization: "SRE Team"
project: "ansible-playbooks"
playbook: "site.yml"
inventory: "GPU Production"
credential: "SSH - GPU Servers"
limit: "gpu_servers"
verbosity: 2
job_tags: "nvidia,docker,k8s"
skip_tags: ""
use_fact_cache: true
survey_enabled: true
survey_spec:
  name: ""
  description: "选择配置范围"
  spec:
    - question_name: "安装范围"
      question_description: "选择要执行的 Role"
      required: true
      type: "multiplechoice"
      variable: "install_scope"
      choices:
        - "all"
        - "nvidia_only"
        - "k8s_only"
      default: "all"

最佳实践与常见坑

最佳实践

实践	说明	示例
幂等性优先	确保重复执行不产生副作用	用 creates/changed_when 控制
Role 模块化	每个 Role 只做一件事	nvidia_driver / docker / k8s_worker 分开
变量分层	defaults < group_vars < host_vars < extra_vars	敏感变量用 Vault 加密
使用 Tags	按 Tag 选择性执行	--tags nvidia 只执行 GPU 相关
Fact Caching	缓存 Fact 避免每次收集	fact_caching = jsonfile 或 redis
Limit 控制范围	先小范围测试再全量	--limit gpu-01 先测试一台
Check 模式	Dry-run 预览变更	--check --diff

常见坑

问题	根因	解决方案
SSH Error: Permission denied	SSH Key 未配置或用户不对	ansible all -m ping 先测连通性
Playbook 卡住不动	SSH 超时 / 目标主机无响应	--timeout=30 + ansible_ssh_timeout
Fact 收集慢	主机多或网络差	gather_facts: no + 手动获取需要的信息
模块找不到	Python 版本不匹配	ansible_python_interpreter=/usr/bin/python3
修改变量不生效	变量优先级覆盖	检查 ansible-inventory --host <host>
Handler 未触发	Task 状态不是 changed	确认 Task 确实修改了目标
apt 模块锁冲突	另一个 apt 进程在运行	先 apt: update_cache 再安装

性能优化

# ansible.cfg — 性能相关配置
[defaults]
# 并发控制
forks = 50                    # 默认 5，大规模可调高

# SSH 优化
host_key_checking = False
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o ConnectionAttempts=3
pipelining = True             # 减少 SSH 连接次数（需 sudo requiretty=False）

# Fact 缓存
gathering = smart
fact_caching = redis
fact_caching_timeout = 86400

# 超时
timeout = 30
ansible_ssh_timeout = 10

# 使用 Mitogen 加速 Ansible（10 倍性能提升）
pip install mitogen
# ansible.cfg
[defaults]
strategy = mitogen_linear
strategy_plugins = /path/to/ansible_mitogen/plugins/strategy

关联知识

• IaC 方法论与实践总览 — IaC 工具全景图，Ansible 在配置管理层
• Terraform 基础设施即代码 — Terraform 创建资源，Ansible 配置资源
• Terraform 生产级实践 — Terraform + Ansible 协作模式
• Packer 不可变镜像构建 — Packer 调用 Ansible Playbook 构建镜像
• ../gpu-cluster-ops/裸金属/GPU 裸金属服务器运维 — 裸金属服务器的 Ansible 配置管理
• ../gpu-cluster-ops/automation/集群自动化部署方案 — 集群自动化部署中的 Ansible 使用
• ../sre/SRE 工具链总览 — Ansible 在 SRE 工具链中的定位
• ../sre/变更管理全流程 — Ansible 变更的安全管控
• ../linux/Linux 内核参数调优 — Ansible sysctl 模块批量调优

参考资源

• Ansible 官方文档: https://docs.ansible.com/
• Ansible Best Practices: https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html
• Ansible Galaxy (Role 市场): https://galaxy.ansible.com/
• AWX 项目: https://github.com/ansible/awx
• NVIDIA Container Toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
• Ansible for Kubernetes: https://github.com/geerlingguy/kubernetes-role

学习时间

阶段	时间	备注
Ansible 基础	2026-08-03	Inventory、Playbook、Module、Role
GPU 配置实战	2026-08-03	NVIDIA 驱动、CUDA、Docker、K8s Worker
AWX 自动化	2026-08-03	Job Template、Workflow、Survey

---

状态: ✅ 已完成 学习时间: 2026-08-03