Managing Kubernetes Nodes with Ansible¶

Repeatable node lifecycle automation for Rancher-managed Kubernetes clusters.

This repository provides Ansible automation for administering nodes in Rancher-managed Kubernetes clusters. Its goal is to make node operations repeatable: discover existing cluster nodes, prepare new servers, join them to a cluster, and run routine maintenance through playbooks instead of one-off shell commands.

Scope¶

Use this repo to manage the node lifecycle around Kubernetes:

• discover existing Kubernetes nodes and build Ansible inventory
• configure baseline packages, users, firewall, and OS prerequisites
• prepare a new node before it joins a cluster
• fetch Rancher registration data through the Rancher API
• execute the Rancher node registration command on the target host
• run routine node operations such as package updates and service restarts

Rancher remains the source of truth for Kubernetes cluster membership. Ansible prepares nodes and generates the Rancher registration command from the Rancher API; it does not create clusters or bypass Rancher's RKE2 join flow.

Use this repo when you want repeatable node operations. Do not use it to create Kubernetes clusters, bypass Rancher, or replace Rancher's cluster management model.

Start with GETTING_STARTED.md if this is a fresh clone.

Operator Workflow¶

For most work, the flow is:

configure local access
-> refresh inventory from Kubernetes
-> choose a target with --limit
-> run the needed playbook
-> refresh inventory again if cluster membership changed

Basic setup:

make configure
make doctor
make graph
make ping

Per-user configuration is stored in config.yml, which is gitignored and auto-loaded by Ansible and the helper scripts. You do not need to source it.

Requirements¶

The runner needs:

• ansible and ansible-playbook
• kubectl
• kubeconfig contexts for the clusters you want to manage
• SSH access to the managed nodes
• for Rancher node joins, a Rancher API token from a dedicated remote/external Rancher user

Managed nodes do not need Ansible installed; they need SSH access and Python for Ansible modules.

Do not use the local Rancher admin account for automation tokens.

Install repo dependencies:

make install

make install installs Ansible, Ansible collections, and local helper dependencies. It does not install kubectl; install kubectl separately from the Kubernetes project documentation or your package manager.

Inventory Model¶

Inventory has two sources:

• static/manual inventory in inventory/*.yml
• dynamic Kubernetes inventory from inventory/k8s-nodes.sh

inventory/k8s-nodes.sh is an Ansible dynamic-inventory script that calls kubectl get nodes for each context configured in KUBE_CONTEXTS on every ansible run. There is no cached file to refresh — adding or removing a node shows up on the next playbook invocation.

Context names become Ansible group names by replacing - with _:

kubectl context: mintfit-cluster-01
Ansible group:   mintfit_cluster_01
group vars:      inventory/group_vars/mintfit_cluster_01.yml

Per Kubernetes node, the generated inventory stores:

• private_ipv4: Kubernetes InternalIP
• public_ipv4: Kubernetes ExternalIP, or mintfit.io/public-ip annotation
• kubernetes_version: kubelet version

The main group hierarchy is:

all
|-- bastion
|-- cluster_nodes
|   |-- mintfit_cluster_01
|   |   |-- mintfit_cluster_01_controlplane
|   |   `-- mintfit_cluster_01_worker
|   `-- mintfit_mgmt
`-- manual
    |-- manual_bastion
    |-- manual_controlplane
    `-- manual_worker

Tag groups such as bare_metal, cloud_instance, privileged, and unprivileged are built from per-tag Kubernetes labels:

kubectl --context=mintfit label node <node-name> \
  mintfit.io/group.bare_metal=true mintfit.io/group.unprivileged=true

The next ansible-playbook (or make graph) invocation picks the new labels up automatically. Side benefit: kubectl get nodes -l mintfit.io/group.bare_metal=true filters directly.

Allowed tag names live in tag-names.yml. Labels with an unlisted tag are ignored with a warning when the dynamic inventory runs.

Connection Model¶

The same inventory works from a local workstation and from an automation runner inside the cluster.

Set network_location with make configure, or override per run:

ansible-playbook playbooks/read/check-ping.yml -e network_location=cluster

Each cluster defines its bastion in inventory/group_vars/<cluster>.yml with bastion_inventory_name.

New Node Lifecycle¶

Use this workflow for a fresh server that should join a Rancher-managed cluster.

1. Add The Node Manually¶

Add the new host to inventory/05-manual.yml under manual_worker or manual_controlplane.

Manual hosts connect as root through their public IP and do not use the cluster bastion.

Example:

manual:
  children:
    manual_worker:
      hosts:
        mintfit-c01-w03:
          public_ipv4: "178.105.200.132"
          private_ipv4: "10.0.0.13"

2. Prepare The Selected Node Profile¶

Each cluster defines one or more node profiles in inventory/group_vars/<cluster>.yml. A node profile is the supported node contract for that cluster: OS family/version, kernel policy, kernel modules, and required node packages.

The selected profile is controlled by node_profile_name:

node_profiles:
  ubuntu_24_04_rke2_calico_longhorn:
    os:
      distribution: Ubuntu
      major_version: "24"
    kernel:
      install_version: "6.8.0-106-generic"
      allowed_regex: "^6\\.8\\.0-106-generic$"
    required_kernel_modules:
      - br_netfilter
      - overlay
    required_packages: []

node_profile_name: ubuntu_24_04_rke2_calico_longhorn
node_profile: "{{ node_profiles[node_profile_name] }}"

Roles still consume simple compatibility variables. The cluster vars derive those from the selected profile:

node_kernel_version: "{{ node_profile.kernel.install_version }}"
rke2_prep_allowed_distribution: "{{ node_profile.os.distribution }}"
rke2_prep_allowed_distribution_major_version: "{{ node_profile.os.major_version }}"
rke2_prep_allowed_kernel_regex: "{{ node_profile.kernel.allowed_regex }}"
rke2_prep_required_kernel_modules: "{{ node_profile.required_kernel_modules | default([]) }}"
rke2_prep_required_packages: "{{ node_profile.required_packages | default([]) }}"

If the node does not run the selected profile's kernel, run:

ansible-playbook playbooks/cluster-node/rke2/prepare-kernel.yml \
  --limit mintfit-c01-w03 \
  -e target_cluster=mintfit_cluster_01 \
  -e node_kernel_auto_reboot=true

For mintfit, the current selected profile installs and allows:

node_profile_name: ubuntu_24_04_rke2_calico_longhorn
node_profile:
  kernel:
    install_version: "6.8.0-106-generic"
    allowed_regex: "^6\\.8\\.0-106-generic$"

Package holds are disabled by default. Enable them only when you intentionally want to freeze kernel packages:

ansible-playbook playbooks/cluster-node/rke2/prepare-kernel.yml \
  --limit mintfit-c01-w03 \
  -e target_cluster=mintfit_cluster_01 \
  -e node_kernel_auto_reboot=true \
  -e node_kernel_hold=true

3. Configure Rancher API Access¶

The join playbook normally uses the Rancher API to fetch the cluster registration token and build the registration command itself.

In inventory/group_vars/<cluster>.yml:

rancher_node_join_rancher_url: "https://rancher.mintfit.hamburg"
rancher_node_join_cluster_name: "mintfit"

Store rancher_node_join_api_token in Ansible Vault, automation runner secrets, or another secret backend.

For local development, copy .rancher.env.example to .rancher.env and set:

export RANCHER_API_TOKEN="token-xxxxx:yyyyy"

The token must come from a dedicated remote/external Rancher user with the smallest permissions needed to read the target cluster and its registration tokens. Do not generate it from the local Rancher admin account.

4. Join Nodes Through Rancher¶

Run the join playbook with an explicit --limit. Lifecycle playbooks refuse to run without a limit unless explicitly overridden.

The playbook runs:

node_baseline
-> users
-> rke2_prep
-> rancher_node_join

rke2_prep performs profile preflight checks before Rancher registration. For mintfit, it currently blocks nodes that do not match the selected profile's known-good kernel pattern:

rke2_prep_allowed_kernel_regex: "{{ node_profile.kernel.allowed_regex }}"

For a new Rancher custom cluster with separated control-plane and worker roles, register at least one node for each required role category before waiting for Kubernetes readiness. Rancher may keep machine plans empty until it has seen an etcd, controlplane, and worker registration.

First register one control-plane/etcd node:

ansible-playbook playbooks/cluster-node/rke2/join.yml \
  --limit mintfit-c01-cp01 \
  -e target_cluster=mintfit_cluster_01 \
  -e rancher_node_roles=etcd,controlplane \
  -e rancher_node_join_wait_for_kubernetes=false

Then register the first worker immediately; do not wait for the control-plane node to become Ready when it has no worker role:

ansible-playbook playbooks/cluster-node/rke2/join.yml \
  --limit mintfit-c01-w01 \
  -e target_cluster=mintfit_cluster_01 \
  -e rancher_node_role=worker \
  -e rancher_node_join_wait_for_kubernetes=false

Watch Rancher create non-empty machine plans and then start RKE2 on both nodes:

kubectl --context=mintfit-local get machines.cluster.x-k8s.io -A -o wide
kubectl --context=mintfit-local get secrets -n fleet-default | grep machine-plan

Once the first control-plane and worker are running, verify the downstream cluster API and nodes:

kubectl --context=mintfit-cluster-01 get nodes -o wide

Add additional control-plane nodes one at a time, waiting for each to become Ready before the next:

ansible-playbook playbooks/cluster-node/rke2/join.yml \
  --limit mintfit-c01-cp02 \
  -e target_cluster=mintfit_cluster_01 \
  -e rancher_node_roles=etcd,controlplane

Add additional workers with the worker role:

ansible-playbook playbooks/cluster-node/rke2/join.yml \
  --limit mintfit-c01-w02 \
  -e target_cluster=mintfit_cluster_01 \
  -e rancher_node_role=worker

If you intentionally want the first control-plane to be schedulable as a worker too, use -e rancher_node_roles=etcd,controlplane,worker. For the standard production layout with dedicated workers, keep control-plane nodes as etcd,controlplane and workers as worker.

5. Verify And Remove Manual Entry¶

After the node appears in Kubernetes:

make graph

Then remove the temporary host from inventory/05-manual.yml. The generated Kubernetes inventory owns the node from this point on.

Reset Nodes For Join Testing¶

This playbook is destructive. It requires both --limit and rancher_node_reset_confirm=true.

It removes the Kubernetes node object when present, stops Rancher/RKE2/K3s services, runs vendor uninstall scripts when present, unmounts kubelet pod mounts, and deletes local Rancher/RKE2/K3s state. It does not remove normal baseline management state such as team users, SSH keys, MOTD, sysctl files, or installed baseline packages.

Before deleting a Kubernetes node object, the role checks the Rancher/CAPI management cluster for a matching Machine. A confirmed reset deletes the matching Machine first so Rancher is not left with NodeDeleted / NodeNotFound status. The lookup also matches Rancher bootstrap objects by node address so partially registered machines without a Kubernetes NodeRef can still be found.

For local runs, set the management-cluster context explicitly when your kubeconfig does not have a context named local. In this environment the Rancher management context is mintfit-local:

ansible-playbook playbooks/cluster-node/rke2/reset.yml \
  --limit mintfit-c01-cp01,mintfit-c01-w01 \
  -e target_cluster=mintfit_cluster_01 \
  -e rancher_node_reset_management_kubectl_context=mintfit-local \
  -e rancher_node_reset_confirm=true \
  --check

When the downstream cluster API is broken or not bootstrapped yet, skip Kubernetes drain/delete and use reset for local node state cleanup only:

ansible-playbook playbooks/cluster-node/rke2/reset.yml \
  --limit mintfit-c01-cp03,mintfit-c01-w01 \
  -e target_cluster=mintfit_cluster_01 \
  -e rancher_node_reset_management_kubectl_context=mintfit-local \
  -e rancher_node_reset_confirm=true \
  -e rancher_node_reset_drain_node=false \
  -e rancher_node_reset_delete_kubernetes_node=false

For a failed brand-new Rancher custom cluster showing an RKEControlPlane message like no etcd machines exist that have plans, delete and recreate the Rancher downstream cluster first, wait for stale Machines/RKEBootstraps to be gone, reset the nodes locally, then run the normal join sequence again.

After reset:

make graph

Then add the host back to inventory/05-manual.yml if it is no longer present there and run the normal node lifecycle again.

Rancher Registration Helper¶

To inspect the command generated from Rancher API data:

scripts/rancher-registration-command.sh \
  --target-cluster mintfit_cluster_01 \
  --worker

With .rancher.env configured:

make rancher-registration-command

List visible Rancher clusters:

scripts/rancher-registration-command.sh --list-clusters

Use a cluster id when names are ambiguous:

scripts/rancher-registration-command.sh \
  --cluster-id c-m-xxxxx \
  --worker

Include explicit node addresses:

scripts/rancher-registration-command.sh \
  --target-cluster mintfit_cluster_01 \
  --worker \
  --address 178.105.200.132 \
  --internal-address 10.0.0.13

The printed command contains a node registration token. Treat it as sensitive.

Routine Operations¶

Always use --limit for mutating playbooks unless you intentionally want a larger scope.

Read-only playbooks:

Mutating playbooks:

Ops and emergency playbooks:

Examples:

ansible-playbook playbooks/read/inspect-system-info.yml --limit mintfit-c01-w03

ansible-playbook playbooks/read/report-node-config.yml \
  --limit mintfit-c01-w01

ansible-playbook playbooks/maintenance/upgrade-apt-packages.yml \
  --limit mintfit_cluster_01_worker \
  -e auto_reboot=true

Use -e dist_upgrade=true only when you intentionally want apt dist-upgrade instead of a normal apt upgrade.

Use playbooks/read/report-node-config.yml when comparing node configuration. It prints live routing, networkd/netplan/resolver state, selected service status, OS facts, package versions, sysctl/module/firewall state, and selected config files from /etc/netplan, /etc/systemd/network, /run/systemd/network, /etc/cloud, /etc/rancher/rke2. Token-like fields and kubeconfig certificate/key data are redacted by default.

The default stdout mode is a compact summary. To print every collected command and config file to the Ansible output:

ansible-playbook playbooks/read/report-node-config.yml \
  --limit mintfit-c01-w01 \
  -e node_config_stdout_mode=full

Report sections can be enabled or disabled independently. All are enabled by default:

ansible-playbook playbooks/read/report-node-config.yml \
  --limit mintfit-c01-w01 \
  -e node_config_section_os=false \
  -e node_config_section_packages=false \
  -e node_config_section_network=true

Available section vars are:

node_config_section_os
node_config_section_services
node_config_section_packages
node_config_section_network
node_config_section_firewall
node_config_section_kubernetes
node_config_section_config_files
node_config_section_journals

To also fetch the full report tarball, including recent service journals:

ansible-playbook playbooks/read/report-node-config.yml \
  --limit mintfit-c01-w01 \
  -e node_config_save_report=true

Saved reports are written to diagnostics/node-config/. Use -e node_config_include_sensitive=true only if you intentionally need raw values in the printed or saved report.

Make Targets¶

Run make help to list available targets.

Common targets:

The Make targets are convenient shortcuts for routine default runs. For node lifecycle operations, use direct ansible-playbook commands so required values such as --limit, target_cluster, and confirmation flags are explicit in the command history.

Repository Layout¶

ansible/
|-- README.md
|-- GETTING_STARTED.md
|-- Makefile                    # operator shortcuts
|-- ansible.cfg                 # inventory, callbacks, fact cache, role paths
|-- requirements.yml            # Ansible collection requirements
|-- config.yml.example          # template written to config.yml by make configure
|-- tag-names.yml               # allowed inventory tag groups
|-- .rancher.env.example        # local Rancher API token template
|-- inventory/
|   |-- 00-static.yml           # static group hierarchy and bastions
|   |-- 05-manual.yml           # temporary/fresh hosts before dynamic inventory owns them
|   |-- k8s-nodes.sh            # dynamic inventory: queries `kubectl get nodes`
|   `-- group_vars/             # global, manual, cluster, and role-group vars
|-- playbooks/
|   |-- read/
|   |   |-- check-ping.yml
|   |   |-- inspect-system-info.yml
|   |   |-- report-node-config.yml
|   |   `-- collect-node-diagnostics.yml
|   |-- bootstrap/
|   |   `-- bootstrap-ansible-user.yml
|   |-- configure/
|   |   |-- configure-users.yml
|   |   |-- configure-sshd-hardening.yml
|   |   |-- configure-node-baseline.yml
|   |   |-- configure-firewall.yml
|   |   |-- configure-unattended-upgrades.yml
|   |   |-- configure-longhorn-disk.yml
|   |   |-- configure-static-ip-hcloud.yml
|   |   `-- configure-static-ip-metal.yml
|   |-- cluster-node/
|   |   `-- rke2/
|   |       |-- prepare-kernel.yml
|   |       |-- prepare-host.yml
|   |       |-- prepare-cis.yml
|   |       |-- join.yml
|   |       `-- reset.yml
|   |-- maintenance/
|   |   |-- upgrade-apt-packages.yml
|   |   |-- reboot-node.yml
|   |   |-- restart-kubelet.yml
|   |   |-- restart-containerd.yml
|   |   `-- restart-systemd-networkd.yml
|   |-- cluster/
|   |   `-- upgrade-rancher.yml
|   |-- tasks/
|   `-- templates/
|-- roles/
|   |-- firewall/
|   |-- node_baseline/
|   |-- node_kernel/
|   |-- kube_bench_prep/
|   |-- rancher_node_join/
|   |-- rancher_node_reset/
|   |-- rke2_prep/
|   |-- unattended_upgrades/
|   `-- users/
|-- files/
|   |-- README.md
|   `-- keys/                   # managed users' public SSH keys
`-- scripts/
    |-- configure.sh
    |-- doctor.sh
    |-- install-deps.sh
    |-- lib-config.sh
    `-- rancher-registration-command.sh

Troubleshooting¶

Run make doctor first. It checks the common local setup problems.

Useful one-node test:

ansible-playbook playbooks/read/inspect-system-info.yml \
  --limit mintfit-c01-w03 \
  --check \
  --diff

Inspect resolved inventory:

ansible-inventory --host mintfit-c01-w03
ansible-inventory --graph