Add longhorn distributed storage to the k3s-cluster

This change adds longhorn, an addition to Kubernetes that adds the
ability to use distributed storage over all nodes to the cluster.

Note, that I tried that in December already but due to very high load on
the machines I rolled _everything_ back. Though, it turned out that the
high load was not because of longhorn, but instead because of bad
configuration of the server, as described in the see-also commit.

Reference: https://longhorn.io/
Reference: https://longhorn.io/docs/1.10.1/deploy/install/install-with-helm/
See-also: 4b8a3d12c4 Use etcd instead of sqlite for k3s-server

README.adoc

@@ -118,6 +118,7 @@ Ansible installs and maintains all cluster software, including:
 
 * Routing and SSH setup on servers
 * A full k3s Kubernetes cluster
+* Distributed block-storage via https://longhorn.io/[longhorn]
 * Foundational cluster services
 
 [NOTE]
@@ -140,6 +141,23 @@ Backup your config if you manage multiple clusters.
 The Kubernetes setup requires an `inventory.ini` file, which Tofu creates automatically.
 So, make sure to apply the infrastructure at least once before running Ansible.
 
+=== Longhorn
+
+The setup installs https://longhorn.io/[Longhorn], which provides a distributed block-storage system for the Kubernetes cluster.
+
+Longhorn exposes a default storage class named `longhorn`.
+This storage class is backed by replicated volumes distributed across multiple nodes,
+reducing dependency on node-local ephemeral storage and allowing workloads to be rescheduled more reliably.
+
+Longhorn also provides a web-based dashboard for inspecting volumes, replicas, and node health.
+
+To access the dashboard, forward the service port:
+
+[source,bash]
+kubectl port-forward -n longhorn-system --address 0.0.0.0 service/longhorn-frontend 8000:80
+
+Then open http://localhost:8000/ in your browser.
+
 === Installed Foundational Services
 
 https://cert-manager.io/docs/installation/helm[cert-manager]::
@@ -167,6 +185,8 @@ You can restrict playbook scope to specific areas using `--tags`.
 `add-server`:: Add a new k3s server node
 `add-agent`:: Add a new k3s agent node
 `update`:: Upgrade Kubernetes or system packages
+`longhorn-compatible`:: Ensure longhorn-compatibility
+`longhorn`:: Deploy longhorn
 `config`:: Update local kubeconfig
 `k8s`:: Deploy foundational services
 
@@ -216,4 +236,5 @@ Ansible::
 * Configure routing on all servers
 * Install and maintain k3s
 * Keep system software updated
+* Install longhorn
 * Deploy foundational services

config.yml

@@ -2,6 +2,8 @@ all:
   vars:
     api_endpoint: "{{ hostvars[groups['server'][0]]['ansible_host'] | default(groups['server'][0]) }}"
 
+    longhorn_state: present
+
     cert_manager_state: present
     cert_manager_version: v1.18.2
 

roles/longhorn-compatible/tasks/main.yml

@@ -0,0 +1,22 @@
+- name: Ensure longhorn dependencies are installed
+  package:
+    state: present
+    name:
+    - open-iscsi
+    - nfs-common
+
+- name: Ensure iscsid service is enabled and started
+  ansible.builtin.systemd_service:
+    name: iscsid
+    enabled: true
+    state: started
+
+- name: Ensure iscsi_tcp module is added
+  community.general.modprobe:
+    name: iscsi_tcp
+    state: present
+
+- name: Ensure nfs module is added
+  community.general.modprobe:
+    name: nfs
+    state: present

roles/longhorn/defaults/main.yml

@@ -0,0 +1,3 @@
+longhorn_state: present
+longhorn_namespace: longhorn-system
+longhorn_chart_version: 1.10.1

roles/longhorn/tasks/main.yml

@@ -0,0 +1,25 @@
+- name: Ensure longhorn chart
+  kubernetes.core.helm_repository:
+    name: longhorn
+    repo_url: https://charts.longhorn.io
+
+- name: Ensure longhorn is installed
+  kubernetes.core.helm:
+    name: longhorn
+    chart_ref: longhorn/longhorn
+    chart_version: "{{ longhorn_chart_version }}"
+    release_namespace: "{{ longhorn_namespace }}"
+    release_state: "{{ longhorn_state }}"
+    create_namespace: True
+
+- name: Ensure longhorn namespace
+  kubernetes.core.k8s:
+    state: "{{ longhorn_state }}"
+    definition:
+      apiVersion: v1
+      kind: Namespace
+      metadata:
+        name: "{{ longhorn_namespace }}"
+        labels:
+          name: "{{ longhorn_namespace }}"
+

site.yml

@@ -27,6 +27,23 @@
   - init
   - config
   - update
+- hosts: k3s_cluster
+  roles:
+  - role: longhorn-compatible
+  tags:
+  - init
+  - add-server
+  - add-agent
+  - longhorn-compatible
+- hosts: localhost
+  gather_facts: no
+  roles:
+  - role: longhorn
+  tags:
+  - init
+  - add-server
+  - add-agent
+  - longhorn
 - hosts: localhost
   gather_facts: no
   roles: