Autoscaling — how it works¶
After the install, the worker pool becomes elastic:
- From the console: Compute › MachineSets › worker MachineSet › Edit Machine count.
- Automatically: the ClusterAutoscaler adds workers when pods are
Pendingbecause they fit nowhere, and removes idle workers again. - On demand: a worker VM only exists in Proxmox while OpenShift uses it.
Every step ends up as a call to the Proxmox API.
Architecture¶
flowchart TB
subgraph ocp[OpenShift]
CA[ClusterAutoscaler + MachineAutoscaler]
UI[Console: Edit Machine count]
MS[worker MachineSet]
M[Machine]
BMH["BareMetalHost worker-auto-N<br/>(Metal3 / Ironic)"]
MC["MachineConfig 99-worker-openshift5-hosts<br/>/etc/hosts + hostname by MAC"]
REP["CronJob autoscale-bmh-repair"]
end
subgraph host["Proxmox host (or runner LXC)"]
RF["proxmox-redfish<br/>Redfish BMC (gunicorn)"]
DH["proxmox-redfish-dhcp<br/>static leases per MAC"]
CON["proxmox-redfish-console<br/>serial console logs"]
end
subgraph pve[Proxmox VE]
VM["VM worker-auto-N.cluster<br/>created on demand"]
end
CA --> MS
UI --> MS
MS --> M --> BMH
BMH -- "redfish-virtualmedia (HTTPS)" --> RF
RF -- "Proxmox API: create, power, boot order,<br/>ISO as virtual CD, delete" --> VM
VM -- DHCP --> DH
VM -. serial .-> CON
REP -. re-registers broken hosts .-> BMH
| Piece | What it does |
|---|---|
Metal3 (built into OpenShift with platform baremetal) |
One BareMetalHost per worker slot worker-auto-0..N. A slot waits as available until the MachineSet claims it; Ironic then installs RHCOS on it. |
proxmox-redfish (roles/proxmox_redfish) |
A Redfish BMC for Proxmox VMs: the sushy-tools emulator with a Proxmox driver, served by gunicorn on HTTPS with basic auth. Power, boot device, UEFI, virtual CD (downloads Ironic's ISO and uploads it to Proxmox). Only worker-auto-N.<cluster> VMs are visible to it. |
| On-demand VMs | A slot has no VM. The first command that changes the machine creates it (UEFI, q35, cpu: host, fixed MAC, serial port). A reaper deletes the VM once its host is released again. |
| proxmox-redfish-dhcp | dnsmasq with static leases for the slot MACs only: fixed IP, hostname, gateway, DNS. Other machines on the network are ignored. |
MachineConfig 99-worker-openshift5-hosts |
Puts all nodes in /etc/hosts and gives a new worker its hostname by MAC, so no DNS records are needed. |
| ClusterAutoscaler + MachineAutoscaler | Scale the worker MachineSet between autoscale_min_workers and autoscale_max_workers. |
| autoscale-bmh-repair | CronJob that re-registers autoscale hosts stuck in an error state. |
| proxmox-redfish-console | Records each worker's serial console to /var/log/proxmox-redfish/. |
Scale-up¶
sequenceDiagram
participant MS as MachineSet
participant BMH as BareMetalHost
participant RF as proxmox-redfish
participant PVE as Proxmox
participant VM as worker VM
MS->>BMH: claim an available slot
BMH->>RF: insert virtual media (Ironic ISO)
RF->>PVE: create VM (first time), upload ISO, attach as CD
BMH->>RF: boot from CD, power on
VM->>VM: Ironic agent writes RHCOS to /dev/sda
BMH->>RF: eject CD, boot from disk, power on
VM->>VM: DHCP → Ignition → update to cluster OS → reboot
VM->>MS: kubelet CSR (auto-approved) → node Ready
About 15–25 minutes from scale-up to a Ready node.
Scale-down¶
- The node is drained and the Machine deleted.
- Ironic deprovisions the host and powers the VM off; the host is
availableagain. - Once the VM has been off for
autoscale_vm_delete_grace(5 minutes), the reaper deletes the VM, its disk and its boot ISO. No idle VMs in Proxmox.
Before deleting, the reaper asks the OpenShift API (read-only token) whether the host is really released. If it can't ask, it deletes nothing.
Why not the Talos / Cluster API projects?¶
talos-proxmox-kaas and proxmox-capi-talos solve the same problem for Talos: a Proxmox operator creates VMs, Sidero keeps them as a pool, and the cluster-autoscaler scales a Cluster API MachineDeployment.
Their code can't drive OpenShift: nodes must be RHCOS configured by the Machine Config Operator, and OpenShift's Machine API isn't Cluster API. This project uses OpenShift's own equivalents of the same pattern:
| Talos stack | Shiftmox |
|---|---|
| Proxmox operator | proxmox-redfish (or kubemox) |
| Sidero server pool | Metal3 BareMetalHosts |
| MachineDeployment | worker MachineSet |
| cluster-autoscaler | OpenShift ClusterAutoscaler |
kubemox mode¶
With autoscale_vm_provisioning: kubemox, kubemox
pre-creates every VM up front as a stopped spare pool (VirtualMachine CRs in
proxmox-pool). Faster scale-up, but VMs sit in Proxmox all the time.
on_demand is the default; switching to it removes kubemox VMs that no worker
runs on.