Platform External + Proxmox CCM (external)¶
Experimental
Uses OpenShift's platform External (the path partners such as Oracle Cloud use for their own cloud integration) with the community Proxmox cloud-controller-manager.
With install_method: external the cluster is installed with the agent-based
installer, like agent, but tells OpenShift that a cloud-controller-manager
(CCM) for the infrastructure will be there:
Every kubelet then runs with --cloud-provider=external, and each new node
waits (taint node.cloudprovider.kubernetes.io/uninitialized) until the
Proxmox CCM has found its VM. The CCM is part of the install manifests, so it
is running before the nodes need it.
What the CCM does¶
| finds the VM | by name (VM master-0.ocp5 for node master-0) and SMBIOS UUID |
spec.providerID |
proxmox://<proxmox_region>/<vmid> |
| labels | topology.kubernetes.io/region = proxmox_region, topology.kubernetes.io/zone = Proxmox node, instance type |
| node lifecycle | a Node whose VM is gone is deleted from the cluster |
It runs as a DaemonSet on the masters with host networking (so before the pod
network is up) in namespace proxmox-ccm, with the privileged SCC.
Load balancer¶
Platform External has no keepalived VIPs. api, api-int and *.apps
must point at a load balancer. Shiftmox installs HAProxy on the
external_lb_hosts host (default: the runner LXC; ./ocp.sh on the Proxmox
host uses the host itself):
| Port | Backends |
|---|---|
| 6443 | masters (/readyz check) |
| 22623 | masters (machine config server) |
| 80, 443 | workers (masters on a compact cluster) |
DNS for external:
| Record | Points to |
|---|---|
api.<cluster>.<domain>, api-int.<cluster>.<domain> |
external_lb_address |
*.apps.<cluster>.<domain> |
external_lb_address |
For SNO no load balancer is installed; all three records point at master-0. The DNS preflight checks the right addresses for each case.
Settings¶
In group_vars/all/install.yml:
| Variable | Default | Meaning |
|---|---|---|
external_lb_address |
runner_lxc_ip |
address HAProxy binds to; DNS points here |
external_platform_name |
proxmox |
platform.external.platformName |
proxmox_ccm_chart_version |
0.2.32 |
Helm chart, rendered with helm template into the install |
proxmox_region |
<cluster>-pve |
region name (shared with the CSI driver) |
proxmox_k8s_api_url |
https://<autoscale_proxmox_api_host>:8006/api2/json |
Proxmox API as seen from the cluster |
proxmox_k8s_token_id / _secret |
the autoscale token (= deploy token by default) | token the CCM uses |
The CCM needs Sys.Audit and VM.Audit. ./ocp.sh token now also grants
the read-only PVEAuditor role; for an existing token run it once more, or:
Limits¶
- No Machine API: no MachineSets, no BareMetalHosts, no
./ocp.sh autoscale(it refuses). Adding workers later needs the agent "add nodes" flow, which Shiftmox doesn't automate yet. - Changing
install_methodneeds a fresh cluster. - Whether the agent installer accepts a
platformNameother thanocidepends on the release; ifagent create imagerejects it, setexternal_platform_nameto what the error message allows.