Patching a CloudStack cloud while tenant instances keep running

When a security advisory lands for the platform that runs other people's servers, there are two bad instincts. One is to wait, because touching the cloud feels riskier than the vulnerability. The other is to upgrade everything at once and hope. Apache CloudStack rewards a third approach: understand its layers, update them in the right order, and let the evidence decide which steps actually touch tenants.

A cloud is several things to patch, not one

A CloudStack deployment is a stack of components with different blast radii:

• The management servers and their database. The control plane: portal, API, orchestration. While it is down, users cannot create or change resources, but running instances do not depend on it.
• The agents on each KVM host. They connect hypervisors to the management server. Restarting an agent does not stop the instances already running on that host.
• System VMs. The secondary storage VM handles templates and snapshots, the console proxy serves browser consoles. Restarting them briefly interrupts those functions, not tenant workloads.
• Virtual routers. These sit in the data path of tenant networks, providing DHCP, DNS, NAT, firewalling and load balancing. Restarting one interrupts that network's services for a moment.

The order of work follows from this list: the parts tenants never feel first, the parts they might feel last, and the last ones only when there is a real reason.

Step one: read the advisory against your own configuration

Most advisories affect a specific feature, often one that is optional. Before planning a window, find out whether the affected feature is enabled in your cloud. The global settings are visible through the API, for example with CloudMonkey:

cmk list configurations name=<setting from the advisory>
cmk list zones
cmk list hosts type=Routing

This tells you how urgent the update is for you, and gives you a record of the state before the change.

Step two: back up what you would need to go back

Before any package changes, take a consistent dump of the CloudStack databases and copy the management server configuration files off the box:

mysqldump --single-transaction --routines --triggers cloud > cloud-before-update.sql
mysqldump --single-transaction cloud_usage > cloud_usage-before-update.sql
tar czf cloudstack-etc-before-update.tgz /etc/cloudstack

Keep the dumps off the management server.

Step three: the management servers

After updating the management packages, check two things. Then check two things after any update. First, that the service is enabled to start at boot, as you intend it to be. Second, that it really finished starting, not just that the process exists:

systemctl is-enabled cloudstack-management
systemctl status cloudstack-management
journalctl -u cloudstack-management -f
cmk list capabilities

Wait for the log to show startup complete and the API to answer before going further. If you run several management servers behind a load balancer, update them one at a time so the API stays available throughout.

Step four: agents, one host at a time, with a brake

Update the agent package on one KVM host, restart the agent, and wait until the management server sees that host as Up again before moving to the next. Script it if you have many hosts, but give the script a brake:

for h in $(cat kvm-hosts.txt); do
  ssh "$h" 'dnf -y update cloudstack-agent && systemctl restart cloudstack-agent' || exit 1
  until cmk list hosts name="$h" filter=state | grep -q Up; do sleep 10; done
done

If any host fails to update or to reconnect, the loop stops there, with one host to investigate instead of the whole cloud. Instances keep running on their hosts throughout, because the agent is not in their path.

Step five: system VMs, then routers by evidence

System VMs carry scripts and packages from the management server. After the update, restart or patch them through the API so they pick up the new code, then confirm they come back Running and connected:

cmk list systemvms

Virtual routers deserve more thought, because they sit in tenant traffic. The platform may mark routers as needing an upgrade, but that flag can be set when the code inside them has not meaningfully changed. Compare the scripts the router would receive after the update with the ones it already runs. If the router's code is unchanged, leave it alone and spare your tenants the interruption. If it did change, schedule router restarts per network, tell the affected tenants, and do them one at a time.

cmk list routers requiresupgrade=true

Step six: prove the end state

Run the same checks as in step one and compare: every host Up, every system VM Running, every router in its expected state, the advisory's setting where you want it, and a test instance created, started and destroyed through the API. Then note the time from first change to green, which tells you how long the next window needs to be.

How we do it

We build Apache CloudStack clouds per client, to the client's specification, on KVM hosts on HPE and Dell hardware with enterprise SAN storage underneath and ShapeBlue's commercial support behind the platform. Security advisories follow the procedure above. When an advisory was published for CloudStack, we applied the fix to a production cloud the same week this way, and every running customer instance stayed up throughout.

Credit where it belongs: the Apache CloudStack community and ShapeBlue do the hard work of finding, fixing and documenting these issues. If your own cloud has an update waiting because the procedure feels risky, we are happy to walk through it with you.

Talk to an engineer

Related: Dedicated cloud on Apache CloudStack, your spec · Private cloud on KVM with OLVM · Private Kubernetes on bare metal, OpenShift and OKD