An engineering simulation cluster on Azure scale sets

The situation

The client's engineers run fluid dynamics and structural simulations: large MPI jobs that want many cores, a low-latency link between nodes, shared storage and licensed solvers. They ran them on Azure VM scale sets of HB-series machines, which carry InfiniBand alongside normal Ethernet. MPI jobs exchange data between nodes constantly, so for this work the interconnect matters as much as the core count.

An existing cluster was in place when we took it on in mid-2023. It mostly worked, but it depended on a hosts file that went stale whenever the scale set restarted, and on build steps that lived in people's heads. In October 2023 we rebuilt it: new head nodes and compute nodes, InfiniBand configured from scratch, shares served over the fabric, and the scheduler, its authentication and its accounting set up again. Users confirmed the first rebuilt cluster was running later that month, and the old clusters were decommissioned in November 2023.

There was no infrastructure as code here. The clusters were built with the Azure CLI and shell scripts, and acceptance was the engineers' own test jobs, run before an old cluster was switched off.

What we did

Fixed placement before the first node existed

After the rebuild, one compute node could not mount the shared home directory. Everything else on it looked healthy. The difference was one field in ip link: the node's InfiniBand broadcast address, which carries the partition key, did not match the other nodes. It had landed on a different part of Azure's fabric. That value is read-only on the node, so the fix had to come before the node existed.

We rebuilt the scale set with a proximity placement group and a single placement group, and moved that decision to the top of the build runbook. A minimal version of the create command:

# placement is decided at creation and cannot be changed on a running node
az vmss create \
  --resource-group rg-hpc --name vmss-hpc \
  --vm-sku Standard_HB120rs_v3 --image <hpc-image> \
  --ppg ppg-hpc --single-placement-group true \
  --orchestration-mode Uniform --upgrade-policy-mode Manual \
  --subnet <hpc-subnet> --computer-name-prefix hpc- \
  --tags owner=<engineering-lead> support=<hpc-admin>

Gave the fabric predictable addresses

Azure assigns the InfiniBand address at boot, and its default mask overlapped the client's corporate networks. We narrowed the fabric to its own /24 and, in a later version of the build, derived each node's fabric address from the last octet of its Ethernet address in a boot-time unit that also mounts the shares. Node .17 on Ethernet is .17 on InfiniBand on every boot, and the head node's fabric address never moves.

Made the scheduler survive scale-set behavior

Scale sets rename nodes when they restart. Users would submit a job, get a job number, and nothing would run, because the controller could not resolve the new names. We kept one canonical hosts block per cluster on the shared mount and wrote a short recovery runbook. Two other recurring faults turned out to be configuration: "MPI is broken" was a scheduler config file that ordinary users could not read, and lost data mounts after reboots came from the platform agent rewriting /etc/fstab, which we locked with chattr +i. Both fixes went into the build scripts, so new nodes started correct.

Scheduled licenses like cores

Solver licenses were the scarce resource. A job that starts without a free seat holds a large node and then fails. We registered each solver's licenses in SLURM's accounting database as resources backed by the license server, so jobs ask for a seat and wait in the queue until one is free:

# count solver seats the way the scheduler counts cores
sacctmgr add resource name=solver-a count=<seats> \
  server=license-host servertype=rlm type=license
sacctmgr add resource name=solver-a server=license-host \
  cluster=cluster-a percentallowed=100

# a job then asks for a seat and queues until one is free
sbatch --licenses=solver-a:1 run-case.sh

Different solvers also needed different Intel MPI versions. We kept four side by side as environment modules, so a user picks one with module load instead of asking for a node rebuild.

Moved node builds onto a golden image

Between November 2023 and July 2024 the clusters were rebuilt or re-imaged five times. The causes were mostly outside the cluster: OS or image end of life, InfiniBand fabric faults, and new VM sizes or quota changes. Each rebuild repeated a long manual build of drivers, scheduler, authentication, MPI modules, solver dependencies, users and mounts.

The build matured in three steps: scripts during the 2023 rebuild, a two-stage quick setup in August 2024, and in November 2024 a golden image captured from a working node. New nodes now start from that known-good image, with a short first-boot list to finish them. One ordering rule survived every version: restore home directories from blob storage before creating users, or every account gets a new, empty home.

Results

• Two SLURM clusters running the engineers' simulations on HB-series scale sets, with shared storage over InfiniBand.
• Solver licenses scheduled as resources, and four MPI stacks available as modules.
• Faults that used to need an administrator (stale hosts, unreadable config, lost mounts) fixed in the build itself.
• Five rebuilds in about nine months, followed by a golden image that new nodes start from.

We did not measure node build time before and after the image, queue wait for licenses, or monthly cost.