GlyphAI When Thousands of Servers Fail at Once
Large GPU clusters fail every few hours. The root cause of a fleet-wide incident should take minutes to find, not weeks.
When thousands of servers fail in the same second, the ones that died instantly could not write down why. Their logs are a storm of identical errors. The cause is usually in the log of the one machine that lived a second longer.
I have stood in the middle of such an incident. Every log said the same thing: a processor catastrophic error. The vendor guide, written for one server, said to update firmware or swap the CPU. Server by server, that meant replacing thousands of processors. The real answer was one firmware version, and finding it took weeks.
What Meta, Google and Microsoft have published
| Company | Finding |
|---|---|
| Meta, research clusters | A 1,024-GPU job fails every 7.9 hours; projected every 1.8 hours at 16,384 GPUs and every 14 minutes at 131,072. Hardware failures hit 0.2 % of jobs but cost 19 % of GPU runtime. |
| Meta, Llama 3 405B | 419 unplanned interruptions in 54 days on 16,384 H100s, about one every three hours; 61 % came from the GPU and its memory. |
| Google, Gemini | Silent data corruption expected every week or two; goodput of the largest job raised from 85 % to 97 %. |
| Microsoft, Philly cluster | 31 % of jobs were killed or failed and used 55 % of GPU time; infrastructure failures were rarer but struck late and cost the most. |
In Azure, a host whose firmware fails TPM attestation is cut off from the fleet until its root cause is found. When a vendor's firmware is wrong, thousands of healthy servers can drop out at once, and the clock runs until someone finds the common cause.
Why it takes weeks
The storm hides the signal. Every vendor writes the same event differently. Vendor guides are written for one server. A GPU that fell off the bus goes silent instead of logging. And the common cause, a firmware version or a rack, lives in the inventory, not in the logs.
Compress meaning, not bytes
GlyphAI Fleet RCA turns every log line, from any vendor, into a glyph: one symbol per meaning. Storms become one glyph with a counter. Servers that told the same story collapse into one signature, and the survivor stands alone because its story has one more chapter. Crashed servers are joined with the inventory to find the common cause; a GPU off the bus is confirmed by three witnesses: the BMC, the kernel and the PCIe tree. Dell, Lenovo and HPE messages are read from the vendors' own guides.
On a synthetic fleet of 6,000 servers and 4.1 million events, it found the single survivor and the firmware version behind the crash in about 98 seconds on one CPU core. Stated limits: synthetic data so far, causality inferred from topology and timing, and nothing to read where a server wrote nothing.
When your cluster goes dark
If you run a GPU cloud or a large cluster and have an incident whose cause is still open, send the logs under NDA. I will run the analysis at no cost on the first incident, in exchange for honest feedback.
Sources
- Meta: Revisiting Reliability in Large-Scale Machine Learning Research Clusters (2024)
- Meta: The Llama 3 Herd of Models (2024)
- DatacenterDynamics: Training Google's Gemini
- Microsoft: Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads (2019)
- Microsoft Learn: Firmware measured boot and host attestation