MLPerf Inference Results

How to read these tables

• Workloads — llama2-70b-99 (Meta Llama 2 70B chat), llama3.1-405b (Meta Llama 3.1 405B — the largest tracked model), mixtral-8x7b (Mistral Mixtral mixture-of-experts), stable-diffusion-xl (text-to-image). The -99 / -99.9 suffix is the accuracy track (see below).
• Server vs Offline — MLPerf's two scenario types. Server simulates real-time traffic with per-query latency limits (the system has to keep up with arriving requests). Offline is batch processing with no latency constraint — pure peak throughput. The same hardware will score differently on each.
• Engine — the inference / serving software (TensorRT-LLM, vLLM, LLMBoost, etc.). Throughput often differs more between engines than between hardware variants of the same chip — a top result on LLMBoost or a vendor-internal stack means the buyer would need that same stack to reproduce. Pulled from MLCommons' Software field; version numbers stripped.
• # GPUs — total GPU/AI chips the submission used (per-node count × nodes). The system column counts the whole rack; the per-GPU column divides by this.
• Tokens/s/GPU vs Tokens/s (system) — the per-GPU column is the buyer-comparable metric (what does one chip do?). The system column is the absolute total MLCommons reports — useful for capacity planning but rewards bigger clusters on a leaderboard. Rows here are sorted by per-GPU rate. Different workloads use different units: LLMs report tokens/s, image generation samples/s, classifiers queries/s.
• Accuracy: 99.0% vs 99.9% — MLPerf's two accuracy tiers per workload. Both mean the submission cleared a quality bar relative to a reference baseline; 99.9% is the stricter "very-high-accuracy" track. This is not the model's absolute accuracy — it's which accuracy track the submitter chose.