ZenDNN 6.0: FP16 Inference and MoE Acceleration on AMD EPYC™ CPUs

Jul 10, 2026

Introduction

ZenDNN 5.2.1 moved quantization from experimental to production-grade, with asymmetric WOQ, INT8 dynamic quantization, and standardized accuracy validation. That release proved that AMD EPYC™ CPUs could deliver serious quantized LLM inference at scale. ZenDNN 6.0 takes the next step: FP16 functional support for the upcoming  6th Gen AMD EPYC™ Server processors, Mixture-of-Experts (MoE) model optimization, and the broadest vLLM compatibility window for ZenDNN yet.

ZenDNN 6.0 is the first release to add FP16 functional support targeting the upcoming 6th Gen EPYC™ processors. Every major operator (MatMul, BatchMatMul, Embedding) now has a dedicated FP16 code path, laying the foundation for half-precision inference workloads. Add to that fused MoE operators in zentorch, optimized attention kernels, and vLLM parity all the way to v0.23.0, and ZenDNN 6.0 is well aligned with the modern LLM serving stack.

Why This Matters: From Current Gen to Next Gen

ZenDNN 5.2 introduced the re-engineered ZenDNN runtime with Low Overhead API and vLLM V1 integration. ZenDNN 5.2.1 brought quantization to production grade. ZenDNN 6.0 builds on this with a dual focus: expanding model support and improving inference performance on the current generation 5th Gen AMD EPYC™ processors, while also laying the groundwork for next-generation hardware. On the current-gen side, fused MoE operators, optimized attention kernels, and broader quantization options enable better throughput and support for newer model architectures like Mixture-of-Experts. Looking ahead, FP16 functional support prepares the software stack for the upcoming 6th Gen EPYC™ processors and their hardware FP16 capabilities. vLLM compatibility through v0.23.0 keeps both current and future deployments aligned with the fast-moving open-source ecosystem.

What’s Under the Hood

ZenDNN 6.0 builds on the modular multi-backend architecture introduced in 5.2, extending it with FP16 operator support, MoE-specific fusions, optimized attention kernels, and deeper quantization integration across the zentorch plugin and ZenDNN runtime.

Key Highlights at a Glance

  • FP16 Functional Support: Native FP16 data path for the upcoming 6th Gen EPYC™ processors across MatMul (with DLP), BatchMatMul, and Embedding operators in the ZenDNN core library and zentorch plugin.
  • Mixture-of-Experts (MoE) Optimization: BF16 Fused MoE operator, quantized MoE support (DA8W8, INT4 WOQ) in the vLLM pipeline, and Group MatMul optimized for expert parallelism.
  • Expanded vLLM Plugin Compatibility: Version parity from vLLM 0.20.0 through 0.23.0, maintaining the zero-code-change acceleration philosophy.
  • Quantization Pipeline Enhancements: LLM-Compressor integration, upstream quantized linear dispatch to vLLM, new TorchAO configuration presets, and DA8W8 support for MoE architectures.
  • zentorch Features and Optimizations: Optimized attention implementation in the ZenDNN core library, AOTI (CPP_Wrapper) support for Ahead-of-Time Inductor compilation, and targeted BF16 LLM throughput optimizations including matmul overhead reduction and runtime API improvements.
  • Modernized Framework Stack: PyTorch 2.12.0, TensorFlow 2.21.0 (backward build compatibility TF 2.16–2.21), Python 3.10–3.13, and vLLM 0.20.0–0.23.0.
Figure 1: ZenDNN Framework landscape at a glance
Figure 1: ZenDNN Framework landscape at a glance

Performance

Through ZenDNN 5.2.1, performance was measured using offline benchmarking, where a fixed batch of prompts is processed without simulating concurrent users. With ZenDNN 6.0, we move to online serving benchmarks, where requests arrive continuously at varying rates, reflecting real-world deployment conditions.

We also adopt multi-instance serving as the primary benchmarking mode for LLM inference on AMD EPYC™ CPUs. Rather than running a single vLLM process across an entire socket, we run N smaller vLLM instances, each pinned to a disjoint block of CPU cores. This keeps each instance’s working set NUMA-local, avoids cross-instance interference, and lets aggregate throughput scale closer to the available core count. Load is generated using GuideLLM, which sends chat workload requests at configured concurrency levels to an NGINX load balancer that round-robins traffic across the instances. Each run captures standard serving metrics (median and p95): request latency, time to first token (TTFT), inter-token latency (ITL), time per output token (TPOT), and output throughput (tokens/s). Page cache is dropped between runs for cold-start parity, and the load generator is NUMA-pinned so the client never contends with the instances under test.

Single-Socket Analysis

Figure 2: Single-socket analysis
Figure 2: Single-socket analysis: vLLM Online Serving – single instance of vLLM-native vs single instance of vLLM accelerated with ZenDNN on 5th Gen AMD EPYC™ CPU. See Footnote (ZD-063)

Dual-Socket Analysis

Figure 3: Dual-socket analysis
Figure 3: Dual-socket analysis: vLLM Online Serving – 7 instances of vLLM-native vs 7 instances of vLLM accelerated with ZenDNN on 5th Gen AMD EPYC™ CPU. See Footnote (ZD-064)

Accuracy

vLLM-zentorch quantization preserves model accuracy within tight margins of the BF16 baseline. Quantized models are validated using the LM Evaluation Harness with 5-shot prompting across standard benchmarks including GSM8K and ChartQA.

 

Table 1: Accuracy comparison of BF16 and DA8W8 configurations across popular LLM models on 5th Gen AMD EPYC™ CPU. See Footnote (ZD-065)

S.No.

Model Name

Tasks

(num_fewshots=5)

BF16

DA8W8

DA8W8 vs BF16
1 Llama-3.1-8B-Instruct

GSM8K

0.8438

0.8415

-0.27%

2

Phi-4-mini-instruct

GSM8K

0.8127

0.8143

0.20%

3

Qwen2.5-VL-7B-Instruct

ChartQA

0.8600

0.8604

0.05%

4

GPT-OSS-20B

GSM8K

0.8901

0.8757

-1.62%

5

Mixtral-8x7B-Instruct

GSM8K

0.6588

0.6543

-0.68%

Note: Negative (%) values indicate a drop in accuracy vs. the BF16 baseline; positive (%) values indicate an accuracy gain vs. the BF16 baseline. Accuracy losses greater than 5% should be evaluated for the specific deployment use case.

Conclusion

ZenDNN 6.0 features across AMD 5th Gen and upcoming 6th Gen  EPYC™ processors:

  • 5th Gen AMD EPYC™ processors: Fused MoE operators, optimized attention kernels, and expanded quantization options improve throughput and bring support for newer model architectures.
  • 6th Gen AMD EPYC™ processors: FP16 functional support lays the software foundation for half-precision inference on upcoming hardware.

Our Commitment to the Ecosystem Our upstream-first philosophy continues. The quantized linear dispatch in ZenDNN 6.0 has been upstreamed to vLLM (v0.22.1+). The expanded vLLM compatibility (now through 0.23.0) helps ensure that AMD EPYC™ CPU users can adopt the latest open-source inference releases without losing ZenDNN acceleration.

Run More with Less FP16 functional support will establish the foundation for half-precision inference on the upcoming 6th Gen EPYC™ processors. Combined with DA8W8 and INT4(WOQ) quantization, ZenDNN 6.0 offers the widest range of precision options for balancing throughput, memory, and accuracy. For organizations running AMD EPYC™ infrastructure, ZenDNN 6.0 makes it possible to run more AI workloads on existing hardware

Call to Action

Download the updated AMD ZenDNN Plugin for PyTorch (zentorch) and AMD ZenDNN Plugin for TensorFlow (zentf) via pip install or from GitHub and try the latest optimizations on your own workloads.

Try it today:

We’d love to hear about your results. Open an issue or start a discussion on our GitHub pages!

Acknowledgements

AMD team members who contributed to this effort: Arjit Mukhopadhyay, Avinash Chandra Pandey, Naveen Thangudu and team.

Resources

Footnotes

Footnotes

  • ZD-063: Testing conducted by AMD Performance Labs as of June 26, 2026 on the ZenDNN 6.0 software library, on a test system comprising of:
    AMD System: 2P AMD EPYC™ 9755 128-core processors, single socket used, SMT on, 24x96GB DDR5-6400 (running at 6000 MT/s), BIOS WVOT4807N, mitigations=off, single vLLM instance pinned to 96 cores, 2 NUMA nodes per socket. Software: Python 3.12, PyTorch 2.12.0+cpu, vLLM 0.23.0+cpu, zentorch-2.12.0.2 (ZenDNN 6.0).
    Workload: vLLM online serving, load generated with GuideLLM; input 512 tokens, output 512 tokens, num_prompts 1000; SLA TTFT p95 ≤ 15 s and TPOT p95 ≤ 400 ms; metric output_tokens_per_sec (higher is better).
    Results and configurations below are in the format of: [processor], [sockets], [SMT], [instances], [cores/instance], [software stack], [workload], [output_tokens_per_sec].
    Normalized Results (Base = vLLM-Native-BF16 @ 1.000); output_tokens_per_sec, higher is better:
    Model Software output_tokens_per_sec Normalized
    • Llama-3.1-8B-Instruct vLLM-Native-BF16 168.63 1.000
    • Llama-3.1-8B-Instruct vLLM-zentorch-BF16 202.64 1.20
    • Llama-3.1-8B-Instruct vLLM-zentorch-DA8W8 252.9 1.50
    • Phi-4-mini-instruct vLLM-Native-BF16 332.46 1.000
    • Phi-4-mini-instruct vLLM-zentorch-BF16 491.23 1.48
    • Phi-4-mini-instruct vLLM-zentorch-DA8W8 629.54 1.89
    • Qwen2.5-VL-7B-Instruct vLLM-Native-BF16 215.52 1.000
    • Qwen2.5-VL-7B-Instruct vLLM-zentorch-BF16 255.46 1.19
    • Qwen2.5-VL-7B-Instruct vLLM-zentorch-DA8W8 285.33 1.32

Performance may vary based on use of latest drivers and other factors.

  • ZD-064: Testing conducted by AMD Performance Labs as of June 26, 2026 on the ZenDNN 6.0 software library, on a test system comprising of: 
    AMD System: 2P AMD EPYC™ 9755 128-core processors (Turin), dual socket, SMT on, 24x96GB DDR5-6400 (running at 6000 MT/s), BIOS WVOT4807N, mitigations=off, 7 vLLM instances each pinned to a disjoint 32-core block (NUMA-local), NGINX round-robin load balancing, 2 NUMA nodes per socket. Software: Python 3.12, PyTorch 2.12.0+cpu, vLLM 0.23.0+cpu, zentorch-2.12.0.2 (ZenDNN 6.0).
    Workload: vLLM online serving, load generated with GuideLLM; input 512 tokens, output 512 tokens, num_prompts 1000; SLA TTFT p95 ≤ 15 s and TPOT p95 ≤ 400 ms; metric output_tokens_per_sec (higher is better). 
    Results and configurations below are in the format of: [processor], [sockets], [SMT], [instances], [cores/instance], [software stack], [workload], [output_tokens_per_sec]. 
    Normalized Results (Base = vLLM-Native-BF16 @ 1.000); output_tokens_per_sec, higher is better: 
    Model Software output_tokens_per_sec Normalized 
    • Llama-3.1-8B-Instruct vLLM-Native-BF16 223.83 1.000 
    • Llama-3.1-8B-Instruct vLLM-zentorch-BF16 306.12 1.37 
    • Llama-3.1-8B-Instruct vLLM-zentorch-DA8W8 629.76 2.81 
    • Phi-4-mini-instruct vLLM-Native-BF16 634.63 1.000 
    • Phi-4-mini-instruct vLLM-zentorch-BF16 659.58 1.04 
    • Phi-4-mini-instruct vLLM-zentorch-DA8W8 1313.3 2.07 
    • Qwen2.5-VL-7B-Instruct vLLM-Native-BF16 351.26 1.000 
    • Qwen2.5-VL-7B-Instruct vLLM-zentorch-BF16 447.8 1.27 
    • Qwen2.5-VL-7B-Instruct vLLM-zentorch-DA8W8 675.11 1.92 

Performance may vary based on use of latest drivers and other factors

  • ZD-065: Testing conducted by AMD Performance Labs as of June 26, 2026 on the ZenDNN 6.0 software library, on a test system comprising of: 
    AMD System: 2P AMD EPYC™ 9755 128-core processors (“Turin”), SMT on, 24x96GB DDR5-6400 (running at 6000 MT/s), BIOS WVOT4807N, mitigations=off,  Software: Python 3.12, PyTorch 2.12.0+cpu, vLLM 0.23.0+cpu, zentorch-2.12.0.2 (ZenDNN 6.0). Accuracy validated with the LM Evaluation Harness using 5-shot prompting on GSM8K and ChartQA, comparing DA8W8 quantization to the BF16 baseline.
    Accuracy results (LM Evaluation Harness, 5-shot; higher is better; delta = DA8W8 vs BF16 baseline): 
    Model Task BF16 DA8W8 Delta 
    • Llama-3.1-8B-Instruct GSM8K 0.8438 0.8415 -0.27% 
    • Phi-4-mini-instruct GSM8K 0.8127 0.8143 +0.20% 
    • Qwen2.5-VL-7B-Instruct ChartQA 0.8600 0.8604 +0.05% 
    • GPT-OSS-20B GSM8K 0.8901 0.8757 -1.62% 
    • Mixtral-8x7B-Instruct GSM8K 0.6588 0.6543 -0.68%

Performance may vary based on use of latest drivers and other factors

Disclaimers

The information presented in this document is for informational purposes only and may contain technical inaccuracies, omissions, and typographical errors. AMD reserves the right to revise this information and to make changes from time to time to the content hereof without obligation of AMD to notify any person of such revisions or changes.

AMD, the AMD Arrow logo, EPYC, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies.

Share:

Related Blogs

View All Blogs