ZAYA1 AI Model Trained on AMD GPUs Reaches Major Milestone

Zyphra, AMD, and IBM spent a year testing whether AMD's GPUs and platform can support large-scale AI model training — and the result is ZAYA1.

In partnership, the three companies trained ZAYA1 — described as the first major Mixture-of-Experts (MoE) foundation model built entirely on AMD GPUs and networking — which they see as proof that the market doesn't have to depend on NVIDIA to scale AI.

The model was trained on AMD's Instinct MI300X chips, Pensando networking, and ROCm software, all running across IBM Cloud's infrastructure. What's notable is how conventional the setup looks. Instead of experimental hardware or obscure configurations, Zyphra built the system much like any enterprise cluster — just without NVIDIA's components.

💡 Zyphra says ZAYA1 performs on par with, and in some areas ahead of, well-established open models in reasoning, maths, and code — offering businesses a second option that doesn't require compromising on capability.

⚙️ How Zyphra Used AMD GPUs to Cut Costs Without Gutting AI Training Performance

Most organisations follow the same logic when planning training budgets: memory capacity, communication speed, and predictable iteration times matter more than raw theoretical throughput.

MI300X's 192GB of high-bandwidth memory per GPU gives engineers breathing room, allowing early training runs without immediately resorting to heavy parallelism — simplifying projects that are otherwise fragile and time-consuming to tune.

Zyphra built each node with eight MI300X GPUs connected over InfinityFabric, each paired with its own Pollara network card. A separate network handles dataset reads and checkpointing. It's an unfussy design — the simpler the wiring and network layout, the lower the switch costs and the easier it is to keep iteration times steady.

🚀 ZAYA1: An AI Model That Punches Above Its Weight

ZAYA1-base activates 760 million parameters out of a total 8.3 billion and was trained on 12 trillion tokens in three stages. The architecture leans on:

• 🔹 Compressed attention for efficient context handling
• 🔹 A refined routing system to steer tokens to the right experts
• 🔹 Lighter-touch residual scaling to keep deeper layers stable

The model uses a mix of Muon and AdamW optimisers. To make Muon efficient on AMD hardware, Zyphra fused kernels and trimmed unnecessary memory traffic so the optimiser wouldn't dominate each iteration. Batch sizes were increased over time, depending heavily on storage pipelines that deliver tokens quickly enough.

📊 ZAYA1 competes with larger peers including Qwen3-4B, Gemma3-12B, Llama-3-8B, and OLMoE. Its MoE structure means only a sliver of the model runs at once — helping manage inference memory and reducing serving cost.

A bank, for example, could train a domain-specific model for investigations without needing convoluted parallelism early on. The MI300X's memory headroom gives engineers space to iterate, while ZAYA1's compressed attention cuts prefill time during evaluation.

🛠️ Making ROCm Behave With AMD GPUs

Zyphra didn't hide the fact that moving a mature NVIDIA-based workflow onto ROCm took work. Instead of porting components blindly, the team spent time measuring how AMD hardware behaved and reshaping model dimensions, GEMM patterns, and microbatch sizes to suit MI300X's preferred compute ranges.

InfinityFabric operates best when all eight GPUs in a node participate in collectives, and Pollara tends to reach peak throughput with larger messages — so Zyphra sized fusion buffers accordingly. Long-context training, from 4k up to 32k tokens, relied on ring attention for sharded sequences and tree attention during decoding to avoid bottlenecks.

Storage considerations were equally practical. Zyphra bundled dataset shards to reduce scattered reads and increased per-node page caches to speed checkpoint recovery — vital during long runs where rewinds are inevitable.

🛡️ Keeping Clusters on Their Feet

Training jobs that run for weeks rarely behave perfectly. Zyphra's Aegis service monitors logs and system metrics, identifies failures such as NIC glitches or ECC blips, and takes corrective actions automatically. The team also increased RCCL timeouts to keep short network interruptions from killing entire jobs.

⚡ Checkpointing is distributed across all GPUs rather than forced through a single chokepoint. Zyphra reports more than ten-fold faster saves compared with naïve approaches — directly improving uptime and cutting operator workload.

📈 What the ZAYA1 AMD Training Milestone Means for AI Procurement

The report draws a clean line between NVIDIA's ecosystem and AMD's equivalents:

The authors argue the AMD stack is now mature enough for serious large-scale model development. None of this suggests enterprises should tear out existing NVIDIA clusters. A more realistic path is to keep NVIDIA for production while using AMD for stages that benefit from MI300X's memory capacity and ROCm's openness — spreading supplier risk and increasing total training volume without major disruption.

✅ Key Takeaways & Recommendations

• 🔸 Treat model shape as adjustable, not fixed
• 🔸 Design networks around the collective operations your training will actually use
• 🔸 Build fault tolerance that protects GPU hours rather than merely logging failures
• 🔸 Modernise checkpointing so it no longer derails training rhythm

It's not a manifesto — just a practical takeaway from what Zyphra, AMD, and IBM learned by training a large MoE AI model on AMD GPUs. For organisations looking to expand AI capacity without relying solely on one vendor, it's a potentially useful blueprint.