Prime Intellect has introduced Prime Mega MoE, marking the release of a backward pass for DeepSeek’s fused Mega MoE kernel. According to reporting by MarkTechPost, the new development makes the kernel trainable and accelerates Qwen3 MoE training steps by up to 34 percent within prime-rl, while delivering a 3.11 times faster forward pass.
Understanding Fused Mixture of Experts Kernels
Mixture of Experts architectures divide model capacity into multiple sub-networks, or experts, activating only a subset for each token. Fusing operations inside these kernels typically reduces memory overhead and improves hardware utilization during processing. By adding a trainable backward pass to DeepSeek’s existing fused architecture, developers can now update model weights during training more efficiently rather than relying solely on optimized forward inference.
Unanswered Technical Questions
While the performance metrics outline significant speed improvements for specific workflows, the announcement did not specify the exact hardware configurations used to benchmark the 3.11x forward pass speedup or the 34 percent training step acceleration. Furthermore, broader compatibility requirements and implementation steps for other model architectures remain unstated in the available details.
Developers and researchers will need to monitor further technical documentation from Prime Intellect to evaluate integration ease across diverse infrastructure setups.
Lead image: panumas nikhomkhai / Pexels
