Research recently hosted on the arXiv preprint server indicates that the common assumption regarding the determinism of greedy decoding in Large Language Models (LLMs) is incorrect when switching between numerical precisions. While greedy decoding is typically expected to produce the same result for a given prompt, researchers found that switching between Bfloat16 (BF16) and Float16 (FP16) precisions causes significant output divergence, even when using identical hardware and the same decoding algorithm.
The Impact of Precision Divergence
The study evaluated six models ranging from 1.1 billion to 7 billion parameters across four different model families, with additional testing conducted at the 12-billion parameter scale. Across three benchmarks, the researchers observed that between 49% and 100% of prompts resulted in divergent outputs when the precision format was changed. This phenomenon occurs because a single “token flip”—where the model selects a different word or character due to minor numerical variations—often cascades into a completely different sequence of text. This suggests that the reproducibility of AI outputs is highly sensitive to the underlying mathematical format used during inference.
Remaining Uncertainties in Error Propagation
The researchers developed an empirical error-propagation analysis to understand how these differences emerge. They identified that approximately 22 layers of accumulated error contribute to these flips, though the report indicates that this accumulation alone does not fully distinguish why specific tokens flip while others do not. The announcement did not specify which particular model families or benchmarks were used in the testing. Furthermore, the report did not name the specific hardware used beyond stating it was identical for both precision tests. It remains to be seen how these findings will affect standard industry practices, where BF16 and FP16 are often used interchangeably during model deployment.
This research highlights a fundamental challenge in achieving consistent AI behavior across different computing environments and precision standards.
Lead image: panumas nikhomkhai / Pexels
