A recent experiment involving the Qwen3.8 27B model has explored how well it can compute addition problems and return the answers spelled out in words. Reported by Simon Willison, the test was inspired by an older experiment run with GPT-4o on increasingly large numbers. To test the behavior in a fully controlled environment, the new evaluation was executed locally using a DGX Spark.
Revisiting Numeric Word Generation
Large language models typically process numbers as tokens rather than strict mathematical values. Asking a model to perform arithmetic and output the result in words rather than digits often reveals unexpected failure modes. Running such evaluations on local hardware allows researchers to isolate the model’s behavior from any API-side modifications or third-party wrappers, providing a clearer view of its raw capabilities.
Unknowns and Future Testing
The available information does not specify the exact prompt format used for the Qwen3.8 27B experiment, nor does it detail the complete error rate across different digit lengths. Furthermore, the announcement did not specify whether similar tests will be conducted on other open-weights models in the near future.
Further details regarding the exact parameters and full dataset results have not yet been published.
Lead image: Nicolas Foster / Pexels
