Researchers have developed a new fine-tuning approach to improve how large language models handle speech translation. As reported by arXiv cs.CL, the study identifies a significant performance gap in models that use transcription-based chain-of-thought (CoT) processing. This gap occurs because models are typically trained on perfect human-provided transcripts but must generate their own transcripts when used in the real world. If the initial transcription contains errors, the subsequent translation often fails because it was never trained to handle imperfect inputs.
Joint Recognition and Reward Learning
To solve this discrepancy, the researchers applied Group Relative Policy Optimization (GRPO) to the Qwen2.5-Omni-3B model. This technique allows for joint fine-tuning of both speech recognition and translation. Unlike standard methods that treat these as separate steps, this approach scores both the transcript and the translation simultaneously. Crucially, the translation quality is evaluated based on its relationship to the model’s own generated transcript, forcing the system to become more resilient to its own internal errors. The team tested three different token advantage strategies to determine the most effective way to reward the model for accurate outputs across four different languages.
Missing Data and Implementation Details
While the study outlines the methodology, the announcement did not specify which four languages were used during the evaluation phase. Additionally, the researchers did not provide specific numerical benchmarks or percentage improvements to quantify how much better this method performs compared to standard supervised fine-tuning (SFT) or direct speech translation (Direct ST). It is also unclear whether this joint reward learning approach requires significantly more hardware resources or time than traditional training methods. The report did not mention if the modified Qwen2.5-Omni-3B model or the training datasets would be released for public use.
By aligning the training environment with actual inference conditions, this method seeks to make LLM-based translation more reliable for practical applications.
Lead image: Anna Pou / Pexels
