A new research paper published on arXiv cs.CL introduces a method for reducing the size of the encoder in OpenAI’s Whisper model. Titled “Six Layers Less: Encoder Pruning for Whisper with Label-Free Recovery,” the study addresses a specific area of model compression that has seen less development than its decoder-based counterparts. While transformer-based automatic speech recognition (ASR) models have been pruned for speed in the past, this approach specifically targets the encoder to further streamline performance.
The Shift from Decoder to Encoder Pruning
In the current landscape of ASR optimization, much of the focus has been placed on the decoder. Existing variants like whisper-large-v3-turbo have successfully reduced the decoder from 32 layers down to four, while Distil-Whisper has reached a two-layer configuration. These modifications have historically provided the most significant end-to-end transcription speedups. However, the encoder has remained relatively large in most widely adopted versions of the model. The researchers note that encoder pruning has likely seen less adoption because custom inference implementations are often necessary to take advantage of the reduced architecture. This new approach aims to bridge that gap by focusing on a six-layer reduction of the encoder component.
Missing Data on Performance and Implementation
The report does not specify the exact impact this pruning has on word error rates (WER) or general accuracy across different languages. Furthermore, the announcement did not specify the technical details of the “label-free recovery” mechanism mentioned in the title, nor did it define the specific datasets used to validate the recovery process. It remains unclear if the custom inference requirements mentioned will be made available as open-source code or if they will require specialized hardware to function. Observations of future benchmarks will be necessary to determine if this method maintains the robust performance associated with the original Whisper architecture.
This research highlights a move toward more comprehensive compression of ASR models. If the six-layer reduction proves stable, it could allow for faster speech-to-text processing on devices with limited computational resources.
Lead image: Júlio Riccó / Pexels
