Researchers have investigated whether the justifications provided by large language models (LLMs) for rejecting specific candidates in a selection task are grounded in the actual content of the text. When asked to choose between options and explain their reasoning, models frequently cite the absence of specific facts—such as a missing director or a date of death—in a candidate‘s profile. To test the validity of these explanations, a study reported by arXiv cs.AI used a method where the “missing” fact was inserted into the profile to see if the model’s decision changed. This process utilized greedy decoding to ensure consistency in the model’s output during the re-evaluation.
Evaluating Model Reasoning Integrity
The study addresses a core concern in AI interpretability: whether a model’s natural language explanation reflects its underlying decision-making process or is merely a post-hoc rationalisation. By inserting the exact information the model claimed was missing, researchers could measure if the explanation actually influenced the outcome. To ensure the results were not simply due to increased text length or placement, the experiment included two controls: a length-matched irrelevant sentence and the insertion of these sentences into a third, unmentioned option. In the largest of three experimental runs, six open-source models were evaluated. This approach provides a concrete way to verify claims about text content without requiring a human judge to interpret the model’s intent.
Unknown Variables in Model Selection
The available report does not specify which six open-source models were included in the largest run, nor does it list the full names of the researchers involved. The specific types of candidates being evaluated—whether they were historical figures, fictional characters, or job applicants—were not detailed in the initial summary. Furthermore, the specific results of the three runs, including how often the models successfully updated their choices after the missing facts were provided, remain unstated. It is also unclear how these models might perform under non-greedy decoding settings where randomness is introduced.
As AI models are increasingly used for recommendation and selection tasks, verifying the honesty of their explanations remains a critical step for deployment. Future research may clarify if this behavior persists across larger proprietary models.
Lead image: Bibek ghosh / Pexels
