A new research paper published on the arXiv preprint server evaluates legal document processing performance across multiple artificial intelligence systems. Researchers compared a model called Jev against nine other language models using ContractNLI, a benchmark designed for legal document inference. The study analyzed how models perform on legal contracts under varying request conditions, including changes to hypothesis visibility, requested outputs, and output order, while keeping the underlying contract text and target judgment fixed.
Evaluating Consistency Beyond Aggregate Accuracy
In legal contract analysis, systems must render reliable judgments across shared documents. However, the study highlights that high aggregate accuracy scores can conceal underlying volatility in individual decisions. Furthermore, the researchers noted that consistent agreement across repeated requests is not a reliable indicator of accuracy, as a model can repeatedly return the wrong answer. Among the evaluated systems, Jev achieved the lowest operational cost and median response time.
Unclear Details and Industry Outlook
The report leaves several technical and practical specifics unstated. The initial summary did not specify the exact identities or architectural details of Jev or the nine comparison language models. Furthermore, specific monetary costs, exact latency figures, accuracy percentages, and full prompt templates were not disclosed. The publication also did not specify whether these stability findings on ContractNLI extend to other legal domains beyond contract inference.
As AI deployment grows in the legal sector, researchers emphasize that evaluating model stability under varied prompt conditions may be more informative than relying solely on overall accuracy metrics.
Lead image: https://kaboompics.com/ / Pexels
