Large language models can grade complex computer science exams with greater consistency than human evaluators, but minor changes in prompt instructions can cause severe system breakdowns, according to a paper published on the arXiv cs.CL preprint repository. Researchers evaluated 171 model configurations across open-weights and closed-weights architectures using a practical Computer Vision exam taken by 570 dual-graded students.
Evaluating Precision and Prompt Sensitivity
Evaluating long-form academic exams typically requires hundreds of hours from qualified graders, presenting a bottleneck in large university courses. In testing, the top-performing model configuration achieved a mean absolute error of 1.64 out of 35 points. This outperformed the disagreement rate between two human graders evaluating the same submissions, who showed a mean absolute error of 2.61 out of 35 points against each other.
However, the experiments demonstrated extreme brittleness when prompts were altered. Adding a short preamble instructing systems to act as a “strict grader” drove 14 out of 17 open-weights models out of the acceptable grading band, resulting in a mean absolute error of 8 or higher. Three of those open-weights models stopped generating grades entirely under the modified prompt.
Unanswered Questions and Operational Limits
The study leaves several technical and logistical details unaddressed. The paper did not specify the names of the open-weights or closed-weights model families evaluated, nor did it identify the specific academic institution where the exam was administered. Additionally, the report did not specify the total compute cost required to run 171 separate grading configurations, or how the models parsed visual diagrams within the computer vision exam. It remains unclear whether targeted fine-tuning could prevent open-weights models from failing under strict prompt directives.
As educational institutions consider automated grading tools to manage teaching workloads, the results indicate that high accuracy is achievable, but operational reliability remains highly dependent on prompt design.
Lead image: RDNE Stock project / Pexels
