A new research paper reported by arXiv cs.CL outlines an evaluation of how agentic artificial intelligence models handle complete data science workflows. The study tests seven different models from five developers—Anthropic, OpenAI, Google, DeepSeek, and Z.AI—in roles where they act as both creators and evaluators of technical projects. During the test, a writer model uses specific tools to download, clean, and analyze a dataset before writing a short paper about the findings. An execution environment runs the project, and a grader model assesses the results. The writer model then has the opportunity to revise its submission over a total of three rounds to see if iterative feedback improves the final output.
Evaluating Multi-Round AI Collaboration
This research addresses a key question in modern machine learning: the capability of models to function autonomously across multi-step tasks rather than simply answering isolated prompts. By setting up a loop where models generate, execute, grade, and revise data science tasks, the study provides insight into how agentic systems handle complex, sequential problem-solving. Evaluating models from major developers like OpenAI, Anthropic, Google, DeepSeek, and Z.AI allows for comparisons across different architectures and training methodologies, particularly regarding their ability to learn from critique within a closed loop.
Unresolved Details and Future Steps
Despite outlining the general framework of iterative creation, grading, and revision, the available report leaves several operational specifics unanswered. The announcement did not specify the exact datasets used for the tests, nor did it disclose the specific identities of the seven models evaluated across the five developer organizations. Additionally, the criteria used by the grader models to score the data science projects were not detailed in the available text, leaving open questions about the exact metrics driving the revision process.
As research into multi-agent feedback loops continues, observing how these models handle broader technical tasks will remain a key area of study for the artificial intelligence community.
Lead image: Abdul Kayum / Pexels
