A new research paper reported by arXiv cs.AI introduces DAYJOB, a benchmark designed to evaluate AI systems on long-horizon professional tasks. Professional assignments frequently begin with minimal instructions, requiring human workers to independently determine necessary actions, relevant documents, and the validity of the initial premise. To test whether autonomous agents can handle this complexity, researchers built DAYJOB using 130 tasks crafted by domain experts.
The Structure of DAYJOB
The benchmark comprises 50 healthcare tasks and 80 finance tasks. According to the announcement, these assignments require significant time investment, with an estimated average completion time of 13.6 hours for healthcare tasks and 16.6 hours for finance tasks. Each task is delivered as a containerized Harbor environment. Evaluation is handled by an agentic judge that applies a strict rubric of binary criteria to the final delivered files. The median number of binary criteria per task is 47.5 for healthcare and 57.5 for finance, and an attempt is marked as passing only if all requirements are successfully met.
Open Questions and Future Outlook
While the paper outlines the scope and evaluation framework of DAYJOB, the announcement did not specify which particular AI models or autonomous agent architectures were tested using the benchmark. Furthermore, the publication details regarding when the benchmark will be publicly accessible for broader research use were not specified in the initial wire report. Observers tracking AI evaluation methods will need to watch for future updates regarding baseline scores and accessibility.
DAYJOB represents a shift toward more rigorous, time-intensive evaluations for professional AI agents.
Lead image: https://kaboompics.com/ / Pexels
