RE-Bench: Evaluating frontier AI R&D capabilities of language model… — ml4se — TG.ME

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts The authors presented RE-Bench, a suite of environments that measure the ability of AI agents to automate AI R&D tasks. They compare humans to several public frontier models through best-of-k with varying time budgets and agent designs, and find that the best AI agents achieve a score 4x higher than human experts when both are given a total time budget of 2 hours per environment. However, humans currently display better returns to increasing time budgets, narrowly exceeding the top AI agent scores given an 8-hour budget, and achieving 2x the score of the top AI agent when both are given 32 total hours.