An adaptive method for choosing validation tasks used substantially fewer search resources in reported benchmark experiments, while its results were close to full-set search, according to an arXiv preprint.
How Task-CoEvolve works
Task-CoEvolve combines variance-weighted subset selection with full-set accuracy estimation from sampled evaluations.
The comparison included Meta-Harness full search, Naive fixed-subset search and Random-Resample, which resampled subsets at each iteration.
Results varied with the evaluation budget
The online text-classification experiments used 130 validation examples and 412 test examples. The search ran for 20 evolution iterations and generated 60 candidates. At evaluation budgets of 7% and 20%, Task-CoEvolve was reported as the highest-accuracy method, with 47.6% and 49.3%, respectively. Its reported advantages over Naive were 2.4 percentage points at 7% and 2.1 points at 20%; at 7%, it used 16 times fewer samples than full-set search.
The paper reported that Task-CoEvolve approached full-set search at a 7% budget and outperformed it at a 20% budget.
The estimate did not always find the winner
Against full-validation scores, the Spearman rank correlation, a measure of how closely candidate orderings matched, was 0.62 at 20% and 0.13 at 7%. The selected candidate ranked 12th of 60 at 20% and 10th of 60 at 7%; its full-validation scores were 51.3% versus 54.7% for the best candidate, and 46.4% versus 47.8%, respectively.
Terminal-Bench results were close to full search
On Terminal-Bench 2.1, the search used one rollout per task per iteration, ran 10 evolution iterations and generated 10 candidates. At a 20% budget, the reported average pass rate over all 89 tasks was 51.7% for Task-CoEvolve, compared with 52.8% for full search, 47.2% for Naive and 48.4% for Random-Resample.
Task-CoEvolve was below full search but above both subset baselines on this average. The same 89 tasks were used for search and final evaluation, so the result was not an independent held-out-task test.
Search costs were lower in two setups
In the Luna setup, Task-CoEvolve used 80% fewer input-token search resources; reported elapsed time was 11.5 hours, versus 22.2 hours for full search. In the Qwen setup, input-token use was 246 million versus 741 million, and reported elapsed time was 20.5 hours versus 38.0 hours. The paper reported cost reductions of 67% to 80%, while execution time depended on infrastructure and parallelism.
Reported means in a component comparison
At a 20% budget, the reported mean accuracies were 47.2% for Naive, 48.2% for resampling, 48.8% for estimation and 49.3% for variance-weighted selection. The highest reported mean was 49.3%.
The evidence has clear boundaries
Task-CoEvolve fixes the number of tasks evaluated per candidate before seeing results. It cannot stop early for clearly worse candidates or add evaluations when candidates are hard to distinguish.
Taken together, the reported results indicate potential search-cost savings in these benchmark settings, but they do not show that estimated-score selection always identifies the true best candidate. Nor does the Terminal-Bench result provide an independent held-out-task test, because the same 89 tasks were used for search and final evaluation.
Paper data and sources
Original title: Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
Authors: Atsuyuki Miyai, Kiyoharu Aizawa, Toshihiko Yamasaki
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text