A new preprint reports benchmark tests in which Pandora’s Router achieved the best or tied-best average trade-off between routing regret and inspection cost across mathematical reasoning, retrieval-augmented factoid question answering and multi-model routing. It queried the costly estimator selectively rather than for every prompt.
The main score combined routing regret, meaning the penalty for a less effective model choice, with the price of inspecting the more expensive estimator. The researchers swept that assumed price to trace the cost-performance frontier.
Paying for a better estimate
The setup used two estimators. The cheap estimator, f, retrieved nearby calibration prompts from prompt embeddings and averaged their rewards; the costly estimator, g, was a fine-tuned language model using tailored, domain-specific context.
The paper frames the choice as a Pandora’s Box problem: a system can inspect a candidate for a price, but it need not do so if the expected benefit of extra information is too small. Under a Gaussian, or bell-curve, signal model, it derives direct formulas for the reservation price, meaning the highest price worth paying, and for the value of information.
When inspection was optional, the implementation used Monte Carlo, or repeated simulation, to estimate the payoff of each candidate policy, using 100 samples to decide which candidate would be held out. The evaluation compared the Router with cheap-only, expensive-only, Top-2, Coin Flip, Random-Npr and Margin-Npr controls.
The more accurate estimate cost much more
On held-out calibration data, g had lower mean-squared error (MSE), a measure of prediction error, in all three domains: MATH’s MSE was 0.096 for g versus 0.154 for f; RAG’s was 0.109 versus 0.175; and EmbedLLM’s was 0.198 versus 0.266.
The corresponding g-to-f cost ratios were 5.8 in MATH, more than 7,000 in RAG and 1.6 in EmbedLLM. These held-out comparisons were reported without uncertainty intervals.
Averaged across tested query costs, the Router’s combined regret-plus-inspection-cost totals were 0.105 in MATH, 0.118 in RAG and 0.386 in EmbedLLM. The MATH total tied Margin-Npr’s, while the RAG and EmbedLLM totals were the lowest reported.
At c_g = 0.001, the Router nearly matched g-only routing with much lower inspection cost. At c_g = 0.1, it never queried g and matched f-only in both routing performance and cost.
Per-cost analyses used paired bootstrap tests with p < .05 as the threshold. The paper says intermediate-cost differences were usually not statistically significant against all alternatives.
A second design lets specialists bid
The study also tested a decentralized version, Pandora’s Bidder. In a leave-one-out posted-price mechanism, one strategic specialist at a time faced the best costly estimate from the other specialists and could refine its own estimate before accepting or declining. The held-out specialist was rotated across models.
Across the cost sweep, the Bidder stayed near whichever fixed strategy had the lower regret: using f only or using g every time. It refined more at low costs and relied on cheap estimates or declined at higher costs.
But the result changed when the other specialists used weaker estimates. In RAG and EmbedLLM, competing posted prices generated by f rather than g improved individual surplus regret at most cost levels while increasing efficiency regret. In plain terms, a specialist could do better while the overall allocation got worse.
The evidence has a narrow boundary
The evidence is benchmark-based: learned value estimators, simulated routing decisions and a leave-one-out posted-price auction were tested across MATH, RAG and EmbedLLM. It does not show how the design would perform in a production system.
One central caveat concerns the opened-value calculation. The paper says its reduction is exact for calibrated posterior decision values, but only approximate when implemented with learned G scores.
The routing results depend on the specified inspection-price assumptions, while the Bidder’s welfare results depend on the accuracy of competing estimates.
Paper data and sources
Original title: Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation
Authors: Adam Fisch, Shubhendu Trivedi, Fantine Huot et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text