A theoretical framework proposes a way for firms to limit what third parties can infer from online experiments without giving up all of the random testing needed to learn. In simulations, a privacy strategy that changed over time pulled further ahead of a constant strategy as tests grew longer, while allocating a firm's privacy budget according to regret often produced more clicks than splitting it evenly.
The work is an arXiv preprint built from mathematical analysis and simulations, not a newly recruited participant study or a live deployment. Its inputs include click-through rates from a website-design RCT, a recommendation field experiment and 78 online RCTs from ASOS. The framework extends privacy risk from individual visitors to single experiments and then to a firm's full portfolio.
A trade-off between privacy and learning
At the heart of the proposal is differential privacy, a mathematical way to limit what an outside party can infer from an experiment's displayed output, paired with randomized response, which deliberately adds randomness to that output. The model then uses a segment-specific epsilon-greedy rule that balances testing alternatives with choosing a leading option.
When the model's ξ parameter is 0, the output is fully randomized; as ξ approaches infinity, the policy approaches pure exploitation. The analysis reports that, when privacy randomization matches the exploration needed for learning, the order of the regret bound—a measure of the learning penalty—does not change. Stricter privacy can increase regret by forcing extra exploration.
Longer tests favored a changing budget
For the website example, the authors used click-through rates from an RSM MBA website RCT comparing abstract and concrete language. The reported rates were 0.295 for the abstract version and 0.324 for the concrete version. They simulated tests of 10,000, 100,000 and 1,000,000 visitors across privacy budgets of 0.05, 0.5, 1, 3 and 5, using 1,000 bootstrap repetitions and 95% confidence intervals.
At longer horizons, the dynamic strategy's advantage over the constant strategy became more pronounced. Mean differences were reported as statistically significant in the 100,000- and 1,000,000-visitor simulations, although the supplied text does not give the p-values or the endpoints of the confidence intervals.
The pattern held differently in recommendations
A separate recommendation application drew on a seven-day ZOZOTOWN field experiment with 1,374,327 visitors and 80 arms, or competing recommendation options. Visitors saw three fashion-item recommendations selected uniformly at random. The simulations varied the number of arms between 2, 4 and 8 and used horizons of 100,000 and 1,000,000 visitors.
In those simulations, constant-strategy performance initially improved as the privacy budget increased, then leveled off or could decline at higher budgets. Dynamic-strategy performance increased with the budget and eventually plateaued at the optimal-performance benchmark.
Regret elasticity—the sensitivity of regret to the privacy budget—offers a way to locate an ideal constant-strategy budget: its zero crossing shifted upward as visitor count or arm count increased. With the dynamic strategy, further budget increases had no effect on expected regret once the privacy cap no longer bound the policy; that cutoff increased with both the test horizon and the number of arms.
A firm-wide budget changed the allocation question
At the firm level, the simulations used 78 ASOS online RCTs. The experiments averaged 21 million visitors, ranging from 69,000 to 149,197,471, and averaged 43.5 days, with a maximum of 131 days. Arm click-through rates averaged 19.55%; the interquartile range was 5.69% to 24.83%, and the full range was 2.65% to 77.24%.
The portfolio analysis tested 10 firm-wide privacy budgets, from 0.01 through 500, summarizing 100 repetitions by their medians and showing interquartile-range error bars. Regret-based allocation often delivered more clicks than equal splitting, with the biggest gains at intermediate budgets including 5 and 20. At very large budgets, both approaches approached the upper benchmark.
What the simulations leave unanswered
The supplied analysis describes the evidence as analytical derivations and simulations based on prior experiments, rather than a direct test of a deployed system. It does not report direct measurement of tracker belief updating, and the authors caution that simulation results may not generalize to live systems.
The firm-wide privacy constraint is conservative because it assumes the same customer could appear in every linkable experiment, even though actual exposure may be lower. The privacy guarantee covers the experimental output itself, not a customer's later interaction or clicks, and the model does not vary privacy sensitivity by arm.
Paper data and sources
Original title: A Privacy Budgeting Framework for Online Experimentation
Authors: Gilian R. Ponte, Alina Ferecatu
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text