Title: Discrepancy-Based Design for A/B Testing Experiments
时间：2018年6月13日10:00-11:00
地点：数学楼二楼报告厅
Abstract:
The aim of this paper is to introduce a new design of experiment method for A/B tests. A/B tests (or “A/B/n tests”) refer to the experiments conducted to estimate the treatment effect(s) of a two-level or multi-level controllable experimental factor. To conclude whether the treatment effect is significant, the common practice is to use a completely randomized design and perform the hypothesis test on the sample difference-in-mean estimate. However, such estimator is not always accurate when the covariates of the test units affect the responses, especially for the small to medium-sized experiments. To overcome this issue, we propose the discrepancy-based design which significantly improves the accuracy of the estimates of the treatment effects, as shown both theoretically and through simulations. More importantly, the design approach is model-free, and thus it makes the estimation robust to the model assumption. Also, it can be applied to both continuous and discrete/categorical types of responses. We develop two optimization procedures to minimize the discrepancy criterion for both offline and online experiments.