As our latest guest for our Research Seminar Series we welcomed Eric Schwartz from the University of Michigan. He shared his recent work on A-B testing in online advertising experiments.

To test user responses to creative elements in online ads, advertisers and researchers typically rely on randomized experiments. To ensure internal validity of the experiment, the experimental groups are supposed to be similar to each other. In his presentation, Eric pointed out potential issues with using the tools for randomized experiments offered by online advertising platforms. The platforms’ targeting algorithms optimize the assignment of different ads to the users. In a simulation Eric and his co-author Michael Braun show that there is a potential bias in estimating the performance of an ad on aggregate level due to heterogeneous user responses. This bias can lead to an incorrect conclusion about which ad performs better and can even create an undetectable Simpson’s reversal. A Simpson’s reversal occurs if all unobserved heterogenous groups of users prefer a specific ad over the other, but one would infer the opposite from the biased test result of the platform tool.

Thank you, Eric, for your visit and sharing your work with us!