A/B test on model - split on results

I developed a predictive model that assigns the best product (P1, P2, P3) for each customer. I wanted to compare the conversion rate using this model VS the as-is deterministic assignment, so I applied an A/B test: I decided the product between P1, P2, P3

  • using the model on 50% of my users
  • using the deterministic rules on the other 50%

and then I compared the different conversion rates.

My question is: is it correct to split the analysis on the test results by product? So I'd like to compare

  • test CR on P1 VS control CR on P1
  • same on P2, P3

I have a doubt since I'm using the model's results (so the product assignment) as the splitting feature. Is it correct, or am I introducing bias in the analysis?

Topic ab-test predictive-modeling

Category Data Science

1
answered at

Provided everything is coacher, and you're doing the analysis of the conversion rate in a way that is devoid of any bias or error, I don't see an explicit problem with either approach. I would do both

  • Compare the conversion rate across all products
  • Compare the conversion rate on a product by product basis

That will give you an idea of aggregate and specific performance, which is useful. You may also want to do a third:

  • compare the conversion rate across meaningful customer subsegments

Maybe your model is better at converting certain types of users, or users in certain areas? The more granular you get, the more you need to worry about confirmation bias (correlation does not imply causation, and if you go digging around you will find compelling trends based on chance), but these tests will allow you to know where you should further improve your approach, and may inspire additional meaningful study.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.