I am going to perform A/B tests for ML models. However I am not sure how long should I run it online in order to see significant differnce. What would be the right time frame ? and what will be the reason behind the time frame ? The A/B test will run againts the None ML systems. Usally we run for none ML features for 2 weeks max. Thank you
Topic ab-test machine-learning
Category Data Science
First it's not a matter of duration, it's a matter of number of data points that can be collected: if there are only a few users every day, it's going to take much longer to collect enough data points than if there are millions.
Now how many data points do you need? There's no simple answer since it depends what exactly you are going to test and how big the difference turns out to be between the two cases.
A good workaround is to start with some hypothetical scenarios: imagine for instance 100, 1000 or 10000 users and an outcome with a large, a medium or a small difference. Run the test in all these different scenarios and check when a significant difference is obtained. Based on this, choose a target number of users which would be sufficient to obtain a clear result in most of the scenarios.
Finally the duration can be approximated based on the average number of daily users.
Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.