How to create A/B test segements for highly variable data

I have a data in which there is a high degree of variability.

My Objective is to do an AB test to check the behavior change due to new changes.

  1. All samples has shown historically high and low performances. This means if I take any 2 cohorts randomly, they show vast historical comparison difference Following is the example for weekly comparison. Same behavior holds true for monthly and daily too.

W1: -10.04%

W2: 3.9%

W3: -4.2%

W4: -3.7%

W5: 5.4%

W6: -2.6%

W8: 0.8%

W9: -2.3%

W10: 1.92% ... .. .

  1. No matter I tried to segment users by their aggregate behavior and see the difference, its still this level of variability is observed.

  2. I'm looking for clearly difference to be consistent no matter in which ever direction. Either its positive or negative. And, with small range of magnitude.

Can someone please suggest the alternative methods to minimize this difference?

Topic ab-test

Category Data Science

1
answered at

Your ability to detect a difference depends on the relative standard deviation of your data. Sometimes the noise is due to seasonal effects, which an AB test does an excellent job in correcting. However if the metric itself is intrinsically high variance, you will need a large sample size, or adjust your significance and power parameters.

Run a check of your standard deviation and do a power calculation to evaluate your situation.

Keep in mind that there are some statistical distributions which have variances approaching infinite.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.