Don't Trust Your Gut
Varianta B pare mai bună cu 2%. Dar e real sau noise? Statistical analysis îți spune când poți lua decizii cu încredere.
Current A/B Test
Test: New Greeting Script
Running - 68% completeControl (A)
“Bună ziua! Cu ce vă pot ajuta?”
Variant (B)
“Bună ziua! Sunt Maria, asistentul dumneavoastră. Cum vă pot fi de folos?”
Statistical Analysis
Key Metrics
Confidence Interval
95% CI for difference:
+2.8% to +18.8%
Sample Size Calculator
Parameters
Required Sample Size
per variant
Total: 3,680 conversations
Decision Framework
Deploy Winner
p-value < 0.05 AND confidence interval doesn't include 0 AND practical significance
Continue Testing
0.05 < p-value < 0.15 OR sample size not reached
No Significant Difference
p-value > 0.15 AND sufficient sample size - keep control
Stop - Variant Worse
Variant significantly underperforms control - revert immediately
Common Pitfalls
Peeking Problem
Verifici rezultatele zilnic și oprești când vezi diferență. Inflates false positive rate la 30%+.
✓ Fix: Pre-define stopping rules
Multiple Comparisons
Testezi 20 metrici simultan - una va fi “semnificativă” by chance.
✓ Fix: Bonferroni correction
Simpson's Paradox
Overall B wins, but A wins in every segment. Caused by unequal segment distribution.
✓ Fix: Segment-level analysis
Survivorship Bias
Analizezi doar conversațiile complete, ignori abandonurile.
✓ Fix: Intent-to-treat analysis