Retention by software build

The standard analysis is to measure Day 1, 3, 7, 14, & 30 retention by what proportion of new users for a build used the app, indexed from Day 0, the day they installed the app. So, the measure for Day d retention is given by:

\[ \frac{\text{number of new users for build on Day d}} {\text{number of users on Day 0 for build}} \]

Notice how the confidence intervals are tighter over minor changes, compared to the smaller-length release patches. Also notice the variability in randomly simulated data, where the underlying assumption is the same retention across builds. So, this industry-standard KPI is quite misleading, especially when stakeholders are talking a difference of 1-2%, even when the confidence intervals don’t overlap.