Is that drop a real break, or just the season?
The Torque Lift team ·
Your weekly active users fell 12%. Before anyone opens the code, work out whether something broke or whether this is the time of year. Here are five checks, in order. Each takes minutes, and the first two settle most cases.
1. Compare with the same weeks last year
Seasonality repeats. If your product dipped in the same week last year, by a similar amount, you are probably looking at the calendar. Say a note-taking app with 10,000 weekly users lost 9.0% in the last week of December last year and 12% this year. That leaves a 3-point gap to explain, not a 12-point one. (Made-up numbers.)
You need a full year of history for this. With less, skip to check 2.
2. Ask who it hit
A season hits nearly everyone. A break hits a slice. Split the drop by device, operating system, browser, country, language and traffic source, and look for one slice that fell much more than the rest.
If every slice fell by about the same share, think season, or something that touched everyone, like an outage. If one slice fell by half and the others barely moved, something changed for that slice: a release, a tracking change, a campaign that ended.
3. Look at the shape in time
A break usually has a date. The line steps down on the day of a release and stays down. A season usually curves: it eases down over days and eases back.
Put your deploys and campaign starts on the same chart. A step that lines up with a deploy is the first thing to read the code for.
4. Check whether new users or existing users changed
Existing users leaving is a retention problem. New users arriving and not coming back is an activation problem. They have different causes, so split the drop into who was new, who stayed, who came back after a gap and who left.
A cohort table shows this well: one row for each week of new users, one column for each week after. If the rows from one week onward are paler than the rows before them, something changed for everyone who arrived then.
5. Check the calendar of the people you serve
Holidays differ by country. If the drop is in one country's users and a public holiday fell in that week, that is a likely cause. School terms and tax deadlines work the same way for some products.
When you still can't tell
If the checks disagree, treat the break as unproven and the season as unproven. The way to find out is an experiment. Fix the metric, the smallest change that counts and the end date first, change one thing, and read the result on the date you set.
How this looks in Torque Lift
Each week, Torque Lift does the splitting in checks 2 and 4: it counts new, retained, resurrected and churned users, splits each by segment, and lists the gaps in your readout. Torque Lift doesn't know about your holidays yet, so check 5 is still yours. A gap is a place to look, not a cause, and an experiment is how you find out.