In the fast-paced world of product management, understanding retention is pivotal—but are we interpreting our retention metrics correctly? In this article, we’ll uncover a hidden fallacy that could be skewing your perception of user retention and highlight actionable strategies to avoid it.
The Problem: Misinterpreted Retention Metrics
Imagine you’re the PM at Bookface, a social network for book lovers. Your CPO is laser-focused on retention and has tasked you with redesigning the onboarding process. Pointing to a chart showing 80% retention by week three, she claims there’s no room for improvement. But is this chart revealing the whole truth?
Retention metrics are among the most fundamental tools for PMs, offering insights into how long users stick around. Most Analytics platforms define retention simply: the percentage of users who perform a specific action (Event A) and return to perform another (Event B) within a set timeframe.
This standard definition seems intuitive, but there’s a catch. What most PMs overlook is that the default "retention" chart might not be measuring new user retention at all—it could be blending in returning users, giving an inflated sense of success.
Retention vs. Churn: What Are We Really Measuring?
Traditionally, retention metrics have been used to track how many users return over time, which is the flip side of churn. However, recent improvements in Analytics tools, like Mixpanel and Amplitude, have introduced advanced options for defining and filtering retention.
These options let PMs:
- Define specific and different events as "initial" and "returns."
- Change retention criteria (e.g., "On or After" vs. "On").
- Define the ‘return’ time segments/buckets as days, weeks or months.
While these enhancements add flexibility, they also introduce complexity. Without careful configuration, the data can mix apples and oranges—new users with returning users. This creates a misleading view of retention.
Why the CPO’s 80% Retention Metric is Misleading
Looking back at the CPO’s claim of 80% week-three retention for new users:
- On the surface, it suggests only 20% of newly acquired users churn within their first three weeks.
- In reality, this metric blends in returning users, who naturally have a lower churn rate.
The Fix: Segment New Users from Returning Users
To uncover the true retention for true new users:
- Filter users by their “first time ever” property (Mixpanel).
- Adjust the retention chart to focus exclusively on users new to the platform during the timeframe.
This updated chart, as can be seen below, paints a very different picture: true new users have a much lower retention rate than returning users.
https://mixpanel.com/s/1k6HEy - We’ve limited users to only those who have done their first time ever property = True new users.
The Role of Returning Users
If we look exclusively at the retention of returning or ‘old’ users, we naturally notice that their retention is much higher:
https://mixpanel.com/s/kyM6Q - We’ve isolated returning users as those who were ‘first seen’ by Mixpanel at least 1 week before our selected time frame. Thus if they fire an event it must be at least 1 week after their 1st visit, which would define it as a returning visit.
For returning users, retention naturally appears higher. Why? These users have already demonstrated their engagement with the product, making them less likely to churn in short timeframes.
The Workaround
Older and less advanced Analytics platforms, such as GA, still have predefined ‘new’ and ‘returning’ user segments. But as those platforms are not commonly used today by PMs (for a very good reason), the manual workaround to compare new vs. returning users would be:
- New users - using the ‘first time ever’ property
- Returning users - by creating a custom event property called ‘Days since 1st seen’, which would attribute to ALL events the number of days between the event taking place and the user first being seen. This property retroactively populates historical data, ensuring old events also reflect this value. As long as the event property value is greater than 0, it indicates the event occurred after the user's first visit (not on the same day).
Here’s how to define the custom property:
Why This Matters for PMs
Understanding the distinction between new and returning users isn’t just a technical exercise—it directly impacts your product decisions. Blurring these segments can lead to:
- Overestimating the success of onboarding initiatives.
- Misallocating resources to features that appeal only to returning users.
- Failing to address critical issues in user acquisition and activation.
Accurate segmentation ensures you’re targeting the right users with the right strategies, driving meaningful retention improvements.
Closing Thoughts
Retention metrics are powerful, but they can be misleading without proper segmentation. By differentiating new and returning users, PMs can unlock deeper insights, make data-driven decisions, and deliver greater value to their products.
At The Product Alliance, we specialize in helping PMs decode complex Analytics challenges like these. Whether you’re working to refine retention strategies or uncover hidden insights, our team is here to guide you.
Call to Action
Did this resonate with you? Have you encountered similar challenges with retention metrics? Comment below or share your thoughts—we’d love to hear from you!
Comment below or reach out to us.
Subscribe to receive these articles as soon as they’re published.
About the Author
Yoav Yechiam is a globally recognized expert in Behavioral Analytics. He serves as the managing partner at The Product Alliance, a strategic consulting firm. They recently launched the GPT-powered 🤖 Analytics Advisor Bot, to help Product Managers select the right Analytics tools. Try it now for free and simplify your Analytics journey.