Chances are if you have ever interviewed for a Product Manager role at a game company, FAANG or any b2c tech company you have been asked some form of this question: “You walk into the office and metric X is down by Y percent. What do you do?”. After you get the obligatory “blame it on the engineers” joke out of the way, you’ll need to dazzle your interviewer with what is known in consulting as root cause analysis. As games as a service continue to dominate the industry, root cause analysis will become an increasingly important skill to have in any live ops role PM role.
Here’s the scenario: you come into the office and your daily revenue from yesterday is down 20% week-over-week (WoW, no not that WoW). What do you do? Let’s start off by looking at our tools available: industry trends, metrics, recent events and properties of your game. Metrics are everything from your tried and true classics like DAU, D1-30, Session Length, etc. all the way to game specific metrics like Spins per User per Day in a slots game or Fights per User per Day in a fighting game. Recent events include version releases and live ops experiments running or events outside of your game like competitor releases. Finally game properties are things like your in-game economy flow or PvP seasons.
Though everyone has their own flavor of root cause analysis, here is Eric McConnell’s current flowchart:
- Start with the top line metric that you are seeing the issue with (ie. Revenue, DAU, Retention etc) and create a metric funnel to identify all the metrics that could be contributing to or causing the issue
- Follow the metric branches of your funnel tree into your problematic metric to find the root cause
- Look at the events over the time period you measured the metric delta and see which events or features could most likely be impacting the lower level metric
- Look at the properties of your game such as feature design, economy imbalances, new feature/content release
1. Build a Metric Funnel
Before we identify the cause, we need to be directed where to put our investigative attention. This is where metrics shine. Although metrics are never the cause, they are the indicators that help you see the entire story. In this instance, our top-line revenue for the week is down.
For any given metric, there is a pyramid shaped funnel of metrics that are used in calculating that metric. DAU (Daily Active Users) can be calculated by adding New Users and Returning Users for that day together. New Users can be calculated by adding together the Installs and Activations from various sources: organics, Facebook, Google, etc.
There is a bit of art to building metric funnels. For instance, in the DAU example above, DAU could also be calculated by adding Payers and Non-Payers for a given day if your funnel was Revenue related. Another point of dispute is how deep should a funnel go. The answer is how deep you need it to. Remember this is a tool for you, the Product Manager, to use. Build your metric funnel to your specific needs.
Back to the Revenue problem at hand, here is an example funnel:
Daily Revenue is broken down to DAU and ARPDAU (Average Revenue per DAU). ARPDAU is further broken down to Revenue per Transaction (size of the average transaction) and Transactions per User (how many transactions is the average user making). We could easily add another layer of reporting to this, breaking down each metric once more, but I believe this is enough to give us an idea of what’s going on.
2. Follow the Problematic Metric Down Funnel
Now that we have our metric funnel, it’s time to plug in numbers. For this situation, Daily Revenue down 20%, we want to compare the metrics for this day to their same metrics a week ago. You’ll find that each day of the week has a different pattern to its performance, making it more effective to compare Monday to other Mondays rather than Monday to Sunday.
Back to our metric funnel, we are comparing yesterday’s metrics to the same metrics a week ago and record the delta, or percentage difference, between the two. Here I color coded the funnel at an 8% threshold, meaning anything +/- 8% is colored red or green respectively.
Daily Revenue’s trouble can be traced down to Repeat Buyers. Daily Revenue is down 20%, Payer DAU is down 33% and finally Repeat Buyer is down 28%. You can see how one, more specific metric, can cause a chain reaction all the way up to Daily Revenue. On the other side, Revenue per Transaction is up 12% WoW which may or may not be related to Repeat Buyers.
3. Events over the time period measured
Since our Daily Revenue is down, we should consider all the events over the last day that could impact Repeat Buyers.
Did your team release a code update to the game?
Did you push experiment data the day before yesterday?
Are there sales running?
Is there hangover from running a previous sale?
Also, look at what was going on in your game a week before yesterday. Repeat Buyers could be down or last week’s Repeat Buyers could have performed abnormally well.
Did you run a sale last week?
Was there a major in-game event a week from yesterday?
Don’t forget looking at ecosystem or competitor related events.
Did a competitor just release a similar game?
Is a competitor running an aggressive User Acquisition campaign?
Lastly, take a look at general trends in the news like major events, holidays and even geopolitics.
Was yesterday the Super Bowl?
Was yesterday a major holiday, election or world event in a key market?
Is there a natural or manmade disaster in a key market (hurricane, tsunami, war, terrorism, protests)?
4. Look at the properties of your game
The last vector to consider are the properties of your game. They can be your code, economy or overall game design.
The easiest place to start with is errors in related to currency or purchasing flows. You, or your related tech lead, can query the Splunk-equivalent for error logs related to IAPs, awarding or spending currencies and general flows to purchasing.
After error logs, move onto your economy in general. In this scenario it may be most important to look at Payer currency wallets. Look particularly at Active Payers over the last week and how much soft or premium currency they have compared to other time periods or even other cohorts.
Next investigate what your Payer engagement looks like. Are Payer session lengths, event participation and overall engagement with the game the same or declining? This is where a robust engagement funnel would be useful.
Finally look at your overall game design. Are there inflection points in your game design that are just surfacing now? Are the lapsed Payers running out of content?
As you can see, determining the cause of a major problem like Daily Revenue declining can be a daunting task with many interwoven variables in play. But if you utilize a Root Cause Analysis framework, this moment of panic can turn into a call for action and the long-term impact of hiccups can be minimized. Root Cause Analysis will be a key tool to leading your game to live-ops success.