Product Management Skills - Analytics and Instrumentation
Product analytics and instrumentation should be a product manager's closest and best friend.
Product analytics. It’s one of the most valuable quantitative tools for making effective decisions because it tells you exactly what your user-base is doing with your product, if properly instrumented.
Yet, I’ve worked with many product managers who have underutilized this value, unable or unwilling to perform this analysis or even consider it a factor in their requirements. Some have legitimate reasons - perhaps their company doesn’t allow it. Others, less so - some view the work as beneath them, or others are simply intimidated.
Yes - intimidated. Because it seems technical and it seems heavy on data, product managers can potentially shy away from digging deeply into learning how to wield analytics. Instead, they toss any analysis needs over to a data science organization, greatly extending their feedback-decision-action loop.
This behavior prevents them from being able to deeply understand their users’ behavior. If all you’re getting is a summary of an analysis, then how do you quickly get answers to ad-hoc questions? How do you truly understand what your users are doing at scale?
A solid understanding of product analytics, instrumentation, and analysis is a must-have for Product Managers to be maximally effective.
Product Managers must know:
The mental models of product analytics
How to identify what to instrument
How to perform basic analysis themselves (funnels, trends, tracing)
How to effectively incorporate instrumentation needs into their requirements
Regardless of the vendor (eg. Mixpanel, Amplitude, Fullstory, home-rolled, etc.), Product Managers must know the fundamentals.
What should you instrument?
It depends, but below is what I consider table-stakes questions for any product manager to be able to answer.
Who is the user performing the action?
What action is the user taking?
What context is the user acting in?
These three questions can help you trace user behavior, and construct trends and funnels for analysis.
Who is the user performing the action?
To identify this, we can break this down into three different categories:
User identity
User demographics
User segmentation and categorization
User identity
User identity are the unique properties that allow you to uniquely identify the user so you can trace a particular user’s behavior as they use the product, and cross-analyze across other datasets if you so desire.
The primary one you should, at minimum, have is:
A record identifier tied to the specific user record in your system (eg. a user_id or account_id)
An internal identifier to identify a specific user across multiple accounts, if needed, such as a randomly generated UUID associated with multiple accounts representing a single person.
Other identifiers can be used as well - just note there may be pitfalls:
Email address is useful to easily find an account, but can change or not be present, especially if you have phone number login or other capabilities.
Phone numbers are useful, but remember that phone numbers can change hands over time.
IP address is not uniquely identifying enough - IP addresses can be shared, can change frequently, and can even represent thousands of other people (eg. a public hotspot).
Regardless of what identifier you use, associate the User Record Identifier and the Internal Identity identifier to the User’s profile for tracking. It’s sometimes helpful to also associate these details with the event as well.
Why a record identifier AND another internal identifier?
Some products require the creation of multiple accounts, which means a single person may have 2, 3, or even 4 accounts. Think multi-tenant systems, or corporate payroll systems segmented by company. Some security policies also mandate the usage of more than one account (eg. separation of admin accounts and user accounts).
If you only have a record identifier, you may get confusing or skewed analysis results.
To correct for this, you’ll need some sort of universal user identity your product generates that is associated with other accounts but separate from the account identity.
Even if you don’t think that’s the case, just remember: users do strange things. Product analytics often lose track of users due to cache, cookie settings, and multi-device behavior.
User demographics
Demographics might include age, income, country of origin, etc. and can often be natural segmentation points where user behavior starts to differ or key information useful for market expansion. Teens are likely to engage completely differently with your product than the elderly. Perhaps you find yourself getting visitors from another country where English isn’t the primary language, and you want to inform whether it’s useful to embark on an internationalization effort.
It’s harder to do that without understanding the demographics of your users.
Just be careful not to over-collect information. If you’re running a financial applicatio, then asking for income might be contextually appropriate. However, users might find it weird if you have a photo-sharing product and ask that same question.
Be judicious and make sure it’s actually relevant and useful information to collect before collecting it.
note: some analytics products auto-collect and interpret information like IP address an turn it into country. It’s imprecise, but accurate enough.
User segmentation and categorization
You’ll probably have some unique characteristics particular to your industry, offerings, and market that you may want to segment on.
Maybe you have some vetted personas that you want to tie all of the users to. Perhaps your company has separated its customer base by a metal level such as bronze, silver, and gold. Maybe you want to track how big of an organization the user is in.
Whatever you want to use to categorize and segment users by, you can track so that you can look at behaviors across you various categories.
Note: Categorization can be anything that’s relevant to your company, product, and industry. In the past, I’ve categorized users by the kind of sport they played, which was strongly correlated with success outcomes in the product I was working on.
What action is the user taking?
Once you have identity, you should also be tracking what the user is actually doing.
There are two kinds of “doing” events:
Interaction events
Business events
Interaction events
Interaction events are things that the user has done to interact with your product or system.
They are typically abstract, generic, and mechanical in nature, such as:
Viewing a page or element
Clicking/tapping/holding a button, link, heading
Hovering on / off an element
You’ll often want to answer questions like “How many times did the user click the thumbs up button?”. Tracking interaction events at a generic, automated level lets you answer those quickly.
These are also useful for constructing interaction funnels - you can quickly chain together a set of views and clicks to see where users are dropping off a funnel.
However, because users can interact with thousands, tens of thousands, or even hundreds of thousands of elements in a single product, you want to track a second, more specific set of events to make you analysis even more useful / faster: business events.
Business events
The second set of events to collect are business events. These are events that are relevant to your business.
They aren’t specifically measuring a user interaction (such as a click or a view) but rather some sort of business-relevant outcome or action which may or may not be triggered by the user via an interaction.
Generic product events might include things like:
Signed up
Logged in
Deleted account
Changed password
Requested password reset
Reset password
Changed email
Subscribed to newsletter
Unsubscribed from newsletter
If you had an eCommerce site, some business events might be:
Item added to cart
Item removed from cart
Item purchased
Item page viewed
Cart viewed
Checkout started
Checkout completed
Checkout failed
Upsell displayed
Return requested
Return denied
Examples for a business dashboard might be:
Chart viewed
Export downloaded
Export requested
Search conducted
Record approved
Record denied
Team member added
Billing info updated
Billing type changed
You get the picture. The important part is to identify what events might be important to you or the business to track - which events lead to outcomes?
What context is the user taking action in?
Finally, you’ll likely want to be able to segment and analyze behavior depending on the context of the user’s actions.
Some products offer a multi-tenant whitelabel capability to customers. Events might occur particular to a specific customer’s environment or whitelabel, and you may want to track these events separately from the global perspective.
Perhaps your application has multiple entry points, and you’ve identified that users behave different depending on which upstream they came from. You can track this as well to establish the context of the user’s interaction with your product.
Useful contextual items include:
The environment or account being acted on (eg. a specific customer environment)
Where the user came from (eg. what page) - also known as a referrer
The device of the user (eg. mobile, tablet, desktop, laptop)
The browser of the user (eg. Chrome, Safari)
Context is, well, contextual - so be sure to think through how you might want to segment and analyze your data.
Add properties to your events so you can analyze them
You’ll always want to make sure that each of your events has a set of properties that describes that you can pivot on to answer your questions.
For example, the following events might have as properties:
page viewed event
page name
checkout completed event
items in cart
total price
payment type
prices in cart
coupon code used
search conducted event
search term used
filters applied
# results returned
search time taken
If we used the above as an example, we could answer questions like:
Which coupon codes resulted in an increase in total price for a checkout vs. a decrease in total price of a checkout?
Checkout completed, summing up total price, broken out by coupon code compared to
Checkout completed, summing up total price filtered by those without a coupon code as a baseline
What are users attempting to purchase or search for that we might want to find sellers to stock?
Search conducted, filtered by those returning no results, broken out by search term used
Is there a correlation between payment method and the total amount of a checkout?
Checkout completed, summing total price, broken out by payment type
Did users who viewed the new “Halloween special” marketing page buy more items than users that didn’t?
Checkout completed, broken out whether there was a Page view for “Halloween special” or not, summing up total price
The questions are obviously limitless. Having the right properties and events will help you answer them. Whatever you might need to be able to answer a question you want to ask is fair game.
Nothing is worse than releasing a product and then wanting to answer a question, only to realize you can’t answer it because you didn’t define the instrumentation for it so you aren’t capturing the events or properties you need.
Note: data never tells you WHY your users did something, only WHAT they did. To get the “why?”, you need to talk with your users. For example - a user may have bought more items on the halloween special because it was a great upsell, or it might be because the halloween special had a defect that added warranty card products to the cart, or maybe it was because the halloween page simply showed more of a catalog than your other landing page and had easier add-to-cart UX. You’ll never know the actual reason through data alone.
Putting events together and analyzing them
With interaction events and business events, you can now start performing analysis of the events - building visualizations of funnels, trends, and behavior across the range of products.
Analyzing a funnel
Funnels are used to identify points in a flow where the conversion rate can be improved.
Funnels are constructed with a sequence of events you expect the user to go through in order, and then you can use the number of people who successfully reached the next step within a set amount of time as a “conversion”.
Analytics products typically have funnel feature built in, but if not, you can construct your own by counting the number of events or users that went through that flow fully, step by step.
For example, if you want to construct an end-to-end funnel for a checkout flow, you could look at these events in sequence:
Item viewed → Item added to cart → Checkout started → Checkout completed
Each column above shows the number of sessions that reached that step. This means you can identify areas with higher than expected drop-off to optimize.
The above example shows a total end-to-end funnel conversion rate of only 23% for events that occurred in a month. That is, 77% of people who view an item do not make it to completing a checkout (“Item viewed” → “Checkout completed”).
Looking more closely, a large portion of drop-off is in the step from “Checkout started” → “Checkout completed”. This area is ripe for improvement: starting a checkout is a high-intent signal, so optimizing this process is likely to significantly improve conversion rates.
Keep reading with a 7-day free trial
Subscribe to Joseph Gefroh to keep reading this post and get 7 days of free access to the full post archives.