How to set up metrics for Product Development?

I have seen many products without clear measures of success. When I asked, “how do you know, that you improved the product?” I got not crystal clear answer, just some “Well…”. I was surprised how much does this happen, considering that Seneca said about 2 thousand years ago, that “If one does not know to which port one is sailing, no wind is favorable.” So, let’s see, how can we ensure that we know what is our port and let’s construct a map to it to have favorable winds.

Step 1: Know your company’s WHY, know what is success for you?

As a first step, you should know, what does success means for your company. As stakeholders can have different definition of success, try to find out your company’s overall WHY first. If you are not familiar with the concept, check here. Once you have your overall WHY, you should talk to important stakeholders and customers to find out what they define as success.

So, let’s see, what can be a why and what can be stakeholder WHYs. Let’s do it for an egrocery company. A WHY can be “making online egrocery shopping simple, great and affordable, so that users can easily switch to it from brick and mortar shops to online”. (You might want to work a bit more on this, but will do as an example. :) )

If you interview different stakeholders, they might add things like:

• General business metrics like: market share, monthly active customers, revenue, profit, average order value etc.
• Customers: easy shopping, good prices, quick delivery, great sortiment etc.
• Commercial team: margin on products, churn rate etc.
• Marketing team: new users, campaign conversion etc.
• Customer support: number of user complaints, delayed deliveries etc.
• Suppliers: sales through your business etc.

We could continue this list endlessly. What you need to be sure is having the most important stakeholders asked and check whether they goals are navigating to the company’s why. E.g. while margin on products is essential, it might be contradictory to your “affordable” goal. Still, making a business sustainable is important, but it should be done hopefully without hurting your big goal.

Step 2: Defining your metrics

At this point you know what is success for the company, what is success for different stakeholders and hopefully you also have an idea, what they would expect from the service you are working on. Based on this, you should define what does success mean for you.

Let’s continue with the egrocery example and let’s consider search as a service. Given the company’s WHY, we can set up some success measures for ourselves. Just to refresh, this was our company’s WHY: “making online egrocery shopping simple, great and affordable, so that users can easily switch to it from brick and mortar shops to online”. If we derive from this, what we should do, it should be making an easy-to-use search providing a great customer experience. What metrics can be set up for this?

For me the first metric to be used for this would be Conversion Rate. Conversion rate is the % of successful searches (e.g. adding a product to cart). If your search is good, the customers find easily whatever they need. Obviously, this is not only on you (e.g. the sortiment can be too narrow or expensive), but you can see how your developments change it. I think that in search normally this should be the North Star (primary) metric — the first to check in vast majority of your developments. Regardless of the field you are working on, you should have a few metrics like this, which can show the progress you made in a snapshot.

Other metrics to consider can be:

• No results searches: Usually having too many no results searches is frustrating, so fixing this should be an important part of the great experience your company wants to achieve.
• Revenues from search / Average order value: Even if it is not in your company’s WHY, you might want to have a sustainable business, so contributing to the company’s revenues can be important. You might want to make this cleaned from seasonal effects and general company trends.
• Median rank in search of the item purchased: The purchased product is hopefully on the top of search results.
• Number of searches: You might need to know the demand for your service.

The list could be continued endlessly. What I would suggest is picking a few (1–3) North Star metrics and a few secondary metrics to avoid being lost in details. Also, from time to time revisit your metrics and adjust them to the new business situation. E.g. if your company is near bankruptcy, increasing revenues might be more important than decreasing no results searches.

Step 3: Setting up logging and metrics generation

Once you have your metrics, you should start logging the data supporting them. Talk to your engineers and data scientists and see what is available and what will you need to develop to get what you need. Pretty often logging has a significant cost, so after seeing the reality, you might want to adjust your initial ideal metrics to match reality more. You can also decide to reduce the initial scope and develop some metrics later. One example for this is measuring search in one of my previous workplaces. We realized that measuring conversion should be harder than we expected, so as a temporary solution, we added a simple “Did you find what you were looking for?” question in the search results page and used the share of Yes as a temporary metric. Imperfect, but might do the job.

After this, you should have an idea how long does it take to set up your logging and you can add it to your development plans. As developing logging and metrics does not generate immediate value, you might need to convince your managers, that it is needed for long-time success.

Once you have your logging online, you will need to develop metrics calculation. When you are doing this, you will realize that some metrics are calculated on the fly, while some of them are delayed as they need post-processing. I would suggest to have them both as in many cases post-processed data can be more punctual, but prompt data helps you to see immediately whether the changes you made are working fine or whether you have a bug. Just to note: metrics are not only great for seeing whether you are navigating towards success, but also in showing whether you have some issues within your service. E.g. if your conversion drops suddenly, you should check immediately what happened.

Step 4: Dashboard and communication

Yay, you have your metrics online. The next thing is utlizing the metrics. You will need to build a place where you can check it frequently and see the changes. A good dashboard should do the trick, but it is also great to have some interface, where you can do additional calculations with the data. If you have Slack, it helps a lot if you do automated metrics posting to a dedicated channel every day, so that you can see what happened yesterday. This saves you the time to go to your dashboard and also keeps your team updated.

Your data does not only help you, but also to your managers checking how you are performing. So, set up regular reports to them, so that they can evaluate your team’s progress. It can also help to update a broader audience (e.g. your stakeholders) about your successes/failures, so that your team is not a blackbox to them, but a transparent group.

Conclusion and Next Steps

In conclusion, setting up metrics for product development is akin to charting a course for success. By anchoring your metrics in the overarching ‘WHY’ of your company and aligning them with stakeholder goals, you create a roadmap that navigates towards your desired destination. The journey involves careful consideration of diverse metrics, with a focus on a select few North Star metrics that encapsulate the essence of your product’s success.

As you embark on this metrics-driven voyage, it’s imperative to recognize the pivotal role of effective data logging. Logging not only facilitates metric calculations but also serves as a historical compass, guiding you through the evolution of your product’s performance.

Once your metrics and logging mechanisms are in place, the journey doesn’t end — it transforms into a continuous cycle of improvement. Regularly revisit and reassess your metrics in the context of your evolving business landscape. Metrics are not static; they should dynamically reflect the changing priorities and challenges your company faces.

In the next steps of your metrics implementation, translate the insights gained into action. Build a robust dashboard for easy visualization, utilize interfaces for deeper data exploration, and consider automated reporting tools for efficient communication within your team. Share your successes and failures transparently with stakeholders, turning your team from a black box into a transparent force propelling towards success.

Remember, the true power of metrics lies not just in measurement but in the actionable insights they provide. By integrating metrics seamlessly into your product development process, you harness the winds of data to steer your ship toward the port of success, as Seneca wisely noted two thousand years ago.

So, set sail with purpose, armed with metrics as your guiding stars, and let the winds of data propel your product development journey forward.