I must admit that the idea of embracing ambiguity didn’t come naturally for me at first: after all, as a computer science graduate I was trained to think binary — zero or one, yes or no, right or wrong. But when evaluating products as a PM, there is usually a spectrum of shades of gray between black and white.
Here is a story where I learned to embrace that ambiguity.
As you might know, I currently service an internal documentation site at Microsoft. About a year ago, we launched an achievement system on the site, so our users can unlock various achievements when they write documentation on our platform. We set the goal of this achievement system to be: to motivate users to use our site more and invest more long-term effort in documenting their work. Now, after about a year, I was tasked to revisit the value of this achievement system and investigate whether it’s worth designing more achievements for users to unlock.
To find out the value of the achievement system, I dived immediately into data. There were two types of data I collected and analyzed: quantitative data (site telemetry data reflecting internal users’ activities on the site) and qualitative data (feedback collected in user interviews). Here is what I learned from both types of data:
- Site telemetry was showing that after the achievements were first published, documentation contribution saw an uptick, across the board in all achievement-earning user activities. However, the uptick disappeared quickly, and site documentation contribution trended back to its normal level and maintained there.
- In interviews, users were telling me that they don’t write documentation just for the sake of unlocking some user achievements. For the documentation veterans, they spent time in documentation simply because it was part of their job; for those who were new to our site or even new to Microsoft, unlocking a few achievements might had been fun at first, but they lost interest once they learned what achievements were available.
So, both the quantitative data and the qualitative data seemed to consistently suggest that although perceived as novel and fun, the achievement system’s positive impact on documentation contribution couldn’t sustain in the long-term. It seemed clear to me that we should divest from it, and not bother coming up with new achievements, since the achievement system was not meeting its goal. That was my thought anyway, until I showed these findings to my manager/mentor, and she asked me this question:
“If the achievement system is not good for bringing up long-term contribution, what is it good for?”
That was the first time I thought about that question: what else is it good for? I was so set to prove or disprove the achievement system’s value in one area, that I completely overlooked its other possible values. So we looked at the data together, and the answer was sitting right there the whole time: the achievement system was being used as a guide book to discover all the cool things you can do on the site. That was why there was a short-lived uptick of usage; those “things you can do” were what the new users were excited to try out when they explored the achievement system.
Suddenly, I saw the shades of gray between black and white. The achievement system was not the “right” solution, but it also wasn’t the “wrong” one. So, I spent time designing new achievements that prompt users to explore more site features, and re-purposed the achievement system to be a step-by-step, level-by-level tutorial on how to use the site. I’ve received very positive feedback about the repurposed achievement system since then, especially from new users.
That was my story of how embracing ambiguity led to novel customer value. Seems like ambiguity can indeed be a PM’s friend!