Skip to content

Ellery Yang

I write about products and the PM role.

Menu
  • Home
  • About me
Menu

Embracing ambiguity as a PM

Posted on March 17, 2019April 25, 2020 by Ellery

I must admit that the idea of embracing ambiguity didn’t come naturally for me at first: after all, as a computer science graduate I was trained to think binary — zero or one, yes or no, right or wrong. But when evaluating products as a PM, there is usually a spectrum of shades of gray between black and white.

Here is a story where I learned to embrace that ambiguity.

As you might know, I currently service an internal documentation site at Microsoft. About a year ago, we launched an achievement system on the site, so our users can unlock various achievements when they write documentation on our platform. We set the goal of this achievement system to be: to motivate users to use our site more and invest more long-term effort in documenting their work. Now, after about a year, I was tasked to revisit the value of this achievement system and investigate whether it’s worth designing more achievements for users to unlock.

To find out the value of the achievement system, I dived immediately into data. There were two types of data I collected and analyzed: quantitative data (site telemetry data reflecting internal users’ activities on the site) and qualitative data (feedback collected in user interviews). Here is what I learned from both types of data:

  1. Site telemetry was showing that after the achievements were first published, documentation contribution saw an uptick, across the board in all achievement-earning user activities. However, the uptick disappeared quickly, and site documentation contribution trended back to its normal level and maintained there.
  2. In interviews, users were telling me that they don’t write documentation just for the sake of unlocking some user achievements. For the documentation veterans, they spent time in documentation simply because it was part of their job; for those who were new to our site or even new to Microsoft, unlocking a few achievements might had been fun at first, but they lost interest once they learned what achievements were available.

So, both the quantitative data and the qualitative data seemed to consistently suggest that although perceived as novel and fun, the achievement system’s positive impact on documentation contribution couldn’t sustain in the long-term. It seemed clear to me that we should divest from it, and not bother coming up with new achievements, since the achievement system was not meeting its goal. That was my thought anyway, until I showed these findings to my manager/mentor, and she asked me this question:

“If the achievement system is not good for bringing up long-term contribution, what is it good for?”

That was the first time I thought about that question: what else is it good for? I was so set to prove or disprove the achievement system’s value in one area, that I completely overlooked its other possible values. So we looked at the data together, and the answer was sitting right there the whole time: the achievement system was being used as a guide book to discover all the cool things you can do on the site. That was why there was a short-lived uptick of usage; those “things you can do” were what the new users were excited to try out when they explored the achievement system.

Suddenly, I saw the shades of gray between black and white. The achievement system was not the “right” solution, but it also wasn’t the “wrong” one. So, I spent time designing new achievements that prompt users to explore more site features, and re-purposed the achievement system to be a step-by-step, level-by-level tutorial on how to use the site. I’ve received very positive feedback about the repurposed achievement system since then, especially from new users.

That was my story of how embracing ambiguity led to novel customer value. Seems like ambiguity can indeed be a PM’s friend!

PM Blog (2017 - present)

  • January 2023
  • September 2022
  • July 2022
  • February 2022
  • September 2021
  • August 2021
  • April 2021
  • January 2021
  • December 2020
  • November 2020
  • October 2020
  • September 2020
  • August 2020
  • July 2020
  • June 2020
  • May 2020
  • April 2020
  • February 2020
  • January 2020
  • December 2019
  • October 2019
  • July 2019
  • June 2019
  • April 2019
  • March 2019
  • December 2018
  • October 2018
  • May 2018
  • March 2018
  • December 2017

A Student's Blog (2015 - 2017)

  • July 2017
  • August 2016
  • July 2016
  • April 2016
  • March 2016
  • February 2016
  • December 2015
  • October 2015
  • September 2015
  • August 2015
  • July 2015
© 2023 Ellery Yang | Powered by Minimalist Blog WordPress Theme