The observer problem and customer experience

Will an increasing reliance on measurement call time on NPS?

When baking a cake, it is commonplace, and from personal experience highly recommended, to measure ingredients accurately. Modern scales are cheap, and produce highly accurate digital readings, but how much of what you’ve just measured in the bowl ends up in the mixture? How much is stuck on the bowl, or the spoon?

This is one example, and there are many, of what’s known as the observer problem, and it exists everywhere where things are measured. Measurement in the real world requires interaction, and interaction changes things. This, for baking, is unlikely to cause really serious problems, but not all measurements are created equal.

KAE has spent the last 12 months looking at feedback surveys from a wide range of different product and service companies. There is scarcely a major business operating today which does not attempt to measure its impact on its customers (or employees, but more of that later). We have seen surveys from car rental agencies, airlines, telecoms providers, retailers and even abstract dance theatre performances.

As CX enthusiasts, it’s great to see the genuinely widespread adoption of this kind of data collection, but as we began to evaluate these surveys more critically, we started noticing the mistakes. A recent survey sent while one of the team was on holiday involved an overseas car rental – the email sent was in English, but the survey was in Italian. A simple, easily avoidable error that cast the company in a bad light – even though the experience was good!

This led us to think about the observer problem. Can experience surveys do more harm than good? Is the insight they’re able to generate worth the reputational risk that comes from a poorly made survey? It’s very difficult to tell of course, but thinking about these questions suggests some simple principles:

  1. Ask as little as possible and at the right time – why, when sending an email survey to one of your customers, are you asking for details that should live in your CRM and be auto-populated? Many experience surveys that we have seen ask consumers for details of the experience that could easily be auto-populated from CRM records. Our recent machine-learning model of the US checking account experience revealed that ‘not feeling valued’ was among the most important painpoints and needing to recount the details of your experience to the provider that sold you that experience is unlikely to help with this.
  2. Predict more, ask less – advanced analytical methods, such as those we at KAE use to calculate the impact of individual painpoints within customer journeys already allow us to calculate the likelihood of negative feedback or a churn event for customers. It is possible to know without asking that a customer who has been on hold for ten minutes is having a bad experience. Asking them to explain how bad that experience was, is probably not going to help. Instead, we could offer them a discount, or at the very least an apology.

These principles lead to a simple workstream for businesses.

  1. Develop a single dynamic customer view with transactions and interactions from as many of the experiences you offer as possible. This view must include structured and unstructured data and should be easily queried for data modelling. Feeds from social media, and where possible online reviews should also be linked.
  2. Understand your painpoints and build systems that tell you when they’ve happened (long wait time on a call is a very simple example).
  3. Identify markers that will predict future change. Then project future conditional probability of customer outcomes based on different interventions.

Reducing the load your experience measurement places on your customers and allowing them to share only the data you need to help improve their experience is certainly a good plan for anyone looking to programmatically improve CX.

However, if we follow this advice, we may see the beginning of a new trend in experience management. The process described above makes it necessary to increase the measurement of both experience and outcome, i.e. what actually happens. Sentiment is used as a proxy for outcome – that high sentiment is better for business is commonly understood, but the sentiment score itself carries no specific dimensions or inherent value.

As the predictability of customer responses to experience grows, we may see a shift in the industry away from the measurement of sentiment to something altogether more sophisticated and powerful. While this will inevitably take some time – consider how long it took for NPS to be adopted in the first place – the recent emergence of holistic experience measurement software and platforms is perhaps the first indicator of this shift. We will be closely monitoring the market for more.