Big Data

For marketers, Big Data is not a new concept, but a new term used to describe what we have been doing the whole of our careers.

The chances are that you know the concept, but are not quite able to relate it to everyday marketing practice. Trust us, Big Data is not just something for mega-brands with tens of millions of customers and budgets to match.

Just finding a definition of Big Data can be a little awkward. Search online and you’ll find more references to Big Data (2bn) than you will for Big Mac, Big Brother, Big Society, and Big Bang Theory combined. The point being that its actual definition is debatable.

This is strange. Depending on who you ask, Big Data is usually described as data generation or data processing. But always in high volumes, high velocities, and often, in new formats. Or just the manifestation of trends that founder of the World Wide Web, Tim Berners-Lee, helped set in motion.

“Opening up data is fundamentally about more efficient use of resources and improving service delivery for citizens. The effects of that are far reaching: innovation, transparency, accountability, better governance and economic growth.”

Actual definition is less important than simply agreeing that Big Data is an increasingly important phenomenon. The real challenge with Big Data is how to make sense out of the tonnes of data being produced every day.

Almost every interaction between people, businesses and government is creating an immensely rich data trail. In fact it is so rich that many marketing opportunities to learn more about customers are being left on the data centre floor, since the volume of data created has recently exceeded the ability to store it.

For marketers, Big Data is not a new concept, but rather a term used to describe what we have been doing the whole of our careers; the organisation and interpretation of raw data, to produce knowledge or insights. It entered the lexicon of data scientists in the 1990s and has become all-things data since.

Commercial scale is important but not the single most defining feature

  • Walmart handles more than 1 million transactions each hour, which feed into databases estimated to contain more than 2.5 petabytes of data — the equivalent of 167 times the information contained in all the books in the US Library of Congress
  • Google processes around 24 petabytes of data per day
  • The BBC’s iPlayer is reported to use 7 petabytes of bandwidth each month.
  • Netflix uses 1 petabyte to store the videos for streaming
  • AT&T transfers about 30 petabytes of data through its networks each day
  • Decoding the human genome originally took 10 years to process; now it could be achieved in one week

Why is Big Data important?

Marketers are panning for gold. Client-side marketers have to make sure they know at least as much about their customers as their new data-savvy competitors do (and they know a hell of a lot). While on the agency-side, a heightened focus on evidence-led approaches that rely on hard data is essential, not least because the era of consultant-led ‘guesstimates’ is long over.

All these examples tell the same story: our environment contains an unimaginable volume of digital information which is growing increasingly vast, ever more rapidly.

Big Data’s marketing successes are not all ‘drowning in data’

  • Wonga, the payday lender, makes credit decisions based on a broad array of data sources, including social media that are well beyond the 15 or so ‘classic’ metrics a typical lender would appraise
  • Mint, the US-based online financial planning tool, draws together all of a customer’s financial information from different accounts (bank, card issuer, savings and investment providers, etc.) to provide an aggregate view, it then matches this with better value products in the market
  • Cancer Research UK’s gamification initiative connects the collective data interrogation capabilities of hundreds of volunteers who analysed a huge dataset to help scientists bring forward the cures for cancers

Making the most of Big Data isn’t easy, but making a start is. Data mining in all its forms – from credit scoring to advert tracking – more often underplays the original role played by market researchers. After all, which other profession has applied so many analytical techniques to understanding vast numbers of consumers?

“Big opportunities for marketers – we just need to recapture some of the debate.”

The ideas behind Big Data have led to some people in the market research sector flagging concerns about its applicability when it risks neglecting sound principles such as using a representative sample, or by being overly concerned with actually handling the huge quantities of data.

This approach may lead to results being biased in some way. Integration across varied data sources – some that are genuine Big Data and others not – presents formidable operational or analytical challenges, but many market researchers rightly argue that such integrations represent some of the most promising new frontiers in marketing science.

Companies invest tens of millions to derive insight from information pouring in from partners or customers. Yet there are scores of major businesses that have insufficient processes and skills to make sense of what these investments tell them.

To overcome this insight deficit, Big Data, no matter how comprehensive or well analysed, needs to be complemented by “big judgment”, according to the Harvard Business Review. Surely this is what professional marketers can offer?

The bottom line is that Big Data is a convergence of trends that market researchers helped set in motion, through commercial application of insight generation techniques. The idea has moved on quickly and risks leaving market researchers behind.

How does all that sound to you? Manageable? Frightening?
A combination of both?