…And if data told you to jump off a bridge?

Jump off bridge

It seems like these days, everyone and everything is data driven.

Political campaigns use data from A/B testing to publish the most compelling election message. Websites and apps gather data on their visitors to be able provide a more personalized experience. And as unromantic as it may sound, some people even use data to find a romantic partner.

But is there such a thing as being too data driven? After all, we use data because we’re interested in the facts, we’re pro-knowledge, pro-information, and pro-facts. How could that possibly be bad?

Well, those things are not inherently bad. It’s relying on the data too much that can get us confused and potentially even making bad decisions.

Dilbert by Scott Adams

 

A narrow window

One of the most obvious problems of using data as the ultimate decision-making tool is its limited scope. You are only going to get the results that are relevant for the data you are looking at.

So for example, if you are A/B testing your landing page, with two variants of call-to-action buttons, you might get a sense for which button gets more clicks. But if you don’t look at the right places, you might not know where visitors are coming from; what they are looking for that made them come to your website and click either button option A or B; how satisfied they were with their experience on your site; or in how many other places they’ve seen your various messaging before they decided to follow your call to action.

Other real world examples of the narrow, misleading focus of focusing on one part of the data include education (teaching for success in a standardized test instead of for cultural literacy and real world skills), startups (founders working hard to impress an investor’s metrics instead of their clients), and of course, the pitfall almost everyone encounters – vanity metrics.

This term refers to the practice of counting website traffic, button clicks, leads generated, and Facebook likes as a true proxy for success – where the real benchmark comes from how many people use your product and get value from it.

Even the numbers that give a truthful answer to “How many active clients does my company have today?” are just the best benchmark we have to answering the question “How many people is my company generating value for today?”

These are fundamentally different questions, and focusing on the data can make us forget about the latter and glorify the former. That’s when your users become numbers instead of people, and when that happens – you need to check yourself.

Narrow window

Analysis paralysis: without an opinion, you’re just another person with data.

There’s this popular quote startups love to use, attributed to William Edwards Deming, that says that “without data, you’re just another person with an opinion.” The idea is of course that there are countless opinions and ideas out there, but the way to validate them is to hold them against this irrefutable measure of reality which good data represents.

Recently, Forbes wrote this article claiming the flipside – that without a theory, you’re just another person with data. The idea behind this lays on the works of Roberta Wohlstetter, who researched Pearl Harbor and the decision making process around it. What she found was essentially that the decision makers involved failed to glean useful information from the piles and piles of data.

There are two parts to understanding this: the first is that the sheer volume of data available makes it impossible to organize it into coherent conclusions. The mere data is not useful to us in its pure form of just a data dump.

The second is that many times, despite having all kinds of data points, we still don’t have the full picture – but we better act nevertheless. As Roberta writes in her book Pearl Harbor: Warning and Decision, “If the study of Pearl Harbor has anything to offer for the future, it is this: We have to accept the fact of uncertainty and learn to live with it. No magic, in code or otherwise, will provide certainty. Our plans must work without it.”

When applying this to the startup world, this rings familiar. How often are startup founders – especially early stage ones – operating with full certainty? Almost never. If we wait for “enough data” to make a move, someone else will probably beat us to it on a hunch.

Similarly, when new medical solutions first surface on the market, there’s often no way to know what the long-term effects of that treatment are until a long time has passed. So you have to choose, without real basis in data, whether you’re prepared to take the risk and try the new medication — or not.

 

What if data told you to jump off a bridge?

When thinking about the future of artificial intelligence, the world seems to divide into two camps: one is celebrating the rise of A.I. and the possibilities it brings for improving our world and the quality of living, and the other dreads it as it anticipates a dystopian world where the machines band together and revolt to destroy mankind.

As much as the latter view seems to stem from a somewhat crazy, technophobic place of fear of progress, have you ever thought about why these doomsday prophecies always depict the robots as deciding to end humanity? Why would they have it in for us so badly? Why would they even care?

I suspect it’s because we feel a deeply rooted guilt and responsibility for many of the negative things happening in the world – many of them rightfully so. Mankind is responsible for pollution, the extinction of hundreds of exotic species, atomic warfare, destruction and depletion of many a natural resource, genocides, poisoning of streams and rivers, and more. If The Machines are so logical, the dystopians reason, their inevitable conclusion for improving the quality of life on earth would be complete elimination of the human race.

What if you were to run all the data in the world through your fancy algorithms, and the data told you it’s definitively better to abort Down’s Syndrome babies, employ only 20-30 year old white males, abandon art and literature, or simply make like lemmings and off with yourself and everyone else? Data is so focused on what happened in the past, that it leaves little room for hoping to make a difference in the future. There is a point in which we must step back from what the data is signaling and compare it with our other systems. Our values is one of those systems, and it’s essential to make sure we’re listening to the metrics that align with it.

 

Why are we so obsessed with data?

This is a difficult place to be in, because we wish there was one definitive truth to follow; we want data to be king, because we want so desperately to be ruled by a king – an all-knowing, objective king who will always make the right decisions for us.

To be sure, the ability to draw actionable conclusions from lots of data has definitely improved some things and can be used to boost our confidence when we hypothesize. But it is far from being that perfect decision-making system.

This isn’t easy, but there are no silver bullets out there for making the right decisions. Not even data. Using all our systems and our good sense in conjunction is the best we’ve got, and giving too much power to data interpretations doesn’t make sense in the real world.

The truth is, we can’t and shouldn’t let ourselves off that easy. We still need to do the hard work of looking at the right data; of acting even in the face of uncertainty due to incomplete data; of making sure our data-driven decisions are aligned with other things we believe to be morally right, while at the same time avoiding hanging on to the status quo in the face of fresh information. Only once we do all that could we come up with truly better decisions.

Leave a Reply