There is no doubt that the vast troves of data residing on the web—“big data” as they have come to be known—hold enormous potential to inform business strategy and boost productivity. One recent report estimates that data could generate gains of 1 to 5 per cent, seemingly modest improvements which could translate into game-changing returns. Amid such great promise, rising demand is spawning dozens of specialized start-ups, several fueled by venture capital—and an entire field of research, computational enterprise analytics.
The biggest challenge right now is how to sift efficiently through this raw information, in order to piece together digestible and meaningful trends and patterns—a process that has been characterized as “wrangling” and “janitor work.” (A task, as you might expect, that can take up most of a typical data scientist’s day.)
“The leap from the tools to the insight is the weak link,” Professor David B. Yoffie of Harvard Business School recently observed. Echoing this sentiment, University of Washington computer science professor Jeffrey Heer cautioned, “It’s an absolute myth that you can send an algorithm over raw data and have insights pop up.”
These scholars raise an interesting question: where do institutional insights come from? Is it, as they seem to suggest, a matter of the scale and the integrity of data, and the sophistication of analysis? Do more data necessarily mean bigger or better insights?
By contrast, in-depth interviews live at the other end of the data spectrum—allowing organizations to learn about themselves and their operating environments by digging where they stand. They provide an opportunity to probe critical issues with the help of people who are knowledgeable and invested. As with all forms of research there is a certain amount of mess involved, but the richness of what organizations learn from such close looking is hard to match.
These conversations can also teach big data a thing or two: they consistently reveal how vital it is to be clear about what you’re looking for from the outset, to formulate meaningful questions, and to think carefully about whom to ask them. The results may be “small data” by comparison, but the insights they generate can be far deeper.