Sorry startups with dreams of data-led disruption, but incumbent companies are in a better position to benefit from the Big Data revolution than you are.
That's according to Kenneth Cukier, data editor at The Economist and co-author of Big Data, speaking at Comptel's Nexterday North conference in Helsinki today.
Startups benefitted from their agility during the early expansion of the web, but larger, legacy companies have a leg up when it comes to Big Data, for one reason: they have the data already.
"In a world of Big Data, what it takes to succeed is access to the data," said Cukier. "Who has the data? Big companies."
However, that head start doesn't guarantee success. Cukier pointed to airlines, which had stacks of data about their customers and the flights they were buying, using it to adjust prices on the fly meaning shopping many months before your travel didn't mean you necessarily got the best price.
Oren Etzioni, the founder of Farecast, realised that he'd paid more than fellow passengers on a flight, and as a computer scientist, wrote an algorithm to figure out the best time to purchase a ticket, Cukier explained. The eventually hoovered up data on every single American flight for a year, some 70 billion data points before it was bought by Microsoft for $100 million and all but killed off.
The idea has been picked up by others, helping consumers buy flights at lower prices the opposite of what airlines were hoping to do with their data. "Never for a second did the airlines think for a second that their own information could be used to undercut their own business," Cukier noted.
Of course, for incumbent companies to make use of the data they already hold requires repurposing it for other uses something data protection regulators aren't always keen on. That balance between privacy and the potential of Big Data to bring value and solve problems needs to be solved for such companies to benefit from the "raw natural resource" that's sitting on their systems, he said.
"Who owns all this data anyway," he asked. "The entity who bothered to collect it or the individual from whom it was taken?" That needs to be clearly decided, with accountability and transparency to ensure data isn't abused just because Uber knows your smartphone battery is low doesn't mean it should bump up prices, something that's technically possible but the company has widely chosen not to implement.
"We need to accept Big Data because it leads to benefits, but it can't be a new alchemy, a black box or a religion it's only a representation of reality," Cukier said. "We need to accept Big Data, but it can't come at the expense of our values."
