How Big is Big Data Adoption?
Data is growing at an ever increasing rate. Analyst organisation Wikibon estimates the global big data market opportunity will touch $53bn by 2017 from $5bn in 2012. At the same time, we are seeing a shift in the debate around big data. No longer is the focus on simply storing data. Increasingly, now companies are ‘doing things with it’, finding ways to process, leverage, analyse and mine the data that they collect to achieve actionable intelligence from it. In other words, they are not just amassing information but increasingly realising the power they can derive from managing and manipulating it.
Big data offers the potential for big value for all organisations. Larger enterprises have been in the forefront of adoption, experimenting with the technologies in order to tap into the vast amounts of valuable information contained within less structured data such as social media, email and other sensors. We are also seeing big data becoming more democratic as smaller businesses with fewer resources look to these technologies to level the playing field with larger enterprises and start to use information to drive agility and competitive edge.
Business First
A new big data adoption survey by Talend, polling the opinions of 231 professionals involved in the delivery of data solutions for their company, confirms this view of a rapidly maturing sector. 41 percent of companies polled in the survey, which was evenly split between European and North American respondents, said they had a big data strategy, highlighting not just the growing adoption of the approach, but the way in which it is increasingly being seen by many organisations as a key business driver.
For companies that already have a big data strategy, it is being driven by several company functions, which indicates that big data as a core strategy has moved past an early adopter stage. 39 percent indicated that big data initiatives are being driven by IT or a bottom up approach to be more efficient in collecting and analysing large data sets. However, 48 percent of big data initiatives are being driven by lines of business or executives, which indicates there are compelling business reasons for big data adoption, such as increased revenue, improved customer satisfaction, or faster time to market.
Its importance is highlighted even more by the fact for those without a formal big data strategy the main reason given (76 percent) was that big data is already part of their overall corporate or total data management strategy.
Drivers and Benefits
As the above findings make clear, businesses today are increasingly seeing big data as a strategic corporate asset and an approach to drive competitive advantage. This stems from the ability of the approach to drive more informed decision-making processes which in turn is driven by the ability to analyse the past and the present to make accurate projections of the future. The survey reveals that the number one business driver for big data, identified by 68 percent of respondents with a big data strategy, was to increase the accuracy and depth of predictive analytics or the ability to analyse current and historical data to make future predictions. Revenue optimisation (51 percent) and new revenue generation (48 percent) were the second and third highest responses as companies seek to do more in depth analysis to maximise market and wallet share e.g. improving cross-selling capabilities.
Once again businesses are increasingly realising that data can be the key differentiator that enables them to gain an advantage over their rivals. In the information arms race, companies that can collect and analyse more information should be able to make faster, better-informed decisions compared to their competitors by targeting new markets, for example, or by knowing when and why customers may leave or by deterring fraud.
For those that have implemented big data projects, 62 percent indicated that they have achieved business benefits with the primary benefit being business process optimisation (28 percent) and improvements in marketing and sales (24 percent).
Companies are looking to achieve business process optimisation benefits through integration of data and applications. IT is integrating existing data warehouses and business intelligences systems with diverse sets of structured and unstructured data for more in-depth analysis. The survey revealed that the most common applications being integrated were financial transactions (48.2 percent) and social media and Internet text (48.2 percent), followed by web logs (35.8 percent) and call detail records (28.4 percent).
By looking at social media and internet text, firms can understand who the “super users” are in any social network or community, i.e. ones that have the most influence over others inside social networks. Also, by correlating financial transactions and call detail records with click streams, one can generate a more complete view of customer buying patterns and behaviour.
Implementation Challenges
The technical challenges in processing big data involve integrating, searching and analysing large data sets. However, like any new paradigm, companies must also find the right skillsets, get budget approval, navigate company politics, and manage the unknowns. According to a recent report from management consulting firm, McKinsey & Co, the United States alone faces a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5m managers and analysts to analyse big data and make decisions based on their findings. Many early big data projects are free of explicit project management structure and over time companies will incorporate standards and procedures just as they have with data management projects. In the survey, 61 percent replied that their primary big data challenge was allocating sufficient time, budget and resources, with just over half (52 percent) reporting a lack of big data in-house expertise.
Looking Ahead
As businesses look to turn their big data vision into a practical reality that delivers the commercial benefits they are aiming to achieve, Hadoop and other open source solutions offer an attractive solution. They are especially effective when coupled with open source data management, data quality and integration software solutions from vendors that understand the space and lower the technical barrier to adoption.
This powerful combination enables even the smallest enterprise to tackle their big data issues; answer complex operational questions cost-effectively, improve decision-making capability and ultimately achieve the holy grail of competitive advantage over their larger business rivals.
The survey by Talend makes clear that businesses are indeed putting in place big data strategies and that they increasingly see the technology as a corporate asset that can help deliver edge. And as we are starting to see powerful, accessible and cost-effective technologies coming on stream, this vision is turning into a practical reality.
