Use of the term ‘Big Data’ has been steadily growing over the last few years, but exploded over the past 12 months. Today it is applied to describe not only the voluminous, diverse and ever-increasing amount of structured, semi-structured and unstructured data companies create, capture and store but also the technologies that are used to store, manage and process this data.
Enterprises are awash with data, having collectively amassed petabytes and even zettabytes of information. According to analyst Wikibon, the big data market is on the verge of a rapid growth spurt that will see it top the $50bn mark worldwide within the next five years – up from just over $5bn in early 2012, based on related software, hardware and services revenue. Data volumes may be growing at an exponential rate but the real issue for businesses today has moved on from how do they manage and store data to how do they harness the power all that accumulated information could have in helping them drive their business forward.
Today, rather than just amassing Big Data, companies are increasingly trying to ‘do things’ with it: i.e. find ways to process, leverage, analyse and mine the data that they collect to achieve actionable intelligence. In other words, the focus and the meaning of the Big Data term have shifted away from the data itself to the way that businesses manage and manipulate it.
The debate around how organisations achieve this is becoming increasingly urgent. Data volumes are growing to the point where companies are being forced to scale their infrastructure, and the traditional “scale up” technologies, legacy systems and licensing models are simply not working. This rapid data growth leads inevitably to challenges with the management and processing of this data. So what’s the solution?
Open Source hits the mainstream
From its onset, open source technology has been at the forefront of massive data management. Indeed, the rate at which the importance and popularity and usefulness of Big Data has grown can be directly attributed to open source. The leading Big Data platform, Hadoop, is first and foremost an open source project managed by the Apache Software Foundation. Today, open source provides the most effective way to address the large scale problem that Big Data presents and get the job done faster and more accurately at a fraction of the price of alternative solutions. Open source effectively shrinks the technology cliff by letting any company adopt and deploy technology regardless of their budget and level of expertise.
Even beyond Hadoop, many of the latest frameworks and databases have their roots in the open source world, where developers routinely create new approaches to problems that haven't hit the mainstream. Today the biggest providers of online communication and data transactions use and contribute to innovative, open development initiatives that are helping to drive the open source industry forward.
The democratisation of Big Data
Thanks largely to open source technology, Big Data is helping to level the playing field for smaller companies that had been disadvantaged previously due to the high cost of processing massive amounts of data. Open source overcomes these concerns by offering reduced adoption and deployment hurdles and access to technical expertise and support.
The removal of the traditional barriers that smaller companies faced is shaking up the industry and changing the way businesses compete and succeed in the new millennium. Smaller companies want to benefit from Big Data without having to put millions of dollars on the table.
Many companies have been doing Big Data for quite some time, but with conventional technologies that are often effective but can also be prohibitively expensive to implement. Open source changes the game, making the collection and analysis of data possible on low-cost, easily scaled, commodity hardware. It is democratising data, turning it into a competitive advantage that is no longer just reserved for the largest players.
