Databricks' new Data Intelligence Platform marks shift away from Lakehouse

Databricks has launched a new Data Intelligence Platform it says will enable customers to more efficiently manage data using artificial intelligence. 

The announcement represents a shift beyond the firm’s long-established Lakehouse data architecture, integrating technology from recently acquired MosaicML to leverage AI on its all-new intelligent data platform. 

AI will be the foundation of the platform, Databricks said, and will use AI models to understand the semantics of enterprise data and provide a raft of features to make day-to-day data operations more efficient.

Introduced by Databricks in 2018, the flagship Lakehouse software provided users with a unified system to manage a wide variety of data types across an enterprise through a singular data management experience. 

The Databricks Lakehouse brought new functionality to enterprise users that traditional data warehouses and data lakes could not. This included providing a single solution through which they could store and manage unstructured data, semi-structured data, and data with high variety, velocity, and volume. 

As an increasing number of businesses continue to rely on collecting and analyzing vast amounts of raw, unstructured data, this functionality has become extremely valuable.

Using data lakehouses, businesses could query all data sources and govern the operations that use this data, such as business intelligence (BI) or AI. 

These platforms were not without challenges, however. One being a significant technical skill barrier limiting who could use the service, as querying data requires proficiencies in SQL, Python, or BI.

Other weaknesses included requiring extensive data curation, needing to ensure adequate governance and privacy of the data, and requiring highly skilled personnel to manage the complex system in order to avoid poor performance.

All of these challenges stem from the platforms themselves lacking an understanding of the data and how it is used.

Databricks eyes deeper customer insights

Acquired for $1.3 billion in July 2023, MosaicML provided Databricks with the ability to integrate sophisticated AI models into a data intelligence layer – DatabricksIQ – which underpins its new data intelligent platform.

The DI platform will use AI models to automatically analyze the data and how it is used, offering organizations new capabilities. 

For example, users – regardless of their level of expertise – will be able to work with internal datasets using natural language prompts. The intelligent platform is able to learn specific company terms and respond to complex questions users may have about the data.

Furthermore, AI systems will be able to optimize how data is organized based on its usage, as well as automatically detect and alert enterprises when their data is being misused. 

Databricks’ integration of AI into its data platforms means end-users will be able to gain insights by easily querying the data while helping data teams manage the data more effectively.

The DI platform will also simplify the development of enterprise AI applications, the firm said. DatabricksIQ will be directly integrated with Mosaic AI, enabling enterprises to create AI applications that can understand their data. 

This will give enterprises the ability to train custom models on their own proprietary data, including building chatbots powered by these models.  

Databricks sees AI as a transformative technology, particularly for data management.

“[T]he impact of AI on data platforms will not be incremental, but fundamental: massively democratizing access to data, automating manual administration, and enabling turnkey creation of custom AI applications.”

It believes the AI-powered features offered through its Data Intelligence Platform will revolutionize data science, where anyone can gain access to data insights, and any business can leverage their data effectively.

“Historically, data platforms have been hard for end-users to access and for data teams to manage and govern. Data intelligence platforms are set to transform this landscape by directly tackling both these challenges – making data much easier to query, manage and govern. In addition, their deep understanding of data and its use will be a foundation for enterprise AI applications that operate on that data.”