What does a data center look like in the AI era?
The AI boom has necessitated major changes in the architecture of data centers, take a look at how AI data centers differ from traditional cloud infrastructure, and the key features required to handle complex, energy-intensive AI workloads


Artificial Intelligence (AI)I has been a hotbed for innovation, promising to transform industries and drive new efficiencies. As such, the infrastructure that supports these advancements has never been more critical. But AI places vastly different demands on this infrastructure, precipitating a transformation in how our data centers are designed.
Traditional data centers are CPU-centric, providing flexible processing power suitable for a number of different computational tasks. These facilities underpin the majority of enterprise applications and data storage workloads, such as web services, database management, and file storage.
The advent of cloud computing forced data centers to become more flexible and efficient such that they could provide compute power for cloud services on demand and with low latency.
The popularity of AI technologies has ushered in a new era for the data center, however, and the workloads these facilities need to compute have become far more complex and power-intensive.
Generative AI relies on training foundation models and performing inference tasks, which requires the ability to perform calculations on vast datasets quickly. This is something GPUs are far better at than CPUs and so the data center has had to evolve once more.
The architecture at the heart of the data center has changed to adapt to the requirements of AI workloads. The CPU-powered racks that previously dominated the data center are replaced with high-performance clusters composed of specialized GPUs such as the Nvidia H100, which provide the power density required to meet the computational demands of AI.
These GPUs are also complemented by hardware accelerators such as neural processing units (NPUs) and tensor processing units (TPUs), which are other types of computational chips designed to tackle complex problems involved in AI training and inference.
Get the ITPro daily newsletter
Sign up today and you will receive a free copy of our Future Focus 2025 report - the leading guidance on AI, cybersecurity and other IT challenges as per 700+ senior executives
These chips are particularly suited to processing large amounts of data simultaneously, with the TPU designed to quickly process machine learning algorithms, and NPUs catering to deep learning neural network operations.
AI systems need access to large amounts of unstructured data to provide high-quality outputs which require new data storage technology as well. AI data centers need higher capacity storage solutions, which also provide low-latency access to data to enable quicker training and inference times.
For example, object storage is a popular approach for storing large amounts of unstructured data, as it is easily scalable as storage requirements grow and it also allows each data point to be stored as an object with a unique identifier.
Keeping up with the energy intensity of AI
This shift in architecture reflects a step change in power density contained within AI data centers, which also necessitates new cooling techniques such as liquid and immersive cooling. Liquid cooling has become something of a must-have for AI data center operators, as traditional cooling methods struggle to manage the level of heat being generated by GPU clusters.
A survey conducted by The Register found more than a third (38.3%) of enterprises expect to employ some form of liquid cooling infrastructure in their data centers by 2026.
AI data centers typically require at least 60 kW per rack, whereas traditional data centers average somewhere in the region of 5 – 10 kW per rack, which illustrates the scale of the new power demands AI workloads are putting on data centers.
Ensuring this power is available is a challenge for operators, who have often opted to commission entirely new greenfield sites near a reliable and cost-effective source of energy instead of retrofitting traditional data centers in less advantageous locations.
Operators also need to be wary of what kind of energy generation they use to supply the power-hungry GPU clusters in their data centers. A number of major players in the cloud infrastructure industry are buying up entire renewable energy generation plants to power their facilities, to ensure they have a cheap supply of clean energy that won’t negatively impact their ESG goals.
Others have begun signing multi-year power purchase agreements (PPAs) with green energy providers to guarantee the supply of carbon-free energy for decades. This is particularly important in giving them the headroom required to keep up with the skyrocketing demand for AI infrastructure.
Scalability is a key concern when designing modern AI data centers, so the facility can add the additional processing power and storage capacity required to meet growing demand and expanding datasets. Additionally, being able to dynamically adjust and reallocate resources according to fluctuating AI workloads helps data centers manage costs more effectively.
The three types of AI data center
Today, AI data centers can be broken down into three primary categories. The biggest facilities are constructed by major organizations to train foundation models and large deep learning neural networks that can be applied to a variety of different AI use cases. These projects are usually model vendors, internet service providers (ISPs), and telecommunications companies.
Slightly less gargantuan AI data centers are used by large enterprises to train their own industry-specific models based on these foundation models, as well as to do the inferencing required once the model is up and running.
A class of smaller data centers are deployed near production centers and populations they serve to deliver inference and fine-tuning with reduced latency.
Edge computing is becoming increasingly critical in supporting AI-driven applications including robotics, autonomous vehicles, and IoT, and the proximity of edge data centers to the end-users can help minimize latency and allow faster data processing, essential for real-time analytics.
Compared to training models, inference is relatively less computationally intensive but these edge data centers still require efficient and reliable infrastructure that can deliver insights in real-time.
Overall, the scalability, computational power, and efficiency of data centers are the competitive edge for enterprise AI. Thus maximizing these areas in your data center will be key to capitalizing on surging demand for AI.

Solomon Klappholz is a former staff writer for ITPro and ChannelPro. He has experience writing about the technologies that facilitate industrial manufacturing, which led to him developing a particular interest in cybersecurity, IT regulation, industrial infrastructure applications, and machine learning.
-
Why are many men in tech blind to the gender divide?
In-depth From bias to better recognition, male allies in tech must challenge the status quo to advance gender equality
By Keri Allan
-
BenQ PD3226G monitor review
Reviews This 32-inch monitor aims to provide the best of all possible worlds – 4K resolution, 144Hz refresh rate and pro-class color accuracy – and it mostly succeeds
By Sasha Muller
-
Does AI level the playing field for small businesses?
Supported Content The odds have always been stacked against smaller businesses, but with AI now so accessible, and affordable, the gains are there to be had
By Bobby Hellard
-
What does a smart campus look like and how far away is it from being a reality?
Supported Content Smart campuses are a growing trend as our educational institutions try to modernize their infrastructure and processes for the modern age
By Keumars Afifi-Sabet
-
How AI and digital transformation are game changers for the finance industry
Supported Content Advances in generative AI technology are enabling financial services institutions to unlock marked efficiency benefits
By Ross Kelly
-
What role will the cloud play in the AI era?
Supported Content For most businesses, cloud technology will be essential for leveraging AI without making huge investments
By George Fitzmaurice
-
How AI innovation is driving educational excellence
Supported Content Generative AI is helping students learn and educators teach, and the classroom of today is shaping the future of work
By Bobby Hellard
-
What role will partners play in realizing our collective AI dreams?
Supported Content When venturing into unknown territory, you should seek to partner up with organizations that can share in your vision and help you implement AI faster
By Keumars Afifi-Sabet
-
Which industries are leading the charge when it comes to AI innovation?
Supported Content From healthcare to finance, industries are not only adopting AI solutions but driving innovation in how the technology can produce tangible benefits for end users
By Solomon Klappholz
-
Dr. AI: How will artificial intelligence and other emerging technologies make for a better healthcare experience?
Supported Content With the right safeguards in place, AI can help healthcare professionals make new discoveries or free themselves up to spend more time with patients
By Rory Bathgate