AI model training costs have been rising steadily over the last two years, but could reach heights of $100 billion over the next decade, according to Anthropic CEO Dario Amodei.
Discussing the state of the industry with Norges Bank CEO Nicolai Tangen on a recent podcast, Amodei said that while many models cost $100 million to train at present, there are examples “in training today” that cost closer to $1 billion – and he predicts that this price tag is going up.
Amodei said AI training costs will hit the $10 billion and $100 billion mark over the course of “2025, 2026, maybe 2027”, repeating his prediction that $10 billion models could start appearing sometime in the next year.
Notably, Tangen asserted the point that “not many people” can participate in a race where the cost of investment is so high, to which Amodei responded that there would be economic viability in different ways.
“I think there's going to be a vibrant downstream ecosystem, and there's gonna be an ecosystem for small models,” Amodei said.
He added that, as the price tag gets bigger for AI models, so too will the capabilities of such models. Once AI models start costing in the regions of $10 and $100 billion, Amodei’s prediction is that their functionality will begin to mirror that of a human.
“I think if we go to $10 or $100 billion … and the algorithmic improvements continue apace, and the chip improvements continue apace, then I think there is, in my mind, a good chance that by that time we'll be able to get models that are better than most humans at most things,” Amodei said.
Cost is a critical pain point in AI
Amodei’s comments point to a growing concern in the tech industry over the surging costs of AI development. While OpenAI's GPT-3 cost $3 million to train in 2023, model sizes have only been expanding since then.
AI developers require thousands of GPUs to run and sustain AI models, driving demand for compute resources and requiring larger and larger amounts of capital to keep pace with model requirements.
The high compute demand of AI also puts a strain on energy use and increases the amount of power required by data centers, in turn driving up prices on this end of the supply chain as well.
One solution may be in the form of ‘local’ machine learning, whereby on-device inference processes could be used to drive down the costs of development in the cloud.
Industry analysts have repeatedly warned that rising costs associated with generative AI could have a detrimental effect on the broader technology industry in years to come.
Speaking to ITPro in October 2023, Ben Wood, chief analyst and CMO at CCS Insight said that the commercial realities of deploying generative AI could soon become untenable for many enterprises.
Wood’s comments followed the publication of research from CCS Insight which predicted a “cold shower” for generative AI investment in 2024 due to mounting development costs.
While this prediction hasn’t fully materialized - especially with regard to investment levels - continued costs could stunt wider adoption rates and create a disparity between hyperscalers and large corporates with the financial capability to invest and the majority of enterprises at the small to midsize range.
