Nearly two years after OpenAI launched generative AI into the public consciousness with the release of ChatGPT, it seems that AI hype may finally be dying down.
Lucidworks’ second annual survey on generative AI, published in June, found that only one in four of more than 1,000 executives surveyed had successfully launched generative AI initiatives in the past year. What’s more, the number of companies planning to increase investment in generative AI stands at 63%, a third down on the 93% recorded in 2023.
For those companies that have had success with generative AI, governance has become a bigger priority than both revenue growth and cost reduction.
While the generative AI gold rush has sparked a surge in spending on both buying large language models (LLMs) and building their own, companies are beginning to realize that the technology is failing to live up to the promises they were sold on.
One of the biggest barriers to widespread AI adoption has long been a lack of trust. It’s why many of the big tech giants have previously agreed to information sharing and the transparent and public reporting capabilities of their AI models. However, generative AI can be difficult to govern because there’s no universal definition and regulation over the technology is likely to spark years of legal headaches.
The gray area in the guidelines of how generative AI should be defined, implemented, and regulated has given rise to AI washing. A bit like greenwashing, AI washing is a marketing tactic where developers of LLMs, chatbots, and AI tools overstate the efficacy of the technology.
The vagueness around generative AI has made it difficult to verify claims and to also separate fact from fiction.
AI hype: Generative AI’s accuracy problem
Above all the myths spread since 2022 about generative AI, those that imply LLMs can outpace human output or are a step towards artificial general intelligence (AGI) straight out of science fiction, are the most damaging.
Ever since the public got its hands on generative AI, and at periodic intervals throughout the release cycles of all the big developers’ major announcements, it’s been clear that generative AI output has a huge trust barrier to overcome. We’ve all become familiar with the term ‘hallucinations’, used to describe the confident falsehoods LLMs regularly produce, and efforts to reduce these have only gone so far to date.
This hasn’t stopped executives from claiming that models can rival human output. Nvidia boss Jensen Huang caused a stir earlier this year when he hinted at the death of coding. Speaking at the World Government Summit in February, he said that, given how quickly AI is advancing, the kids of today might not need coding skills to join the tech sector job market. Matt Garman, CEO at AWS, has also recently suggested that AI could mean developers aren’t writing code within the next two years.
AI has certainly closed the technology divide and developers of AI pair programmers may argue that in the long term, anyone could be a programmer. But these claims also deserve scrutiny, particularly claims that AI could replace human developers.
Evidence suggests coders aren’t going anywhere soon. A May survey of 1,700 developers across Stack Overflow’s community found that 76% are using AI code assistants or plan to use them in the near future. However, many of these developers admitted that their AI code assistants struggled with context, complexity, and obscurity within code. This included, for example, architectural questions.
While generative AI tools are able to carry out simple and repetitive coding tasks, more complex requests require human intervention to oversee them and to correct any mistakes. While 43% of respondents reported trusting AI outputs to at least some degree, 31% either somewhat or highly distrusted the code their tools produced. Only 2.7% placed high levels of trust in AI.
It may be that AI coding tools simply help skilled developers with menial tasks, rather than replace them altogether.
“AI coding might boost the productivity of senior developers, but it may exacerbate the difficulties juniors face in the job market,” says Mądrzak-Wecke. “Fully autonomous coding is anticipated in the future. It’s unlikely to materialize in 2024 or 2025, though, so the potential risks probably won’t be immediate.”
AI hype: Buying versus building an LLM
In the rush to invest in generative AI, one thing that may be overlooked is the actual costs involved in implementing it.
Companies seeking to use LLMs run in the public cloud will pay hyperscalers to inference, train, or embed models on a pay-as-you-go basis. This is a popular format for adopting AI as it negates the need for up-front investment and allows businesses entry into large AI marketplaces with a range of models on offer such as Amazon Bedrock, Microsoft’s Azure AI, Google’s Vertex AI.
However, if you decide to build your own LLM, then you’ll need to consider that it will also be an ongoing investment.
Although building your own LLM may involve higher upfront costs, the investment is likely to pay off in the long run because it can be highly customizable and trained to your specific needs. On the other hand, an off-the-shelf LLM has been built for a wide range of tasks, so there may come a point when it no longer suits your needs and you have to invest in a new one.
“Training is the most resource-intensive and costly process, requiring significant compute and power. But, once a mode is trained, the cost of fine-tuning and inference should be lower,” says Łukasz Mądrzak-Wecke, head of AI at enterprise digital product consultancy Tangent, in conversation with ITPro.
With more and more companies turning to LLMs for a competitive edge, training should be seen as “an ongoing expense,’ he adds.
In the medium-to-long term, these concerns may have been alleviated by retrieval augmented generation (RAG). This process allows models to be grounded in vectorized external data to improve outputs, dramatically reducing the need for additional rounds of training and fine-tuning on off-the-shelf models.
But this comes with its own costs, primarily in the form of data infrastructure, which must also be factored into the price tag of using an LLM for business functions.
AI hype: Energy problems
A lot has been made about generative AI’s immense demand for energy. An often cited statistic, drawn from a paper by researchers at the Allen Institute for AI and the machine learning firm Hugging Face, is that generative AI systems can use up to 33 times more energy than machines running task-specific software.
Microsoft’s emissions rose 29% in 2023 due to AI-enabling data center expansion, while Google has publicly expressed doubt over whether it can meet its net zero by 2030 goal as AI pushes its emissions well above targets.
At present, training accounts for 80% of the energy usage and inference for about 20%, but, in the future, this is expected to flip on its head as the need for inference – passing new inputs through pre-trained models – accelerates. The actual amount of energy consumed is different for each use case. For instance, text classification is less power-hungry than image generation.
One response to these concerns is to house AI models in green data centers, which have far lower emissions and often run on 100% renewable energy.
While the tech industry works on making training efficient and lowering power consumption, there are still ways that LLMs can drive energy savings. Sarah Burnett, a technology evangelist, tells ITPro that if a company deploying an LLM can enable an office of dozens of workers to reduce their laptop usage by 30 minutes a day, then this could help offset their LLM’s net energy consumption.
“The energy cost of LLMs is undeniable, but their potential to transform workflows does present a compelling counter-argument. It's a complex equation,” says Burnett
Beyond energy, developers and hyperscalers will need to do more to reassure customers over the environmental cost of AI in the near future. The $ immense water consumption of data centers, for example, will likely define conversations around technology and the environment in the coming years.
