Meta has unveiled the latest iteration of its Llama model, Llama 3 which it says trumps other open source contenders on performance and looks set to challenge the biggest names in AI.
Llama 3 comes in two parameter sizes, 8B and 70B, both of which beat out models of a similar size such as Google’s Gemma 7B, and Gemini Pro 1.0, as well as Mistral 7B.
Trained on two recently announced custom-built 24K GPU clusters that feature a training dataset seven times larger than that used for Llama 2, the firm boasts these models will take “data and scale to new heights”.
On MMLU, a common benchmark for AI models, Llama 3 8B leads the way with 68.4% over Gemma’s 53.3% and Mistral’s 58.4%, while Llama 3 70B has an MMLU score of 82% that just overtakes Gemini’s score of 81.9% and Sonnet’s score of 79%.
Llama 3 supports an 8K context length that doubles the capacity of Llama 2 and the new model reportedly “excels” at language nuance, contextual understanding, and complex translation or dialogue generation tasks.
It can also handle multi-step tasks “effortlessly” according to the firm, while Meta’s “refined post-training processes” have lowered false refusal rates and boosted diversity in the model’s answers.
"The release of Llama 3 by Meta with its advanced capabilities is poised to set a new benchmark in the open-source generative AI market,” Moses Guttmann, CEO and co-founder of ClearML, told ITPro.
“By pushing the limits of what open-source models can achieve, Llama 3 challenges other contributors to elevate their own offerings, accelerating innovation across the board,” he added.
Meta also teased a more powerful model that it says is still in training called Llama 3 400B+, which looks to be competitive with the biggest proprietary models at benchmarks of 86.1% MMLU and 84.1% HumanEval, a benchmark that tests code generation capabilities.
By way of comparison, Claude 3 Opus operates at 86.8% MMLU, just 0.7% ahead, while OpenAI’s GPT-4 operates at 86.4% MMLU.
Of course, Meta has yet to release the 400B+ version of Llama 3, though these figures do point to an overarching acceleration in open source as a competitor to closed source.
“These latest advancements reaffirm our belief in the potential of open source AI to rival closed-source alternatives,” Guttmann said.
Proprietary models could fall by the wayside for enterprise use
Though OpenAI’s GPT-4 is still the frontrunner in terms of performance, Llama 3 may signal a sea change in the enterprise’s approach to AI, according to CEO and co-founder of SambaNova Systems Rodrigo Liang.
Discover vendors that are the most highly rated by their customers
“Enterprises don’t need to build their own models, because the open-source community is more innovative than any single organization,” Liang told ITPro.
“A community working together will always be more powerful than an individual company, and leveraging this community is the best way for any organization to keep up with a rapidly evolving AI landscape,” he added.
Liang suggests that the ethos of innovation in the open source community will seem a more attractive landscape for the enterprise AI user, and Meta’s latest announcement seems to further solidify this position.
“With each passing week, the trajectory becomes clearer, affirming that open source is the optimal path forward,” Guttmann said.
