Putting small language models under the microscope

We’ve spent the past few years discussing large language models, the huge AI models that power a lot of the generative AI tools that have dominated the headlines.

But small language models are also possible – and rapidly growing in popularity.

What benefits do these tiny AI models offer, and how much use will they get in the coming months and years, and how do they differ from other lightweight, ‘low latency’ models?

In this episode, Jane and Rory take a look at some of the smallest AI models on the market, asking what they're for and if they could be the future of the technology.

Highlights

"Microsoft gave the example of using them for organizations with very limited resources, using them for very specific needs at the edge. So for example, if you wanted to put small language models on smartphones or in a vehicle, they could say very quickly answer user prompts without needing to send any data to the cloud or even have to stop and process the requests."

"Distillation, in short, is taking a small model, typically an open source, quite lightweight, small language model and training it based on outputs from a much larger model."

"It feels very much like you're pulling something apart only to put it back together again in a possibly more expensive or less convenient way. The reality is, if you need the capability of one of the hyperscalers, you probably don't have the capability to do that yourself, or it wouldn't make any economic sense to do so."

Footnotes

• Small language models are growing in popularity — but they have a “hidden fallacy” that enterprises must come to terms with
• Small language models set for take-off next year
• Google’s new ‘Gemma’ AI models show that bigger isn’t always better
• Microsoft wants to take the hassle out of AI development with its ‘Models as a Service’ offering
• Three open source large language models you can use today
• Chinese AI firm DeepSeek has Silicon Valley flustered

Subscribe

• Subscribe to The ITPro Podcast on Apple Podcasts
• Subscribe to The ITPro Podcast on Spotify
• Subscribe to the ITPro newsletter
• Subscribe to the ITPro Podcast on YouTube
• Join us on LinkedIn