The Met Office has turned to the cloud to deal with ever increasing demand for weather data, but many of its core services will remain on-premise for the foreseeable future.
As the number of web apps and weather alert services using the Met Office’s data increase, the organisation decided to move part of its infrastructure into the cloud to ensure spikes in demand didn’t affect its services.
James Tomkins, enterprise architect at the Met Office, tells IT Pro: “It’s an issue of scale for us. The scale of consumption of the data we collect has driven us into the cloud, as well as the customer expectations around how they consume that data as well.”
The organisation’s supercomputer kicks out between one and two terabytes of weather forecasts everyday. “As the number of people who consume that data increases, we were running into bandwidth issues in our ability to transfer that data to the customers,” Tomkins explains.
Latency was also an issue, he added, saying: “If it takes us an hour to transfer that data, it may no longer be any use to the person receiving it. Our ability to transfer these large amounts of data to a large number of recipients quickly was one of the drivers [behind the move to cloud].”
When the weather observation body first took the decision to move into the cloud 18 months ago, it looked to some of the big public cloud providers, and found Amazon Web Services (AWS) to be its best fit after narrowing down its search to the public cloud giant and its nearest rival, Microsoft Azure.
“We’re predominantly a Java/Linux shop,” Tomkins explains. “Amazon had a much stronger ecosystem for that tech stack than Microsoft did and we felt there was more power to leverage in that stack at that time.
“We’ve not done that analysis again, I know Microsoft are making great gains in that space but we’ve never had cause to revisit that decision.”
The Met Office was storing its weather data on an on-premise file system, then using file transfer protocol (FTP) services to push that data to weather aggregation services like Accuweather and The Meteo Group, who would use that to deliver location-based weather forecasts to users.
But the risk was that this on-premise system would be unable to cope with the sheer number of queries from users checking their local weather forecast, and would choke access to the Met Office’s weather warning product, the Public Weather Media Service, which helps major media customers deliver crucial weather warnings to the general public.
With AWS, the Met Office started storing a lot of its weather data in an S3 bucket, allowing location-based weather services to ping this for relevant data, and scaling the bucket to deal with any spikes in demand.
On-premise, “we’d had have to had a server dedicated to hosting an application that could do that, but it might be doing nothing 99% of the time”, Tomkins says.
In the cloud, the Met Office has automated the entire process, so that any data pushed into S3 triggers Amazon SNS (Simple Notification Service) to ping customers with that data.
When the data arrives, AWS Lambda, Amazon’s serverless coding platform, caches it and puts it back into the Met Office’s API, so future requests for the same data can use the cache, rather than trigger a new query, easing demand on the infrastructure.
