Met Office uses MongoDB to predict space weather

The Met Office has chosen MongoDB as a central data warehouse for information that will be used to predict space weather, such as coronal mass ejections (CMEs), solar flares and solar wind.

The project has been undertaken as the affect space weather can have on our modern lives has become more apparent.

"Natural disasters have been recognised as increasingly important by governments," James Tomkins, data services portfolio technical lead at the Met Office told IT Pro.

Tomkins gave the 1989 geomagnetic storm, which resulted from a CME and solar flare and knocked out power in the Canadian province of Quebec for nine hours, as an example of this.

In a worst case scenario power could be knocked out across an entire continent and take years to rebuild, Tomkins claimed.

More commonly, space weather can cause satellites to go offline temporarily or they may need to be replaced altogether.

"President Obama had been looking at space weather quite closely, following advice from NASA, and that made its way across to the UK, where it was put on the Natural Hazards National Risk Register.

"With it being on the Risk Register, they needed to find a risk owner, and the Met Office was deemed to be the best place to work with NASA and develop the kind of predictive capability that we have for terrestrial weather," said Tomkins.

Currently, space weather prediction is in its infancy and particularly with events such as solar flares, which travel at the speed of light, all that can be done is to notify governments and satellite operators that it is happening, how severe it is likely to be, and how long it is expected to last.

It is hoped the Met Office programme, which brings together data from NASA, NOAA, the UK Space Agency, and the British Geological Survey, will accelerate the development of improved space weather models and predictions.

Tomkins said he and his team chose MongoDB after realising their incumbent relational database was not fit for the purpose of collecting, housing and analysing the diverse range of data the project deals with, which can include images and videos as well as raw data.

"During our pilot of the programme, we encountered a few problems in terms of performance and the amount of development effort required to ingest all of that data and then serve it out again in a performant way to the forecasters," said Tomkins.

"So when we came to implementing the project for production, we reviewed our technology choices and while it was our first step into the world of NoSQL, the use case pattern seemed to fit what we wanted to do," he added.

Tomkins said he and his team had chosen MongoDB as it is an open source resource something the Government has been pushing departments to adopt.

"Whereas we might have gone with traditional vendors in the past, the steer from the Cabinet Office meant we did dip out toe in the water a lot more with open source technologies," Tomkins said.

"We have in fact created an end-to-end open source solution, incorporating the Apache Camel integration framework and Apache ActiveMQ messaging system, as well as MongoDB," he added.

With phase one of the project completed it is hoped the Met Office will soon be able to make space weather information available to the public alongside terrestrial weather information on its website.

However, it is still seeking funding for this endeavour.