Whitehall's inadequate tools and IT systems are preventing its data scientists from making the most out of Big Data, claimed its head of analytics today.
Chris Fleming, head of data and analytics at the Government Office for Science, stressed he was speaking in a personal capacity, but also pointed to a lack of requisite skills to tackle Big Data among the government's staff in general, as well as saying newly graduated data scientists are under-served by public sector technology.
Speaking at industry trade body techUK's event, The Value of Big Data: What does it mean for Government?, he said: "We're recruiting people via our fast-stream scheme who have really top skills, straight out of university, they have got data science skills, but because the tool set and the locked down IT systems we often have in our ... teams in government, those skills get blunted."
The government invested 73 million to help 55 public sector projects better harness Big Data last year, claiming the technology could net the economy 216 billion and create 58,000 new jobs by 2017.
But investment in central government Big Data tools appears to be a different story, despite a number of ongoing projects such as one started a year ago by the Office for National Statistics (ONS) to help it analyse unstructured data like social media, which requires a non-relational database to store the data in.
Fleming said the ONS has done well with limited resources.
"ONS have made strides in building their own Big Data lab from scratch, using [Apache] Spark and essentially a bunch of computers they found lying around," he said, adding that other departments are looking into cloud-based analytics platforms.
The Ministry of Defence is one, exploring the potential for a single analytics platform and one set of rules for all its government analysts.
However, Fleming warned that a lack of investment in Big Data technology is leading to a cycle that has a detrimental effect on skills.
"The argument I set out was that somewhere along the line we've stopped investing in technology in a way that's flexible enough to keep us up to date, and that has the effect of plummeting skills and then you get this cycle," he said.
"We become poor customers for being able to update our tools. Fixing the technology is an important question."
The problem of career civil servants
The data and analytics chief referred to a quote from Google that read: "When we've tried to work with government agencies on data analysis there's been a massive skills gap there of statisticians, data scientists and analysts. Without those skills making practical use of the insights of Big Data is hard."
Aside from Whitehall's recent intakes of data science graduates, Fleming admitted government predominantly relies on traditional analysts and researchers with good maths skills but less impressive abilities when it comes to IT.
He blamed the habit of cultivating career civil servants for this, where staff develop skills over time as they take on various roles in different parts of government, rather than focusing on one area.
"A more effective model would be to plan for portfolio careers," he said. "We are never going to meet the private sector salaries for data scientists but we do have fascinating data. [They could] come in for part of their careers and then leave."
Middle management
But the general level of knowledge about the potential of Big Data must also improve if government is to take advantage of the technology, added Fleming.
He said: "We are chipping away at it, one of things we've been doing with [government chief scientific adviser] Mark Walport is sharing seminars from department to department to bring together the leadership to say here's some of our tools, [and ask] what's going on in the department, where are the opportunities."
Big Data success
Despite the challenges, Fleming also highlighted some Big Data success stories in government.
In January 2014, the government established a data science programme to explore demonstrator projects, skills, technology, transparency and ethics.
He pointed to a project that is a collaboration between his department, the Cabinet Office, the ONS, and the Government Digital Service.
Using D3.js, a JavaScript library for manipulating documents based on data, developers created an interactive dashboard to visualise a dataset of 100,000 London Fire Brigade call-outs, showing where and when the incidents occurred, and what kind of call-out each one was.
"You'd be amazed at the traction we got just from building something as simple as this," said Fleming. "Imagine a world where a decision maker in government has to say what's happening in malicious firearms in parts of London.
"They'll typically commission their statisticians for a few days, but with this they can plonk it down right in front of them."
With GDS head Mike Bracken now the government's chief data officer, Fleming believes the future is in good hands however difficult the obstacles appear to be.
"Mike's identified an early priority to address the need for government-wide data standards and a mechanism for enforcing them," he said. "The future looks bright for data in government."
