Cutting through the red tape of government IT

Public sector organisations don’t exactly have a reputation for trailblazing innovation. The history of government IT projects is littered with blunders; they often end up going beyond their original budget and deadline, if they make it to the finish line at all. On top of that, restrictive procurement processes and a general atmosphere of risk aversion usually means government departments are shackled to outdated technologies from legacy suppliers.

It’s somewhat surprising, then, to learn that a Scottish government team has managed to build its IT infrastructure using cutting-edge tools and techniques – more so than many private sector organisations. Not only is this department using DevOps methodologies and cloud infrastructure, but it’s also exploring software-defined data centres, containerisation and custom-built servers.

What’s even more astonishing is that this department isn’t in charge of technology or innovation, or even something as high-profile as trade or finance. In actual fact, the department in question is the Scottish Government’s Agriculture and Rural Economy (ARE) Directorate, responsible for controlling livestock disease outbreaks, maintaining animal welfare and paying out subsidies to farmers.

The person leading these mould-breaking efforts is Neill Smith, the ARE Directorate’s head of infrastructure. His 13-strong team is a mix of civil servants and contractors, and is split between database administrators, developers and both traditional and virtual sysadmins. Their remit is to build and maintain the applications that Scottish farmers use to submit claims for subsidy payments.

Mix and match

Smith’s infrastructure is a bit of a mixture. Most of the department’s workloads run in VMware virtual machines running on top of custom-built Red Hat Enterprise Linux (RHEL) servers, but the department is using a wide variety of different platforms and technologies, including Oracle SPARC, Postgres, DynamoDB, AWS, VMware, Rubrik, Ansible and more.

“We run a small number of [AWS] workloads, but they’re relatively cutting edge,” he says. “One of them is 100% serverless, from end to end. So whilst we're still in our infancy inside the cloud, we're not one of these traditional government departments that took a virtual machine and converted it to an EC2 instance, and went 'yay, cloud'.”

Smith is currently finalising the deployment of software-defined networking infrastructure, which he says is the last piece needed to complete the department’s software-defined data centre model. This, he says, allows him to make heavy use of containerisation and infrastructure as code tools like Terraform in order to implement micro-segmentation.

The directorate’s infrastructure is based on Pure Storage technology, and the vendor’s acquisition of Portworx’s container orchestration platform last year has opened up some interesting possibilities, Smith says.

“The challenge with containers has always been about data, and stateful and stateless applications. And I was really taken by Portworx, because I [thought] 'wow, that almost solves that problem'. And it has so many other features, with the migration, and the disaster recovery, and it can convert from an on-premise VMware instance to AWS Elastic Block Store.”

“So that's something as well that we're planning to do, hopefully round about maybe July or August, to get a proof-of-concept for Portworx and really play about with that. But that really opened my eyes up, and containers and the orchestration of them for us now becomes more strategic, because I can see the bigger picture and how it fits.”

One of the main things attracting Smith to containers in particular is the portability and flexibility they offer, allowing him to run services across multiple environments without his devs having to rewrite the same applications multiple times. Google Anthos is appealing for the same reasons, and a proof-of-concept for Google’s multi-cloud platform is also on the cards for the near future.

The proof of the pudding

Multi-cloud deployments, serverless applications, infrastructure as code and containerisation is an impressively forward-thinking battery of projects for any organisation to be undertaking, let alone a public sector body. Smith credits the progress he’s managed to make on these initiatives to the autonomy afforded to his team, which allows him to run small-scale proofs-of-concept for technologies that he thinks may deliver a business benefit. Once he’s proven its effectiveness, he can then show it to the business, obtain support and move forward with confidence.

“I've got the business buy-in, I've already proven the technology, I can put a cost around it, then I can go to finance and my boss, who's a senior civil servant, and say 'hey, look at what we can do – and the business loves it'. That's my technique. And it's worked, and it continues to work. But I'm lucky – I do have a team that is self-sufficient, where we can go 'hey, look at this'. And people go 'I love it', and you're like ‘great’. And then we can do it.

“Generally what you find is things start to happen a lot more easily, rather than going through the traditional bureaucratic route where it's like going through sludge, and it stifles you and stifles creativity. Then you end up going, oh, we can't do it because we're going to wait for this meeting or this board or this approval.”

However, Smith’s role isn’t without its challenges. Even with a solid business case, he says, cost and reluctance to adopt new technologies can be barriers. In some areas, his team is also at the mercy of other government departments that may not be as agile, and can introduce frustrating hurdles.

“We're very independent inside ARE. We have our own hardware, and we manage everything inside our own VDI. However, when it gets to the network layer – so the physical switching and the firewalls – we have to hand that over to the central government as a separation of duties. The challenge for us can be, we need to make changes quickly, or we need to adapt, and what we have to do is fall into our central system with SLAs. And it can be really tedious, where we can find it really difficult to get changes made.”

Smith says he also faces some challenges around recruitment and retention, particularly in relation to newer skillsets. Part of the reason for this, according to him, is the poor reputation public sector IT has for innovation and agility.

“Everyone wants to recruit AWS people, and that's difficult,” he explains, “particularly for the government. I don't think we pay particularly badly nowadays, but it's still not an attractive option. I still think, when people look at the government, it's like, ‘yeah, they're five years behind the curve’ … that's people's perception. So it's [challenging] being able to attract the right people.”

Like any good manager, Smith says it’s the people who make up his team that really do the work of keeping the ARE directorate’s IT at the cutting edge. His job, as he tells it, is to give them the space and the flexibility to ensure that they’re as free as possible from the red tape and bureaucracy that commonly plague government IT.

“My goal is to minimise operational overhead to let my guys do the cool stuff,” he says, “and that's my sole role, is making the environment and the culture the right place for these guys to excel.”