How embracing flash storage helped Mississippi’s tax authority boost critical apps

Building a robust storage infrastructure that includes flash storage has helped the Mississippi Department of Revenue (DOR) boost the performance of business-critical applications and enhance its backup strategy.

Mississippi’s DOR collects more than $9 billion in tax revenues every year, to support the state and local governments across 82 counties and a population of three million people. The money funds public services such as police and fire departments, road construction and maintenance, public schools, and state universities. 

Two of the DOR’s key applications are Marvin, the vehicle registration system used by car dealers and county registration offices, and the state’s tax-collection system Mars, which supports a taxpayer portal and allows liquor stores to order alcohol. 

Combined, these two applications account for 95% of the state’s revenue – so ensuring they are constantly available is of the utmost importance. But for some time, the DOR has struggled with its storage platform, with old systems keeping its data from being accessible and performant. 

“We have the same problem as everyone else, sprawling data sets,” says Michael Dehaan, infrastructure services manager and enterprise architect at the DOR.

“One of the things that we’ve actually discovered around this, is that our agency disconnected from its data is nothing. What we do as an agency is not possible without that,” he tells ITPro. “So because it is … so important to us, we put a focus on that so we can be more effective as an agency.” 

Embracing flash storage for better backups

One key project for the department was to move Marvin and Mars to the FlashStack converged flash storage infrastructure from Pure Storage and Cisco. 

First to move was Marvin, a mainframe application hosted by the central IT authority for the state, that was being converted into a web app. Following the successful migration of Marvin, the other line-of-business application Mars then followed suit.

On Mars alone, the department found that the improved response times saved hours of staff time every day, which Dehaan describes as a “huge” win. “With the performance gains coming off of a legacy storage system, we saved 39 a half hours a day on loading screens, just spinning wheels on screens across our agency. The time to value of that speaks for itself,” he says.

The next step was to look at updating the department’s backups as its needs continued to grow. Dehaan’s team had to tackle more than a dozen terabytes of data, on which the revenues of the entire state depend, needing to be backed up. 

The DOR had previously been using a purpose-built backup appliance, but experienced issues with this system. In one instance, it was unable to restore data from a backup when it was needed. 

With this incident fresh in mind and in consideration of the comparatively positive experience with Pure during the Marvin and Mars projects, the DOR settled on FlashBlade to improve its database backups. FlashBlade is Pure Storage’s unstructured, flash-based storage platform for file and object workloads. Dehaan says the DOR was one of the early adopters of using FlashBlade as a backup system.

The department uses FlashBlade to run full nightly backups of its SQL databases with snapshots every 15 minutes, allowing point-in-time restores if needed. It also uses the solution for staging, testing, and development environments. For example, the department does weekly restores from its production environment down into its staging environment, for its developers to work on. 

Using previous technologies, Dehaan’s team would kick off the restores on a Sunday morning that would still be running when the developers were clocking in on Monday morning. This was inefficient and could delay the team getting to work. With FlashBlade in place, the restore time has been cut from more than 30 hours to just three. 

Knowing where to put your workloads

Dehaan says FlashBlade has also become the de facto landing place “for the weird stuff”, noting that it's a good place for workloads that aren't going on block storage. “It just eats everything up that we throw at it,” he tells ITPro.

While cloud storage is ideal for some workloads, with Dehaan giving the example of Exchange servers which he says he would never run on-premises, he stresses it’s not necessarily the best for every application.

“Applications that go to the cloud need to be built for the cloud to work,” he says. “So if you just take legacy applications and throw them into a [virtual machine] in the cloud you’re going to be spending more money to run them, you’re going to have to employ new people to run them, because the toolset around it is more difficult. And you’re going to spend a lot more money for those people – to get cloud engineers, DevOps engineers, cloud security engineers, it’s more costly.”

As such, moving applications to the cloud could be much more expensive, prompting Dehaan to question the strategy for specific workloads: “Why would we run a VM in the cloud when we are already killing it on-prem?”

While reliable backup for these applications is hugely important, Dehaan notes that because the system is now reliable to the point that his team has very little to do with it day-to-day: “It’s so low-touch, we are not having to log into it." 

"We use the monitoring for our Pure environment, but it’s not for monitoring the Pure stuff it’s to monitor the stuff running on the Pure. I can go to our [database administrators] and say ‘What’s going on here, why didn’t you tell us about this?’”

For other organizations looking at similar storage projects, Dehaan stresses the importance of an open-ended approach: “Something we are working through our infrastructure, at every touch point we can, is to build modular."

"The more modular you make things, the better off you are going to be in the long run,” he says, explaining this is because it gives you flexibility to respond to business needs.

“One of the things I’ve seen across our organization and others is people always underestimate data sprawl," he adds. "You’ll always need more than you think in terms of capacity and performance.”