Genome project reduces storage requirements

The Duke Institute for Genome Sciences and Policy (IGSP) has revealed it has achieved a dramatic reduction in the virtualised storage capacity of its ever-growing genomic research information.

The Durham-based institute is dedicated to the study of life through scientific advancement in genetics and genomics. It requires substantial amounts of data storage for the creation and analysis of genomic information.

Over a two-year period, this saw its data storage infrastructure requirements rocket from 4TB to 300TB, made up of large Oracle databases on dedicated servers and locally attached storage.

Rob Wagner, Duke IGSP database administrator said: "We just bought more disks and kept giving them to Oracle. Over time this proved to be very difficult to manage. Server sprawl and inefficient data storage use became a constant headache."

In order to solve these issues, the IGSP IT team implemented a server virtualisation project to consolidate 40 physical servers down to just three running the Oracle applications. At the same time, it also migrated and consolidated its locally attached storage to a networked fabric attached storage (FAS) system from NetApp.

The project enabled the institute to use data deduplication techniques to identify and eliminate redundant data across applications, as the technology is a key component of the NetApp core operating architecture, Data ONTAP.

Alan Cowles, Duke IGSP system administrator explained: "Previously, I spent more time managing 60TB to 70TB of direct-attached storage than I do managing the 225TB of NetApp storage we have now."

The current IGSP environment consists of a 40-node high-compute server farm, three VMware ESX servers and 50 mixed-use application servers, where all data is provisioned from a NetApp FAS3070 clustered storage system.

Cowles said deduplication now plays a major role in managing data growth. "When we originally set up VMware, I allocated about 2.4TB for it. With NetApp deduplication, I've been able to shorten that down to less than 700GB. We now see an average of 83 per cent reduction in redundant data on our VMware systems."

A 25-year veteran enterprise technology expert, Miya Knights applies her deep understanding of technology gained through her journalism career to both her role as a consultant and as director at Retail Technology Magazine, which she helped shape over the past 17 years. Miya was educated at Oxford University, earning a master’s degree in English.

Her role as a journalist has seen her write for many of the leading technology publishers in the UK such as ITPro, TechWeekEurope, CIO UK, Computer Weekly, and also a number of national newspapers including The Times, Independent, and Financial Times.