Engineering firm avoids disaster with data backup

WTB Group has successfully recovered key communications data after serious IT failures recently put its business operations at risk.

The UK supplier of civil engineering and building materials carries out most of its business via electronic messages, sending and receiving an average of 25,000 emails every day.

But when a drive on its storage area network (SAN) with one terabyte of data and eight virtual servers failed, the volume group became corrupted when it tried to recover itself. The failure also impacted a series of servers running Microsoft Exchange and SQL, affecting normal business operations for 1,000 WTB internal users.

WTB IT systems manager, Tim Brice said: "Because half of office based users and all remote users were without Email, we obviously missed out on important correspondence with business partners including potential customers. In addition, business operations were severely impacted by having 50 per cent of users unable to access the systems and several key applications unavailable."

He added: "The situation continued to worsen, with blue screens of death appearing on several virtual servers hosted on the SAN and key databases losing data and becoming corrupted. In the end we made the decision to re-build the SAN volume group, rebuild the virtual machines and restore data and configurations from backup."

WTB called on its backup systems provider, Databarracks to help with the SAN volume group re-build, having started using its services last year to protect 3 terabytes of emails, databases and files on 21 servers.

The re-build work took just over 24 hours, using data restored from Databarracks' systems. "We were able to run with much higher availability the following day," said Brice. The resulting delays and costs would have been much worse if Databarracks had not acted so quickly."

Since January's incident, WTB had to call on its backup supplier again last month when work to cable computer sockets in the firm's main computer room shorted a mains circuit. This tripped the main electrical feed into the computer room and the uninterruptible power supply (UPS) automatically shut down to protect itself.

All of the server, storage, and network systems crashed and some data had to be restored through a remote connection to ensure systems were operational once the electrical fault had been fixed.

Brice said: "I used to have nightmares when backing up to tape due to reliability, now I can just forget about backups in the knowledge that backup data is safe and sound in a managed data centre environment." And, although he benefits from the Databarracks systems performing their day-to-day tasks, he added that the service was key to the company's ability to recover operational systems during unforeseen IT failures.