Users are expressing their fury after a human error at the UK's largest web hosting provider, 123-reg, wiped their data and deleted their websites.
A letter from 123-reg to affected customers using its virtual private servers (VPS) admitted that flawed code was behind the bug that caused havoc yesterday, forcing many users to rebuild entirely from backups - if they have them.
The coding error only wiped 67 of 123-reg's 115,000 European servers, but those without their own backups could be in serious trouble, because the firm has no backups of customer data itself, it told the BBC.
The hosting provider told Cloud Pro it is working with data recovery specialists Kroll to try and salvage as much data as it can on a case-by-case basis.
But BookabuilderUK, which offers a database of registered builders and landscapers, told Cloud Pro it had lost "multiple VPS, email accounts, hosting, hundreds of domain names also".
"Terrible service and even worse customer handling skills," the company said. At the time of writing, its website was still offline.
Another source, who wished to remain anonymous, said: "One would have expected 123-Reg to have also been maintaining regular backups of their VPN Servers? And hence, they should be able to restore their customer's websites as part of their Value Added Service?"
Dominic KVSmith, however, told Cloud Pro: "123 have messed up but the other side is - people are not backing up data properly. I was back online yesterday [afternoon]."
The rogue piece of script was written to show 123-reg how many machines are running compared to its database. Instead, an error in the script caused it to show 'zero-records' and effectively deleted customers' virtual machines (VMs).
The letter from 123-reg's brand director Richard Winslow, seen by Cloud Pro, read: "As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers."
An extract from the letter, seen by Cloud Pro
The company continues to advise its VPS customers, the majority of whom are in the UK, to restore their VPS from their own backups if possible, telling Cloud Pro in a statement: "Our VPS product is an unmanaged service and we always recommend that customers implement backups to safeguard against unexpected issues.
"Customers who had purchased 123-reg backups can be online now as can those who are using another solution for website backup. If customers restore from their own backups, this will not overwrite our efforts."
18/04/2016: 123-reg flaw deletes customers' websites
Customers of the UK's largest web hosting firm, 123-reg, have been hit by a catastrophic technical fault that has deleted their virtual servers and websites.
The company, which claims to host 1.7 million websites, first acknowledged there was a problem on Saturday, 16 April, when it stated that some customers on its virtual private server (VPS) platform may be experiencing connectivity issues.
Over the following 48 hours, the company updated its support site repeatedly to both apologise to customers and assure them that its tech teams are working to restore normal service.
However, early on Monday morning, it issued an update saying: "Customers that have a local backup of their VPS are advised to rebuild their servers." It also linked to a support article instructing users how to do this.
"A more comprehensive update regards to the undergoing work bringing the servers back online will shortly follow," 123-reg added.
The firm has so far declined to explain what caused the disruption, nor what will happen to those who do not have a local backup.
However, one of 123-reg's customers, cloud software provider Innmaster, has claimed a piece of faulty code is to blame, and that the error has devastating consequences.
"At 7am on Saturday morning, someone at our hosting provider ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers!" the company said in a blog post.
Innmaster also criticised 123-reg for providing little information as to what had happened and pointed out that rebuilding a server is "a full day's work".
Only a small proportion of 123-reg customers are affected by the error, the provider told Cloud Pro. It declined to confirm whether Innmaster's account is correct, what will happen to those who do not have a local backup, and when a resolution can be expected.
The firm told Cloud Pro: "A small proportion of our customer base has been affected. We have identified the problem as a technical fault and we are working around the clock on data recovery.
"We are working quickly with a team of experts to restore as much as we can. We are addressing customers' needs on a case by case basis. We are speaking with customers individually about their websites and expected timeframes."
Are you a 123-reg customer experiencing problems? Contact our reporter Jane McCallion at jane_mccallion@dennis.co.uk.
