22/04/2016: Just 26 of UK web hosting provider 123-Reg's servers are back online, following a catastrophic gaffe that saw the company wipe 67 servers, inadvertently obliterating customers' businesses and websites.
The company also warned that some of the affected parties would be unable to recover their sites at all.
"Unfortunately, there have been a small number of incidences where we may not be able to restore a customer's data," a statement 123-reg sent to Cloud Pro read.
"We will continue work to do so and we are in contact with every affected customer to keep them informed on the situation."
The company also said that it was "working with expert partners to analyse how the fault occurred and to restore data", but admitted that just 39 per cent of the Virtual Private Servers affected were back up and running.
The news comes after Cloud Pro exclusively reported that a massive security blunder allowed some customers to see others' personal information, including billing information, IP addresses and password notifications, during the incident.
21/04/2016: 123-reg has confirmed a "small number" of its customers were redirected to strangers' accounts when they tried to log into their accounts over the weekend, at the same time as a rogue script erased 67 of its servers, wiping out entire businesses in the process.
Cloud Pro was first made aware of the potential security breach by a number of 123-reg customers, who presented proof they had been presented with other users' names and tickets, including some billing information and IP addresses, when trying to access their own accounts.
The company has since confirmed an incident took place and that it has referred itself voluntarily to the ICO. However, it has denied any sensitive data was exposed.
"There was a brief period of time where a minority of 123 Reg customers were able to see another customer's ticket information while logged into the ticket system. During the time in question, a logged in customer was not able to access any other customer's 123 Reg control panel, where product configuration and sensitive information is stored," 123-reg told Cloud Pro
"We have put technical measures in place to ensure this does not happen again. Our customer support teams are also invoking additional security measures when dealing with customer information. We are confident that no sensitive data was accessed nor that there is any risk to our customers information," it added.
The ICO told Cloud Pro: "We are aware of an incident and are making enquiries."
19/04/2016: Users are expressing their fury after a human error at the UK's largest web hosting provider, 123-reg, wiped their data and deleted their websites.
A letter from 123-reg to affected customers using its virtual private servers (VPS) admitted that flawed code was behind the bug that caused havoc yesterday, forcing many users to rebuild entirely from backups - if they have them.
The coding error only wiped 67 of 123-reg's 115,000 European servers, but those without their own backups could be in serious trouble, because the firm has no backups of customer data itself, it told the BBC.
The hosting provider told Cloud Pro it is working with data recovery specialists Kroll to try and salvage as much data as it can on a case-by-case basis.
But BookabuilderUK, which offers a database of registered builders and landscapers, told Cloud Pro it had lost "multiple VPS, email accounts, hosting, hundreds of domain names also".
"Terrible service and even worse customer handling skills," the company said. At the time of writing, its website was still offline.
Another source, who wished to remain anonymous, said: "One would have expected 123-Reg to have also been maintaining regular backups of their VPN Servers? And hence, they should be able to restore their customer's websites as part of their Value Added Service?"
Dominic KVSmith, however, told Cloud Pro: "123 have messed up but the other side is - people are not backing up data properly. I was back online yesterday [afternoon]."
The rogue piece of script was written to show 123-reg how many machines are running compared to its database. Instead, an error in the script caused it to show 'zero-records' and effectively deleted customers' virtual machines (VMs).
The letter from 123-reg's brand director Richard Winslow, seen by Cloud Pro, read: "As a result of our team's investigations, we can conclude that the issues faced having resulted in some data loss for some customers."
The company continues to advise its VPS customers, the majority of whom are in the UK, to restore their VPS from their own backups if possible, telling Cloud Pro in a statement: "Our VPS product is an unmanaged service and we always recommend that customers implement backups to safeguard against unexpected issues.
"Customers who had purchased 123-reg backups can be online now as can those who are using another solution for website backup. If customers restore from their own backups, this will not overwrite our efforts."
18/04/2016: 123-reg flaw deletes customers' websites
Customers of web hosting firm 123-reg have been hit by a catastrophic technical fault that apparently deletes their virtual servers.
The company first acknowledged there was a problem on Saturday, 16 April, when it stated that some customers on its virtual private server (VPS) platform may be experiencing connectivity issues.
Over the following 48 hours, the company updated its support site repeatedly to both apologise to customers and assure them that its tech teams are working to restore normal service.
However, early on Monday morning, it issued an update saying: "Customers that have a local backup of their VPS are advised to rebuild their servers." It also linked to a support article instructing users how to do this.
"A more comprehensive update regards to the undergoing work bringing the servers back online will shortly follow," 123-reg added.
The firm has so far declined to explain what caused the disruption, nor what will happen to those who do not have a local backup.
However, one of 123-reg's customers, cloud software provider Innmaster, has claimed a piece of faulty code is to blame, and that the error has devastating consequences.
"At 7am on Saturday morning, someone at our hosting provider ran a script that had a catastrophic error in it. The result was that the script deleted the servers and websites of all their customers!" the company said in a blog post.
Innmaster also criticised 123-reg for providing little information as to what had happened and pointed out that rebuilding a server is "a full day's work".
Only a small proportion of 123-reg customers are affected by the error, the provider told Cloud Pro. It declined to confirm whether Innmaster's account is correct, what will happen to those who do not have a local backup, and when a resolution can be expected.
The firm told Cloud Pro: "A small proportion of our customer base has been affected. We have identified the problem as a technical fault and we are working around the clock on data recovery.
"We are working quickly with a team of experts to restore as much as we can. We are addressing customers’ needs on a case by case basis. We are speaking with customers individually about their websites and expected timeframes."
Are you a 123-reg customer experiencing problems? Contact our reporter Jane McCallion at jane_mccallion@dennis.co.uk.
