Vodafone has slammed results from a RootMetrics survey, which concluded the operator had the worst connectivity out of the big four networks in London.
RootMetrics, a Seattle-based firm said it performed over 83,000 tests in London to compare calls, texts, and mobile data connectivity on the four major networks EE, O2, Three and Vodafone.
Each one was rated by category and given an overall score also. EE came out top in all of the tests, but it did share the honours in the call performance category with O2.
Vodafone has hit out at the results, claiming that the methodology is so flawed, the results are rendered "laughable". The network also suggests the results may be skewed because other networks are customers of RootMetrics.
In a statement to IT Pro, Vodafone claimed that RootMetrics showed no signs of following standard industry practices during testing, and outlined four key complaints about the findings.
Different phones were used for testing different operators, which did not provide a like-for-like comparison.
The performance of EE was measured solely using 4G and then compared against 2G/3G for the other operators.
There is no information of the test variables, which Vodafone claims makes it impossible to ensure like-for-like performance is being contrasted.
It was unclear how the final result was calculated. Vodafone claims the ranking does not match the individual Key Performance Indicators that RootMetrics makes available on its website (eg for mobile internet Vodafone is scored 17 points behind Three and 8 points behind O2 despite comparable or better download/upload performance).
In a statement a Vodafone spokesperson said it regularly commissions independent testing of its network by Germany firm P3, which also carries out work for many of the world's largest mobile networks.
"The results that RootMetrics is publicising through the press bear little relation to the experience we give customers every day," the Vodafone spokesperson added.
"Their most recent findings, which are barely a month old, disprove many of the findings of RootMetrics in London."
Update: 7/6/13 RootMetrics has since provided a detailed response to Vodafone:
The firm stopped short of denying it works for any particular network, instead maintaining that it is 100 per cent independent and its goal is to improve the quality of networks.
RootMetrics confirmed that it used the Samsung Galaxy S3 4G to test EE, Three and Vodafone, and decided to use the Sony Xperia Z for O2. The firm claimed it benchmarked phones before testing and then used the device which would put "every operator in the best light".
With regards to why it compared EE's 4G against operators using 3G, RootMetrics said that it wanted to testing to reflect the experience that a consumer can expect. which is why it
"In the specific case cited here, we tested EE on a 4GEE plan, which will default to 2G/3G when 4G is not available (we install no special software to 'force' 4G only testing)," RootMetrics said.
"We found the step down from 4G occurred not infrequently within the London urban centre, 80% of our tests were on 4G (LTE); outside the centre, however, the percentage of tests on 4G fell below 50 per cent. We do not weight results in any way."
In terms of methodology, RootMetrics claims that it explains exactly which tests are performed, how they are conducted and what happens during them on its site. All test on all operators are conducted at the same time and same place to ensure an apples to apples comparison. In the case of London, this encompasses over 80,000 tests across the four networks reviewed.
Finally, RootMetrics noted that its rankings are calculated in the following manner. Call scores account for 45 per cent, mobile internet 45 per cent and text 10 per cent. Each of these scores include elements based on empirical results for instance, for mobile internet, the scores encompass not only the speeds but also factors such as connection success percentage, speed to load a typical web page or connected app experience, time to download 10 emails (and the various success or failure percentages in doing these activities).
RootMetrics claims it does not publish every individual data point because "the 100 KPIs could confuse consumers".
