Imperva anti-virus study "flawed", claims IT security expert

Imperva has been forced to defend the findings of one of its recent security reports, following accusations the methodology used to create it was flawed.

The security vendor released its "Assessing the Effectiveness of Anti-virus Solutions" report last week, which pitted 80 previously non-catalogued viruses against more than 40 anti-virus products.

In the report, the company claimed that less than five per cent of these products were able to detect new viruses and that some took up to a month or more to update their signatures.

In the run up to the report's release, Tal Be'ery, web research team leader at Imperva, told IT Pro the findings were not designed to put people off adopting anti-virus.

They were not exposing the products to threats in the way they would be in the wild.

"[The research shows] malware has windows of opportunities and it shows there are a few weeks before it gets detected by the anti-virus products," he said.

"[Even so] you should definitely have anti-virus to protect against most malwares and [because] after a time it recognises and can protect against them."

The company used the website, VirusTotal, to analyse the samples before each one was tested by the anti-virus products.

This resulted in the creation of a report, which revealed whether or not the sample was picked up by the anti-virus product.

However, the study's findings have been criticised by one security expert for not exposing the products to viruses in the same way they would be "in the wild".

Speaking to IT Pro Rik Ferguson, director of security research and communications at rival security vendor Trend Micro, described the study as flawed.

"Simply scanning a collection of files, no matter how large or how well sourced misses the point of security software entirely," he said.

"They were not exposing the products to threats in the way they would be in the wild."

For instance, where was the email with the malware attached, he asked, or - if the threat was URL-based where was the analysis of its content?

"To decide whether or not a threat would be blocked, it must be processed in a test in the same way it would be delivered to the victim," he added

In a follow-up statement to IT Pro, Be'ery defended the research, claiming the evolving nature of security threats mean Ferguson's recommendations may not work for every testing scenario.

"[The recommendations] address an old threat model in which the attacker would try to infect many possible targets with a single campaign," he said.

"When the old threat model is considered...and all the defences are tested, the same conclusion holds: while anti-virus is effective in fighting widespread malware, for new [threats], there is a good chance it will evade the anti-virus solutions."