Thoughts on Anti-Malware Product Comparison Testing

When I see all the many different so called tests of anti-malware products, all I see is a bunch of advertising. In addition to the fact that no two of these so called tests find the same results, I have some other concerns;

How does one who is testing take into account the fact that all of the variables involved as well as the tools themselves are in a perpetual state of flux? Any real testing takes a bit of time. Even before any test is published, the results will already be invalid. While BrandX is still developing a particular definition, BrandY has already done so. Yet the BrandX definition will be published the next day and possibly better than that of BrandY.

Any real benchmark of these tools must include the study of it's removal routines for each type of nasty currently in the wild. That alone is a daunting study. But without it, there is no value to the testing.

Most F/Ps (false positives) don't occur on a freshly installed system. Removing items falsely can and very often does cripple innocent components. In fact, most every scanner out has been known to cripple innocent apps or even the system at one time or another  to some degree...some more than others. How do you measure the probability of F/Ps?

A true benchmark of the detections must include a validating sampling of targets. Limiting the sampling does not represent a true test at all. I can make any tool look good by simply limiting the sampling for the test. In all of the testing that has been published by online magazines and other so called professionals, this limiting has caused every single one to have different results. This has often been used as a deliberate marketing strategy.

In addition to studying the detections and removals, what other features are offered by the tools? What proactive features exist and do they work as pitched? Almost every scanner advertises that it protects the system. Do they really?...to what degree?...how much of it is just bloat?

Should any testing done by those affiliated with a particular tool be considered viable? How does anyone reading a test result know if a test was done by someone who is affiliated or has an interest in a particular tool?

I have yet to see a real viable comparison test/benchmark. IMO, the methodology to perform a real comparison does not exist. Also, I believe that 99% of the so called tests published to date are simply advertising ploys and have absolutely no truth to them. I believe the other 1% are just done by well meaning folk who just simply don't have the understanding or expertise required in order to perform such testing.

Ask yourself; Why doesn't any two published comparison testings report the same results?