Comparing performance is a game of how to do apples-to-apples comparison and yet make your system look better. The simplest trick is to chose a hardware or a software that favors your system. And use the same platform for the competing system. Then under identical conditions, using same benchmark, your system would obviously perform better. If you look at the experiment as a whole, it’s a clean apples-to-apples comparison. Many competitive papers published or sponsored by vendors may be such studies.
To understand the real game, one needs to know more about how the two particular softwares are designed. How they work with different underlying hardware. Each system has unique features which are designed to perform better under specific applications. For example, one may be using on-board cache to give better performance while other may be using multiple paths to the storage to give better performance. Any differentiating features between the two softwares can be exploited to tilt the level of the playing field. By ‘correctly’ choosing the underlying hardware one of them can be shown in better light than the other.
That is why even though system performance is a very rigorous subject, it’s application to compare performance of commercial systems is a subtle art.
So, the next time you come across a ‘performance comparison’, be sure to look at the conditions under which the performance measurement was carried out.