Corrected Intel Benchmarks Are Much More Favorable to AMD

Last week, Principled Technologies, under contract from Intel, released a whitepaper that claimed to profile the performance of AMD’s Ryzen 7 2700X and several Threadripper CPUs. Such documents are always assumed to be taken with a grain of salt — whenever a company performs an analysis of its competitive advantage against another company, they’re doing so to highlight the areas where they perform well, not poorly. At the same time, such documents are usually accurate within the scope of what they claim to measure. There’s a difference, in other words, between simply making up numbers to make one’s competition look bad and picking tests or scenarios that accurately depict an advantage for one’s own product.

In short, Principled Technologies’ first whitepaper release should’ve been a useful document for reviewers and interested parties, but not a major event. But the company tested the Ryzen 7 2700X with Game Mode enabled via AMD’s Ryzen Master overclocking and enthusiast utility. This was in direct opposition to how AMD recommends the Ryzen 7 2700X be configured (Game Mode is intended for Threadripper only, on an AMD Ryzen 7 it just turns off half the CPU cores). In an email to ExtremeTech, the head of Principled Technologies, Bill Catchings, wrote:

Specific to AMD CPUs, we started the testing on Game Mode for AMD Ryzen Threadripper processors. Those results did indeed show Game Mode overall yielded the best gaming performance on Threadripper. For consistency, we then used Game Mode on all of the AMD processors. We have now added results from our testing of the AMD Ryzen 7 2700X in its default mode (i.e., Creator mode) as well. That mode overall yielded the best gaming performance on the 2700X. We apologize for not testing both modes in the initial report.

There’s a small-but-significant change to the whitepaper that reflects one of our major criticisms — the configuration of the memory subsystems on the testbeds themselves. According to sources we’ve spoken to, DOCP and XMP were enabled on all systems, despite the whitepaper originally omitting this information. The revised whitepaper now states that the respective capabilities were enabled in all systems.

Do we know, in some absolute sense, that this was the case? No. But if I’m being honest, I can believe the omission was accidental. Principled Technologies provided an unusual amount of detail about its system configuration and test conditions in this whitepaper. On the one hand, that’s part of what got the company into trouble — the press was quickly able to zero in on details that were causing problems for the paper. But on the other, we have to acknowledge that the company did provide the information that made it possible to identify the discrepancies in the first place, and most people who are attempting to flatly conceal data don’t bother putting it in their papers to start with. When you’re writing a list of literally every single step you took when configuring a platform, it’s easy to skip one, especially when you’re writing that list down for the fourth or fifth time.

I’d still quibble with describing the memory configuration Principled tested as “parity,” because using different RAM clocks on different CPUs isn’t parity — but it isn’t invalid, either. Intel and AMD respectively certify their CPUs for maximum RAM clocks and there’s nothing wrong with testing systems using those clocks even if enthusiasts frequently adjust them. And there’s a point to be made in all of this. Configuring systems for maximum officially supported RAM clocks might indeed present a small intrinsic advantage to one manufacturer over another. This is the kind of place where manufacturers will usually try to find conditions that allow them to show performance differences that reflect well on their platforms precisely because it’s a defensible point.

How Much Performance Did AMD Regain?

Principled Technologies’ new whitepaper can be found in full here, but we’ve run the math on their overall benchmark results so you don’t have to. On average, across all titles, the Ryzen 7 2700X picked up 8 percent performance. That may not sound like a ton, but it’s quite serious when companies are facing off against each other with their flagship products. More to the point, the 8 percent average obscures the fact that gains were larger in more than one title. Here’s a graph of the biggest shifts:

Assassin’s Creed Origin scores also improved by 17 percent, from 10,531 to 12,354, but the “Score” metric of that benchmark breaks our graph if included above. These results are also a cheeky look at just how many games can make meaningful use of more than four cores, at least on AMD’s side of the equation. Out of the 19 games PT tested, six of them showed gains of more than 10 percent when moving from Game Mode to Creator Mode and, by extension, from 4 to 8 cores.

We’ll have more to say about how AMD compares against Intel’s Core i9-9900K when that chip drops on Oct. 19. We’re glad to see that Principled Technologies took its retesting seriously and that the retested results give a more accurate view of the overall competitive situation between AMD and Intel microprocessors.

Now Read: Principled Technologies Responds to Intel Whitepaper Controversy, We Test Performance Claims, Intel Announces New Core i9 Family, 9th Generation CPUs, and Intel May Have 10nm Hardware In-Market Faster Than Expected