Intel Doesn’t Want to Talk About Benchmarks Anymore
At Computex last week, Intel CEO Bob Swan spoke about the overall state of the computer and technology industry and the impact that the pandemic has had on computing. He also called on the industry to move away from benchmarking and towards different metrics.
We should see this moment [the COVID-19 pandemic] as an opportunity to shift our focus as an industry from benchmarks to the benefits and impacts of the technology we create. The pandemic has underscored the need for technology to be purpose-built so it can meet these evolving business and consumer needs.
And this requires a customer-obsessed mindset to stay close, anticipate those needs, and develop solutions. In this mindset, the goal is to ensure we are optimizing for a stronger impact that will support and accelerate positive business and societal benefits around the globe.
Let’s talk about that.
Evaluating Swan’s Idea on the Merits
Before we talk about why Intel is making this argument now, let’s examine the argument itself, which I’ll word as follows: Technology coverage should focus on positive (or even transformative) use cases rather than emphasizing benchmark results.
The fact is, Swan has a point. I can publish five hundred tests showing relative performance between two laptops, but I can’t benchmark how well you’ll like the keyboard response, whether the edges of the machine will cut into your wrists or arms when you type on it, or if you’ll like the trackpad. I can tell you how light a notebook is, but I can’t tell you if it’s light enough to avoid triggering your carpal tunnel if you carry it around much of the day. The best I can do is to describe my own experiences and hope that the description will help guide your own purchase.
Benchmarks have another weakness: They don’t capture the happiness of having a new system that solves previously annoying problems. I can tell you that Laptop A gets 6 hours of battery life while Laptop B gets 12 hours, but that doesn’t say anything about the surge of satisfaction you might feel if your flight was hit by a four-hour delay and you still didn’t have to go rooting around in your bag for a charger or fight for space at one of the public charging stations.
There’s also a distinct satisfaction that comes from being able to run a game or application that ran poorly before that reviews don’t always capture. Reviewers tend to compare previous top-end hardware against current top-end hardware, effectively missing the user experience of someone who leaps from, say, a GTX 680 or Radeon 7970 to an RTX 2080 or Radeon 5700 XT.
As a hardware reviewer, I agree with Swan and always have. Benchmarks don’t capture the entire experience of using a product.
Of course, that’s why reviews have text in them in the first place, instead of just benchmarks. A review comprised solely of benchmarks would, in fact, have more results in it — I’d have more time to run them. The reason we don’t do that is because text is how we convey contextual information of exactly the sort Swan wants the industry to focus on.
Evaluating Swan’s Claim Against Intel’s Current Competitive Position
As much of a point as Swan has regarding the limitations of benchmarks, there’s no way to disentangle his comments from Intel’s current competitive position vis-à-vis AMD. Currently, it’s not that great. This isn’t a repeat of 2004-2005 when Intel was saddled with Prescott and Smithfield, but AMD has taken the overall lead in the CPU market for any use-case beyond gaming — and the gap in gaming is pretty marginal.
Bob Swan knows this, of course, which is at least part of why there’s a remark about how COVID-19 is a reason to shift away from benchmarking. The idea of emphasizing the benefits and impacts of technology on a work from home situation make good sense, given that WFH is very new to a large group of people — but it’s best deployed alongside test metrics, rather than in lieu of them.
I entered tech journalism just as the first websites were really getting started. At the time, there was a tremendous surge of excitement over leaving the tyranny of column inches behind. No more fighting with copy editors to get graphs made instead of charts. No more fighting for every scrap of space. You want 50 graphs in a story? Put 50 graphs in it.
And we did.
That didn’t happen by accident. There was a tremendous hunger in the market for this kind of coverage because techies and enthusiasts wanted to see more performance information about the products they were considering buying.
Benchmarking is how we catch cheaters. Benchmarking is how we find thermal problems. Benchmarking is how we discover that one laptop will burn your legs if you try to game on it, while another won’t. Benchmarking is how we discover if your laptop’s power-saving mode is working properly, or whether the battery life is actually an improvement over the system you’re using currently. Want to know if your smartphone got slower over time, or if a new Android or iOS version is slower? Benchmark it. Need to figure out which system component, peripheral, or device is causing high latency across your system during audio processing workloads? Benchmark it.
Benchmarking has its limits. Within the scope of those limitations, it is nothing short of a titanic force for good.
The last thing to be aware of is that this kind of concern for performance rankings is really, really common. Every time Intel, AMD, or Nvidia drops behind in performance rankings, we can expect to see a discussion of whether the benchmark practices and tests reviewers currently rely upon are actually capturing what they need to capture. Sometimes, these discussions lead to real and fruitful improvements to benchmarking as a whole, but they’re a common tactic for companies to call for.
Intel undoubtedly would like to de-emphasize benchmarking at the moment. Fortunately, it’s not going anywhere.
Now Read:
- Some AI Advances in the Last Decade May Have Been Illusory
- Intel Core i9-10900K CPU Review: Comet Lake Paints a Target on AMD’s Matisse
- Intel May Have Reserved Its Top-End 28W Ice Lake CPUs Exclusively For Apple
