UL Delists Huawei Benchmark Scores After Cheating Discovered

UL, which owns 3DMark, has issued its own statement on Huawei’s benchmarking practices after the company was discovered to be artificially inflating its benchmark performance by pushing its SoCs out of thermal bounds when benchmarks were added to a whitelist that allowed the SoC to run at clocks that actually match those listed on a phone’s packaging. In all other cases, devices run far below their listed clocks to stay within thermal limits.

Huawei may have come up with this idea as a means to avoid being accused of overclocking its hardware — you can’t argue something is overclocked if it’s only hitting its specified frequencies — but the larger problem is that the company’s hardware tweaks completely blow the power budget and only apply to whitelisted benchmarks.

Data by Anandtech

Huawei and UL have had several conversations on this topic in which Huawei attempted to explain that this wasn’t cheating, but reflected new AI optimizations being used to ensure maximum performance. At the same time, however, these AI optimizations are only being used when the handset is benchmarked in specific tests as opposed to in games — and they’re pushing the devices so far out of bounds, they’re overheating in some cases. Since the entire point of a benchmark is to capture useful information about the performance of a device, a benchmark mode that only runs when the device is being tested is the exact opposite of helpful.

UL writes:

UL understands the intent of Huawei’s approach, but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require a device to run the benchmark as if it were any other application.

Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device…

To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode.

This kind of problem is far from new in mobile benchmarking, and while there’s evidence that Huawei is indeed trying to develop a new approach to improving its GPU performance through the use of AI, what the company did here appears far more like a standard whitelist trick to boost performance in the apps people use to test phones by blowing the SoC’s power curve. The device can’t practically operate for any length of time while drawing 8W of power, and that’s what matters — far more than any question of AI optimization or feature usefulness.

Now Read: Samsung Goes Legit, Stops Cheating on Android Benchmarks, Tests Appear to Show the OnePlus Five Cheating on Tests, and Lies, Damned Lies, and Benchmarks