Tweaking a phone to optimize benchmark apps might produce some numbers that make a small subset of nerds drool, but those numbers don’t correlate to the actual experience of using the phone. What’s funny about all of this is that benchmarks don’t really matter that much. Others have been caught doing this in the past This timing is particularly unfortunate for Huawei, since just weeks ago it got caught trying to pass off a DSLR photo as a photo from one of its phones. Samsung got busted for the same behavior on its flagship phones in 2013, and just last year, OnePlus was found to have done the same.
Huawei is far from the first company to get caught toying with benchmark results. That indicated that the phones weren’t actually smart enough to identify high performance demands on their own, which meant the benchmark score wasn’t an accurate reflection of how the phone would handle a typical app without special attention from Huawei.Īs punishment, 3DMark has removed these phones’ rankings from its leaderboard and adorned their listings on its website with a note that the phone’s “manufacturer has not complied with UL benchmark rules.” Many of their results have been removed as well. When UL ran an internal version of 3DMark, which Huawei’s phones couldn’t recognize the name of, the phones performed worse in the test. That’s what Huawei seems to have done, according to UL, which is behind the 3DMark software. While phones can adjust their performance as part of their typical behavior under high workloads, they can’t be hard coded to maximize their behavior just because a specific benchmark app is running. The OnePlus 3T was idling at 0.31 GHz, the way it does in most apps, rather than at 1.29 GHz for the big cores and 0.98 GHz for the little cores like it does in the regular Geekbench app.Huawei said it was AI, but UL found it wasn’tīut the way that Huawei implemented that behavior isn’t allowed. Immediately upon opening the app, the difference was clear. This version of Geekbench 4 is designed to avoid any benchmark detection, in order to allow Geekbench to run as a normal application on phones that are cheating (going beyond the package renaming that fools most attempts at benchmark cheating). Thanks to the substantial changes between Geekbench 3 and 4, the “Mini Golf” version had to be rebuilt from the ground up specifically for this testing. Most notably, the OnePlus 3T was looking for Geekbench, AnTuTu, Androbench, Quadrant, Vellamo, and GFXBench.Īs by this point we had fairly clear evidence that OnePlus was engaging in benchmark cheating, Primate Labs built a “Bob’s Mini Golf Putt” version of Geekbench 4 for us. The initial testing included a ROM dump which found that the OnePlus 3T was directly looking for quite a few apps by name. We brought a OnePlus 3T to Primate Labs’ office in Toronto for some initial analysis. We reached out to the team at Primate Labs (the creators of Geekbench), who were instrumental in exposing the first wave of benchmark cheating, and partnered with them for further testing. Our hypothesis was that OnePlus was targeting these benchmarks by name, and was entering an alternate CPU scaling mode to pump up their benchmark scores. Upon first seeing this we were worried that OnePlus’ CPU scaling was simply set a bit strangely, however upon further testing we came to the conclusion that OnePlus must be targeting specific applications. This is quite strange, as normally both sets of cores drop down to 0.31 GHz on the OnePlus 3T when there is no load. When entering certain benchmarking apps, the OnePlus 3T’s cores would stay above 0.98 GHz for the little cores and 1.29 GHz for the big cores, even when the CPU load dropped to 0%.
0 kommentar(er)
