Nobody reviews hardware more exhaustively than chip gurus over at AnanadTech â€" for these guys, no detail is too small and nothing escapes their scrutiny. When the publication set out to review the Galaxy Note 3, it shamed Samsung by spotting some tweaked code which artificially inflates the device’s scores on popular benchmarks by anywhere between twenty to fifty percent.
Not only did it provoke a reaction from Apple’s marketing head honcho Phil Schiller, the discovery has prompted AnandTech to really scrutinize other device makers. Shockingly enough, turns out most Android OEMs pad their results by increasing CPU/GPU clock speed during benchmarking to make them look good.
So, who are the cheaters? Almost everyone, except Apple and Motorola. Samsung, HTC, Asus, LG and many others are all dirty and guilty of gaming the benchmarks.
If that’s not the biggest wake-up call the benchmarking industry has ever seen, I don’t know what is. I myself feel dirty even blogging about this, but the numbers don’t lie and that’s the sad state of things. Just who are the cheaters? Go past the break for the full disclosure…
Mostly every Android smartphone OEM cheats by using performance-enhancing code that overclocks the CPU/GPU and raises the chip voltage to its highest state when a popular benchmark is detected, AnandTech has discovered in its extensive analysis [1] .
This involves LG, Asus, HTC and other big name gadget makers, with the exception of Google/Motorola and Apple. Notably, the iPhone maker has never faked benchmark scores, AnandTech noted.
The hilarious part: these nonsensical performance “enhancements†yield modest five to ten percent speed gains, which hardly absolves the cheaters from wrongdoing.
“I can’t stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors,†wrote authors Anand Lal Shimpi and Brian Klug.
AnandTech rightfully calls the issue a “systemic problem,â€Â a pandemic among Android licensees that won’t go away simply because it’s not what hardware vendors want.
We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop.
With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization.
It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior.
The table below says it all (‘Y’ is for Yes, ‘N’ stands for No).
The publication doubts much can be done to rectify the situation because beefing up the benchmark scores runs rampant at virtually every major Android OEM licensee.
Virtually all Android vendors appear to keep their own lists of [benchmarking] applications that matter and need optimizing. The lists grow/change over time, and they don’t all overlap.
With these types of situations it’s almost impossible to get any one vendor to be the first to stop. The only hope resides in those who don’t partake today, and of course with the rest of the ecosystem.
As for Samsung, the company has an established history of messing with benchmarks.
Back in July, AnandTech caught Samsung cheating using the performance-enhancing code to fudge benchmarks for the then-new Galaxy S4 smartphone. Though it’s been shamed twice, the South Korean chaebol now has the nerve to announce its own benchmarking [2]  efforts.
Like anyone would believe the scores produced by such a targeted benchmark. I’m sorry if all of this sounds harsh, but AnandTech’s findings really got me enraged and I refuse to accept them as a fact of life.
This isn’t about the if-Apple-had-done-this at all.
It’s about the bulk of buyers who base their purchasing decision in part on benchmark scores. With that in mind, no one in their right mind would condone â€" and I certainly don’t feel good about it â€" big name corporations shamelessly taking advantage of customers by misrepresenting device capabilities.
Has our industry really come to this?
And who’s to blame?
AnandTech’s discovery is nothing if not an appalling admission of dirty tactics Android manufacturers resort to in absence of real innovation.
The irony?
If ordinary users had any say, it should never be about the speeds and feeds!
Before signing off, let me get something else off my chest.
Everyone and their brother think they’re entitled to bash Apple these days â€" from the defensive Samsung suggesting that Touch ID is a fad to chip maker Qualcomm’s silly statement that the 64-bit A7 chip is but “a marketing gimmickâ€Â (of course it’s not [3] ).
At the same time, Qualcomm â€" an Apple supplier and partner â€" goes all out promising to bring out its own 64-bit mobile processor at some point in the future, because “the OS guys will want itâ€.
It seems to me that jelly competitors just can’t keep their mouth shut.
Restrained tech media certainly isn’t helping by politely giving haters a lip service and spewing Apple doomsday scenarios when the company is at the top of its game. Last time I checked, only Apple thus far has managed to deliver the world’s first 64-bit mobile chip.
And only Apple has the world’s first 64-bit mobile operating system with the kernel, libraries, APIs and frameworks fully-optimized for 64-bit computing â€" and the company is now accepting 64-bit submissions  from its registered developers .
Apple’s doing all of that without resorting to the murky benchmark-gaming tactics. And Apple doesn’t lie to its customers about the benchmark scores of iOS devices like most Android backers do.
The full report [4] is well-worth reading: you’ll never look at benchmarks the same again.
So, what do you think and where do you stand on this issue?
Links
- ^ AnandTech has discovered in its extensive analysis (www.anandtech.com)
- ^ announce its own benchmarking (sammyhub.com)
- ^ it’s not (www.mikeash.com)
- ^ The full report (www.anandtech.com)
No comments:
Post a Comment