The HTC One (M8) Review
by Anand Lal Shimpi & Joshua Ho on March 26, 2014 7:00 PM EST- Posted in
- Smartphones
- HTC
- Mobile
- HTC One
Subtle Cheating: New Benchmark Optimizations
We’ve been tracking the state of benchmark optimizations among Android OEMs for a while now. It wasn’t too long ago that we published a piece calling out nearly all Android OEMs for optimizing for benchmarks. The optimizations are pretty crude. Upon detecting a whitelisted benchmark by APK identifier alone, most OEMs would enable a performance mode that would:
1) Plug in all CPU cores
2) Run all cores at max frequency
3) Raise thermal limits to eliminate/reduce throttling
Defeating the optimization was just as simple: thorough renaming of the benchmark and all internal application IDs. For the past several months we’ve been working with benchmark, silicon and handset vendors to curb the behavior. Although we found the optimizations to have minimal impact on our test results, it’s still a messy practice that isn’t worth doing.
We’ve seen early (encouraging) indications that some vendors have reconsidered their position on benchmark optimizations. Unfortunately HTC isn’t quite there yet.
The M8’s Android 4.4.2 build includes a new, more subtle form of benchmark optimization that we hadn’t seen in previous devices. Benchmarks are still detected according to their application identifier, but instead of hot plugging in all CPU cores and driving them to max frequencies, everything appears to be normal at launch.
Here’s the state of the CPU cores after launching the Play Store version of any optimized benchmark:
Everything looks just fine. But look at what happens if we monitor CPU frequency over time on the Play Store and a special renamed version of 3DMark:
Average CPU frequency is about 15% higher while running the Play Store version of 3DMark. I still need to run some thermal analysis on the device but I don’t think HTC is raising thermal limits. Instead what appears to be happening is HTC is simply more aggressively tuning the governor response to performance demands, allowing for higher frequencies. Note that the frequency response latency is now so low that I couldn't even grab the 300MHz screenshot above in the Play Store version of 3DMark. As soon as the device detected a button press it would ramp up to 1.7GHz.
The impact on performance goes hand in hand with the increase in average clock speed. I measured performance during 3DMark’s Physics test (which is CPU bound). The difference was about 15%.
I also tracked GPU clock speed over time. Thankfully the optimization seems limited to CPU frequencies alone:
The list of optimization targets has also expanded since we last looked at HTC. The latest versions of GFXBench, BaseMark X and BaseMark OS II are now included in the benchmark whitelist.
HTC made one small concession - it’s allowing users the ability to run their device in this high performance mode at all times. Under developer tools (tap on the build number 5 times in Settings > About > Software information > More), you’ll find an option to enable high performance CPU mode. Checking that box will put your device in the same mode that’s enabled when a whitelisted benchmark is detected.
 |
 |
I do appreciate that HTC is exposing the optimization control, the only thing missing is the ability to toggle the benchmark optimization off (not to mention that I’d prefer if it was disabled to begin with). I fear that HTC’s justification in all of this is that everyone else is doing it so why opt out. The reality seems to be trending the other direction however. We’ll have to see what Samsung does with the Galaxy S 5, but I have a feeling that HTC is going to end up on the wrong side of history with this move. All of our benchmarks are already immune to the optimization, so it’s really a matter of sacrificing integrity for no real gain. There’s nothing more to say other than I’m disappointed.
Facebook
Twitter
RSS
222 Comments
View All Comments
jonup - Friday, March 28, 2014 - link
My point exactly! I was not arguing that the One has the best or better camera than the other flagships. All I said is that people a bashing the phone over a camera that is good enough (ok if you wish) in the day light but it (the phone) is exceptional in any other way (save for the wasted real estate by the logo). I would give up 600mah of battery to get rid of the black stripe underneath the display. Which brings me to the other Samsung sponsored reviewers argument - on screen keys vs capacitive key. On-screen keys is the Google way. Not the iOS Samsung copy cat way. I wish there is more custom-ability in the GEL and in the OEM laucher to set height of the nav bar, the width of the buttons, and the choice of buttons. As usual my two cents.mkozakewich - Sunday, March 30, 2014 - link
It doesn't take steady video, which is where it's more important. All the regular photos in this article were still shots, except for the video demonstrating the difference between the M7 and the M8.sephirotic - Sunday, March 30, 2014 - link
4k recording is worse than gimmick on any 1/3" sensor. Compression efficiency on smartphones is very low and unoptimized for video, there is simply no way to retain actual 1080p motion resolution with currect processors and pixel pitch of 1/3 sensors. Z2 samples makes this obvious, Actual true resolution on Z2 is worse than a semi-professional 720p video on a 2010's GH1. I'd rather have a more efficient 1080p compression and slow motion than 4k. EVEN IF there was actualy improvement from 1080p videos on 4k recorded from smartphone, 99% of regular costumers woudn´t even have a 4k screen to display it. Using for mastering casual videos? Even less likely.4k video on smartphone is just plain idiotic.
However I do agree 4mp is a little on the low side even for 1/3" sensor but no doubt pushing anything beyond 8mp is pure gimmick. it's a shame that sony kept pushing the megapixel race for so long even after the other manufactures stoped with this nonsense and now this is back on Smartphones and Video.
I do agree removing OIS is sad drawback.
CoryWeston101 - Monday, March 31, 2014 - link
Let's get some facts straight. NO ONE NEEDS 4K recording on a smartphone or period. Less than 1 percent of the poulation has 4K TVS or monitors making 4K recording a useless gimmick. You need it that badly go by an actually 4K camera from Sony or Cannon or someone. It will do a better job. But 4K recording is a stupid useless gimmickThey took OIS out because of the 2 camera set up. And also with a 2 camera set up YOU DON'T NEED OIS as it essentially does the same thing only better. The camera is a giant leap forward And no that's not why it is getting knocked. It is getting knocked because of the uneducated people out their like.
You bring the Moto X up? The Moto X camera sucks. The M8 camera makes the X camera look like a childs toy.
The M8 has one of the best cameras on a smartphone. They did not regress. They progressed. You are just to daft to understand.
CalaverasGrande - Monday, March 31, 2014 - link
4k recording on your smartphone?What phone DOES do that now.
Anand R - Tuesday, April 1, 2014 - link
You want 4K recording? What for? Do you have a 4K screen? Besides, even if you do, why not use an actual camera for 4K recording (which would be better in every way) rather than use a phone camera?purerice - Saturday, April 12, 2014 - link
Complaining about lack of recording 4K... on a cellphone... with 16GB total storage...Please tell me you forgot the "/sarcasm" tag.
Few cell phone cameras can compare to even a low end PaS camera in good light. Reduce the light level, increase the distance, add some motion, and the point&shoot wins by leaps and bounds. Olympus and Panasonic make decent all-weather cameras that fit in your pocket for 1/2 the price of a flagship cellphone.
If the phone has a good enough camera for video calling and quick snaps, it's good enough for me. I would no more replace a real camera with a cellphone because the cellphone can remedially take photos, than I would replace a computer with a cellphone because my cellphone can remedially edit spreadsheets.
doosh bag - Thursday, March 27, 2014 - link
I totally agree. The camera is more than capable enough in the opinion of professional photographers, but these nudniks out here, these spec snobs can't seem to find anything better to do than regurgitate the same tired, flimsy rhetoric, they read from some meshuggah reviewer, again and again. Far as I can gather from the comments on this first page, none of these yentas even own a M7 or a M8. It's all third person narrative. I own the M7 and let me tell you something, that camera is boss. For example, I was driving down the interstate in the desert, around 70 mph, driving into a beautiful sunset. I held the phone up to the side window and held the shutter button for a few seconds. Got about 20 shots. In one shot there was a semi truck going by, in the opposite direction, in the farthest lane away from me, traveling roughly the same speed, about 200 feet away, in failing light. So, combined speed between the two of us, about 130 mph. I snapped these photos in portrait mode. The picture looked as if everything were perfectly still. The detail so sharp that, without zooming the photo at all or flipping it to landscape view, I can clearly see and count all the lug nuts on the front wheel of that truck. There is nothing wrong with that camera. That camera does everything it's supposed to do and it does it exceptionally well. It's every bit as good as most, and almost as good as some. Comparison after comparison has proven it to be more than adequate. Someone mentioned 4K recording. Let me explain something to you people. 4K is a gimmick. It's a joke. 4K went be commercially viable for another 5 years, if then. They don't even have proper codecs to deal with it. Still using HD codecs to render Ultra HD source code. No phone can render the 4K video it shoots. Televisions and monitors are extremely expensive.puremind - Saturday, March 29, 2014 - link
I also totally agree. The M7 convinced me more in terms of shooting clean blur free, fast focused pictures. Other phones have let me down big time. MP give you sharper pictures if you are lucky... That's 30% of the time. I have regretted not being able to crop but I liked thetrade off of faster shooting.Death666Angel - Friday, March 28, 2014 - link
"If you are into photography that you care so much about picture detail, you won't be using you phone to take pictures."That is such a stupid argument. Considering that smartphones are alround devices capable of many things, you can say that just about every feature;
- if you care about high speed browsing, you will use a laptop/desktop
- if you care about listening to music, you will use headphones/DAC/dedicated mobile players
- if you care about video chat, you will use a 4k camera and a high end beamer
That do a lot and thus nothing really spectacularly. But that's not the point here, is it?