Remark on Benchmark Cheating

Last year, I uncovered some of Samsung’s habits in manipulating benchmark applications by tweaking  SoC DVFS parameters and tipped off Brian who went on to publicize what ended up being a embarrassment for many vendors. We hoped this would discourage vendors from continuing the practice, and indeed, since then we’ve seen some improvement as Samsung, for example, has abandoned all tampering with benchmark applications. Josh recently pointed out at Huawei’s practices on the Ascend P7 and made it clear that Huawei wasn’t deterred from playing dirty. For the Honor 6, we see the same behavior.

You might ask yourself why I only mention this in the battery section as we already took a look at benchmark score of device. In the case of the Honor 6, Huawei doesn’t allow any higher DVFS states or enables additional cores for benchmark runs, but instead takes advantage of the kernel’s HMP parameters to try to improve scores. I already mentioned that setting the performance profile to “optimal performance” does little in actual gains and adversely affects battery life. While that profile uses threshold values of 75% up and 25% down, versus Smart’s (rightfully) conservative 90% up and 50% down, Huawei’s “optimization mechanism” automatically detects benchmarks and lowers these down to 30%/15%. The effect is that you might gain about half to a full frame per second in 3D benchmarks and a bit of a boost in CPU-spiky loads such as the 3DMark physics test. On the other hand, you completely and utterly destroy big.LITTLE’s power management scheme. This is especially visible in GFXBench's battery test where, if ran with the cheating mechanism in place, would drop the battery scores down by a staggering 20%.

I was a bit shocked at this because what Huawei has done here is detrimental and not worth the small boost in benchmark scores. I found that the mechanism detected all current popular benchmarking applications, but there's one extra addition I was baffled by: Chrome. Huawei forcefully sets the HMP parameters inside of Google's Chrome browser to equal the "Performance" profile in the battery settings, even if you have it set to the "Smart" profile. More intriguing is that the stock browser is not affected by this and retains the set parameters that the user chooses. We see this as a targeted attempt in trying to manipulate our (and other site's) benchmarking suites inside the browser as Chrome is chosen as an apples-to-apples environment for web tests.

In the end, it sadly seems that the only way to deter OEMs of including such mechanisms and avoid benchmark manipulation is to continue to having to report on it. Huawei's attempt is, bluntly said, just stupid, as they achieve nothing more than handicapping themselves in battery benchmarks.

As this article is also a SoC evaluation piece where we look into the technical aspect of the device, all benchmark tests were run without being affected by the cheating mechnaism. 

Battery life

While performing all battery tests we normalize the brightness of all devices at 200cd/m² to be able to compensate for display power discrepancies.

Web Browsing Battery Life (WiFi)

The Honor 6 lasted 10.7h on the Wi-Fi web-browsing test, which places itself quite competetively among other flagships such as the Galaxy S5 or HTC One M8. When you consider the device's 3000mAh battery which has a 200mAh advantage over the S5 and 400mAh advantage over the M8, then it ends up less efficient. The M8 is especially a good comparison device as it has the same size 5" 1080p screen, yet the Huawei Honor 6 only manages to trail it while having a 15% advantage in battery capacity.

BaseMark OS II Battery Life

BaseMark OS II's battery test is mainly CPU loading, and due to Huawei/HiSilicon's lack of a power budgeting driver for the Kirin 920 we see one of the lowest battery scores we've ever seen in recent smartphones. The Cortex A15's are simply running wild without any power limitations other than thermal limits. Of course you'd be hard to find any kind of real-world use-case where an application would load the device to such figures.

Anand had argued for 2+4 designs as "optimal" for big.LITTLE, where we would have only 2 big cores available in a SoC; I don't necessarily agree with that, as beyond the cost and increased die size of the chips there is no negative impact on the end-device. Cores can always be turned off, but can't be added. That theory does not work though when the software stack fails, and therein we see big.LITTLE's greatest weakness. The above graph is a perfect illustration of the impact that a misconfigured big.LITTLE SoC can have on a device. 

GFXBench 3.0 Battery Life

The GFXBench T-Rex battery test also the puts the Honor 6 among the bottom of current generation devices. Again, we go back to the platform power consumption figures to make sense of it: As theorized, the Mali T628 has roughtly half the perf/W of the Adreno 330 when normalizing for the same performance. The fact that the A15 cores are being fired up at all in GFXBench's very light CPU load doesn't do it any favor in terms of power. 

GFXBench 3.0 Performance Degradation

I'd like to mention that the "Battery Performance" chart we've used until now has been renamed to "Performance Degradation" to better describe what it represents as I've seen instances where people would confuse our chart with GFXBench's own "Long term performance" figure. The difference being that we measure the performance on the last run while GFXBench averages for the whole duration of the battery test.

The Honor 6 here behaves well in comparison to other devices. It's able to maintain its instantaneous 17fps performance figure until the end of the battery test. I think this is due the phone having quite good thermal dissipation charateristics. This also ties in with the bad battery life figure, the GPU doesn't throttle much if at all. I did extract the device's thermal policies, and we see that the GPU is only throttled in the second thermal throttling state, which is enacted when a sensor passes 47°C. I wasn't able to determine exactly which sensor this is, but it does not seem to be located on the SoC silicon as there we can see temperatures rise up to 75°C. Huawei running its thermal policy on an external sensor may also explain the lack of much thermal throttling on the device, as its dissipation is adequate enough that it never gets enacted very often.

Battery Charging

Integrated in its body, we find an embedded 3000mAh battery running a 3.85V chemistry. The actual capacity is a little bit higher as the typical figure is mentioned at 3100mAh. I also could see that the fuel-gauge has a capacity tracking mechanism, for this unit I saw it measure a value of 3050mAh. This is definitely a useful feature to have for tracking battery degradation. Many OEMs nowdays prefer to implement voltage tracking fuel-gauges as they are simpler and cheaper to implement, this is definitely a plus in favor of HiSilicon's own charger IC.

The device comes with a 5.0V 2.0A charger delivered. I didn't have any plug converter available to use the US style plug on the supplied charger so I used one of Samsung's 2.0A chargers. I could see no sign of any kind proprietary signalling and the device's drivers setup a charging rate of around 1750mA.

In the charging graph we can see where the device switches from its full charge rate to trickle-charging, which is around 80% of capacity or at the 75 minute charging mark. The power input from that point on steadily lowers itself in three stages until it reaches only about of 150mA of input current.

Charge Time

From zero to 100% the phone charges in a little over 2 hours, a pretty good figure for a phone with an embedded battery of its size.

GPU performance Display
POST A COMMENT

59 Comments

View All Comments

  • semo - Friday, September 12, 2014 - link

    Why no user replaceable battery? I've heard all the reasons why I shouldn't care but I still want one and wonder if planned obsolescence is the only reason for not including it.

    Also, is Qi an option for this phone?
    Reply
  • Andrei Frumusanu - Friday, September 12, 2014 - link

    There is no Qi charging option. Reply
  • semo - Friday, September 12, 2014 - link

    Thanks Andrei. I think it is worthwhile pointing out this missing feature in the article just like you did with the non-removable battery. It is important for some! Reply
  • marcokatz - Friday, September 26, 2014 - link

    Well said. Also it's important to point at that this is an Apple-wannabe that no way can match up to some of the really highly rated phones out there. /Marco from http://www.consumertop.com/best-phone-guide/ Reply
  • Excerpt - Wednesday, October 08, 2014 - link

    Yeah bro, you tell em. And Andrea why U no learn Chinese man? I wanna know what that extra stuff does. I'm going to China in like, a minute, like everyone else. What, you don't care about me bro? I love you man.

    And does it have haptic feed-back for goodness snakes? I want them good vibes in me fingas.

    What about a sit test? Most of us reading here have fat arses, do you know sit (test)? I don't know sit (test) but I wanna hear your experience with sitting, maybe try with a heavy object like a cow, yeah a bull sit test.

    That'd be great keep up the good work. <3
    Reply
  • Murloc - Saturday, September 13, 2014 - link

    1. takes up space
    2. makes the phone structurally weaker
    2. planned obsolence/forced service you have to pay a lot for are good ways to make money
    Reply
  • Alexey291 - Saturday, September 13, 2014 - link

    Takes up literally no space seeing how Samsung S4 and S5 are both smaller and thinner than HTC One (M7 and M8 respectively)
    Arguably the back that's able to pop off makes the phone LESS prone to breaking as it allows the force to dissipate somewhat better than in the case of a rigid structure which simply breaks.
    And lastly I am going to play the world's smallest violin for the POOR POOR manufacturers trying to make a quick buck from planned obsolescence or paid-for battery replacement.

    I mean I know anandtech is all about manufacturer interests but I'll care about their concerns and problems the moment they stop earning millions upon millions in profits.
    Reply
  • Intervenator - Saturday, September 13, 2014 - link

    Are you really saying that the S4 and S5 are thinner than the HTC One because of the replaceable battery? And that it takes up "literally no space"? Really? Reply
  • arsjum - Saturday, September 13, 2014 - link

    Not because of, in spite of. Reply
  • Alexey291 - Saturday, September 13, 2014 - link

    Thank you sir. That's precisely what I meant. Its thinner and yet it has a replaceable battery. Impossiburu /o\

    To me personally all that "the replaceable battery is inefficient" nonsense is just that - nonsense. Some companies are just better at making thinner phones and some try to cut corners (and costs) at every possible stage.

    And yes Anandtech has historically supported nothing but the manufacturer interests. They have been walking on these eggshells for a long time and that's precisely why they never directly criticise any dubious or greedy decision made by their sponsors :) In fact they sometimes go a step further and tell consumers (like myself) that we are wrong in wanting things like replaceable batteries and microsd slots because they are so "inefficient"

    /sigh
    Reply

Log in

Don't have an account? Sign up now