POST A COMMENT

363 Comments

Back to Article

  • ancientarcher - Wednesday, October 02, 2013 - link

    Good one!
    You need to keep the vendors honest and this is the best way to do it - name and shame! And not just the vendors, the silicon suppliers as well - Intel for the AnTuTu rigging, Samsung for all the shenanigans it has played with Exynos. What about Apple? Are they above all this?
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    So we tried the name and shame thing in July, that didn't work and I don't suppose it will anytime soon. We need a better option and I think we've got it. We'll continue to track this stuff publicly, but I outlined the real mitigation in the conclusion:

    "The best we can do is continue to keep our test suite a moving target, avoid using benchmarks that are very easily gamed and mostly meaningless, continue to work with the OEMs in trying to get them to stop (though tough for the international ones) and work with the benchmark vendors to defeat optimizations as they are discovered. "

    Apple doesn't do any of the frequency gaming stuff, no.
    Reply
  • solipsism - Wednesday, October 02, 2013 - link

    Unfortunately I think this is far too technical to gain enough traction to affect sales amongst consumers if they continue, and none of the companies have enough mindshare to cause enough blind outrage. Reply
  • mugiebahar - Monday, October 07, 2013 - link

    I agree with your assumption that while us techies understand the information, the average user will not (which is a shame as being ignorant seems to be ok nowadays). What really would hurt these guys is if all the tech community got together and every month published a list with nothing other then vendor name, and then beside it cheater or honest. Nothing else written as it complicates the underlining rule if the vendor is cheating or not. Send the list to all major sites and news sites that will publish it or post it. This puts vendors as either or not kinda or not so bad offender, thus calling them out right if they are cheaters. I still love the in-depth info and love the articles so please never stop for us nerds, but a simple list of cheaters non-cheaters in 1 list makes it easy for most, if they want to know more then they can read the attached article. I dunno if you guys agree but its easy to send and read. Reply
  • loluvz - Tuesday, July 01, 2014 - link

    You all clearly do not understand.. Reply
  • Solandri - Wednesday, October 02, 2013 - link

    Name and shame just doesn't have enough visibility to make an impact. People see and remember the graphs, while the text where you name and shame doesn't make as much an impact and is soon forgotten.

    Flip this around. Since the maximum gain you're measuring from cheating is ~10%, simply deduct 10% from the benchmark of any device which cheats. Put a * after the graphs for those devices and in the small print below explain that because the device cheats on this benchmark its score was reduced to make it fair for the devices which don't cheat. Now cheating offers no benefit or even hurts on the memorable graphs, while the forgettable text is what explains actual performance of the cheating device may be better than indicated by the graphs.
    Reply
  • BC2009 - Wednesday, October 02, 2013 - link

    I love that idea of just adjusting all reviews accordingly. Reply
  • hlovatt - Wednesday, October 02, 2013 - link

    Really like the idea of systematically pinging anyone who cheats on *any* benchmark 10% on *all* benchmarks and 'naming and shaming' them with a star and some small print. They want the better benchmark at any cost and therefore systematically docking their score is necessary to stamp this evil out. Reply
  • SIRKGM14vg - Thursday, October 03, 2013 - link

    If a device manufacture, or a chip manufacture is willing to do this, what other areas are they doing this in? What other areas are they selling you one thing, and giving you something else. Accept it or not it's the truth. 10% less the benchmark, that doesn't make any sense considering that most of these aren't scalar result sets. Reply
  • BARTHOLOMAUS - Monday, October 14, 2013 - link

    Samsung invented the panel lottery, one round of panels go to reviewers & display models and the next round of production are a crapshoot from four or five different chinese factories.

    This is criminal behavoir.
    Reply
  • loluvz - Tuesday, July 01, 2014 - link

    Nope. Reply
  • Tanclearas - Thursday, October 03, 2013 - link

    I was thinking the same thing, and I agree with the proposed methods. Dock 10% from ALL benchmarks if the OEM is found to be cheating at any of them. This of course needs to be done per device (ie: the Nexus 4 shouldn't be penalized because LG games the benchmarks on other devices). Reply
  • TheEvilPenguin - Friday, October 04, 2013 - link

    I don't think I'd feel comfortable with Anand doing this. One of the things that really impresses me about AnandTech is how everything is evidence based and quantified. I think that to just subtract an average number from benchmarks detracts from this. It feels a bit petty, and still rewards those who cheat more than average.

    I would be much happier if writers continue to try to bypass the cheating to get true figures while also noting which manufacturers are cheating.
    Reply
  • Murloc - Saturday, October 12, 2013 - link

    yes, just state in the comparisons if the benchmark in question is subject to cheating or not. Reply
  • themossie - Wednesday, October 02, 2013 - link

    For future reviews: Highly recommend you asterisk or otherwise mark known cheaters in benchmark graphs.

    Hope this calms some of the bizarrely... aggressive comments from mostly new commentators on the Note 3 review.

    Thank you Anand and Brian for this article and most excellent table heading!
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Our plan is to just switch to privately named tests, both old and new. If the OEMs don't know what we're running, they can't optimize for it.

    We do something similar on the SSD side, and honestly it's where I'd like to end up, where we have benchmarks that even the OEMs don't have. It's a taller (and more expensive) order for mobile but something I'm still trying to pursue.
    Reply
  • name99 - Wednesday, October 02, 2013 - link

    The obvious thing to do is to yoke battery to performance tests.
    Run the performance tests repeatedly till exhaustion, and report the battery life for each test.
    (Note that the PERFORMANCE result you report will have to be the TOTAL performance across all runs, otherwise vendors will start playing games like running the first 20 iterations fast, then all subsequent iterations at low speed.)
    Reply
  • theuglyman0war - Sunday, October 06, 2013 - link

    If the same optimizations exist on the shipped models...
    that would at least suggest the results are representative of benchmark app performance??
    It seems that if such optimizations were explicitly marketed as an enthusiast performance feature which
    could be turned on manually "as well" on any game, graphic app etc...
    That such ( automatic bench optimizations ) could then be openly touted as
    representative examples of that "feature's" benefit.

    Being a lemming hardware enthusiast myself...
    instead of feeling cheated with the last review of Samsung's Galaxy Note 3,
    I am instead finding myself hopeful, that this Anand expose results in a firmware update that
    exposes new performance settings that allows the same optimizations.
    ( in addition to the favorable Anand review I would probably open up my wallet asap fer a new max-CPU enabled Note 3 )

    Not that I agree with the under handidness of the doctored bench results.
    Just think as much would have been "no problem" if presented in the honest light of day:

    "it's a feature!"...
    Reply
  • dylan522p - Wednesday, October 02, 2013 - link

    Will you guys ever write your own benchmarks to bring up the benchmarking scene for phones so it doesn't suck like you guys did with SSDs? Reply
  • MrSpadge - Wednesday, October 02, 2013 - link

    Be careful that they don't simply exchange the file name for a checksum. Then renaming won't help. Might be enough to hard code the file name or some random string somewhere.

    > but what I expect to happen next are code/behavior detects and switching behavior based on that

    This leads to some interesting thought: tell them not to do this, as the only way to beat your new benchmarks would be careful observation of the load followed by rapid power delivery to the blocks which need it the most, accompanied by quick and efficient voltage & clock state changes. And once they get there - voila, finally the perfect power management! Then we just need to hack these routines to apply them to all workloads at a sane power target.. they'll probalby forget about this in their chase of benchmark scores ;)
    Reply
  • anactoraaron - Wednesday, October 02, 2013 - link

    If I recall right, ARS simply changes the name of the benchmark application since that's where this cheating is optimized - by the name of the application running. It shouldn't be too hard to change the name of the apk, no? Simply post the results as '3DMARK' or 'Vellamo' but use the results of the name-changed application in the review. Reply
  • ettohg - Thursday, October 03, 2013 - link

    It really doesn't take much time to write some code that automatically repackages APKs with a randomly generated name. Also, once you've done it a single time, you can apply it to any benchmark app you want.
    I mean, even I could do something like that and I'm mainly a Windows developer.
    Reply
  • varase - Thursday, October 03, 2013 - link

    Do do realize that the next step will be to CRC check the binary "to make sure it's okay", right? Then do a compare and branch on CRC match.

    This will make the cheat harder to detect, and will needlessly complicate launch.
    Reply
  • patrickjchase - Thursday, October 03, 2013 - link

    Many devices compute hashes for apps at *installation* time (as part of signature verification) so I doubt it would impact launch for any real customer workload. Reply
  • HansCPH - Thursday, October 03, 2013 - link

    Anand please, - could you look into whats going on with the many mid 2012 Macbook Air Toshiba SSD´s starting to fail en masse´. Reply
  • patrickjchase - Thursday, October 03, 2013 - link

    I suspect that they'll be able to circumvent renaming easily enough, if you're just renaming the stock benchmark executable. At the very least they can store the hashes of known benchmark executables and detect benchmarks based on hash rather than name. In fact it wouldn't surprise me a bit to learn that some of the devices that you think are "good" are doing exactly that - It's blindingly obvious and simple to implement.

    Of course you can get around that easily enough (there are plenty of ways to manipulate an executable such that the hash is changed but its behavior is not), but then the "optimizers" can switch to hashing specific file segments or doing behavioral analysis using performance counters etc.

    This has all the hallmarks of a classic arms race.
    Reply
  • jasonelmore - Thursday, October 03, 2013 - link

    oem's probably wont like this at all, and are likely to not send you samples. We shall see. Reply
  • tim851 - Wednesday, October 02, 2013 - link

    Philosophically speaking, I don't think it's cheating.

    If people evaluate a phone by how good it does on synthetic benchmarks, a phone vendor should make sure their product does good in synthetic benchmarks.
    Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    It's not cheating simply because it's the benchmarks that are broken. No need to invoke philosophy!

    Any benchmark that runs for only a few milliseconds does not represent a real workload. Anything that runs long enough (several seconds at least) will always run at the maximum frequency (and thus gives the correct score without any boosting). Ie. the boosting is only required to get the correct result in broken benchmarks that run for short periods.

    So rather than incorrectly calling out Samsung and others for cheating, we should call out those rubbish benchmarks and retire them: AndEBench, AnTuTu, Sunspider, the list is long...
    Reply
  • virtual void - Thursday, October 03, 2013 - link

    What is more similar to the workload that most users will actually run on their phones:
    * long running tasks that has the CPU and/or GPU pegged at 100%
    * short events that need to deliver the result as fast as possible

    I know you hate AnTuTu, but that benchmark runs things that tries to emulate the behavior of mobil applications. Not something you can use to compare different CPU architectures, but it tells you more about your user experience compared to benchmarks like Geekbench (which btw has an average L1$ hit rate of 95-97% on Sandy Bridge, not really what you would see on "real" applications).
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    Rendering a typical complex webpage is a long running task which can keep multiple CPUs occupied. It's certainly not something that just takes 25ms before it is finished like some of these micro benchmarks.

    AnTuTu doesn't run anything at all like mobile applications. For example the integer test is simply the 20 year old ByteMark which was never regarded as a mobile phone benchmark, let alone as a good benchmark. The fact that the developer doesn't even understand the difference between addition and geomean really says it all.

    No benchmark gives an approximation of the "user experience" of a GUI. That's something that is extremely subjective and typically completely unrelated to CPU performance (people want things like an immediate feedback when they click on something and smooth animations - these are all GUI design related).

    So all we can do is measure maximum CPU and GPU performance, and that is exactly what current benchmarks try to do, for better or worse. Geekbench is certainly one of the best ones we have right now, and AnTuTu one of the worst (especially given all the cheating we have seen there).
    Reply
  • akdj - Wednesday, October 16, 2013 - link

    Understand your concern with AnTuTu. And totally agree. However, others like G/B and their crew that's 'understanding' the need and desire for benchmarking mobile performance in a 'meaningful' way--@ least one that can be 'felt' and immediately observed by the end user. Fluid GUI. Instant response from the 'tap', no lag while surfing, reading, multitasking or the transition from putting the caller on speaker and looking up show times on Flixter being as easy to do as it sounds (this is remarkably fluid, fast and efficient on AT&T with the iPhone 5 & 5s...my Note 2 though has challenges). Geekbench is upping the ante so to speak and responding to the reviewers and concerned hobbyists and technical folk
    That said...we are literally 50 months into 'reasonable and comparable' performance in mobile to really GAS, right? I mean....benchmarking the original Treo or the older blackberry devices...was that an actual thing? Or at that time didn't we chose our platform and just get what we could afford? I'm 42. They weren't around in high school...and when they became affordable and reasonable and Nokia ran the world, I bought it. Probably <20 years ago. Never bought a phone. Just got the new 'free' with contract flip phone. I even remember wondering why a camera on the phone was necessary as short a 7 or 8 years ago.
    In the past 4 years, Moore's Law hasn't necessarily left the 'computer' world but it's certainly evident in the mobile sector...and a reason Intel is in hyperdrive to create and build out high performance /low energy silicon and SOCs. Something they've become adept to with the iGPU in the last several years
    These tests are still benchmarks AND still relevant PRECISELY because of 'where and when we are' in mobile technology. Hard to believe just 6 years ago....we didn't consider our Blackberry or Treo an actual computer in our pocket...they had shitty and slow processors, data was at a snail's pace....no apps not provided by the carrier or OEM (Snake, anyone?).
    We are in the infancy of mobile computing. Phones and tablets alike. Processing both CPU and GPU as well as memory control continue to 'double ever 12-18 months'. Apple has started something I'm sure Google and Samsung and Sony will all grab on to over the next 5 years. Creating their own chips and instruction set...hopefully Google will standup to the carriers and strip TouchWiz from Sammy....lots of changes in the interim
    For the time being though...Sunspider is an extremely valuable test as these ARE pocket computers. Accessing sites, graphic performance and central processing...and geekbench's awareness, we'll get there. You're a familiar responder. I don't know that I've read an Anand review in three years that doesn't mention if not beg for better and more specific testing. For today though, we have what we have and it's (as the guys and gals at Geekbench will tell you) not a lucrative business. Just take a gander at any specific machine, CPU or GPU and notice the '32 bit' majority. Even getting folks to pay for the 64 bit version....maybe $10/$12? It's like pulling teeth. Only the true 'geeks' and 'real' reviewers will spend the money. Would be awesome if someone here, keeping up to date with technology and readying their master's thesis would come up with, code and develop a true 'cross platform, cross SoC' benchmarking app. Until then, nothing will EVER compare to end user experience. Lack do issues. Quantity of quality apps and last but certainly not least, accessible and helpful customer support
    Again, for many, their smartphone HAS become their personal computer. In the truest sense of the word....when you look at what AOL was doing to those that 'didn't get it's just a decade ago. More software than any point in computing history. Gaming at reasonable prices. Extremely good battery life and amazing displays.
    While I love reading a lot of the reviews and comments here at Anandtech...sometimes I have to remind myself....benchmarks, while an objective measure of performance is nothing when compared to the analysis and the 'words' outside of the little b/m charts that matter. The 'subjective' user experience felt and shared by the author
    Tl/dr--- we have what we have. It's early in mobile...and it's the wave do the computing future for most folks outside of 'work'. Until someone codes better tools to measure the objective performance, subjective performance will have 'to do' ;-)
    Reply
  • KPOM - Thursday, October 03, 2013 - link

    I disagree. Few people are going to base a purchase solely on a benchmark, but it is still a valid datapoint if we can rely on them. Benchmarks are supposed to replicate real world behavior. Obviously everyone's mileage will vary, but gaming benchmarks serves no valid purpose whatsoever. The way Samsung did it, no developer can benefit. Samsung just identifies the app and if it detects a benchmark disables all the power management features to make the phone seem faster than it really is. Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    You're thinking of it from the wrong way. Samsung is actually enabling to run the benchmarks correctly at their real speed. So the phone does not really seem faster than it really is, as it is the correct speed. The reason the original benchmarks run slower is because they are badly written (for example when benchmarking professionals always do a warmup run first, run for at least several seconds, run several times and then take the fastest run). Reply
  • bah12 - Friday, October 04, 2013 - link

    Sorry but you are the one that has it backward. Samsung is ONLY letting benchmarks run at this speed. Technically yes it is the fastest speed the chip can run, but the code only let's that particular app do it. It is presenting data about the device that is not attainable by a 3rd party app. Reply
  • Wilco1 - Friday, October 04, 2013 - link

    No. Any application can run at the maximum clockspeed. Whether it is boosted or not makes no difference at all. This is exactly why it is not cheating - there is no overclocking going on.

    The boosting simply means that the benchmark runs at the maximum speed from start to finish rather than letting frequency increase from whatever it previously was to the maximum (this takes some time if you were idling before starting the benchmark). You can see this clearly happening in the AndEBench graph in the article. The non-boosted version runs ~80% of the time at the maximum frequency, the boosted 100%. The performance difference due to this was 4.4%.
    Reply
  • akdj - Wednesday, October 16, 2013 - link

    "Any application can run at the maximum clockspeed"
    Really? In the real world? Why then have the only 'triggers' to the CPU governor been specific to the only major benchmarking suites used ubiquitously by all of the review sites, tech blogs? It's impossible to get that same power from Asphalt 8? Editing and manipulation of photos and videos...why? Maybe if because 'if allowed' the battery would be dead in 48 minutes of gaming....or an hour and a half playing in Instagram? I'm genuinely curious why you seem to be the only one defending this type of bullshit? Genuinely
    Reply
  • thesglife - Saturday, October 05, 2013 - link

    Yes, an asterisk would be appropriate. Reply
  • shadowii - Wednesday, October 02, 2013 - link

    I think it would be wise to stop using the term "optimization" because it dilutes the term from what it traditionally means to be inclusive of what is basically cheating. Already I see comments on other sites wondering what the big deal is over optimizing. Ferrari was caught doing the same thing when loaning out demo cars to reviewers, where the cars were either upgraded over OEM, or specially configured for the reviewer's test course, so basically review cars were performing beyond how they would straight out of a dealership.

    I can understand the journalistic desire to avoid loaded terms, so maybe "cheating" is too strong, but "optimization" is definitely too soft. It doesn't easily convey in one word what they are actually doing in the way "cheating" does. "Rigging", "deception", something, who knows, but optimization isn't it.
    Reply
  • dylan522p - Wednesday, October 02, 2013 - link

    They are cheating. Not as huge of a difference when it is 5-10% but it is still cheating. Reply
  • HisDivineOrder - Wednesday, October 02, 2013 - link

    You could just try... not reviewing the product of companies that do this. If everyone did it, then no one would talk about their stuff except to say, "We do not review [insert name} products here because they are lying cheaters who cheat and lie their way to get slightly better reviews. As such, we do not condone buying their products or using their products."

    That might have an impact.

    Or you can keep trying to stay ahead of their optimizations and probably not succeed forever.
    Reply
  • dylan522p - Wednesday, October 02, 2013 - link

    Yes, lets not review the phones of the largest, 2nd largest and 3rd largest OEM of Android. That sure will work.............. Reply
  • CBone - Wednesday, October 02, 2013 - link

    How were you able to tell about the Apple benches? It's not like they have same chipset devices to bench against or allow changing the benches. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    We have some internal tools to poke around at frequency, A7 appears to behave honestly. I explored this issue with engineers at a couple of different companies who were also interested in finding out the answer. Finally, much of what I used to characterize the A6/A7 aren't publicly available, so there's no gaming there.

    The bigger concern is how much optimization for the web browsing tests exists in Browser.apk, Chrome and Mobile Safari. Like I said in the article, this is going to get a lot more difficult to detect.
    Reply
  • CBone - Wednesday, October 02, 2013 - link

    I thought the big differences there were their JavaScript engines. I would be interested in reading that article. Thanks for the fast reply! Reply
  • errorr - Wednesday, October 02, 2013 - link

    The problem is that the browser source code isn't available so it is a lot harder to figure out what are pure optimizations that are informed by using the browser as a target and help performance everywhere or what is an optimization that either is only useful in that benchmark alone and doesn't help in other applications or even negatively affects other use cases. Reply
  • eiriklf - Wednesday, October 02, 2013 - link

    The javascript benchmarks are more dependent on the browsers JS engine than anything else, I'm seeing about 40% to 80% improvements in performance on my nexus 4 in less than a year. Reply
  • Khato - Wednesday, October 02, 2013 - link

    I take it that the extra poking around at frequency on the A7 was in part the result of the fact that the tools available at the time incorrectly reported frequency for the A6 back when it was launched?

    Assuming that the tools are correct this time 'round, what does A7 frequency look like versus time as a benchmark is run? Constant 1.3 GHz? And potentially going a step further to confirm that the tools are indeed working correctly, would it show a reduction in frequency if the iPhone 5s under test was intentionally heated to the point where reduced scores were observed. (I've been curious about this since the original article since the geekbench sub-scores versus the A6 fit expectations pretty much perfectly if there was a ~30% frequency bump.)
    Reply
  • Beautyspin - Friday, October 04, 2013 - link

    I think this is a vague answer.When you are accusing all the android vendors of cheating.. you cannot just say A7 appears to be honest. It is not enough you think A7 is honest. Show us the tests and results and the process used for the tests and let us decide if A7 is honest. Reply
  • ricardoduarte - Wednesday, October 02, 2013 - link

    This is bugging me quite a lot, since the apple has full control over the OS, how is it possible to know for sure that Apple isn't in this mess also. In my view it would easy for apple to move whatever files, or just get functions calls from the sdk (private or not) to give whatever values they want, to things like the processor speed, voltage, etc. to a different file in iOS, and just allow access to fixed values that apple wants the reviewers and testers to see?
    Before people start bashing I am an iPhone user, and i feel that Apple has a big advantage in this department as it is harder to verify actual results from iOS, when compared to android (even thought these manufacturers could also go to great lengths to make it almost impossible to check if they are faking it or not), as long the checks are done in the software side.
    Whati would like to know, how can these companies know for sure that Apple is not involved in this messing around, and the mess with android manufacturers isn't bigger than what it seems?
    Reply
  • tim851 - Wednesday, October 02, 2013 - link

    Why would Apple do that?

    They have no competition in iOS-land. Between the different eco-systems and the different hardware priorities (esp. screen size), synthetic benchmark performance is the last thing people look at when comparing devices.

    There are a million posts on the web from people bashing the iPhone and telling the world why they wouldn't buy it. I have literally NEVER read anybody cite performance worries.
    Reply
  • Beautyspin - Friday, October 04, 2013 - link

    Why would Steve Jobs come on stage and say you are holding it wrong when the problem is clearly with the phone? (It lost a class action suit, remember and it also sacked the Antenna Engineer)..Was that ethical? Pls do not talk about Apple ethics.. Reply
  • ESC2000 - Tuesday, October 08, 2013 - link

    Apple may not cheat (or at least currently appear to cheat) but their "ethics" are questionable in that they steal money from their customers with their ridiculous profit margins. There's a reason they have $150 billion in the bank even though they hold only small slivers of market share in everything except tablets.

    The truth is that it makes little sense to put any of these companies on a pedestal or endow them with an ethical spirit because under the law (in the US) their only duty is to shareholders, so they will do anything they can do within the bounds of the law to increase profits - ethics is a distant consideration.
    Reply
  • muyaad - Wednesday, October 02, 2013 - link

    Jus because almost all dirty Apple should do it?
    You've said it all Apple control its OS and devices that work with it. Apple flagship shipped with 1GB of RAM, 2 cores processor and 8-MP compare this with another Android flagship in the reality test, you'll find out that Apple will outperform it. This is real optimization not benchmark cheating.
    Even try to compare a picture taken by GS4 with one by iPhone 5, you will find out that the iPhone 5's picture resembles the real image more than GS4's which always recreate the image by throwing in colors and tones.
    Reply
  • MrSpadge - Wednesday, October 02, 2013 - link

    You could measure power consumption in benchmark scenarios. That's a lot of work.. but if a vendor burns way too much power for slightly higher benchmark scores they'd look worse in battery life. Which is something people easily understand and care about. Of course there'd be the argument that such sustained load on a phone would be unrealistic - but the message would remain: "the cheater is less efficient". Reply
  • iluvdeal - Wednesday, October 02, 2013 - link

    Great investigative work on this. It's a shame Apple isn't doing this though as the outcry and media attention would be much louder if Apple was the one cheating on benchmarks. Reply
  • jerrylzy - Wednesday, October 02, 2013 - link

    It is a great idea, but isn't it better to test gaming performance instead of benchmarks? Reply
  • Granny - Thursday, October 03, 2013 - link

    You might want to monitor the freq while running sunspider and (surprise) after the benchmark is done. Reply
  • Thorburn - Thursday, October 03, 2013 - link

    Do you know for a fact Apple don't do this, or is it an assumption? How do you go about measuring it on iOS? Reply
  • kmj3943 - Thursday, October 03, 2013 - link

    Perhaps you're just another Apple fanboy that motivated your test? Reply
  • JJGotit - Friday, October 04, 2013 - link

    You can't prove Apple does or doesn't do this because their OS is "closed source" and you can't disassemble their code without being sued for sure. Reply
  • nicmonson - Wednesday, October 02, 2013 - link

    "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization."
    - Quote from the article
    Reply
  • nicmonson - Wednesday, October 02, 2013 - link

    Technically, they should have included NVIDIA too according too their own chart as one of the non-cheaters... but then again, they barely make a blip on the map right now. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    I didn't because I didn't go back and test the older Tegra 3 devices, but as you point out there's not a ton of Tegra 4 stuff out right now. Reply
  • inighthawki - Wednesday, October 02, 2013 - link

    Do you have access to any pre-release Surface 2's that you can run benchmarks on using Windows? It would be interesting to know the difference not just between hardware, but OS as well. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Not yet. I suspect any Surface 2 data will happen under NDA :) Reply
  • takeship - Wednesday, October 02, 2013 - link

    Technically, NVIDIA cheats just as much as everyone else through their "Plays Best" program, but they're open about it, and it usually improves actual game & app performance rather than just benchmarks i.e. Tegra exclusive graphics effects/extensions. So on the "cheating vs optimization" line they're much closer to the optimization side. Or cynically, they've learned from their GPU experience how to hide it better. Reply
  • watzupken - Wednesday, October 02, 2013 - link

    Personally, I see a difference between cheating and optimizing. Nvidia's GPU program would be what I think as an optimization since it improves performance, and everyone using an Nvidia card benefits from it when you play those games in the program. Samsung cheats cause they boost performance in benchmarks only, which does not benefit anyone at the end of the day. No doubt that this the full potential of the phone, but they made no effort to use the full potential of the phone on a day to day usage level. How many people actually buy a cellphone to bench everyday? Reply
  • klagermkii - Wednesday, October 02, 2013 - link

    There was quite of dumping on the Moto X for it's mid-range performance, now I wonder if that's just skewed perception based on the rest of the industry cheating. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    We didn't dump on the Moto X because of its performance. On the contrary, we showed why Motorola did extremely well having only two cores: http://www.anandtech.com/show/7235/moto-x-review/7 Reply
  • eiriklf - Wednesday, October 02, 2013 - link

    Mostly because everyone saw two cores and only a 720p screen and decided that was enough to call it mid range. Reply
  • Klimax - Wednesday, October 02, 2013 - link

    Just a thing, I don't think anybody yet produced evidence that Intel's compiler cheated in AnTuTu benchmark. All that was shown, was broken benchmark long before even compiler entered picture. (Frankly, benchmarks should be open unless a good reason can be provided why not, though no sure what reason could be) Reply
  • errorr - Wednesday, October 02, 2013 - link

    First is benchmarks aren't cheap to make and second if they were truly open then they are just that much easier to game. Also, what if the companies optimize the code for their devices and start arguing over which version of the test is the best written one. Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    Honestly I can't see what all the fuss is about. This not at all like the AnTuTu cheating which gave more than 2x speedup. We're talking about simply allowing very short duration benchmarks to run at the maximum frequency from the start rather than half-way through the benchmark. And that is a bad thing somehow???

    Let me tell you this: The first thing any benchmarking professional does is switch the CPU governor to performance mode. You want stable, repeatable results at a known frequency. I use a Chromebook, and have it switched to performance mode all the time (I do a lot of benchmarking at work). I can't see how one could ever consider benchmarking without that. So the governor "hacks" actually show the correct performance, and without it performance is slightly lower due to the benchmark being broken.

    Yes, so the real issue is the benchmarks themselves: they run for extremely short durations, there is no warmup run, no use of multiple runs and taking the fastest one. The graph for AndEBench shows it takes 30ms to run - that's MILLISECONDS. Anything that takes less than 5-10 seconds to run should not even be called a benchmark...

    Seriously, if you want to stop the practice, all you need to do is to stop using all those horrible micro benchmarks. Anand, why don't you start by not showing any benchmark which takes less than 5 seconds to run?
    Reply
  • The Von Matrices - Thursday, October 03, 2013 - link

    Thank you. I agree completely with your post. The duration of these runs is way too short. Unless this is a benchmark analyzing SoC turbo response, then they should be long enough that as few factors as possible can sway their score. Eliminate the micro benchmarks and these "cheats" that are described are completely avoided. Even looping a micro benchmark to extend its length is unrealistic because no phone/tablet usage scenario repeats the same miliseconds of code over and over indefinitely (unless the device is locked up).

    The only issue might be that these phone and tablet SoCs have a tendency to overheat and then throttle after a short duration running at full load. So which run would you pick a representative benchmark score in a set of multiple runs where thermal throttling causes the performance to keep decreasing with each successive run? It's really part of the reason why benchmarks aren't directly translatable to real usage, and there should be more emphasis making benchmarks of realistic length and scope compared to complaining that the current benchmarks are being "cheated".
    Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    True overheating is an issue, especially on phones. Still I don't think a few seconds will be an issue, but you certainly wouldn't be able to run something for minutes without getting throttled. So you could run a single benchmark and insert a delay before the next run to cool down again, and then take the fastest of several runs.

    In normal typical usage you wouldn't expect any throttling, certainly not on lightly threaded workloads. So I'm not so sure it would be reasonable to let a benchmark throttle and use that as a real score. Also different manufacturers may use different temperature settings even when using the same SoC, so then becomes a matter of how close do they let it get to actually melting or giving burns...
    Reply
  • neaoon - Monday, October 07, 2013 - link

    I just want to add my story. I get paid over $87 per hour working online with Google! I work two shifts 2 hours in the day and 2 in the evening. And whats awesome is Im working from home so I get more time with my kids. Its by-far the best job I’ve had. I follow this great link >>>> www<DOT>BAY89<DOT>COM Reply
  • darwinosx - Tuesday, October 08, 2013 - link

    It says in the article you are commenting on but didn't read that Apple doesn't do it. Reply
  • dishayu - Wednesday, October 02, 2013 - link

    Hilarious heading for the table. :D

    As Anand said in an earlier podcast, the PC industry giants seem to have grown out of "cheating" in benchmarks whereas the Mobile players insist on repeating the same mistakes.
    Reply
  • jgrnt1 - Wednesday, October 02, 2013 - link

    I'm sorry, but this article does not begin to fix the issues of your Note 3 review. Optimizing for particular programs or games is often done to make those particular programs run better. "Optimizing" for a particular set of benchmarks is done only to make a product look better than its competitors, when real world performance doesn't show the same. It is cheating, plain and simple. You need to call it that, call it out loudly in the review, and show adjusted results. The only way to stop this behavior is if there is enough bad press about it. Instead, the results you put in your review will further enable this abhorrent behavior, because the article will be read by many who are not aware of the issues which make the results worthless.

    Fix the review and save what's left of this site's integrity.
    Reply
  • smartthanyou - Wednesday, October 02, 2013 - link

    Well said.

    Personally, what I think should be done is when I device is caught cheating, no benchmarks of any kind should be published with the article. Take them all out and explain in the article why.

    Unfortunately, AT and sites like it lack the courage to do such things.
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    I can remove the impacted graphs, there's no courage required. I just didn't want to mask what was going on.

    If the readers prefer that we remove the affected graphs I have no issues in doing so, I just preferred to be up front/transparent about the whole thing here.
    Reply
  • ASEdouardD - Wednesday, October 02, 2013 - link

    Personally, I'd prefer that you keep them in. We can always go back to this chart to try to figure out what the real numbers are.

    To put it more clearly : Either update them all (which takes way to much time) or keep them in. Often, even with cheating, the results are still informative (as a 5% difference between devices really is insignificant in day to day use).
    Reply
  • Klimax - Wednesday, October 02, 2013 - link

    No. People will often miss context, miss what went on and thus will be unaware that they are looking at cheated benchmark.
    At bare minimum those graphs need to have big bloody note that result is obtained under unrealistic benchmark specific conditions, if you re not going to fix them properly like rerunning renamed benchmarks. (With including clearly marked results from both runs.)
    Reply
  • Death666Angel - Wednesday, October 02, 2013 - link

    Yeah, we should all cater to the lowest and stupidest among us... ugh.
    Keep the reviews as they are, they are perfectly fine.
    Reply
  • Drumsticks - Wednesday, October 02, 2013 - link

    Agree with angel. Keep them in, put an asterisk or something even more noticeable if necessary, but don't remove them. There would be nothing left to benchmark, and reviews would be nothing more than a glorified hands on. Reply
  • Klimax - Thursday, October 03, 2013 - link

    Note: I didn't call for removal, I called for updated or at least clearly marked. (If not enough time)

    Also I have seen number of times direct link to graph and nothing else. (Not hotlink)

    And it is not catering to stupid. It's doing bloody hell your job properly. (BTW: That was in itself very stupid remark, because it is not true nor applicable and uses more assumptions then is healthy)
    Reply
  • CBone - Wednesday, October 02, 2013 - link

    Keep them in. The people that care about mobile device benchmarks enough to come here and read the graphs know what's going on. As long as they are running the same tests as the competition for the review, the scores are valid. This isn't baseball. No one cares what an old phone with different hardware, different drivers, different OS version, and probably a different bench version benched at back in the day. There's no Hall or record book that anyone would care about. There's no reason to salt the earth over it. Reply
  • DanNeely - Wednesday, October 02, 2013 - link

    i vote for either removing them or indicating that the devices are cheating on the graph itself. My ideal option would be to list both the cheat and non-cheat scores for the device being reviewed and non-cheat numbers for the comparison devices.

    It was the fact that, aside from a single sentence which implied it wasn't a big deal, there wasn't any indication of it in the GT3 article that had me incensed yesterday.
    Reply
  • Sushisamurai - Thursday, October 03, 2013 - link

    I vote a systematic deduction for all cheaters for such benchmarks, as this situation is dire and requires a more offensive approach for change to occur. As for what viewers want... Use a poll! Reply
  • SophieD - Thursday, October 03, 2013 - link

    I'd prefer the result left in, with overlay text like "cheating; uncorrected" (if the cheat was not worked around) or "cheating; correct" (if the cheat was undone). Reply
  • ESC2000 - Tuesday, October 08, 2013 - link

    Keep them in with an asterisk. I think you've handled this issue well, so don't listen to the naysayers who are claiming you lack integrity. Reply
  • identity - Wednesday, October 02, 2013 - link

    I don't get you people. You bitch and moan when Anandtech doesn't mention the boosting in the review. When they released this, you complain and moan even more. Anandtech, just keep doing what you're doing. Screw the haters who'll never be satisfied no matter what you do to improve your benchmarking and site. Reply
  • tryptsoft - Wednesday, October 02, 2013 - link

    I couldn't agree more, Anand is one of the good guys here. Reply
  • Death666Angel - Wednesday, October 02, 2013 - link

    +9000! :D Reply
  • flatrock - Wednesday, October 02, 2013 - link

    I strongly disagree with this. I think both the falsely optimized and more reliable benchmark data should be provided so people can see what is going on and make up their own minds about it.

    It shouldn't be up to this site or other such sites to censor things out when a company does something they don't like. Instead they should present all the facts along with their opinions. The facts so that people can make up their own minds about the issue, and their opinions because their opinions also have value. But they always have more value when they are backed up with facts.
    Reply
  • errorr - Wednesday, October 02, 2013 - link

    Well I rather have the info than not. Also, Anandtech is a review site. They need to make money and drive traffic. A lot of us are interested and loyal readers but if somebody less savvy is looking for info they will just end up at a website with inferior testing if Anand doesn't weigh in. The manufacturers would probably prefer to deal with the crap testers than with the sites that know what they are doing. Reply
  • JumpingJack - Saturday, October 05, 2013 - link

    "Personally, what I think should be done is when I device is caught cheating, no benchmarks of any kind should be published with the article. Take them all out and explain in the article why."

    Why on earth would you want that.... the moment a site states something along the line "the benchmark data is tainted by special boosting mechanisms that does not represent the capability of the device, there fore we will not show the results" next thing you know is a fanboy like you will scream "where's the proof"....
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    I updated the Note 3 piece when I published this, pointing out where cheats were active, who cheated, and the magnitude of the cheats.

    We've also been working with literally all of the impact benchmark developers to fix this going forward, at least as far as our suite is concerned.

    I disagree that bad press is the only way they're going to stop. We did that in July, will keep doing it, but that alone won't change things. We need a technical solution as well, which is what we're doing.
    Reply
  • melgross - Wednesday, October 02, 2013 - link

    A problem is that the mainstream computer sites such as Pcmag, which have far more readers, and c/net, etc. just do their skimpy tests, and don't know about these problems, or just don't care. It's hard to fight that, especially as Android, as aw hole, and Samsung in particular, seem to be the favored OS and companies these days. Reply
  • muyaad - Thursday, October 03, 2013 - link

    Cnet has reported about Samsung cheat again and that was yesterday but I don't think that have posted this new finding yet. Reply
  • tryptsoft - Wednesday, October 02, 2013 - link

    Thanks for your efforts and opening up this to the masses. Thinking back to the SSD trim article of years past, I love when you do an in depth piece that really opens up the story of whats going on. Reply
  • SophieD - Thursday, October 03, 2013 - link

    A single spike of bad press won't dissuade them, but a consistent pointing out of cheats might. If your (highly regarded!) reviews and graphs clearly indicate cheaters as such, over time that knowledge will diffuse throughout the wider tech "review" world, and to the wider public from there. That state, I think, will dissuade the cheaters. Reply
  • willis936 - Wednesday, October 02, 2013 - link

    This is almost big enough to make a front page article rather than a pipeline story. OEMs should get as much public flak as possible for bad marketing decisions. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Done. Reply
  • RRFuze - Wednesday, October 02, 2013 - link

    If it is Apple who got caught about cheating, it will be the biggest news in the world for whole month. Media won't report anything thing bad about Androld or Samsung. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    We don't play those games :) Reply
  • Tetracycloide - Wednesday, October 02, 2013 - link

    You can't be serious... Reply
  • CBone - Wednesday, October 02, 2013 - link

    Dude. Apple and Samsung are the only makers that really move the needle. Everything they do gets coverage! If it's "bad" they get more! Reply
  • steven75 - Wednesday, October 02, 2013 - link

    How many people in the US (for example) know the current CEO of Samsung was convicted of bribery and embezzlement.

    Now how many people do you think would be aware if Tim Cook or Steve Jobs was convicted of those same things?

    That's what I thought.
    Reply
  • CBone - Wednesday, October 02, 2013 - link

    How many people in the US care about a foreign CEO vs an American anything?

    That's what I thought.
    Reply
  • muyaad - Thursday, October 03, 2013 - link

    but many Americans care about foreign products.
    If they care about Sammy devices they should care about Sammy CEO and if that isn't the case then something is wrong with the media.
    Reply
  • SophieD - Thursday, October 03, 2013 - link

    I don't know.

    But many Europeans (myself included) are quite aware that Steve Jobs was suspected (not even convicted!) of wrongdoing regarding the dating of some of his options. OTOH, if I was aware of Samsung's CEO felonies, I had apparently forgotten about it.
    Reply
  • willis936 - Wednesday, October 02, 2013 - link

    Qualcomm is bigger than you think. They never show up in consumers minds but that doesn't mean they don't have a lot of weight in the industry and they are perhaps the least afraid to throw it around. What they lack in vertical integration they more than make up for in sheer volume. Reply
  • haruhiko - Wednesday, October 02, 2013 - link

    This happens here in Hong Kong too. Any "bad" news about Apple will be put to the front page of "International News" section of popular newspapers and Samsung on the other hand is literally untouched. I don't know whether it's because Apple news generates a lot of sales or clicks, or simply because Samsung is one of their biggest client (advertisement). Reply
  • muyaad - Thursday, October 03, 2013 - link

    I can't agree more but the reason is Apple is in the central of media, whatever Apple does got popularized whether it's bad or not. Apple really drives attention more than any mobile company. Reply
  • akdj - Wednesday, October 16, 2013 - link

    "Apple really drives attention more than any mobile company." Apple is one of the biggest, most profitable companies in the world. It single handedly changed mobile communication and brought tablet computing to the masses. Their phones continue to outsell the competition. As well as it's past flagship. They're kicking ass right now. That'll typically garner 'attention'. While Samsung makes phones...and some good ones, they also make refrigerators, TVs, blenders and microwaves...the list goes on. I think recently they may have done a bit better branding themselves as a 'player' (heavy advertising) but Apple continues to lead the lack (by a significantly large margin) when it comes to profit, mystique and innovation. Those are the facts (I also own both Android and iPhone/iPad) Reply
  • gregounech - Wednesday, October 02, 2013 - link

    Very good job.

    The name of the cheating table made me cry and laugh at the same time :)
    Reply
  • rhughesjr - Wednesday, October 02, 2013 - link

    Ha! Great caption on the first table. :) Reply
  • zanon - Wednesday, October 02, 2013 - link

    The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell.

    Anand, regrettably I'm not sure that your contention that this is without value is true. Looking at your own charts there, in many instances 5-10% would be plenty to swap positions at the top of the chart, and that might well have value. On GLBench for example, it'd be plenty to push the Galaxy Note 3 above the iPhone 5S, and "We're number one in performance, the iPhone is relegated to second" is marketing that could resonate, even if the actual difference is tiny and even if it's due to cheating. There's plenty of real psychology when it comes to "1st" vs "2nd".

    Of course, I suppose that just makes vigilance all the more important, but as part of your efforts you should consider that they may not agree with you that it simply isn't worth the trouble on its own merits. In turn, shaming (along with the continuous benchmark evolution you discuss) would have some value even if it's not enough by itself, as it somewhat counters the marketing value of the practice. Technical solutions matter the most as part of changing the cost/benefit equation, but I don't think you'll necessarily simply be able to appeal to their better natures if they genuinely do think it's helping their sales pitch.

    Anyway, thanks for the investigation into this as always. Fighting the good fight is always an ongoing thing but it's appreciated.
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    I look at the shame detriment vs. performance benefits and I can't see any of this being worth it, but I recognize your point. Regardless, the right solution is just to make the suite untouchable - at least by the easily detectable, low-hanging fruit. And that's nearly all complete at this point. Reply
  • dugbug - Wednesday, October 02, 2013 - link

    "I look at the shame detriment vs. performance benefits and I can't see any of this being worth it"

    That's because you are unfortunately bound by ethics. Think of someone that works in marketing :)
    Reply
  • GauravDas - Thursday, October 03, 2013 - link

    Haha. Best comment ever. Reply
  • admiralpumpkin - Wednesday, October 02, 2013 - link

    Anand,

    I don't understand why you are willing to post "gamed" benchmarks in your reviews. These OEMs should get an NA that shows an inability to get an accurate result.

    Your reviews should only post accurate numbers.

    What if an OEM was able to use GPS to determine your location and modify battery behavior so that it appeared to get amazing (but gamed) battery life? Would you still post this erroneous data?

    Basically, why are you posting inaccurate data when you now know it's inaccurate?

    In my opinion, all gamed data should be retracted in all reviews.

    You have the most rigorous and thorough tech reviews on the web, knowingly leaving this bad data in undermines all that you do here.
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    We're transitioning to that exact point. There's a lot of historical data out there, changing it all is unfeasible but we can at least re-run everything that is gamed, repopulate the database and clean things up. Reply
  • ciparis - Wednesday, October 02, 2013 - link

    How about loading up the Note 3 charts in Skitch, and slapping a nice red "CHEATING" on all of the graphs where they got to the top through this nonsense? Reply
  • admiralpumpkin - Wednesday, October 02, 2013 - link

    I'm glad to hear that, but I still feel like your response has been fairly analytical, i.e. "let's see just how much these optimizations affected things"; as opposed to big picture: "these OEMs are trying to undermine PRECISELY that which makes my reviews valuable". Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    I'd hope that what makes our reviews valuable is more than our ability to run a few benchmarks ;)

    I know how to work around these cheats, these ones are simple and we're almost totally protected now going forward (at least from this stupid optimization, the ones to come will be *much* worse).

    The problem is all of this was a known quantity to us before we published in July. There's literally nothing new here. What I needed to quantify was how much the stuff impacted our suite since there was concern that we were talking about huge deltas in performance. Since that's what was new, that's what I focused on.

    Is this dumb? Yeah I hope I've called it that enough in the text, but it's no longer an issue. This optimization is dead as far as we're concerned. It barely impacted our CPU tests to begin with and will impact literally 0 going forward.

    We're in a tricky spot too. Two things will happen if you make a huge stink about it: 1) they'll stop, or 2) they'll get better at hiding it. I hoped for the former, but it's clear the latter is what will happen. I didn't want to force that hand until we had a better collection of tests to move to, but here we are.

    I've been engaging a few parties over the past few months with the hopes of building a real, cross-platform suite of our own control. It's expensive, and not everyone wants in, but that's an option I'm still exploring.
    Reply
  • Tetracycloide - Wednesday, October 02, 2013 - link

    I'm not sure I'd agree that the benchmark comparisons are what makes their reviews valuable. There's so much more in the review articles beyond the charts. It undermines the charts but benchmarks are a pretty small part of the review picture on handheld devices. Reply
  • admiralpumpkin - Wednesday, October 02, 2013 - link

    Tetra, I agree. I overstated that point a bit.

    For me, the deep technical knowledge and rigorous evaluation procedures of Anand (and his team) are what draw me to this site and reviews. These select OEMs are attempting to undermine _part_ of the objective evaluation—the benchmarking—of their devices.
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    If you only knew all of the stuff that we deal with that doesn't get published :-P Literally everyone is dirty/shady in some way.

    I try to keep things pretty drama free here, this isn't a tabloid and we don't sell based on sensationalism. I want to keep it that way. If a manufacturer oversteps their bounds I try to solve the problem first, anyone can raise a stink - I prefer solutions. I turn to the power of you all if I can't get things done on my own.

    I remember telling someone this: you can raise hell online, but I'd much prefer calling up the person in charge of making these decisions and getting them to change. The former is fun, but the latter can typically get quicker and more meaningful results. The question is do you care about the drama or do you care about the problem being solved? In the media there's a preference for the former as you can drive page count there. I don't consider us journalists (the term has a negative connotation in my mind), journalists are out for the story, I just want to make things better.
    Reply
  • charleski - Wednesday, October 02, 2013 - link

    Anand, do you honestly think Samsung is going to change if you have a quiet word with them?

    There was a lot of talk after you exposed the cheating on the S4 back in July. But here we are in October and they're still doing exactly the same thing. They know they can get away with some critical comments on enthusiast sites and I see no evidence that they care. They're going to keep doing this until the noise gets loud enough that it percolates through to the mainstream press.

    'Everyone is dirty/shady' is no excuse for keeping quiet, even if it were true. I'm rather concerned at that sort of cynicism.
    Reply
  • gorash - Wednesday, October 02, 2013 - link

    I think that the journalists are mostly fine... It's mostly the editors who F things up. Reply
  • SophieD - Thursday, October 03, 2013 - link

    If "literally everyone is dirty/shady in some way", I'd love to understand each case. Ethical considerations do guide some of my buying decisions, and I'd prefer not to unwittingly encourage misbehavior with my wallet. Reply
  • Beautyspin - Friday, October 04, 2013 - link

    Seriously, did you find any of these bugs when you reviewed the iOS7 "So extensively"..Not a single one..

    http://www.phonearena.com/news/iOS-7-a-bug-infeste...

    And you still want people to believe you when you say Apple does not cheat in the benchmarks without any proofs?
    Reply
  • ESC2000 - Wednesday, October 09, 2013 - link

    I don't think it's possible to definitively prove apple isn't cheating because there are so many possible ways it could be happening that it would take a prohibitively long time to check them all, especially since apple is vertically integrated which gives it more opportunities to cheat. Reply
  • PC Perv - Wednesday, October 02, 2013 - link

    You forgot to bold "Y" of Galaxy Tab 10.1 (Atom Z2560) / AndEBench. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Fixed, thank you :) Reply
  • noeldillabough - Wednesday, October 02, 2013 - link

    Bravo! I was expecting this article, given your comments over the last few weeks but this is better than I expected! Nice that you expose the cheaters, or make everyone cheat the same so relative is relative. Reply
  • tential - Wednesday, October 02, 2013 - link

    I was looking at the Galaxy 10 2014 edition and I was like "WOW these benchmarks are INSANE!!!!!"

    Then, I see the review and it's laggy/slow (on engadget haven't read it on Anandtech yet). This article sums it up perfectly though. This is annoying now because I LIKE to have numbers and now I can't have number evidence.
    Reply
  • nerd1 - Wednesday, October 02, 2013 - link

    You just cannot trust any review from engadget/gizmodo/verge etc. Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    Add anandtech to list Reply
  • Krysto - Wednesday, October 02, 2013 - link

    Yeah, Intel is not cheating in benchmarks. They're doing it right in front of you with stuff like Turbo-Boost and SDP measuring. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Turbo isn't a cheat, we went through this back in the Lynnfield days - there are years of evidence pointing to turbo being attainable/usable in real world cases (monitor turbo frequencies while using any Intel machine, you'll see them regularly hit regardless of workload). Literally all CPU vendors are at varying stages of pursuing similar opportunistic clocking based on thermal conditions, AMD and Intel simply have the most experience here.

    SDP/TDP is something that only impacts the OEM, not our tests or end users.
    Reply
  • dylan522p - Wednesday, October 02, 2013 - link

    If turbo is cheating then every ARM vendor is cheating as they are effectively advertising the boost clock. The S4 is cheating anytime it goes above 925mhz because that is it's nominal clock. Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    ARM CPUs do not have a turbo mode or a nominal clock. Eg. Exynos 5250 varies between 200MHz and 1700MHz depending on the load. An OS could force it to a nominal clock of course, but that is less efficient than lowering clock all the way to the minimum if the load is low.

    The issue with Intel's Turbo mode is not so much that it increases frequency, but that it actually increases the TDP as well, and thus cannot be sustained for long periods. It makes Intel's marketing claims very dubious as they never mention that the TDP is increased significantly during Turbo mode and often cite the Turbo speeds only. Basically it allows them to claim a low TDP at an artificially low "nominal" clock and then boost the hell out of it at a much higher TDP. And they hope that people will believe that you can run at that maximum boost frequency at that low TDP. Forget it!
    Reply
  • ezorb - Wednesday, October 02, 2013 - link

    Can you add the shield to those charts, you mention it for not cheating and it would be interesting to see how an unconstrained device that is thermally unconstrained compares. Reply
  • Da W - Wednesday, October 02, 2013 - link

    " nearly almost all Android OEMs are complicit in creating this mess"

    That's it. I'm getting a Windows phone!
    Reply
  • Braumin - Wednesday, October 02, 2013 - link

    At least you won't have to worry about your phone being reviewed on AnandTech! Reply
  • melgross - Wednesday, October 02, 2013 - link

    Or have anyone read the review in case it is. Reply
  • ESC2000 - Wednesday, October 09, 2013 - link

    I would read it. I'm actually excited for the Nokia tablet coming out Oct 22. Reply
  • A5 - Wednesday, October 02, 2013 - link

    Have any WP devices come out besides the fleet of Lumias? Reply
  • Braumin - Wednesday, October 02, 2013 - link

    Are the fleet of Lumias not enough? Apple repackaged last year's iPhone into a plastic shell and it got a full review... Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    I have not read the review of iphone5c.did they excellent score of new innovation .aka colors Reply
  • krumme - Wednesday, October 02, 2013 - link

    Great work guys. Keep pushing!

    We are supporting you.

    (Lads push the adds)
    Reply
  • foxalopex - Wednesday, October 02, 2013 - link

    I have a bad feeling this isn't going away because your typical consumer doesn't care as much as we would believe. Most consumers are not interested in benchmarks but rather in situations such as will this game play, will this movie play or can it surf the web at reasonable speed. Features drive Cellphone sales more than performance does. Until smartphones become high performance computing platforms, I'm not sure anyone will be really interested in the benchmarks except for top tier users like us. Reply
  • SZeal - Wednesday, October 02, 2013 - link

    How about if these performance benchmarks took into account power consumption. Given that most of the platforms are running same hardware, any performance / watt will expose this cheating. Thoughts? Reply
  • MFK - Wednesday, October 02, 2013 - link

    Anand.
    I've been following your site since the very early days.
    I think the first review I read on your site was of the Geforce 200 MX.

    I must say that you've been very accommodating of your readers in the way you have handled the feedback to this issue. I think this is what keeps me coming back to this site.
    But I would also like to caution that you mustn't do everything that the readers demand.
    Your own direction is also what we come here to see.

    So in conclusion, just keep up the good work and may this site receive my patronage for 15 more years.
    Reply
  • nerd1 - Wednesday, October 02, 2013 - link

    This kind of article makes the anandtech shine - and it is actually quite funny that ars techinca blamed note 3 for 'cheating', comparing it with G2 (which also cheats :D ) Reply
  • CBone - Wednesday, October 02, 2013 - link

    Ars didn't go the next step before blasting Samsung. Anandtech did. Reply
  • WaltFrench - Wednesday, October 02, 2013 - link

    “This also helps explain why the Nexus 4 performed so slowly when we reviewed it - this mess was going on back then and Google didn't partake.”

    “…it’s almost impossible to get any one vendor to be the first to stop.”

    So maybe those 5%–10% gains aren't so trivial, after all. We could be talking about many hundreds of millions of dollars of revenues shifted from Vendor A to Vendor B, simply because all else equal, B looked faster.
    Reply
  • FunBunny2 - Wednesday, October 02, 2013 - link

    How do you "rename" the benchmarks? I assume that you parse the object code? Then what? Replace "Foo" with "Bar"? Which is to ask, a smart cheater will know where, and what strings, to look for. That should also mean there has to be a known non-"Foo" value one can substitute? Is it "Bar"? How do you know? Reply
  • klagermkii - Wednesday, October 02, 2013 - link

    Appreciate the effort in putting this together and for shining more light on it. I know that in the past I've looked at benchmarks and thought the Nexus devices were slower than other vendors, and while it isn't a single overriding factor, performance is a consideration in what I buy. At least now I know I can't trust Samsung vs Nexus benchmarks. Reply
  • Thermogenic - Wednesday, October 02, 2013 - link

    This really makes the consistent wins for the iPhone 5S all the more impressive. Apple seems to be quite a bit ahead of the other ARM vendors, which is pretty impressive. Reply
  • ESC2000 - Wednesday, October 09, 2013 - link

    The 5S may be faster (that is one thing I remember from my iPhone 5 days: it was generally fast) but what it has in speed is overshadowed by what it lacks in features. You can't use that speed to do much! Reply
  • akdj - Wednesday, October 16, 2013 - link

    Interesting. I don't think I've found anything I can't do (fast) on my 5s. Features? Lacking? I've got the Note 2 and can't get out of the contract quick enough. The speed of the 5s is extremely evident. Apps. Software. That's what these are for. That speaks volumes when it comes to 'features' Reply
  • Klimax - Wednesday, October 02, 2013 - link

    Interesting, it looks as if all Asian makers do it. (Asus, LG, Samsung, HTC) while sole NA maker Moto is out. (I don't think there are any left in Android space anyway.)

    Question is, do Chinese do this as well? (My prediction based on pattern would be yes...)
    Reply
  • CBone - Wednesday, October 02, 2013 - link

    Check out mikecanex.wordpres.com. The chinese tablet and phone makers are shameless in their cheating. Reply
  • CBone - Wednesday, October 02, 2013 - link

    That's mikecanex.wordpress.com. Need an edit button. Reply
  • Klimax - Thursday, October 03, 2013 - link

    Any particular post? He got quite few of them... Reply
  • gorash - Wednesday, October 02, 2013 - link

    Could it be because those countries have only recently emerged as developed countries, so the laws against false advertising aren't as stringent? (Korean and Taiwanese). But Anand said "pretty much all OEM cheated". Which companies didn't? Reply
  • Klimax - Thursday, October 03, 2013 - link

    Moto apparently. (Based upon table, though not complete obviously)

    Now, follow up would be tablets. (You get more players there like Dell, Microsoft, ...)
    Reply
  • xodusgenesis - Wednesday, October 02, 2013 - link

    Thank you Anand and Brian for a great article. All of this just makes Apple's A7 seem amazing performance wise. If Android OEM's spent as much effort optimizing silicon as they did trying to optimize benchmarks, they would probably be neck and neck in performance. Reply
  • Gravy Jones - Wednesday, October 02, 2013 - link

    You guys sure know how to go out of your way to call overclocking something else like "Raising the thermal limit" or pushing the voltage/frequency limit. I get a good hearty laugh out of it because it is smoking the current offering of Apple. You would laud other platforms, like the Intel Core-I series for having adaptive overclocking baked into the system. Being able to overclock certain applications is a real dream, if you ask me. Sometime battery life has to play second fiddle to raw power. I use a program that was originally for iOS but now for Android called Sky Safari Pro and though it works fantastically on my S4 and Nexus 7, just being able to overclock it would be something I'd like to try. Cheers. Reply
  • CBone - Wednesday, October 02, 2013 - link

    ""With the exception of setting mandatory parameters specific to systems with multiple GPUs, such as AMD CrossFire or NVIDIA SLI, drivers may not detect the launch of the benchmark executable and alter, replace or override any parameters or parts of the test based on the detection. Period."

    If I'm reading it correctly, both HTC and Samsung are violating this rule. What recourse Futuremark has against the companies is up in the air, but here we at least have a public example of a benchmark vendor not being ok with what's going on."

    I don't think you are reading that correctly. They aren't changing the test, that phrasing is from the Nvidia 3Dmark cheating where they weren't rendering correctly.
    Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    I was being a little coy there. Futuremark considers this practice to be in violation of that clause, although it was originally from the driver based cheating. Reply
  • thunng8 - Wednesday, October 02, 2013 - link

    Your conclusion says the CPU cheats amount to 5% or less. Arstechnica found it was 20% on Geekbench3 on the Note 3 and up to 50% on linpack. A not insignificant number. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    All I can speak to is how we test and what we test :) Reply
  • Deelron - Wednesday, October 02, 2013 - link

    And even 5% can cause a switch in the rankings, making one "win" over another. Clearly there's more value to cheating then not financially, otherwise they wouldn't risk the negative press. Reply
  • eiriklf - Wednesday, October 02, 2013 - link

    Anandtech doesn't use geekbench or linpack to compare devices AFAIK, so in that sense it's not one of their "CPU tests".

    I do think this raises some concerns for the scores they have shown from geekbench, however.
    Reply
  • Zink - Wednesday, October 02, 2013 - link

    Thanks. Leaving the Note 3 review how it was would have set a very bad precedent. Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Bingo! I had a whole section about the embarrassment that is software DVFS before culling it to keep the whole thing manageable. Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    Actually the real problem is the extremely short duration of the micro benchmarks, not the DVFS. Proper benchmarks run for more than a few milliseconds. Reply
  • Mugur - Thursday, October 03, 2013 - link

    I would have loved to see that... That's even an issue with say SpeedStep and Xeon cpus on server loads, I remember of reading an article about that some time ago and I see it's a bigger issue in mobile space now.

    All in all, the benchmark "optimizations" pointed out this real situation.
    Reply
  • speconomist - Wednesday, October 02, 2013 - link

    What is BSG? Reply
  • Anand Lal Shimpi - Wednesday, October 02, 2013 - link

    Battlestar Galactica, a reference to the terrible ending to a show that started out so very well :) Reply
  • speconomist - Wednesday, October 02, 2013 - link

    Thanks... Should have known. Reply
  • Roland00Address - Wednesday, October 02, 2013 - link

    Thank You Reply
  • eastyy123 - Wednesday, October 02, 2013 - link

    i know with android probably not as much choice when it comes to benchmarks but always felt with PC they should really get rid of synthetic benchmarking and just concentrate on real world applications like games or creation software as i don't really care for example how well a cpu does in 3dmark but would be very interested in how it runs the witcher 2 or handbrake Reply
  • darkcrayon - Wednesday, October 02, 2013 - link

    Good point. In any case Anandtech should try adding a few productivity app tests, especially since we actually have lots to choose from now. Might not help as much with cross platform tests, but should be really useful in-platform. Reply
  • Deelron - Wednesday, October 02, 2013 - link

    Most of the time on PC side there are plenty of less synthetic benches, the problem is with the mobile example they would just whitelist those applications and it'd be the same end result. Reply
  • jbelkin - Wednesday, October 02, 2013 - link

    BTW, Samsung is claiming that it's overclocked when apps require it - nevermind that if you change the benchmark names, Samsung phones do not trigger the overclocking. Perhaps Samsung should spend more resources on improving their phones by offering OS upgrades or other such nonsense ... but I guess a programmer can get a raise for improving test scores 4% versus some other non quantifiable measurement ... it's also hilarous the people defending samsung with 'it's no big deal, no one cares about benchmark scores.' Nice ethical arguments. What's next? It's only a small battery fire? It's not like it's gamma radiation ... Reply
  • CBone - Wednesday, October 02, 2013 - link

    I can't imagine that setting up the lists took more than a day's work by a competent engineer.

    The line is pretty simple. Benchmarks => no one gets hurt or maimed or disabled. Don't get absurd with it.
    Reply
  • BC2009 - Wednesday, October 02, 2013 - link

    Anand, what about Sony? Reply
  • ZeDestructor - Wednesday, October 02, 2013 - link

    Still nothing (I've been asking about Sony hardware since the G2 review). Looks like Sony is still not providing them any devices...

    That aside, I'm happy to have a go if I can get access to both the unmodified packages and the renamed packages (or a nice guide to rename them)
    Reply
  • itpromike - Wednesday, October 02, 2013 - link

    Anand, I know you're busy and I'm not even sure if you're still monitoring the comments in this article anymore but would you be open to re-running all the benchmarks with the cheating code disabled? Or at least run them for the top 10 or 15 devices... GS4, Note 4, HTC One, G2 etc...? I have both android and iOS devices but I'm a big fan of the engineering skill of Apple when it comes to leer frequency chips outperforming much higher freq chips. I'd love to see the 5S in a chRt with the top phones with all the cheats turned off as an example of what can be done with honest to goodness engineering know how and hard work without cheating. Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    How we turn cheat off on iPhone..closed os Reply
  • PeteH - Wednesday, October 02, 2013 - link

    Did you miss the part where Anand specifically cited Apple and Motorola as the only two OEMs that never cheated? Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    Ahem, we don't know that at all! Another shoddy conclusion not based on facts. Yes Anand couldn't find it, but that doesn't prove anything. They may recognise benchmarks in less obvious ways - there are a million ways to detect a benchmark binary without looking at its name. And I bet this is exactly what all future devices will do.

    Note it's not cheating - the real issue is that the benchmarks do not run long enough to be seen as a real workload by the OS. So the boosting is a workaround to fix broken micro benchmarks which should never have been written, let alone used widely.
    Reply
  • Graag - Thursday, October 03, 2013 - link

    This is just another case of Apple Derangement Syndrome. "The fact that we have seen no evidence that Apple is cheating doesn't mean that Apple isn't cheating." Followed by, "Also, this isn't really cheating."

    Seriously?
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    Ever heard of logic? "I couldn't prove X, so !X must be true" is incorrect. The hard fact is that we simply don't know whether Apple is "cheating" or not. Whether you consider boosting frequency cheating or not is an orthogonal issue altogether. Seriously, is the IQ of people getting lower all the time these days? Reply
  • SpacedCowboy - Sunday, October 06, 2013 - link

    Ever heard of reading ? Towards the beginning of the comments, Anand points out that he has tools that show the frequency-scaling characteristics of the A6/A7, and those tools do not show the cheating behavior that Samsung et al use.

    And yes, it is cheating. Despicable scum-of-the-earth cheating.
    Reply
  • akdj - Wednesday, October 16, 2013 - link

    Why do you continue to respond and declare this to NOT be a 'cheat'? Anand specifically talks about working with other engineers and poking and prodding the a7 for cheats. There aren't any to be found? Is that not enough for you? Why don't YOU set out to prove Apple is cheating? Where is YOUR IQ? Seriously. You're the ONLY respondent that's convinced Apple is tricking you. WTF? Are you here as a paid shill from Android? Samsung? Good lord. Let it rest. Dollars to donuts, Anand and Brian's individual IQ's, knowledge and experience far out weigh your ridiculous drivel.
    They're cheating. Pure and simple. Cheating....look it up. The definition. It's EXACTLY what they're doing and in some cases, 5-10% will indeed move that specific piece of gear up a notch or three in the 'charts'
    You're a trip man. It's amazing your incredibly obstinate view on this subject. You're alone bud.
    Good luck though. Cynicism in life sucks. Bad way to live your life
    Reply
  • ExodusC - Wednesday, October 02, 2013 - link

    This is another reason I don't even care to look at silly benchmarking applications on mobile devices. I care almost exclusively about real world speed and fluidity. Anand, you've talked about your philosophy of a mobile device being an appliance as opposed to a computing platform. I think I lean mostly in that direction. I don't care about mobile gaming performance and such (at this point in time). I just want a device that works well, is fast in general applications, has good battery life, a big screen, etc. Reply
  • errorr - Wednesday, October 02, 2013 - link

    Just want to reiterate that I found it amazing that Anand predicted this exactly in the podcast. I was bored a while back and was listening to the older podcasts around the time this stuff came to light.

    On episode 15 at 1:04:58 Anand says: "Is this when they start to do app detects and wait for someone to fire up Antutu or something and, and then just go all A15s?" While he and Brian were discussing the 5410.

    The sad part was that this was so predictable in a way. While specs don't matter much in the US the spec sheet is vital in certain markets like China. The Pentium 4 space heater is still a formative experience for me and this all seems as dumb as quak 3 was at that time.
    Reply
  • Quizzical - Wednesday, October 02, 2013 - link

    This is why you measure performance and power consumption (sometimes indirectly, via battery life) in exactly the same benchmark. That way, a vendor that cheats at one will get artificially worse results in the other.

    Or rather, most tech sites don't do that. But I think you should. If vendors know that benchmark A will always be used to measure performance only and never power consumption or battery life, while benchmark B will always be used to measure power consumption or battery life but never performance, then it's easy to cheat.

    Trying to shame vendors isn't going to work unless the attempts at shaming them are more prominent than your reviews. Posting benchmarks that make their hardware look bad because you're using the benchmark for the opposite of what the cheaters expected as your official benchmarks on the device has a better chance of working.
    Reply
  • jwcalla - Wednesday, October 02, 2013 - link

    I keep telling you people not to take benchmarks seriously. Most of what is benched here is testing software and compiler differences. You can't benchmark hardware accurately unless the software stack is identical across the board.

    I know, I know... I'm wasting my breath.
    Reply
  • tk11 - Wednesday, October 02, 2013 - link

    So what... This has been as issue since artificial benchmark scores first became valuable as promotional material. As always it's the responsibility of the benchmark software to produce results consistent with real world performance. Maybe it's time for Anandtech to drop some of the benchmarks that fail to produce results that are consistently indicative of real world performance. Calling out anyone other than the makers of the benchmark software is never going to have any effect on the situation. Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    I don't think Anand wants to drop all his beloved (but totally useless) JavaScript benchmarks. Sunspider is always the first benchmark shown. Interestingly it is also one of the shortest and most cheated on benchmarks ever. Reply
  • jwcalla - Thursday, October 03, 2013 - link

    I think I mention in every mobile benchmark article posted here that SunSpider is nothing more than a browser benchmark.

    Deaf ears.
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    And not even a good browser benchmark at that as JavaScript is only a tiny proportion of your browsing experience... But SunSpider perfectly encapsulates the state of benchmarking on AnandTech. Reply
  • akdj - Wednesday, October 16, 2013 - link

    Ya know...last time I'll comment to you Wilco. This is a phenomenal board. An incredible site. Free for you, me...my wife and anyone in the world to read. You're acting like an asshole. If you don't like Anand's methods, reviews, benchmarks, et al....move along. Go away. Feed your spider, move out of your mother's basement and start breathing through your nose as well (as your mouth). What is it that you're contributing to our society?
    Sunspider is a ubiquitous test .... Desktops, laptops, now phones and tablets and it most certainly is a relevant browsing test. Shockingly it's not only relevant but evident when you go to use said 'fast' Sunspider units for browsing. Imagine that?
    Wow....just can't believe what a dick you are and have been attempting to discredit Anand and his crew and his site. Please....grace us with your exit

    Jeremy
    Reply
  • teiglin - Wednesday, October 02, 2013 - link

    Bwahaha, best table title ever. Reply
  • xype - Wednesday, October 02, 2013 - link

    Much as I criticized tge Note 3 article, I have to applaud this one. This is the kind of journalism I come to AnandTech for. Well done and thanks for the very quick turnaround time on writing it all up. Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    you come here to see that ur beloved iphone is being praised.ofcourse it is ,because that what apple pay them for.let me tell you some thing.your iphone should not even be compared to note or s4 or htc one.these phone are league of there own.iphone should be compared to the andriod sold on prepaid services. Reply
  • bpondo - Wednesday, October 02, 2013 - link

    Go to sleep Samsung, you're drunk. Reply
  • Deelron - Wednesday, October 02, 2013 - link

    Hey he could have been a Motorola zealot! Reply
  • akdj - Wednesday, October 16, 2013 - link

    If you get a chance, try an iPhone out. It's great at helping you spell, use punctuation and construct sentences.
    Ouch. That was painful!
    Reply
  • kira023 - Wednesday, October 02, 2013 - link

    Nexus 7 is Qualcomm Snapdragon S4 Pro SoC not Qualcomm Snapdragon 600 right?!!
    I doubt myself if it's really that.
    So motorola and all nexus are free of cheat codes.
    Well, I don't feel bad for all of this but better not cheating.
    Reply
  • eiriklf - Wednesday, October 02, 2013 - link

    Qualcomms snapdragon 400, 600 and 800 names (and previously S4, S4 Pro and so on) do not really refer to a specific architecture, it just gives you an idea of it's performance. The SoC in the nexus 7 has the same architecture as snapdragon 600 devices, but it is clocked lower so quallcomm calls it an S4 Pro. IMO snapdragon 600 is just marketing anyway so it doesn't matter, but I guess the authors didn't want to confuse people by referring to part numbers. Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    anand
    if you have problem with these companies cheating.then you or any other site should not review their product.i know you people wont because that what pays your bills.
    i dont find any of this cheating.because,if the cpu is advertised 2.3ghz and its running 2.3 while running benches then its not cheating even if system see a bench app and awake all cores before starting app.it will be much easier to fix this issue if the maker of the bench app put a code to awake full system before taking benchmarks.just like a warm up at basketball game.
    Reply
  • Phasenoise - Wednesday, October 02, 2013 - link

    That's a wildly flawed analogy of course. A better one using your scenario is if a player could consistently play better in practice while scouts were watching, but never while actually playing a real game.
    You know, just like a benchmark and an app which would actually use the power.
    Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    You know good and well if the player is no good in practice front of scouts ...he will never play a real game.Samsung is like iron man.when he puts the suit on he is powerfull .without suit above average..aka apple Reply
  • PeteH - Wednesday, October 02, 2013 - link

    You misunderstood his point. It doesn't matter how well you do in practice if you can't execute in a real game.

    Regardless, I think the issue here is pure deception. Consumers have an expectation that performance on benchmarks correlates to performance in everyday use. For devices that cheat to improve the benchmark score that's not true. Cheaters like Samsung are lying to consumers.

    I think the only way you'll ever see this stuff stop is if there's a class action lawsuit on benchmark cheating being effectively deceptive advertising. Money seems to be the only thing that makes companies change.
    Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    The issue is not one of deception but one of broken benchmarks. These benchmarks run at a lower clockspeed because they last so briefly that they don't appear to the OS as tasks that need performance and thus the maximum clock frequency.

    In the real world tasks do indeed run the CPU at its maximum frequency if they run longer than a few milliseconds. Therefore the performance of the benchmarks at maximum frequency with the governor optimization correlates better to real-world performance than Anand's renamed benchmarks. So he is only making things worse...

    The only way to stop this stuff is by fixing the broken benchmarks. There is no other way. Anand should immediately stop showing his rubbish JavaScript "benchmarks", especially SunSpider - the most cheated benchmark ever.
    Reply
  • The Von Matrices - Thursday, October 03, 2013 - link

    Thank you. Most of these commenters read "cheating" and jump to conclusions that accuser is absolutely innocent. This is not true. These benchmarks are purely synthetic tests and have almost no correlation to real-world usage. They shouldn't be used to judge whether a phone should be recommended or not since they present no correlation to any scenario in which a phone would be used. Therefore, this cheating is meaningless. These synthetic benchmarks are purely a curiosity for raw comparison. It's like assuming that people will buy the latest processor because it beat every other processor in pi computation speed. That's great, but test something that actually matters. Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    No he is 100% right there. All professional benchmarking is done with the CPU governor set to performance. Properly written benchmarks do indeed do a warmup run and take a bit more than a few milliseconds to run. So yes, it is exactly like warming up and stretching before sporting. Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    That's the word I was lookin for...warmup run.
    I'm afraid a7 won't make it that long to finish benchmarks being stressed that long.you know how iOS user use their device.mostly text
    Reply
  • kascollet - Friday, October 04, 2013 - link

    Please, stop posting. You're embarrassing yourself. Reply
  • sunnysingh - Friday, October 04, 2013 - link

    That was a joke..but on serious note.why they even bring apple in this.iPhone is nothing more then old razors...popular fashion item. Reply
  • ddriver - Wednesday, October 02, 2013 - link

    What about the state of cheating at apple? Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    dont even mention apple.they are gods.they just got punished for ebook price fixing and yet anand says that apple dont do any of this.apple might be worse then any of these. Reply
  • bpondo - Wednesday, October 02, 2013 - link

    Samsung astro-turfers representing the crazy tinfoil hat brigade. Reply
  • SpacedCowboy - Sunday, October 06, 2013 - link

    Hey, at least they don't pay people to AstroTurf... Reply
  • PeteH - Wednesday, October 02, 2013 - link

    Anand specifically called out Apple and Motorola as the only two OEMs that never cheated. Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    Failure to prove they did "cheat" is not prove they didn't "cheat". There are a million ways to recognise a benchmark without using its name. And showing the correct performance of a benchmark is not cheating. Reply
  • lilo777 - Wednesday, October 02, 2013 - link

    I am a little disappointed that AT succumbed to fanbois pressure to declare Samsung a cheater in Galaxy Note 3 review. It appears that the situation is not as white and black as some are trying to present it. I would expect a more nuanced approach from AT. Unlike in some "classic" cases of benchmark cheating, nobody disputes that these Android phones are actually technically capable of running benchmark and all other applications at the speed reported by the benchmark apps. Sure, some benchmark apps get "preferential" treatment. I believe it was AT who originally discovered this issue on SGS4 and back then you reported that not just benchmarks but some other apps were executed in accelerated mode (camera app was one example). Now, lets take a broader look. What we are dealing with here is the approach where resource allocation uses application name as one factor. Is this a new approach? It is not. Remember how Nvidia Optimus used application name lists for choosing which applications should be run on discrete GPU and for which ones IGP was good enough? Was it cheating too?

    Sure, Samsung could offer users to decide which applications should be executed at full "speed" (frequency, voltage whatever). Would that be a good idea? I am not sure. Given the number of apps and the type of customers who use smart phones that could be problematic. They might work with major app developers to allow third party apps to get the benefits of CPU potential where it's appropriate.

    In any case, to call it "cheating" is a disservice to people who want to know the actual facts.
    Reply
  • ddriver - Wednesday, October 02, 2013 - link

    Calling this cheating is technically incorrect. While increasing the thermal limit does sound like cheating, when Intel processors do overclock (especially in single threaded heavy workloads) it is a feature? I am fairly certain that overclocking impacts performance significantly more than what your test indicate for this "cheat".

    Sending an explicit hint to the CPU for a known performance critical workload - it does not make the CPU any faster than it already is. This is as much cheating as a programmer using an explicit prefetch to warm up data that spans across multiple lines for faster execution from the cache and not wait on the ram. It is an optimization, and if anything, it ENSURES the realistic possible maximum, which is what a benchmark is used for. If anything, every benchmark should explicitly hint the platform itself to be on its toes for its duration. Calling this "cheating" in your little "unbelievable" childish chart... somehow really doesn't suit you.

    What you are vastly overlooking is the main reason for the performance delta - the duration of the workloads. The problem of most of those benchmarks is that they consist of a series of short tests, so what happens is between each test the CPU goes to a lower state, and when the next test runs, the time it takes for the platform to analyze the workload and to boost the clocks is too long relative to the way too short test. So it ends up eating a lot of "score...

    Needless to say, this does not happen in the real world. Number crunching is 99% of the time a sustained, continuous workload. This means the impact of power management will exponentially diminish as the duration increases. CPUs are benchmarked to determine how high performance is. And how high is performance is measured in HPC conditions. Real world performance workloads are continuous, not short pulses.

    But don't take my word for it, I employ you to head over to stackoverflow and ask about whether a 25 msec loop test can provide what may be considered adequate measure of performance. The 25 msec your own graph reveals. It is really the benchmark implementers fault for not doing that absolutely mandatory optimization in the first place. You cannot chain together a bunch of random short tests and pretend for this to be adequate measure at real-world performance, which is what is impacted mostly by power management - stuff that takes too little time already. There isn't much of a difference if a tab will open in 5 or 10 msec, but there is a significant difference if say your video edit takes 5 or 10 minutes, which is where performance is really needed, and where power saving features won't impact performance. That would be an adequate measure of how high performance is in a real world scenario.

    When people run at the Olympic games, is there a discipline in which the result measured in such miniscule magnitude? There is a good reason those guys and galls run no less than 100 meters. This kind of CPU benchmarks combined with power management is relative to making Olympic runners run 50 cm. Run 50 cm, stop, run 50 cm, stop.... does this give a good measure of at athlete's performance? Is it even a good idea? I don't think so, because this measures how fast you can start and stop, not how fast you can run. If anything, it should be a benchmark about how well power saving features work and at what grain and latency they analyze and kick in. Not about the actual maximum chip performance.

    Sorry to have to say it, but this looks like anti-PR, in which case would be highly unethical, especially if endorse financially or otherwise by direct Samsung and overall Android competitors. But your justification is not technically sound. It may pass for most of your readers, but certainly won't go unnoticed. Looks like you are trying to exploit poorly written benchmarks to influence the buyer’s mind, especially now, during this bonanza of new products and spikes in sales, with re-energized cash flow and an industry eager to get a bigger part of it, including “anti-competitive” competition that involves harming the competitors to be “relatively better” instead of simply besting them. As for suspects - I'd look at the few vendors that are not present in this "witch hunt" even though are known to have plenty of dirt on their hands already, the main benefactors from this "public shaming". Which makes this whole deal kind of hypocritical in two ways - the vendors resorting to such practices considering their own track record, and the attempt to hide such a dishonest act under a veil of a plea for honesty.

    Is this a site about tech, or a site that exploit tech to drive corporate interests? I think I gave plenty of technical argumentation on the subject. I used to be a reader years ago, recently I returned to find this place significantly changed... The world doesn't need another Engadget. One is one too many...
    Reply
  • watzupken - Wednesday, October 02, 2013 - link

    Quoting this,
    Calling this cheating is technically incorrect. While increasing the thermal limit does sound like cheating, when Intel processors do overclock (especially in single threaded heavy workloads) it is a feature?

    I think you are pretty confused over the difference. Intel chip boost when loaded and when there is thermal headroom, that is true. But it boost across a range of applications, and does not selectively boost when running benchmarks only. The Exynos is programmed to wake all core and stay at full speed when it recognizes a benchmark being launched. The former boost from Intel gives you actual improvement in user experience across the board. Like I can be doing video conversion, and I am benefiting from the boost. The latter, only benefits benchmark results, and will not make your day to day usage on the phone any better. The motive is clear.
    Reply
  • ddriver - Wednesday, October 02, 2013 - link

    I agreed to a certain extent increasing the thermal envelope can be considered cheating, since it robs you out of battery life, which is not normal, something that doesn't matter on a desktop system that much, and probably the reason ARM chips are not providing the feature in the first place. Considering the workload is 3D gaming, it is not something you rush into completing, so it is a little unfair to boost unless the it makes the difference between choppy and smooth experience, which is the only case that justifies the extra battery drainage, even if the CPU is not the major power consumer in a typical mobile device. But this only applies to the GPU tests, the claim the CPU clock hint is a cheat is entirely ungrounded. Which is mostly what this whole story is all about.

    And if you want improvement "across the board" improvements due to this hack, you should look at developers, samsung cannot go after every one of them and do it for them, especially considering it should be done internally and much more finely grained, not on OS level and determinable by the file name, which is quick and dirty, but certainly not cheating, samsung and the rest just took care of the applications whose numbers influence people. The very fact this hack is applied indicates that Samsung did their homework and analyzed and detected a weakness in the benchmark design that can easily be improved. But even their resources don't stretch infinitely do it for every application, considering in most it is not really possible without wasting battery life, but for those for which is possible, why not do it? The battery drainage will be negligible which is the cost of learning how fast the CPU is on itself.
    Reply
  • ddriver - Wednesday, October 02, 2013 - link

    What is worse, ever since the note 3 popped out into existence, there has been an excruciating amount of effort to tarnish it as a product. While it does seem at this point the software is still buggy, it is just dirty corporate tricks, a tad more dirty than the amount of "cheating" caught.

    The funny part is that even without the "cheats" this device destroys pretty much every thing on the market, while not the best at anything particular, easily the best general purpose all-rounder, feature and future proof and usage versatile device. So this only goes to show how desperate the competition is, unintentionally admitting to the qualities of the product they try to tarnish in this critical time of consumerism shopping frenzy.
    Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    Actually when Intel CPUs boost they also increase TDP. That's perhaps not cheating per se but certainly misleading as Intel doesn't tell you that (it's hidden in their datasheets).

    You seem to imply that other applications do not benefit from the maximum clock speed on Exynos (and other ARM) devices. This is completely false. The boost is not required for real workloads. Micro benchmarks which only run for a few milliseconds are not being recognised as real loads by the OS, and deservedly so. So if only these benchmarks ran for a few seconds (rather than a few milliseconds...), they would not need to be boosted so that they show the correct CPU performance.
    Reply
  • Wilco1 - Wednesday, October 02, 2013 - link

    Amen to that. I can tell you are also a professional in the field. As I wrote earlier in this thread, when I benchmark I always set the CPU governor to performance. The real issue is the micro benchmarks that run for ridiculously short timespans, not the workarounds put in by the OEMs to get the correct score. Reply
  • Arbie - Wednesday, October 02, 2013 - link

    I commend Anandtech on doing so much, and think people questioning AT's "integrity" ought to go find sites where that really is an issue.

    I also think the cheaters should not get an unadorned bar on the chart, because the chart bars rule. Few people doing quick comparisons later will see all the background.

    --> One good suggestion here is to chop 5-10% from the perfomance bar for any cheaters, with a footnote as to why. The idea of marking the bar as suspect isn't as good. Cheaters shouldn't get their big bar in the first place. At least not on AT.

    Shame on Samsung especially. And kudos to Google.
    Reply
  • imaletufinish - Wednesday, October 02, 2013 - link

    Bottom line: It doesn't make it right, but they all do it (apparently except Apple). I'm still buying the Note 3 based on software features, form factor, and hardware. Benchmarks have never and will never be a factor in my purchases. Reply
  • Deelron - Wednesday, October 02, 2013 - link

    And Motorola. If the companies that do this deserve shame then everyone who doesn't should at least be recognized for playing it straight. Reply
  • techtock - Wednesday, October 02, 2013 - link

    Good Article.
    Put them on notice of 12-month benchmark suspensions. Send violators to the bottom of the graph, company name highlighted in red, BENCH CHEAT - INELIGIBLE FOR BENCHMARK UNTIL (DATE) so they get called out right on the graphs and automatically are shown only at the bottom.
    Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    This site is not doing favors for Samsung to reviews their device.I'm pretty sure that AT begs them to send them a device..ask an and about this.........Samsung is big company and don't give shit what these tech sites think of them. Reply
  • PeteH - Wednesday, October 02, 2013 - link

    Samsung's $300M marketing budget says you're wrong, they care very much how they're perceived. Reply
  • repoman27 - Wednesday, October 02, 2013 - link

    I can't believe Anandtech CONTINUES to give Samsung a pass.

    I knew something was up the moment I saw the INSANE T-Rex HD frame rates that Brian posted for the Galaxy Gear. Obviously Samsung is detecting GFXBench and plugging in the second core of that Exynos 4212. I mean how else do you explain 1.7 fps at 320x320? Samsung better enjoy their brief moment at the top of the smartwatch performance charts before Apple or somebody else calls them out on their shenanigans.
    Reply
  • jerrylzy - Wednesday, October 02, 2013 - link

    LoL. I highly doubt the results of NVIDIA Shield. I hope Anandtech's editors can compare scores between Google Play Antutu and Antutu X, the version released by Antutu to prevent 'key word' cheating. NVIDIA Shield clearly cheats in at least Antutu, especially 3D parts. I found some words like 'NVCPU_BOOST' in framework and NVCPLSvc.apk is highly considered the culprit of cheating, which contains an encrypted txt file called tegraprofs.txt under /res/raw folder.
    I sincerely hope Anandtech can check its results of NVIDIA Shield.
    Reply
  • nbnb001 - Wednesday, October 02, 2013 - link

    Shield is also cheating in benchmarks. Launch antutu or 3dmark, check /sys/kernel/edp/vdd_cpu/edp you may find shield disables thermal-throttling and makes quad a15 run at 1.9GHz even at 100 centidegree . Antutu had released a X edition without a keyword in package name, which can be downloaded from Google play. I have tested on my shield and find the total score dropped from 38000 to around 31000 , mostly in CPU and 3d score. Reply
  • aatjitra - Wednesday, October 02, 2013 - link

    Well done!!

    Show a proof and name them then publish it...

    Cheating with any reasons even for small increment as results still not acceptable and it shows dishonest intention from them... as the damage caused by fake results may be so huge for other brand...
    Reply
  • Sreekanth - Wednesday, October 02, 2013 - link

    How about Nokia devices? Just curious why weren't they part of this exercise? Reply
  • sunnysingh - Wednesday, October 02, 2013 - link

    Is anyone using Nokia.tech sites only call out companies sells lots phone..anything under 1000 don't count. Reply
  • Sreekanth - Thursday, October 03, 2013 - link

    Come out of your cave and get more sun. It is not enough if you just have name as "sunny". Windows phone market share is in increasing in many markets. Google before you blabber. Reply
  • sunnysingh - Thursday, October 03, 2013 - link

    I am aware of windows phone market share.that whole 1%.its them cheap crap only cheap people use.I bet they don't browse internet on that. Reply
  • kyuu - Thursday, October 03, 2013 - link

    You'll notice all the listed phones are Android phones. Nokia doesn't make Android phones.

    He mentioned Apple in passing, but Anand says he only knows that Apple doesn't cheat due to some behind-the-scenes poking around he's been able to do.
    Reply
  • rocky12345 - Wednesday, October 02, 2013 - link

    OMG just get over it already this is the tend nothing any one can do about it. The way I see it for those comp[anies that do this by maxing their device to it's limit is showing what said device is able to do in a benchmark when it is running maxed out. I say oh boo hoo to the companies that are to lazy to take the time to put settings into their hardware to actually show what each given device can fully do.

    ALso the consumers do not care whether their page loads half a mil second faster or not all they want is good battery life from their device. So no you will never most likely ever see what a device can fully do unless the companies take the time to put scripts or profiles into their device to get that little bit of performance out of a device.

    Everyone is so upset at a problem that actually is not a problem at all. This is some guy notice that lets say device A got a bit better score than device B & he then started crying to the world that something was not right. Well boo hoo maybe like I said if a company has a device that is not measuring up to another device but has the same hardware then maybe that companies needs to rethink their plan & make their device run better. There is nothing anyone can say that will change my stance on this so don't even try. Point is a lazy company does not deserve to be in the same league as a company that spends a bit more time to make their device appear to be better than the next device. It is called spit & polish people maybe it will make the lazy boys & girls wake up & work for that extra 300 points in a benchmark the end result is we get better products because it forces these lazies to work harder & in turn release a more polished product. NUFF said. Nothing else to see here move a long.
    Reply
  • rocky12345 - Wednesday, October 02, 2013 - link

    Sorry I guess there is no edit for this but anyone that can read will be able to get by the few mistakes & fully understand what I was saying. The big point is There are far bigger things to worry about in this world than some device makers trying to squeeze a couple hundred points out of a meaningless device. Reply
  • rocky12345 - Thursday, October 03, 2013 - link

    Also wanted to add that it wouls be cheating if a company actually raised the limits of the hardware by overclocking it past what the spec sheet says. If the spec sheet says 2.3G for the CPU & the company sets it to performance mode which causes the CPU to run at it's top speed and not past that then it is still within the spec of what the specs have listed it to run at. Some say well in real world apps you won't see that type of speed from the CPU actually you do but not for very long because it chews up the battery a lot faster. So samsung & other that are doing this are just making sure you see the hardware for all it is worth & what it can do. The biggest problem come in when the smaller companies do not take the time to do the same thing & rush a product out. I do it on my own phone when benching it. It is rooted & I set it to performance mode & do see a jump in scores and it shows me that my hardware is actually faster than what I read in reviews on sites like this.

    Oh by the way I would bet Apple does it too but most likely harder to catch because they design their own chips so it is most likely buried in the hardware as extra instructions to tell the CPU to beef itself up because here comes another bench test.
    Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    Agreed, this is way too overblown. It's not even cheating as the issue is the stupid micro benchmarks. And it doesn't compare to the real cheating of Intel in the AnTuTu benchmark. Reply
  • pandemonium - Wednesday, October 02, 2013 - link

    While this information is welcomed - cheaters be damned - I really don't take synthetic benchmarks with any weight, what-so-ever. Battery life, display quality, build quality, camera quality, telephony quality, and even the amount of "smooth user experience" are more valuable to my user experience.

    You guys can keep your synthetic benchmarks that tell little about how the phone actually works in the end.
    Reply
  • thebeastie - Thursday, October 03, 2013 - link

    Maybe its a bit of a trollish statement but Android worshiperz tend to be all about specs compared to an overall view.
    Anandtech finally coming out with an article about this stuff must be having a crushing effect on how such people see the world.
    I was wondering when you guys were going to write an article about this but it really is disgusting to see some spec head raving about some GPU benchmark on a phone when you know it's cheating.
    Reply
  • Crsytral - Thursday, October 03, 2013 - link

    Where's Sony? I believed that they dun do what Samsung did, but still i would love to find out. Reply
  • XFire99 - Thursday, October 03, 2013 - link

    Did all devices runs at OC speeds then its original spec?
    Exynos 5410 CPU is at 1,6Ghz and GPU 532Mhz. So how is this cheat when it runs at thiers max speed? As long its runs at max speed and not auto OC CPU/GPU at benchmarks. Then I dont see how is this cheating?
    Android governor, downclock cpu/gpu speeds, whenver not using the device and then its cheating too? Since we paid for the device to trust it will run at it specced.
    Reply
  • few - Thursday, October 03, 2013 - link

    Good find!

    A straightforward way to resolve this is to link performance and battery life more directly: Record performance statistics for the test-suite, running the test suite from full power (or 95%) down to a drained state (or 5%). List the run-time as the functional battery life. They can either game the performance by clocking faster or game it to save battery life, but those two are a direct tradeoff which are both critically reviewed by potential customers in the mobile space. My guess is that the manufacturers will self-adjust back to a balance.
    Reply
  • darkich - Thursday, October 03, 2013 - link

    So what's the difference between this and the Turbo mode in Intel's chips? Reply
  • thope - Thursday, October 03, 2013 - link

    Intel will do it for all applications regardless of OS even. its a hardware feature. The issue discussed here is where the phones detect certain benchmark apps in software and go into a higher power envelope than normal to get a higher score.

    The CPU/OS will not do it when you run any other app because it would severely degrade battery life. So, if you were browsing/Playing a game you will not see this performance and therefore it is being viewed as cheating.
    Reply
  • Wilco1 - Thursday, October 03, 2013 - link

    This is not true. A real application workload will also run at the maximum frequency. Therefore there is no difference in power envelope - it just avoids the overheads caused by power management on short-duration benchmarks. Reply
  • BillBear - Thursday, October 03, 2013 - link

    You could always threaten to run your device battery life tests with one of the "optimized" apps running in the background but not otherwise doing anything.

    Wouldn't that prevent the device from dropping it's CPU cores to a power saving idle state as much as possible?
    Reply
  • Adam-James - Thursday, October 03, 2013 - link

    As far as what Futuremark could do in response to HTC and Samsung violating their policies and gaming the benchmarks, how about...

    if (Build.MANUFACTURER.equals("Samsung") || Build.MANUFACTURER.equals("HTC")) {
    throw new RuntimeException("Optimize this!") ;
    System.exit(0) ;
    }

    Seems like it would do the trick.
    Reply
  • Vhysics - Thursday, October 03, 2013 - link

    Very good work :)

    But how about for Sony products? Thank you very much. :)
    Reply
  • anger999 - Thursday, October 03, 2013 - link

    the "I Can't Believe I Have to Make This Table" table had me on the floor laughing Reply
  • mr_tawan - Thursday, October 03, 2013 - link

    I've just got my Xperia Z1. How can I test against that one so I can tell weather or not it's also cheating ? Thanks. Reply
  • JakeLee - Thursday, October 03, 2013 - link

    The term "optimization" is more than misleading for this kind of dishonorable practices. As a professional system optimizer, I even feel insulted by this.

    Optimizing is something really really positive that everyone benefits from. But what the Android OEMs did is nothing but making fools of the press and their (potential) customers.

    I'm very aware of the case 3DMark vs nVidia where the term "cheating" had to be taken back, and the same might happen to you as well if you did which would be more than unpleasant.

    Even though I fully understand you being careful with terms regarding this case, the term "optimization" is way off IMO. What about "trickery"?

    BTW Anand, if you intend to write/extend your own benchmark, I'd be gladly assisting you by providing some customized, OS-independent, hand written assembly routines that can be used to measure the CPU's Integer, Float, SIMD, and cache performances.

    Just tell me what you'd need.
    Reply
  • alison_lenihan - Thursday, October 03, 2013 - link

    what Eric said I am shocked that some people can profit $4550 in one month on the computer. see post....... CuttR.it/tvtmbce Reply
  • Diorarat - Thursday, October 03, 2013 - link

    That's pretty quick. Everybody was bashing about benchmark, anandtech and Samsung on the note 3 the other day and anandtech followed it up quickly with this article. Pretty impressive. Reply
  • KeeA328 - Thursday, October 03, 2013 - link

    Users don’t care about this kind of stuff. As long the device is working fine and dandy, it shouldn’t affect them. Reply
  • AnnihilatorX - Thursday, October 03, 2013 - link

    I think it is not possible for apks to rename itself on the fly after it was packaged. But it is certainly possible to compile it beforehand with whatever name, as avast anti-theft module does it and allow user to pick any name before downloading. Reply
  • StigtriX - Thursday, October 03, 2013 - link

    What about Sony phones? Z1? Reply
  • JNo - Thursday, October 03, 2013 - link

    "The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 - 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell."

    The thing is that the tech press, including the likes of anandtech, do play a part in this whole mess, even if unconsciously. Most reviews *will* mention that extra 5% of speed as an advantage in a phone review or it will contribute to a better overall impression, even though 99% of apps wouldn't benefit from it anyway. And the OEMs can't just new silicon out *that* much faster so 5% in a cut throat world is meaningful.

    I'm not saying I like the whole situation but we as consumers can obsess over the speed of silicon, as can review sites and hence, so too the OEMs. We all need to sit back and take a reality check. Well done for exposing it though anandtech.

    On a small note, your "can't believe I'm doing this" table would really benefit, in terms of readability, from some basic colour coding - ideally with psychological connotations, e.g. red background in the Y boxes and green in the N boxes. Just a suggestion.
    Reply
  • DucSportClassic - Thursday, October 03, 2013 - link

    Let me preface this post by saying I am not a developer, programmer , etc. and am totally ignorant in this area.

    I visit Anandtech because as a typical guy who loves "tech stuff" I want to learn more about the products I intend to buy. Learn the specifics about a product from those way more knowledgeable in this area than I and provides with me something more than just a review of physical specifications. But the more I read, the more I notice the bias towards certain companies. I purchase what offers me the best performance, features, etc. that I am looking for. In this case , I have owned iphones, Samsung , Nokia and Windows phones etc. and will jump ship in a heartbeat if I find something better. There is no brand loyalty with me.

    But I have to agree with others here such as lilo777 ( page 19 ) , ddriver ( page 19 ) , The Von Matrices ( several pages starting on 5 ) and also Wilco1 , who started on page 2 and has continued throughout the entire thread.

    lilo777 said it best;

    "In any case, to call it "cheating" is a disservice to people who want to know the actual facts.

    Anand,

    You responded to quite a few comments posted in this thread, all of them in support of your article , but not one response to these posters. Why is that?
    Reply
  • ddriver - Thursday, October 03, 2013 - link

    Why bother when you are not in the position to be objective... I can predict with 99% certainty what a response from Anand will consist of. Pretty much the same type of answer you will get from a politician. Reply
  • Graag - Thursday, October 03, 2013 - link

    Because they are Android fanbois who are not interested in honest discussions of facts, but only in spinning facts to make their favored devices look better, and to make Apple look worse.

    Samsung's cheating program boosted performance when it knew that a benchmarking program was being run, and at no other time. That is cheating, period.
    Reply
  • lilo777 - Thursday, October 03, 2013 - link

    How do you know that performance was not boosted at any other time? Any evidence> In fact AT already reported that SGS4 actually did boost performance for camera app. Reply
  • Wilco1 - Friday, October 04, 2013 - link

    Stop claiming people are biased when they are actually experts in the field. I don't own any Android or Apple device. As I've said before I never run any benchmark without setting the governor to performance. It's how things are done by the professionals.

    The reason this is not cheating is because other applications do run at the same frequency as you can see for yourself in the article. The graph clearly shows that the renamed AndEBench runs at max frequency for ~80% of the time. The boosting simply ensures it runs at max frequency for the whole (extremely short) duration of the benchmark. If the benchmark ran for 1 second rather than 30ms then the boosting would have no effect at all.
    Reply
  • kapg - Thursday, October 03, 2013 - link

    +1 DucSportClassic

    I am pretty much the same as you and did consider Anandtech to be the gold standard for any tech news/review but it seems I need to rethink my position.

    Everyone here seems to quickly to be jumping to a final conclusion about "cheating" and tend to use blanket statements without due consideration.

    Adding to the confusion I'd like to know if the whole question is about being 'fair' then why is the native screen resolution of each device metioned in the chart where CPU/GPU prowess is being depicted. Is it 'fair' to compare a 1920x1080 screen in the same chart as that of a 1280x720, specially when comparing GPU/CPU without even mentioning the difference???
    Reply
  • kapg - Thursday, October 03, 2013 - link

    *not mentioned in the chart Reply
  • Hoggleboggle - Thursday, October 03, 2013 - link

    I hope you get a chance to dig into Apple's code as well to see where they are optimizing their benchmark results, although I could imagine that it would be trickier to find due their closed system. Reply
  • kapg - Thursday, October 03, 2013 - link

    Also please someone do something to improve the comments system here. One should be able to promote or demote comments n all replies should be collapsible...... Reply
  • Deelron - Thursday, October 03, 2013 - link

    It was pretty easy to notice these, as the clock frequency/core behavior changed depending on what was run, it didn't require a digging in the code to find the problem at all. Reply
  • JEREMIAH13 - Thursday, October 03, 2013 - link

    Hi Anand, Can I just clarify about the Table 1's numbers in brackets for the Note 10.1 2014?! Surely that cant be factor increase?! Reply
  • Jaaap - Thursday, October 03, 2013 - link

    Anand, did you consider running these benchmarks in a hot room 10 times and using only the last 3 scores?
    That would defeat some of the cheating (higher freqs/voltages), wouldn't it?
    Reply
  • Rickschwar - Thursday, October 03, 2013 - link

    In case anyone is wondering what app was used to expose this, it's called Trepn Profiler and it can be downloaded for free from this page: https://developer.qualcomm.com/mobile-development/...

    You can see it in action in the first screenshot of this article.

    - Rick
    Reply
  • ChrisMars - Thursday, October 03, 2013 - link

    Considering that AnTuTu and Vellamo are enabled in ALL Snapdragon 600/800 of our table (excluding Nexus-devices naturally), couldn't it be that tuning for those 2 apps is part of Qualcomm's SW-release towards the vendors...?

    I'm just wondering, did you find a Snapdragon 800 device so far which is NOT reacting to those two benchmarks?
    Reply
  • ChrisMars - Thursday, October 03, 2013 - link

    *your, not "our" of course ;-) Reply
  • chandu6119 - Thursday, October 03, 2013 - link

    what about sony smartphones?does they cheat? Reply
  • redhotiron2004 - Thursday, October 03, 2013 - link

    Why no Sony mobile have been included in these kind of tests. Are they completely white?
    Or do you consider them as a pretty small brand to not matter at all?
    Reply
  • Braunfelt - Thursday, October 03, 2013 - link

    The optimization of the CPU/GPU is basically an overclock with throttling, so even with the optimization with throttling are they able to sustain that with out cooking the device or over heating? If so than all of the above does not matter. If it can only run for short durations I can understand the statement. But given all PC cpu's and gpu's now use thermal throttling how is this any different? Reply
  • Wilco1 - Friday, October 04, 2013 - link

    No this is definitely not an overclock - that would be cheating indeed. And yes the boosting can be sustained for the duration of the benchmark without overheating. A long running benchmark would still throttle eventually of course, and the boosting has no effect on that. Reply
  • ncsaephanh - Thursday, October 03, 2013 - link

    "I Can't Believe I Have To Make This Table" - Best table title ever. Reply
  • lightshapers - Thursday, October 03, 2013 - link

    Hello Brian and Anand. I read your article with great interest since months now.
    Working in this area in a great soc manufacturer, I could say that thermal policy is what is guiding max frequencies and the time cpus can maintain it. No hi performance soc can maintain its max frequency for ever, and throttling always happen, so using a benchmark trick is not far from tweaking the thermal governor so that is can sustain a bit higher freq (but time it maintain it will be shorter) or maintain max freq for a longer time.
    For the latets smartphone, the thermal governor is guided by the case temperature, because is can reach unconfortable temperature quite fast as user is holding the phone in the hand, and to avoid this, cpu is throttling around 1.3GHz after a short minute (S600 or CA15 based soc)... This can be easily found out by monitoring frequency in android once rooted.
    What we can say about this, is that for some benchmark, phone vendor tolerates that case is getting a bit hotter for benchmark. I don't think it is really cheating, we all want to have the best performance.
    It's like a car race where car engineer are paid to get the best performance in regards to the rules of the competition. Here it is a score at the benchmark, what is heavily used by technophile's website like yours, so we could say that this behavior is more or less induced by the charts you and your mate (bgr, engadget, ...) are showing since now some time. Phone vendors just want to score the best, and play with the phone physical margin (and hands of the user...).
    My é cents on this subject is just that it is probably more than "a cheat code" or good or bad people.
    As an exemple, publishing a benchmark that advantages a platform or another is cheating, but preparing a smartphone to score best in a competition is just "showing the best".

    Hope that gives another point of viez on this...

    Greg
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    I agree this is simply a case of "showing the best". As far as we know there is no overclocking or increasing of thermal limits, it's simply benchmarking at the maximum frequency. Setting the governor to performance is routine when one does benchmarking on Linux.

    As I'm sure you must know too, there are far more questionable techniques being used to get good scores for eg. SunSpider. Then there was the AnTuTu debacle, and that was actual cheating. Analysing these things is obviously far beyond the technical capabilities of most, including AnandTech.
    Reply
  • Brakken - Thursday, October 03, 2013 - link

    How about any device with 'adjusted' software being excluded from review? If they get no air time, wouldn't this result, eventually, in more honesty? Reply
  • petergreyhill - Friday, October 04, 2013 - link

    I am not sure why all reviewers are calling this cheating - no one added external chemicals for higher combustion as in a racing car - these are smartphones so in my opinion, I don't care how they get full cpu optimized mode, I want the fastest most powerful I can get - maybe the benchmark guys need to update their software. Most engineers and business professionals will continue to purchase the fastest no matter what - Samsung has built a beast of phone - with laptop capabilities, 3 g of ram and a very fast and deep chip - LG also has a power beast of a phone - when did everyone become old ladies screaming ?

    No overclocking no harm no foul just flat out speed - I want the fastest.
    Reply
  • xype - Friday, October 04, 2013 - link

    You don’t get the fastest, though. Unless you spend 99% of your time running the couple of rigged benchmarks. If you don’t, you get performance throttling—and get it all the time. Like everyone else.

    I’m constandly amazed at how hard of a concept this seems to be.
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    Correct - there is no overclocking going on. The boosting simply allows badly written benchmarks to run at the maximum frequency 100% of the time rather ~90% of the time. Reply
  • DucSportClassic - Friday, October 04, 2013 - link

    Cheating.. really?

    Did Samsung provide false performance numbers the phone was never able to achieve? Did they provide review sites with special mocked up phones so they can achieve these numbers?

    Personally I would prefer my phone to take advantage of the speed available for the more intensive tasks.
    Reply
  • thunng8 - Friday, October 04, 2013 - link

    But it is not .. have you even read the article? It only affects benchmarks, not real applications. Reply
  • Wilco1 - Friday, October 04, 2013 - link

    Not true, any application can reach the same maximum frequency. A well-written benchmark would show no difference between boosting or not. So the underlying problem is with the badly written benchmarks which run for too short timespans. Reply
  • thunng8 - Friday, October 04, 2013 - link

    Anand,

    Are you sure that Samsung are not cheating in GFXBench 2.7 with the Note 3? The DVFSHelper.java file has an entry for GLbenchmark27.
    Reply
  • csotany - Friday, October 04, 2013 - link

    These comments are hilarious:
    - Fanboys with a reality distortion field derping in pain
    - Fanboys who "want maximum performance" dismissing the accusation, because they either incapable or unwilling to understand what the problem with this practice is
    - Notorious astroturfes defending the undefendable / attacking Anand
    - Some manufacturer is aware of Sony's cheating, and dispacthed more astroturfers
    Reply
  • ddriver - Friday, October 04, 2013 - link

    "They're (Almost) All Dirty" - and blowing that out of proportion to serve corporate interests isn't dirty - it is highly moral and squeaky clean, just like all the vendors with impeccable records spared in this pathetic little witch hunt...

    All corporations are greedy immoral crooks, this is no secret, despite that some might view me as s Samsung sympathizer for my comments, I do have my share of resentment toward the company. But this is not what this is all about, and it is a very sad fact that most people immediately fall for such a cheap ploy and don't see what it is really all about - hurt the sales of brands to the benefit of other brands, whose asses are commonly thoroughly kissed here at AT...

    Tomorrow some samsung executive will pick his nose and it will be all over the news, because samsung has products that worry its competitors...

    If the corporate competitive model is allowed in the Olympic games, competing there will include breaking limbs, drugging or otherwise sabotaging competitors, exploiting stolen personal or classified information to extort into result fixing, implanting moles as trainers to hamper others' performance, legally bribing regulatory bodies or even installing your own people there to judge in your favor. This is the reality of contemporary capitalistic competition, and sadly this site has turned into just another small gear in this immoral and corrupt machine. Demonizing selectively some brands to make others appear better, even if they are just as dirty or even more.

    Samsung's marketing model was never based on numbers, they did not claim to have the fastest product or whatever (unlike some other vendors). Samsung "cheated" by optimizing a lamely written benchmark, just because they are aware there are plenty of fools whose understanding of technology boils down to comparing how big numbers are relative to each other - stop the presses and shout everyone is dirty, besides the vendors who get preferential treatment at AT of course - o lordy lord, what a surprise.
    Reply
  • csotany - Friday, October 04, 2013 - link

    This sort of astroturfing is really counterproductive, because people intelligent enough to read your long posts will not buy into your whitewasing / Anand-bashing.

    In fact, it seems that the only people agreeging with your comments are fellow astroturfers.

    It's really sad.

    Now, I call you an astroturfer because you're too hard working for a fanboy, and also smart enough to follow the logic of technical conversations here, yet incapable of seeing why your value system suffers from double standards.

    I mean, seriously, how pathetic is that you're trying to smear Anand for uncovering, measuring, proving what you could by any objective standard call cheating, and then go on for four paragraphs protecting Samsung?
    Reply
  • ddriver - Friday, October 04, 2013 - link

    No, yours and Anand's value system suffers from double standards, because the reality is that they are ALL dirty, Anand including, maybe not dirty for cheating at a particular set of tests, but as dirty as it gets nonetheless. The only reason this is being done today is because it is convenient and lucrative for the brands that get preferential treatment at AT and US "tech" sites in general.

    I don't smear Anand for their analysis, but for the purpose they misuse it for and the fact they blow it completely out of proportion, which is what is really pathetic. As I already mentioned, I've only owned a single samsung device in my life and it was a crappy TN LCD - hardly something to inspire brand loyalty. I couldn't care less if the entire industry perishes tomorrow, including samsung.

    Your lame insinuations are the product of the simple fact that you cannot substantiate your claims with anything besides insulting people and making stuff up.
    Reply
  • ddriver - Friday, October 04, 2013 - link

    This "cheating" is nothing, Apple's targeted profit margins has driven Foxconn into exploiting people so hard as to push them to suicide. Samsung has poisoned a number of people in its own plants. WE are talking human lives here, and that's what's really dirty. What AT does is actually helping one of those dirty companies to make more money at the expense of the other dirty companies, which is dirty on its own. Reply
  • Wilco1 - Friday, October 04, 2013 - link

    For short duration tasks it doesn't matter much whether they take 20ms or 25ms - as long as it is faster than the human response time then you cannot notice the difference. You could indeed create a benchmark that tries to benchmark DVFS response, but you'd have to measure power consumption as well. Note that race-to-idle is actually not as power efficient as a slower DFVS response: higher frequencies increase power consumption quadratically while giving less than linear increase in performance. So you actually use less power when running at a lower frequency for short duration tasks. And that's exactly what appears to happen with these benchmarks. So a good DVFS would give a lower score on short duration benchmarks, but that's not how people want to interpret it - they want a higher score, but that means burning more power. Reply
  • lightshapers - Monday, October 07, 2013 - link

    simple benchmark would be maintaining the max demand for more than a minute and start a benchmarck after this, so as to ensure the frequencies achieved the balanced status, where power emmited is reaching the platform dissipation power. A second benchmark would then consists in max performance at start, meaning the absolute max at cold like setting the performance governor and run a test.
    A third would be linked to response time, meaning measuring the time it takes to the system to reach the max performance level.

    i's right now that measuring brutal performance is just good for making sensational stuff, and very easy to "cheat on"...
    Reply
  • BradM - Friday, October 04, 2013 - link

    I know this is a hardware geek out fest and we do care about the details of the numbers, but at the end of the day, I care about how well the device performs on normal workloads. Presumably cheating on the benchmarks is going to make the device run hot with reduced battery life. So, one approach might be to (a) note whether or not the device is "fast enough" off an arbitrary subjective, but normalized impression, and (b) what the temperatures and battery life were while running the benchmarks. Cheaters will get lousy scores for burning your hand and wasting battery life, and non-cheaters will get better scores for usefulness. Reply
  • Ma Deuce - Friday, October 04, 2013 - link

    Well done. Hopefully exposing some of these companies will make a difference for the better.

    It's amazing the number of people actually trying to defend this. With companies doing things like this it makes benchmarking worthless to the normal consumer.

    Thank you Anandtech. This is one of the reasons I keep coming back to this site and one of the reasons I stay away from the biased garbage like Tom's...
    Reply
  • Beautyspin - Friday, October 04, 2013 - link

    I see you vehemently denying Apple doesn't do this but where is the proof? I am not seeing any proof for that assertion. Any reason you could not find any? Reply
  • Ma Deuce - Friday, October 04, 2013 - link

    Vehemently? lol...

    "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. "

    Maybe, and this might be a long shot, they didn't find any proof because Moto and Apple aren't doing it?

    But you didn't mention Moto, just Apple... I suspect you wouldn't be buying an Apple device anyway, so what would it matter?
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    That's not a proof that's flawed logic. I can't prove you are male, so you must be female. QED. Reply
  • Ma Deuce - Friday, October 04, 2013 - link

    What proof are you looking for? I don't recall trying to prove anything... Reply
  • Wilco1 - Sunday, October 06, 2013 - link

    The proof for the claim that Apple, Motorola, Sony etc are NOT boosting in any way. All that Anand said was that RENAMING the application didn't make any difference. He didn't prove that there is definitely no boosting, just that they don't use the same trick recognizing the name of the benchmark. So his words you quoted:

    "With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. "

    is not a proof that Apple and Motorola are not boosting. They certainly could do so but in a DIFFERENT way than looking at the name of the application - after all there are a million ways to achieve the same effect in software. Got it now?
    Reply
  • Beautyspin - Friday, October 04, 2013 - link

    I did not mention Moto because it is already in the table so that means it was tested and the results are there for us to see. I want to check on Apple mostly as it is a closed sourced system end to end and I would like to know what tests Anand conducted to determine that Apple did not cheat.. I would also like to see that these tests are fair.. I cannot believe that Anand just says Apple has not cheated without showing any metrics for this conclusion..

    You are right about one thing though. I would not buy an iPhone right now with all the bugs that are so in the face but Anandtech did not find of the design blunders in their extensive interviews. This post will be deleted so I am going to take a screen shot and post it on twitter...
    Reply
  • versesuvius - Friday, October 04, 2013 - link

    All benchmark vendors must build in the cheat into their software then. That will at least put every product on the same level. That is, rev up everything to max and report the heat and power consumption as well. Granted the result is a bit removed from reality of the device, but as long as there is that kind of cheating involved every product might as well be given the advantage. On the other hand, and I am not sure if it is done, some developers may try to offer the same cheat as an standalone program so the user can add programs that he likes to run faster to its list and have them run faster on all cores and max frequency. Reply
  • Wilco1 - Sunday, October 06, 2013 - link

    It would be good indeed if Android added an API so that an app can say it is CPU intensive and needs the maximum frequency immediately. That would avoid the need to have this special list of apps to boost.

    However if the benchmarks did a warmup run and ran for a bit longer than 20ms there would be no difference between boosting and no boosting. That's because the OS will automatically boost to maximum frequency after a while. So the major issue is that the benchmarks are badly written.
    Reply
  • coolxenu - Friday, October 04, 2013 - link

    So if I am looking at car specs and it says the car's top speed is 200kph that's cheating since there is nowhere in Canada I can travel at 200kph? Reply
  • Ma Deuce - Friday, October 04, 2013 - link

    Really? I was just in Canada and I drove over 260 kph. Is it all up hill where you live?!?!

    Now the car that I was driving has a super charger that gives me about 15psi of boost. If the manufacturer gave Car&Driver a car with 20psi of boost to benchmark, but only let me get 15psi you would have a better analogy...
    Reply
  • Wilco1 - Friday, October 04, 2013 - link

    No the analogy would be a gadget that ensures the engine revs in such a way that that it gives the maximum boost when you want it most so that you get great 0-60 acceleration. Cheating?

    Or a feature called launch control?
    Reply
  • Ma Deuce - Friday, October 04, 2013 - link

    Dead wrong, because you don't get the boost from the cheating suppliers when you want it. Only when they want it.

    Unless all you want to do is run benchmarks, then I suppose you got me there.
    Reply
  • Wilco1 - Sunday, October 06, 2013 - link

    No, you can get the boost yourself as well if you want it for your own apps, there are countless Android apps that allow you to do so. Remember also that a well-written app doesn't actually need it as the OS will boost it automatically. Reply
  • Ma Deuce - Sunday, October 06, 2013 - link

    Sorry, I thought you were comnenting about this article. Can you please link to the one you've been reading? Reply
  • Wilco1 - Sunday, October 06, 2013 - link

    Please read the whole article again. Anand states at no point that there is any overclocking going on. Any application can run at the maximum clock speed even without boosting. He even provided a graph showing exactly that. Reply
  • Ma Deuce - Tuesday, October 08, 2013 - link

    Exactly, please read the whole article again. Not the one that you are reading about how some Androids apps can get around this limitation or the "over-clocking" you are talking about.

    Or, maybe you could be so kind as to make a list for all of us to let us know which manufactures allow what software to run at full speed and what software they limit?
    Reply
  • Wilco1 - Tuesday, October 08, 2013 - link

    By definition all software can run at the maximum speed for as long as thermal limits allow. Again, Anand showed clearly there is no overclocking going on. Reply
  • Ma Deuce - Tuesday, October 08, 2013 - link

    Again, you keep talking about overclocking... Why are you so stuck on overclocking? It's not what this is about. Reply
  • reddog007 - Saturday, October 05, 2013 - link

    I don't really get how it is cheating. They are running at advertised clock speeds. All they are doing is basically turning off the CPU governor. Otherwise the scores will be artificially low. The point about benchmarks is to get the absolute most performance and never represents real world. I guess the Moto X is a cheater in everyday life because its governor is geared more aggressively?
    Cheating would changing driver settings to reduce resolution, reduce IQ, increasing clock speeds beyond advertised (which some do), skipping frames, whatever, etc.
    I don't think its cheating running at advertised clock speeds!
    Reply
  • Ma Deuce - Saturday, October 05, 2013 - link

    It's cheating because they don't let you run at your so called "advertised clock speeds." They only use those speeds when it benefits the manufacturer and not the consumer.

    If these speeds actually were used to benefit the consumer it would be a non issue. I would be upset if I spent money on one of these phones after doing research on how well they benchmarked.

    If I buy a pint of beer, I want to be able to drink it all and not leave 10% in the bottom of my mug.
    Reply
  • Wilco1 - Sunday, October 06, 2013 - link

    Completely incorrect. All devices DO run at their advertised clock speeds, whether boosted or not. Boosting simply runs at the advertised clock speed for LONGER. It's as simple as that. Reply
  • Ma Deuce - Sunday, October 06, 2013 - link

    Samnsung themselves has stated that they don't allow all apps to run at full speed. Are you correct, or is Samsung correct, or is Anand correct? Reply
  • Wilco1 - Sunday, October 06, 2013 - link

    Samsung has not said anything like that. Anand has not said anything like that - and even provided a graph that shows AndEBench running at the maximum clock frequency without boosting. So you are incorrect in claiming that it is about overclocking when it isn't. Boosting is about running at the maximum clockspeed for longer than usual. That's all. How hard is that to understand? Reply
  • Ma Deuce - Tuesday, October 08, 2013 - link

    "the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode." -samsung

    Please work on your reading comprehension. You keep bringing up "overclocking" but this has nothing to do with overclocking.

    The graph you are talking about shows that AndEBench is being targeted by these shenanigans. Once again, reading comprehension...
    Reply
  • Wilco1 - Tuesday, October 08, 2013 - link

    Nope, you need to work on your reading comphrension. The boosting we are talking about does not go over the maximum frequency. That includes Samsung's statement (which is about downclocking to avoid overheating) and the AndEBench graph. At no point is there any overclocking. That means that any application can run at that same maximum speed (as the AndEBench graph clearly shows). And because of that, it is not cheating. Reply
  • Ma Deuce - Tuesday, October 08, 2013 - link

    Once again, you are the only one talking about overclocking... Slow down and try to digest the information about what's happening with AndEBench. When you are done, read the rest of the article.

    You can deny it all you want, but it's not going to go away.
    Reply
  • Mostrou - Saturday, October 05, 2013 - link

    Does anyone know if Sony is gaming with benchmarks too? Reply
  • Wwhat - Saturday, October 05, 2013 - link

    Amusing to include the nvidia shield making it out as if nvidia is above this, when in reality the shield is a unique device so it does not yet have to worry about competition.
    And the reality is that nvidia is a company who has a long long list of engaging in shenanigans exactly like this..

    And to specifically mention that companies like intel do not endorse this, when in fact intel also is known to spike cards if they can in the past, to the point of facing prosecution as I recall.
    Reply
  • Beautyspin - Sunday, October 06, 2013 - link

    I am comfortable with a vendor who releases the source code for the kernal. You cannot really call it cheating when he knows everybody can easily find it out. Now I am not comfortable with Anand telling that he is sure Apple is not gaming the scores and repeating the same again and again like a child without any proof. This is specially egregious as the Apple source code is closed and they could be gaming the scores in a wide variety. Also the alacrity with which Phil schiller responded to these allegations looks like these studied were orchestrated and hence the doubts.. Reply
  • ruthan - Sunday, October 06, 2013 - link

    I think that there are not iOS tools to detect it.. so Anand cant say, that they dont cheating too. Only truth is that iOS is too closed to discover such things. Reply
  • ruthan - Sunday, October 06, 2013 - link

    I think, only way to mitigate such crap, as overclocking for better bencmark results is monitor device temperature in during every test and will temperature is too big, simply say ok.. Score was reached over our temperature limit and ad to result row some exclamation mark or use red buring font etc..

    To push device manufactors to stop cheating is only way, dont show device result at all and simple to comparision table write cheaters, and to post score at all.
    Reply
  • kkr123123 - Sunday, October 06, 2013 - link

    What do you Gain by doing all these Junk study ??
    I think none of these manufacturers, except crAPPLE, agreed to pay you, so you writing crap about them.
    Anyways all the products you have mentioned have the capacity to run at much higher speeds. If end users want higher speeds, they can flash the custom ROMs and can use the full capacity of the hardware.
    Please dont write articles as if these manufacturers have committed a blunder.
    Just stop posting these crap articles to gain viewers to your junk website.
    Reply
  • mysword - Monday, October 07, 2013 - link

    can anybody tell me the apps'name which could monittor GPUfrequency? I want try this on my mobile.thanks. Reply
  • stalemate.black - Monday, October 07, 2013 - link

    And yet, this "revelation" still won't make me buy an Apple smartphone. Reply
  • Ev1l_Ash - Tuesday, October 08, 2013 - link

    This is a cheat for marketing so how about a marketing solution to a marketing problem. Here in Ireland stuff like "Guaranteed Irish", "Bord Bia Approved" (Food Board) and meat traceability help sell mid -> premium level food products.
    Since most of these cheats are for mid -> premium level phones, anandtech and the benchmark vendors could / should produced such a Marque / Label /Standard to indicate no gaming of benchmarking occurred (I'll leave the title /snappy phrasing to someone in marketing).
    It then becomes a battle between the marketing departments in the different companies to
    1) secure this Label (or face stigma)
    and
    2) gain the highest scores with it (i.e. without gaming).
    As a Fandroid I'd hate to hand Apple an advantage, but if you get them (and possible Motorola / Google) to carry it and make a big deal about it, the other OEM should fall in line.
    Reply
  • clsmithj - Saturday, October 12, 2013 - link

    So because Samsung's mobile device operate at a battery optimized state during normal use, you're upset that because the device's software automatically enables it's hardware's full performance on the detection of benchmark software use so you can benchmark's your device for the purpose to see it's full potential? You're deeming the results as fraudulent?

    Am I missing something here? Have some you fellow tech nerds gone mad. Did you forget the whole point of mobile device's power saving CPU states are counteractive to benchmarking software designed to rate your system at it's full performance.

    There's no controversy here, just stupidity.
    Reply
  • BARTHOLOMAUS - Monday, October 14, 2013 - link

    This is blatant cheating, just like the samsung panel lottery was designed to mislead consumers into paying top dollar for rubbish panels Reply
  • martixy - Saturday, October 19, 2013 - link

    Well you just gotta turn on your common sense I guess.
    Unsurprisingly I've managed to avoid all cheaters.
    I have 2 nexus devices and nV's Shield has been on my wishlist for sometime.

    Simply ignore "marketting" and be done with it.
    Reply
  • FauXbenchmarX4phonez - Tuesday, October 22, 2013 - link

    Anand, you indicated that you've spoken to both the hardware mfgrs and the carriers : did these companies respond (directly or indirectly or perhaps not at all) when you presented your findings? Did any of them even acknowledge it? There's an underlying issue of trust at stake here and how they handle their communications will play into that. Of course, by continuing their practice of gaming and corrupting the integrity of these benchmarks results, they've already said a LOT.

    Looking ahead or even now, at the high base level of performance from all mid to high range phones, I wonder how much I'll prioritize 5%-%10 (real, not rigged) differences in benchmark scores in my buying decision. There's still no excuse for the cynical behavior that you've exposed and it's essential that these companies are held accountable. Thanks for all of your hard work!
    Reply
  • Sukesh1090 - Saturday, November 30, 2013 - link

    What about Sony?
    Are they involved in this shame business?
    Reply
  • JordanKwon - Wednesday, March 05, 2014 - link

    http://arstechnica.com/gadgets/2014/03/samsungs-ki...
    No more such sh*t in KitKat.
    Reply
  • loluvz - Tuesday, July 01, 2014 - link

    Uhh, running the CPU at max is just running a Performance Kernel Governer.. Nearly all custom kernels provide this feature. I run my phone with a performance governer 24/7 so it doesn't matter whether I am running a benchmark app or any other app, my CPU will scale up to max freq, regardless. These optimizations are NOT CHEATING!!
    They provide real world speed improvements, and the benchmark programs register that.
    FAKE CELL PHONE MAKERS will mod a version of Antutu that actually alters the scores. THIS IS CHEATING!!
    Have you ever heard of an overclock?? This is a TECH SITE, GET HACKING!! lolz
    Reply

Log in

Don't have an account? Sign up now