The AnandTech Podcast: Episode 26by Anand Lal Shimpi on October 3, 2013 11:45 PM EST
- Posted in
On the list for today is Surface 2/Pro 2, Galaxy Note 10.1 2014 Edition, Galaxy Gear, Galaxy Note 3, Cheating in Android Benchmarks and a preview of our 2013 iMac review.
The AnandTech Podcast - Episode 26
featuring Anand Shimpi, Brian Klug
RSS - mp3, m4a
Direct Links - mp3, m4a
Total Time: 1 hour 44 minutes
Surface 2 - 0:00
Samsung Galaxy Note 10.1 2014 Edition - 0:11
Samsung Galaxy Gear - 0:35
Samsung Galaxy Note 3 - 0:46
Cheating in Android Benchmarks - 0:58
iPhone 5s Camera - 01:26
The new iMac - 1:31
Post Your CommentPlease log in or sign up to comment.
View All Comments
dylan522p - Friday, October 4, 2013 - linkWhy don't you guys make your own benchmark that is actually good? You guys did it in the SSD space and the benchmarking scene there was not nearly as big nor bad.
dishayu - Friday, October 4, 2013 - linkI don't think it's even possible to cheat in SSD benchmarks. I believe they made the SSD benchmark to simulate real-world performance rather than purely synthetic numbers. And I think I read Anand's tweet/comment somewhere that they do plan to make their secret, undisclosed (to phone manufacturers) benchmark in the future.
Anand Lal Shimpi - Friday, October 4, 2013 - linkIt absolutely is possible, which is why I've never given out our tests (or even low level details of our tests) to manufacturers even though they've asked over the years.
The same is true for our smartphone/tablet battery life tests.
Anand Lal Shimpi - Friday, October 4, 2013 - linkIt's extremely expensive :)
What we've done in the SSD space is ideally what we'd want to do here, but it's not quite that simple.
This isn't a no, just means it's something that's going to require a lot more thought.
Kevin G - Friday, October 4, 2013 - linkHave you considered the open source route for a mobile benchmark? While having the source open would let manufacturers know exactly what tests and how they're done, it would enable peer review of the code and allow other site to do a custom build that'd evade basic application detection.
dylan522p - Friday, October 4, 2013 - linkThey would detect certainly code and almost certainly cheat.
Kevin G - Friday, October 4, 2013 - linkThe ability to modify/build it yourself can be used to make this increasingly difficult.
dylan522p - Friday, October 4, 2013 - linkHope you guys can afford one. Maybe kickstarter it and offer nothing in return. Just use the money to build the benchmark. In either case I just hope you don't let anyone touch it so they can't cheat it.
Callitrax - Friday, October 4, 2013 - linkFirst, when discussing how devices feel in hand, I expect analyses of polar moment of inertia from now on.
I think one of the big problem is you need unique benchmark suites for phones, or browsers.
The PC used to have some of these issues, but not so much anymore because most of the programs used to test computer CPUs are the same programs the user actually uses. When you tested Haswell it was with Excel, 7zip, Visual Studio, Photoshop and so on. Those are real programs that people use, so if a vendor "cheats" at those, well it just makes those programs actually faster for users. When benchmarking video cards at this point, its pretty much exclusively with video games (if there are synthetic benches in a video card review I just skip it). At this point its just understood that AMD and NVidia tweak their drivers to improve performance in big name games, but you know what - that means those games are actually performing faster, so its actually a good thing. (as long as they aren't degrading video quality like the old nvidia quake problems)
So really the problem appears to come down to a lack commonly used apps and games on phones? Because then "cheating" on those would just make life better for everyone, well except that the current Android cheating method kills battery life.
teiglin - Friday, October 4, 2013 - linkI think the problem is twofold. First, there is a distinct lack of automation tools. Windows automation is a mature, well-developed market, but this is much less true in the mobile/ARM space. Not to mention any such benchmark could well require root privileges in order to launch other apps and profile them. Is it even possible to write something like fraps as a non-root app for Android or iOS?
The other problem is basically what you mention--a lack of obvious benchmarking targets. Desktop benchmarks hit obvious, common tasks that people do with computers, like file compression, video encoding/decoding, rendering, and code compilation. The only tasks besides gaming that I can see being remotely applicable to mobile from a desktop benchmark are photo editing and maybe spreadsheet calculations. And mobile game developers don't include benchmarking tools, so I have to refer you back to my first point, which is that I at least don't know how you would go about benchmarking mobile games.
I'd love to see Anand's and Brian's approach to a good mobile benchmark, but it's definitely not a simple problem.