Javascript Performance

Although smartphones are clearly headed for a life beyond simple messaging, web browsing and phone duties, we are still lacking the tools to measure performance in areas other than a component of web page rendering. Measuring javascript performance is one component of the entire web page rendering process but it's the most mature in terms of something we can benchmark.

Sunspider is quite possibly the most well known of these javascript tests:

SunSpider Javascript Benchmark 0.9.1 - Stock Browser

We expected Snapdragon S4 to do very well here based on our preview numbers and it did not disappoint. Although it's not quite as fast as Intel's Medfield, it's clearly the fastest smartphone SoC in this test otherwise. It's worth noting that HTC is able to deliver performance that's within 5% of Qualcomm's Snapdragon S4 MDP, a significant improvement over where things were last year.

The Sunspider test makes good use of 1 - 2 cores but there are times when it'll stress all four on a Tegra 3. For the most part however, NVIDIA's extra cores go unused in this benchmark. Krait and HTC's ICS browser definitely offer a significant advantage over the performance of the ICS launch vehicle, Samsung's Galaxy Nexus. This is honestly the hardware that Google should have used as the basis for its Nexus platform, it's too bad that it wasn't available at the end of last year.

BrowserMark is another js benchmark in our suite:

BrowserMark

Here the international and AT&T One Xes trade spots as Tegra 3's extra cores are able to give it the slight edge in performance. Once again we're talking about the fastest smartphones in our test suite.

Low Level FP Performance

Linpack isn't a great indication of smartphone performance, but it is a good test of the floating point capabilities of the CPUs in these SoCs. ARM has steadily been improving FP performance for the past few generations but we're going to see a big jump to Krait/A15. As most client smartphone workloads are integer based and those that are FP heavy end up relying on the GPU, an advantage here doesn't tell us much today (particularly because Linpack isn't running native code but rather atop Dalvik) other than how speedy the FPUs are:

Linpack - Single-threaded

Improved single threaded FP and memory performance is something we get with Krait, and it's demonstrated quite well by the AT&T One X. While you're unlikely to see this magnitude of an advantage in most real world smartphone workloads, this is an architectural advantage of Qualcomm's Krait that's worth mentioning. However these days most FP intensive workloads on smartphones are handled by the GPU, making this performance advantage mostly academic at this point.

Linpack, like many scientific workloads, scales up to multiple cores quite nicely. If we spawn as many threads as there are logical cores (2 for Intel and Qualcomm, but 4 for NVIDIA's Tegra 3) we can see how these SoCs fare in a multithreaded world:

Linpack - Multi-threaded

BaseMark OS

Rightware's BaseMark OS is a general purpose benchmark designed to better simulate overall Android performance. It includes a heavily threaded benchmark, file IO tests, and compression/decompression tasks that all contribute to its overall score. We only have results from the HTC One S/AT&T One X (Snapdragon S4), One X (Tegra 3), Galaxy Nexus (OMAP 4) and the Lava phone (Medfield) here:

BaseMark OS Performance

As expected, the AT&T One X does very well in this general purpose, OS-level benchmark. The device significantly outperforms Samsung's Galaxy Nexus, which isn't a surprise given how dated the hardware was at the time of launch. Again the magnitude of advantage is likely exaggerated by this benchmark, but there's no changing the fact that HTC makes the fastest Android smartphones on the planet for now.

Note that the performance difference between the two One X models is likely exaggerated by BaseMark OS as well. In our regular use we found the two to be fairly similar in performance, with the slight edge going to the AT&T One X but not by a huge margin. You'd be hard pressed to tell these two apart.

Vellamo

Vellamo is a Qualcomm developed benchmark that focuses primarily on browser performance, both in rendering and UI speed. The results are heavily influenced by the browser used on the device being tested. As a whole Vellamo isn't always indicative of whether or not you're going to get a smooth browsing experience, but it's another datapoint that captures more than just javascript performance. The Qualcomm-developed nature of the benchmark is always cause for concern, but even if you exclude the Snapdragon results the benchmark can be useful:

Vellamo Overall Score

Qualcomm's SoCs have always done very well in this Qualcomm-built benchmark, but the results still echo what we've seen in practice. The AT&T One X definitely feels like the fastest Android phone we've used, especially compared to much of what was available last year. The gap between the international and AT&T One X versions is, once again, debatable. The Qualcomm performance advantage is at times perceivable, but I wouldn't consider it to be significant at all.

Flash Rendering Performance

These days nearly all high-end smartphones (I refuse to call them superphones) can render Flash smoothly. The HTC One X is no exception. We're basically at vsync here and will have to move to either more stressful flash content to test or just accept that flash is already performant enough.

CraftyMindFlash Rendering Performance

GPU Performance - GLBenchmark 2.1

As we wait for actual 3D gaming benchmarks to make their way into Android (and hopefully crossplatform) games, we must rely on synthetic tests designed to simulate 3D game performance as best as possible. We start with GLBenchmark, one of the better Android GPU tests on the market today. There are two benchmarks, Egypt and Pro, and each is run in two modes: native screen resolution and offscreen (vsync disabled) at 720p. The latter is more useful for apples to apples comparisons as everything is rendering the same number of pixels, whereas performance in the onscreen tests is determined by the screen resolution of the device along with the performance of its GPU.

GLBenchmark 2.1 - Egypt

Part of the deal in getting Krait to market as quickly as possible required that Qualcomm pair the CPU with an older GPU, in this case the Adreno 225 instead of the newer Adreno 3xx offerings due out later this year. As a result, the AT&T One X can take a back seat to the international One X in GPU performance. When not limited by v-sync, the Tegra 3 based international One X is about 12% faster than the Qualcomm S4 based AT&T One X. For most gaming however, you'll be hard pressed to notice a performance difference.

GLBenchmark 2.1 - Egypt - Offscreen (720p)

GLBenchmark 2.1 - Pro

The Pro offscreen results actually flip flop a bit with the AT&T One X leading the international version. Once again, limited by v-sync (onscreen results) the two are basically identical performers.

GLBenchmark 2.1 - Pro - Offscreen (720p)

Basemark ES 2.0 V1

Rightware's Basemark ES 2.0 V1 is an aging GPU test that tends to favor Qualcomm's Adreno GPUs above almost all others:

RightWare Basemark ES 2.0 V1 - Taiji

RightWare Basemark ES 2.0 V1 - Hoverjet

We're not surprised to see the AT&T One X do so well here as it appears Qualcomm has done quite a bit of optimization work for this particular benchmark. I wouldn't put too much faith in these numbers other than to show you an example of just what can happen with a good amount of optimization on behalf of a hardware vendor.

Battery Life Camera - Stills and Video
POST A COMMENT

137 Comments

View All Comments

  • MrMilli - Tuesday, May 01, 2012 - link

    "On the GPU side, there's likely an NVIDIA advantage there as well."

    How do you get to this conclusion?
    Qualcomm scores a little bit higher in Egypt as in the Pro test of GLBenchmark. I don't know why you would put any importance to the off-screen tests for these two devices since they both run the same resolution (which is even 720p) which takes me to my next points. Actual games will be v-synced and how does the Tegra suddenly become faster than the Adreno even though they both are still rendering at the same resolution as on-screen but just with v-sync off. I've always had a hard time accepting the off-screen results of GLBenchmark because there's no way to verify if a device is actually rendering correctly (or maybe even cheating). Can you imagine testing a new videocard in the same fashion?
    Reply
  • metafor - Tuesday, May 01, 2012 - link

    Results can vary with v-sync because Tegra could be bursting to higher fps values. The offscreen isn't a perfect test either but it gives you an idea of what would happen if a heavier game that didn't approach the 60fps limit would be like.

    Of course, those games likely won't have the same workloads as GLBenchmark, so it really wouldn't matter all that much.
    Reply
  • ChronoReverse - Tuesday, May 01, 2012 - link

    The offscreen test is worthless really.

    If at 720p, the same benchmark, except it puts an image on the screen, shows that the S4 GPU is faster than the Tegra3 GPU, then how useless is the offscreen test showing the opposite?

    Furthermore, neither the S4 nor Tegra3 comes close to 59-60FPS, both tipping at around the 50FPS range.

    It's pretty clear that by skipping the rendering, the offscreen test is extremely unrealistic.
    Reply
  • metafor - Wednesday, May 02, 2012 - link

    It doesn't need to come close. It just needs to burst higher than 60fps. Let's say that it would normally reach 80fps 10% of the time and remain 40fps the other 90%. Let's say S4 were to only peak to 70fps 10% of the time but remained at 45fps the other 905. The S4's average would be higher with v-sync while Tegra's would be higher without v-sync.

    The point of the benchmark isn't how well the phone renders the benchmark -- after all, nobody's going to play GLBenchmark :)

    The point is to show relative rendering speed such that when heavier games that don't get anywhere close to 60fps are being played, you won't notice stutters.

    Of course, as I mentioned, heavier games may have a different mix of shaders. As Basemark shows, Adreno is very very good at complex shaders due to its disproportional ALU strength.

    Its compiler unfortunately is unable to translate this into simple shader performance.
    Reply
  • ChronoReverse - Wednesday, May 02, 2012 - link

    That's still wrong. If you spike a lot, then your experience is worse for 3D games. It's not like we don't know that minimum framerate is just as important.

    As you mentioned stutters, a device that dips to 40FPS would be more stuttery than one that dips only to 45FPS.
    Reply
  • metafor - Thursday, May 03, 2012 - link

    I'm not disagreeing. I'm just saying that v-sync'ed results will vary even if it's not close to 60fps. Because some scenes will require very little rendering (say, a panning shot of the sky) and some scenes will require a lot of heavy rendering (say, multiple characters sword fighting, like in Egypt).

    The average fps may be well below 60fps. But peak fps may be a lot higher. In such cases, the GPU that peaks higher (or more often) will seem worse than it is.

    Now, an argument can be made that a GPU that also has very low minimum framerates is worse. But we don't know the distribution here.
    Reply
  • Chloiber - Monday, May 07, 2012 - link

    Well the benchmark doesn't measure your experience in 3D games but the fps. Reply
  • snoozemode - Tuesday, May 01, 2012 - link

    The ATRIX has a LCD pentile RGBW display, as well as the HTC one S, so LCD is definitely not a guarantee for RGB. Maybe you should correct the article with that. Reply
  • snoozemode - Tuesday, May 01, 2012 - link

    Sorry one s is obviously amoled. Reply
  • ImSpartacus - Tuesday, May 01, 2012 - link

    It's also the passable RGBG pentile, not the viled RGBW pentile. Reply

Log in

Don't have an account? Sign up now