Rightware, makers of MobileMark, BrowserMark, and other mobile benchmarking tools recently launched a new 3D benchmark for mobile devices named BaseMark GUI. BaseMark joins our growing suite of mobile device benchmarks and tests OpenGL ES performance on Android and other platforms with an emphasis on 3D user interfaces. We've run BaseMark on almost all the devices we've got on hand, and will do so going forward on other devices we get our hands on.

We've been making a lot of additions to our mobile benchmarking suite lately in an effort to get the best picture possible of the performance landscape out there for tablets and smartphones alike. First it was GLBenchmark 2.0, now it's Basemark. This new test benchmarks OpenGL ES 1.0 or 2.0 performance on devices through a suite of five feature tests and one overall 3D user interface test. In its current incarnation, we've been given binaries for the Android platform, however Basemark's Kanzi 2.0 engine, which is written entirely in ANSI-C, will be ported to other platforms in the future. For now however, we're focusing on testing Android hardware with Basemark and using the OpenGL ES 2.0 render pipelines since all of the Android devices we have (and coming in the future) are of the 2.0 sort.

Basemark divides benchmarking into two categories - feature tests and the final overall user interface test. Up first are the feature tests. These are named animation, vertex streaming, texture streaming, and alpha blending as shown in the menu above. Each tests a different collection of what Rightware believes are representative graphics tasks that will be leveraged in user interfaces moving forward. The first item in that test menu screenshot above - 3D UI - is an overall graphics performance test designed to simulate a completely 3D user interface with example application interfaces. It's one of the heaviest 3D benchmarks I've seen yet. 

First up among those feature tests is animation, which animates a graphics element (in this case, a robot moving through a set of actions) by stepping through a table of keyframes and interpolating the character's movement between using splines. This particular benchmark uses ES 2.0 APIs and per vertex lighting, but the purpose of this test is to be as CPU-bound as possible and specifically stress floating point performance.


Animation Test

Next are the vertex and texture streaming tests, both of which measure asset streaming performance - according to RightWare, effectively memory bandwidth. The vertex test loads geometry data into GPU memory and frees it when no longer needed. The scene itself involves a lot of geometry - ten repeating city blocks which the camera moves through with increasing speed and draw distance. The test ramps from around 3k vertices to 15k vertices per frame, and 190k to 250k triangles per frame. There's a single texture material, fog, two lights, and OpenGL ES 2.0 shaders which use per vertex lighting.


Vertex Test

The texture test is a bit more straightforward, quickly loading images from RAM into the GPU memory and discarding them.


Texture Test

These asset streaming tests effectively test memory bandwidth from disk into RAM and into to GPU memory. Although Kanzi supports ETC texture compression (and Basemark exposes this option for testing), we've initially tested with the default configuration which has ETC texture compression disabled. 

Blend testing - as its name implies - tests alpha blended rendering by drawing a number of semi-transparent contact cards atop each other. These are overlaid sequentially until we reach a desired number of layers. This test actually runs twice, first with front to back ordering of these contact cards, and then with back to front ordering.

This ordering difference shouldn't be confused with the fact that feature test actually runs in both back to front and front to back rendering orders and are combined later.


Blend Test

Next is the composition test, which composites interface screens rendered off-scene into one large tiled user interface. The example here are applications rendering in the background being composited into one view. This test combines 16 off-scene objects. 


Composition Test

There's also a battery test which loops the composition test indefinitely until battery dies. Going forward we will likely use Basemark's battery test to augment our three other ongoing battery test metholodogies. 


3D UI Test

Last but not least is an overall 3D performance test which emulates a futuristic interface consisting of applications rendered on screens attached to a central... orb. This is definitely the most challenging of the Basemark suite, combining post processing blur for depth of field and basic blur, per pixel lighting with three point lights and one directional light, cube and normal mapping, ambient occlusion mapping, and a shader on the arms holding the displays around the cube which animates a texture map. To start we ran with 4xAA enabled (by Basemark default), though we're running Basemark GUI with AA turned off as well. Remember for a moment that all of the phones we've seen so far are tile-based renderers. 

Screenshots and Performance Results
POST A COMMENT

20 Comments

View All Comments

  • strikeback03 - Tuesday, January 11, 2011 - link

    Not knowing who they get their test phones from (carrier, manufacturer, or both depending on circumstance) they just might not have any Verizon or Motorola phones around anymore. Though that should really be remedied to have a TI SoC or two in tests like this.

    Could it be possible that the tests the Adreno 205 pulls ahead with are ones where the additions Qualcomm has made to the Scorpion CPU help it out over a standard A8?
    Reply
  • metafor - Tuesday, January 11, 2011 - link

    Since this is a GUI test, I suspect that is the case. Only specific functions can be run on the GPU, for most of the more complicated GUI work, the CPU is generally used.

    But I'd need to know more details about the test.
    Reply
  • Brian Klug - Tuesday, January 11, 2011 - link

    thornburg,

    strikeback03 has it exactly right - currently we're sampled by carriers, and most if not all of the time they want their devices back about a month after launch. Unfortunately, the result is that for tests like these that are new, we don't have much we can do other than test everything we have and try to snag older devices for comparison.

    It's one of the things we're still working (a lot) on making better. :)

    -Brian
    Reply
  • DaFox - Tuesday, January 11, 2011 - link

    I loved the music in that video. I had just left it open while it played out in the background. Reply
  • rsj56 - Wednesday, January 12, 2011 - link

    agreed! wonder what/who the title/artist is... Reply
  • Mr Perfect - Tuesday, January 11, 2011 - link

    Fascinating, Captain. Reply
  • austonia - Tuesday, January 11, 2011 - link

    poor EVO... 6 months old and already bottom of the barrel. graphics are good enough for me but then i don't use it for gaming. Hopefully something with Tegra 2 is available this summer (upgrade every year with Sprint Premiere perks). Reply
  • silverblue - Tuesday, January 11, 2011 - link

    ...that the Adreno 205 was a tile-based renderer. I thought only PowerVR offered those. Still, you learn something new everyday. :)

    I also found out that AMD owned what is now the Adreno line until they sold it off to Qualcomm in 2008. I bet they're kicking themselves, especially considering they were good products even back then.
    Reply
  • Huulio - Thursday, January 13, 2011 - link

    I think it is more about benchmarking the whole system; GPU, CPU and disk performance in all together.

    Anyway it looks very nice :)
    Reply
  • Loki7T1 - Thursday, February 03, 2011 - link

    Hmm...this seems like a weak excuse. Seriously, you guys couldn't procure ANY OMAP-based devices? NONE? I would think that at the very least someone on the staff would own one.

    This does explain why all the Droid X benchmarks are with Android 2.1, though.

    AnandTech is a great site and can do better.
    Reply

Log in

Don't have an account? Sign up now