3DMark for Android: Performance Preview
by Anand Lal Shimpi & Brian Klug on April 2, 2013 7:00 AM ESTAs I mentioned in our coverage of GL/DXBenchmark 2.7, with the arrival of Windows RT/8 we'd finally see our first truly cross-platform benchmarks. Kishonti was first out of the gate, although Futuremark was first to announce its cross platform benchmark simply called 3DMark.
Currently available for x86 Windows 8 machines, Futuremark has Android, iOS and Windows RT versions of 3DMark nearing release. Today the embargo lifts on the Android version of 3DMark, with iOS and Windows RT to follow shortly.
Similar to the situation with GL/DXBenchmark, 3DMark not only spans OSes but APIs as well. The Windows RT/8 versions use DirectX, while the Android and iOS versions use OpenGL ES 2.0. Of the three major tests in the new 3DMark, only Ice Storm is truly cross platform. Ice Storm uses OpenGL ES 2.0 on Android/iOS and Direct3D feature level 9_1 on Windows RT/8.
The Android UI is very functional and retains a very 3DMark feel. There's an integrated results brower, history of results and some light device information as well:
There are two options for running Ice Storm: the default and extreme presets.
3DMark - Ice Storm Settings | ||||||||
Default | Extreme | |||||||
Rendering Resolution | 1280x720 | 1920x1080 | ||||||
Texture Resolution | Normal | High | ||||||
Post-processing Quality | Normal | High |
Both benchmarks are rendered to an offscreen buffer at 720p/1080p and then scale up to the native resolution of the device being tested. This is a very similar approach we've seen by game developers to avoid rendering at native resolution on some of the ultra high resolution tablets. The beauty of 3DMark's approach here is the fact that all results are comparable, regardless of a device's native resolution. The downside is we don't get a good idea of how some of the ultra high resolution tablets would behave with these workloads running at their native (> 1080p) resolutions.
Ice Storm is divided into two graphics tests and a physics test. The first graphics test is geometry heavy while the second test is more pixel shader intensive. The physics test, as you might guess, is CPU bound and multithreaded.
Before we get to the results, I should note that a number of devices wouldn't complete the tests. The Intel based Motorola RAZR i wouldn't run, the AT&T HTC One X (MSM8960) crashed before the final score was calculated so both of those devices were excluded. Thankfully we got the Galaxy S 3 to complete, giving us a good representative from the MSM8960/Adreno 225 camp. Thermal throttling is a concern when running 3DMark. You have to pay close attention to the thermal conditions of the device you're testing. This is becoming something we're having to pay an increasing amount of attention to in our reviews these days.
Graphics Test 1
Ice Storm Graphics test 1 stresses the hardware’s ability to process lots of vertices while keeping the pixel load relatively light. Hardware on this level may have dedicated capacity for separate vertex and pixel processing. Stressing both capacities individually reveals the hardware’s limitations in both aspects.
In an average frame, 530,000 vertices are processed leading to 180,000 triangles rasterized either to the shadow map or to the screen. At the same time, 4.7 million pixels are processed per frame.
Pixel load is kept low by excluding expensive post processing steps, and by not rendering particle effects.
Although the first graphics test is heavy on geometry, it features roughly 1/4 the number of vertices from GL/DXBenchmark 2.7's T-Rex HD test. In terms of vertex/triangle count, even Egypt HD is more stressful than 3DMark's first graphics test. That's not necessarily a bad thing however, as most Android titles are no where near as stressful as what T-Rex and Egypt HD simulate.
Among Android smartphones, Qualcomm rules the roost here. The Adreno 320 based Nexus 4 and HTC One both do very well, approaching 60 fps in the first graphics test. The Mali 400MP4, used in the Galaxy Note 2 and without a lot of vertex processing power, brings up the rear - being outperformed by even NVIDIA's Tegra 3. ARM's Mali-T604 isn't enough to pull ahead in this test either; the Nexus 10 remains squarely behind the top two Adreno 320 based devices.
Graphics Test 2
Graphics test 2 stresses the hardware’s ability to process lots of pixels. It tests the ability to read textures, do per pixel computations and write to render targets.
On average, 12.6 million pixels are processed per frame. The additional pixel processing compared to Graphics test 1 comes from including particles and post processing effects such as bloom, streaks and motion blur.
In each frame, an average 75,000 vertices are processed. This number is considerably lower than in Graphics test 1 because shadows are not drawn and the processed geometry has a lower number of polygons.
As you'd expect, shifting to a more pixel shader heavy workload shows the Galaxy Note 2 doing a lot better - effectively tying the Tegra 3 based HTC One X+ and outperforming the Nexus 7. The Mali-T604 continues to, at best, tie for third place here. Qualcomm's Adreno 320 just seems to deliver better performance in 3DMark for Android.
The overall score pretty much follows the trends we saw earlier. Qualcomm's Adreno 320 leads things (Nexus 4/HTC One), followed by ARM's Mali-T604 (Nexus 10), Adreno 225 (SGS3), Adreno 305 (One SV), Tegra 3 (One X+/Nexus 7) and finally Mali 400MP4 (Note 2). The only real surprise here is just how much better Adreno 320 does compared to Mali-T604.
Physics Test
The purpose of the Physics test is to benchmark the hardware’s ability to do gameplay physics simulations on CPU. The GPU load is kept as low as possible to ensure that only the CPU’s capabilities are stressed.
The test has four simulated worlds. Each world has two soft bodies and two rigid bodies colliding with each other. One thread per available logical CPU core is used to run simulations. All physics are computed on the CPU with soft body vertex data updated to the GPU each frame. The background is drawn as a static image for the least possible GPU load.
The Physics test uses the Bullet Open Source Physics Library.
The physics results give us an indication of just how heavily threaded this benchmark is. The quad-core devices are able to outperform the dual-core Cortex A15 based Nexus 10, despite the latter having far better single threaded performance. The Droid DNA/Optimus G vs. Nexus 4 results continue to be a bit odd, perhaps due to the newer drivers included in the Nexus 4's use of Android 4.2 vs. 4.1.2 for the other APQ8064 platforms.
The overall Ice Storm results don't paint a vastly different picture from what we've seen thus far:
We will likely have to revisit this as non-Nexus devices get updated to Android 4.2, especially given the strange delta between the Nexus 4 and Optimus G/Droid DNA. When the One gets updated to Android 4.2.x, it'll be interesting to see if it pulls ahead of the Nexus 4 in 3DMark.
It's also interesting to note that 3DMark seems to value the Adreno 320 more than ARM's Mali-T604 in the Nexus 10. I tossed results from the HTC One into the T-Rex HD benchmark data from GL/DXBenchmark 2.7 and found that the two benchmarks seem to agree on this point as well:
The magnitude of difference between the One (Adreno 320) and Nexus 10 (Mali-T604) isn't as large, but we are looking at two very different benchmarks here.
3DMark includes an optional demo run that will generate a score as well (although these results aren't included in the overall Ice Storm score):
Just for kicks I ran the extreme (1080p, high quality textures and post processing) presets on a couple of devices to see how the numbers changed:
3DMark for Android - Default vs. Extreme Settings Comparison | ||||||||||||
HTC One | Google Nexus 10 | Google Nexus 7 | ||||||||||
Default | Extreme | Default | Extreme | Default | Extreme | |||||||
Ice Storm Score | 11309 | 6190 | 7990 | 5499 | 3539 | 1895 | ||||||
Graphics Score | 11250 | 5942 | 7952 | 5030 | 3130 | 1581 | ||||||
Physics Score | 11520 | 7246 | 8127 | 8165 | 6527 | 6196 | ||||||
Graphics Test 1 | 49.2 fps | 29.8 fps | 36.9 fps | 27.3 fps | 12.2 fps | 7.5 fps | ||||||
Graphics Test 2 | 48.7 fps | 22.8 fps | 32.5 fps | 18.2 fps | 15.3 fps | 6.4 fps | ||||||
Physics Test | 36.6 fps | 23.0 fps | 25.8 fps | 25.9 fps | 20.7 fps | 19.7 fps | ||||||
Demo | 52.5 fps | 25.7 fps | 35.2 fps | 21.5 fps | 16.8 fps | 7.0 fps |
Moving up to the Extreme presets definitely increases the workload on these devices. It's important to note that today's flagships are close to running 3DMark at extreme presets at 30 fps in some cases. On average, we're looking at low to mid 20 fps for the best of the best on Android today.
3DMark for Android seems to be another good data point to have. While the holy grail remains a good collection of actual games with integrated benchmarks representative of game play, at least having a good selection of simulated game tests is a lot better than where we were just a year ago.
The fact that all of 3DMark's benchmarks are comparable across devices regardless of display resolution is a nice touch, although I do appreciate GLBenchmark's ability to target the native resolution of any device (especially when dealing with very high res tablets). The inclusion of a dedicated physics test is very nice, however I'd like to see equal emphasis placed on single (or lightly) threaded performance as the real world tends to not be as thread happy as 3DMark's physics test implies.
Overall, the 3DMark Ice Storm workload is clearly less stressful than GL/DXBenchmark, which makes me wonder how it'll age over time. On the bright side, 3DMark will make a nice companion to GL/DXBenchmark - offering a glimpse at what GPU performance would look like in workloads closer to what is presently available, rather than forecasting what may be possible in the future. We plan on using both benchmarks in our reviews going forward.
56 Comments
View All Comments
Mugur - Tuesday, April 2, 2013 - link
Well... all I want to say is that I regret that OUYA don't have an Adreno 320 GPU instead of a Tegra 3... :-)ChronoReverse - Tuesday, April 2, 2013 - link
What's interesting about this benchmark is that it agrees with benchmarks like Basemark's Taiji test or Epic's Unreal Citadel test in that Adreno does a lot better than traditionally given credit for in GLbenchmark.I'm glad there'll be another tool in the stable to provide a more balanced view of GPUs.
Infy2 - Tuesday, April 2, 2013 - link
The fastest physics score is around 11,000 while my Core7-3770K at 3.9GHz (4 cores+HT) scored only 52,000. That can't be right can it? If it is it would mean a mobile CPU at TDP of few watts scores 22% performance of a 77W TDP desktop CPU!The mobile GPU graphics score seems more plausible at 11,400 to my Radeon 7970 GE's score of 320,000, which makes it some 28 times faster.
thecoldanddarkone - Tuesday, April 2, 2013 - link
On my surface pro it got 24.5k and used 100 percent of the cpu. On my 3820@4.3 it never reached above 50% thread load.thecoldanddarkone - Tuesday, April 2, 2013 - link
Look at that I had 55k with hyperthreading and 65k without. Yea... (all 4 cores were pegged)mwildtech - Tuesday, April 2, 2013 - link
They must have a crappy Droid DNA? Mine scored much higher @ Total score of 8800 in standardPentius_5123 - Tuesday, April 2, 2013 - link
Pretty much the same here, I posted my stats and they seem to be 10 FPS under on everything...KompuKare - Tuesday, April 2, 2013 - link
Can we have the SOC or better the vendor, corec count, GPU and speeds listed in the graphs please?Not everyone follows phone / tablet SOCs in so much detail that they can remember which is which; I know it would make the graphs a bit wide - or maybe have the device take up two lines?
Gambit2K - Tuesday, April 2, 2013 - link
I got 11502 on the Ice Storm with my Lg Optimus G E975 @ stock settings. Nothing tweaked in anyway oter than it's rooted and the bootloader is unlocked. Android 4.1.2. The scores on the the for the OG seems a bit on the low side.Pentius_5123 - Tuesday, April 2, 2013 - link
Going to Cross-Post my stats from a similar Cnet article to provide a bit of insight to the HTC DNA (Internationally, Butterfly 'American Version')Ice Storm Score: 8768
Graphics Score: 10262
Physics Score: 5809
Graphics Test1: 46.1 FPS
Graphics Test 2: 43.2 FPS
Physics Test: 18.4 FPS
Demo: 49.8 FPS
OS Version: 4.1.1
1.5GHz Quad Core Krait
Adreno 320 GPU
2GB Ram