Launching the #CPUOverload Project: Testing Every x86 Desktop Processor since 2010
by Dr. Ian Cutress on July 20, 2020 1:30 PM ESTCPU Tests: Office
Our previous set of ‘office’ benchmarks have often been a mix of science and synthetics, so this time we wanted to keep our office section purely on real world performance.
Agisoft Photoscan 1.3.3: link
Photoscan stays in our benchmark suite from the previous benchmark scripts, but is updated to the 1.3.3 Pro version. As this benchmark has evolved, features such as Speed Shift or XFR on the latest processors come into play as it has many segments in a variable threaded workload.
The concept of Photoscan is about translating many 2D images into a 3D model - so the more detailed the images, and the more you have, the better the final 3D model in both spatial accuracy and texturing accuracy. The algorithm has four stages, with some parts of the stages being single-threaded and others multi-threaded, along with some cache/memory dependency in there as well. For some of the more variable threaded workload, features such as Speed Shift and XFR will be able to take advantage of CPU stalls or downtime, giving sizeable speedups on newer microarchitectures.
For the update to version 1.3.3, the Agisoft software now supports command line operation. Agisoft provided us with a set of new images for this version of the test, and a python script to run it. We’ve modified the script slightly by changing some quality settings for the sake of the benchmark suite length, as well as adjusting how the final timing data is recorded. The python script dumps the results file in the format of our choosing. For our test we obtain the time for each stage of the benchmark, as well as the overall time.
The final result is a table that looks like this:
The new v1.3.3 version of the software is faster than the v1.0.0 version we were previously using on the old set of benchmark images, however the newer set of benchmark images are more detailed (and a higher quantity), giving a longer benchmark overall. This is usually observed in the multi-threaded stages for the 3D mesh calculation.
Technically Agisoft has renamed Photoscan to MetaShape, and is currently on version 1.6.2. We reached out to Agisoft to get an updated script for the latest edition however I never heard back from our contacts. Because the scripting interface has changed, we’ve stuck with 1.3.3.
Application Opening: GIMP 2.10.18
First up is a test using a monstrous multi-layered xcf file we once received in advance of attending an event. While the file is only a single ‘image’, it has so many high-quality layers embedded it was taking north of 15 seconds to open and to gain control on the mid-range notebook I was using at the time.
For this test, we’ve upgraded from GIMP 2.10.4 to 2.10.18, but also changed the test a bit. Normally, on the first time a user loads the GIMP package from a fresh install, the system has to configure a few dozen files that remain optimized on subsequent opening. For our test we delete those configured optimized files in order to force a ‘fresh load’ each time the software in run.
We measure the time taken from calling the software to be opened, and until the software hands itself back over to the OS for user control. The test is repeated for a minimum of ten minutes or at least 15 loops, whichever comes first, with the first three results discarded.
The final result is a table that looks like this:
Because GIMP is optimizing files as it starts up, the amount of work required as we increase the core count increases dramatically.
Ultimately we chose GIMP because it takes a long time to load, is free, and actually fits very nicely with our testing system. There is software out there that can take longer to start up, however I found that most of it required licences, wouldn’t allow installation across multiple systems, or that most of the delay was contacting home servers. For this test GIMP is the ultimate portable solution (however if people have suggestions, I would like to hear them).
110 Comments
View All Comments
jebo - Wednesday, July 22, 2020 - link
Can we get a rundown of the underlying systems being used? RAM etc.Thanks for this!
GeoffreyA - Wednesday, July 22, 2020 - link
Astounding work, Ian! All the best on the project.Kdam - Wednesday, July 22, 2020 - link
Thanks for the effort. I was wondering if it was possible to include a cam benchmark (mastercam or other)nathanddrews - Thursday, July 23, 2020 - link
Would it be possible to add a sort or filter to see 95th percentile frame rates only? A filter by quality level? It would make reading the data much easier. QOLOldTech920 - Thursday, July 23, 2020 - link
Your CPU table (on page 2) is weirdly incomplete for Nehalem and Westmere CPUs. Specifically, it's missing the whole 1st generation Nehalem HEDT parts (aka "Bloomfield" 45 nm chips using the X58 chipset), such as i7-920, i7-940, through i7-975 EE . Combined with a recent GPU, these are still amazingly viable 4-core/8-thread CPUs.Robberbaron12 - Monday, July 27, 2020 - link
THere is no support for X58 and skt 1366 anymore in the latest version of Win 10, so its not possible to install the test suite. I know it still works if you had a 3-4 year old version on Win 10 but you can to clean install now, and I'm pretty sure skt 1156 is going the same way.Oxford Guy - Tuesday, July 28, 2020 - link
Windows 10 is a disgrace.juraj2 - Friday, July 24, 2020 - link
That is a great project. I would like to see as performance per watt has been changing during the years. Also, current benchmarks show for example CPU with 105W, but that is completely false because during the test CPU was consuming much more power. This makes results confusing and mostly in favour of Intel. Intel is cheating a lot in this regard.Oxford Guy - Tuesday, July 28, 2020 - link
Real power consumption is definitely more interesting than the "let's pretend" TDP numbers.alpha754293 - Monday, July 27, 2020 - link
This is fantastic!!!I was the person who asked for the OpenSSL benchmark because I was moving a lot of data around and needed SHA256 to ensure the data transfers completed successfully.
Thank you for putting this together.