Launching the #CPUOverload Project: Testing Every x86 Desktop Processor since 2010
by Dr. Ian Cutress on July 20, 2020 1:30 PM ESTCPU Tests: Office
Our previous set of ‘office’ benchmarks have often been a mix of science and synthetics, so this time we wanted to keep our office section purely on real world performance.
Agisoft Photoscan 1.3.3: link
Photoscan stays in our benchmark suite from the previous benchmark scripts, but is updated to the 1.3.3 Pro version. As this benchmark has evolved, features such as Speed Shift or XFR on the latest processors come into play as it has many segments in a variable threaded workload.
The concept of Photoscan is about translating many 2D images into a 3D model - so the more detailed the images, and the more you have, the better the final 3D model in both spatial accuracy and texturing accuracy. The algorithm has four stages, with some parts of the stages being single-threaded and others multi-threaded, along with some cache/memory dependency in there as well. For some of the more variable threaded workload, features such as Speed Shift and XFR will be able to take advantage of CPU stalls or downtime, giving sizeable speedups on newer microarchitectures.
For the update to version 1.3.3, the Agisoft software now supports command line operation. Agisoft provided us with a set of new images for this version of the test, and a python script to run it. We’ve modified the script slightly by changing some quality settings for the sake of the benchmark suite length, as well as adjusting how the final timing data is recorded. The python script dumps the results file in the format of our choosing. For our test we obtain the time for each stage of the benchmark, as well as the overall time.
The final result is a table that looks like this:
The new v1.3.3 version of the software is faster than the v1.0.0 version we were previously using on the old set of benchmark images, however the newer set of benchmark images are more detailed (and a higher quantity), giving a longer benchmark overall. This is usually observed in the multi-threaded stages for the 3D mesh calculation.
Technically Agisoft has renamed Photoscan to MetaShape, and is currently on version 1.6.2. We reached out to Agisoft to get an updated script for the latest edition however I never heard back from our contacts. Because the scripting interface has changed, we’ve stuck with 1.3.3.
Application Opening: GIMP 2.10.18
First up is a test using a monstrous multi-layered xcf file we once received in advance of attending an event. While the file is only a single ‘image’, it has so many high-quality layers embedded it was taking north of 15 seconds to open and to gain control on the mid-range notebook I was using at the time.
For this test, we’ve upgraded from GIMP 2.10.4 to 2.10.18, but also changed the test a bit. Normally, on the first time a user loads the GIMP package from a fresh install, the system has to configure a few dozen files that remain optimized on subsequent opening. For our test we delete those configured optimized files in order to force a ‘fresh load’ each time the software in run.
We measure the time taken from calling the software to be opened, and until the software hands itself back over to the OS for user control. The test is repeated for a minimum of ten minutes or at least 15 loops, whichever comes first, with the first three results discarded.
The final result is a table that looks like this:
Because GIMP is optimizing files as it starts up, the amount of work required as we increase the core count increases dramatically.
Ultimately we chose GIMP because it takes a long time to load, is free, and actually fits very nicely with our testing system. There is software out there that can take longer to start up, however I found that most of it required licences, wouldn’t allow installation across multiple systems, or that most of the delay was contacting home servers. For this test GIMP is the ultimate portable solution (however if people have suggestions, I would like to hear them).
Facebook
Twitter
RSS
110 Comments
View All Comments
Sootie - Tuesday, July 21, 2020 - link
Any chance of a crowd sourced version of the bench? People with unusual CPU's could run a cut down version of the bench with only software that does not require a license and heavily disclaimed that it was not an official run just to add a few more data points of rare devices. I have a whole museum of old servers I can run some tests on but it's not practical to send them elsewhere.I'm a big fan of all the work you have done and are doing on the bench though I use it constantly for work and home.
Tilmitt - Tuesday, July 21, 2020 - link
Phenom II X6 and X4 would be cool to see if the "more cores make future proof" narrative actually holds up.lmcd - Tuesday, July 21, 2020 - link
X6 outperformed early Bulldozer 8 cores by a notable bit if that's of any interest.loads2compute - Tuesday, July 21, 2020 - link
Dear Ian,Wow! What a nice idea to test all these legacy processors on modern benchmarks. I think it is a great idea!
But als wow, what an enormous effort you are taking on automating all that stuff, starting from scratch and using autohotkey as your main tool. It seems like going to an uninhabited island, starting civilization from scratch and taking a tin opener as your main tool.
In my line of work (bioinformatics) we have to automate a load of consecutive tasks. Luckily there are frameworks for this, which make the work a lot easier.
Luckily there is already a framework for automated testing and benchmarking which happens to work on Linux, Mac and Windows (and even BSD). It is called the phoronix test suite http://phoronix-test-suite.com/. It can be extended with modules, so you could integrate all your desired tests in there. There is even paid support available, but since they guy who runs this (Michael Larabel) is working on a fellow tech outlet (phoronix.com) I am sure you can work something out to your mutual benefit. No doubt he is interested in all these old processor benchmarks too!
The phoronix test suite also comes with phoromatic, which according to the website : "allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface."
So please do not start from scratch and do this yourself! Use this great open-source tool that is already available and consequently you will be able to get a lot more work done on the stuff that actually interests you! (I take it AHK scripting is not your hobby).
Ian Cutress - Tuesday, July 21, 2020 - link
Scripts are already done :)The issue is that a lot of tests have a lot of different entry points; with AHK I can customizer for each. I've been using it for 5 years now, so coding isn't an issue any more.
Fwiw, I speak with Michael on occasion. We go to the same industry events etc
eek2121 - Tuesday, July 21, 2020 - link
Was procuring a new GPU really that hard? I am going to blame your owner on this one. If you were an independent website I honestly would have purchased a 2080ti and donated it to you. It honestly seems like not being independent is hurting you more than it is helping. Without going into specifics, I know of websites smaller than AT that can afford at least 3 good full time writers and a bunch of awesome hardware.I have toyed with the idea of starting an alternative site where all hardware is procured in the retail channel. I know what advertising rates are like and I know that using affiliates, sponsorships, and advertising more than cover the cost of a few models per generation. Maybe it’s time AT staff strike out on their own. Just a thought.
Outside of that, I look forward to future endeavors.
Ian Cutress - Tuesday, July 21, 2020 - link
Procuring a GPU is always difficult, as we don't have the bandwidth to test AIB cards any more.Fwiw AT only has 2/3 FT writers.
If we were to spin back out, we'd need investors and a strategy.
Igor_Kavinski - Tuesday, July 21, 2020 - link
Request: Core i7-7700K DDR3 benchmarks (There are Asus and Gigabyte mobos that allow DDR3 to be used) to compare with Core i7-7700K DDR4 benchmarks. Thanks!Xex360 - Tuesday, July 21, 2020 - link
Very fascinating.dad_at - Tuesday, July 21, 2020 - link
Pls include HEDT Sandy Bridge E: one of Core i7 3960X, 3970X, 3930K, etc. Once it was present in the CPU bench, but you removed it since 2017...