Testbed Setup and Testing Methodology

Thunderbolt's adoption, till now, has mainly been driven by the Mac ecosystem. Solutions for DIY PC enthusiasts are not widespread. GIGABYTE has a few motherboard models with Thunderbolt integrated, while Asus's Thunderbolt-capable motherboards need an add-on PCIe card. Our Windows-based testbed for direct-attached storage (DAS) systems is based on the latest Z97 chipset from Intel. The build originally started out as a multimedia editing machine. The requirements of a DAS testbed fit in nicely with the original purpose of the build. The following table details the configuration of our testbed.

AnandTech DAS Testbed Configuration
Motherboard Asus Z97-PRO Wi-Fi ac ATX
CPU Intel Core i7-4790
Memory Corsair Vengeance Pro CMY32GX3M4A2133C11
32 GB (4x 8GB)
DDR3-2133 @ 11-11-11-27
OS Drive Seagate 600 Pro 400 GB
Optical Drive Asus BW-16D1HT 16x Blu-ray Write (w/ M-Disc Support)
Add-on Card Asus Thunderbolt EX II
Chassis Corsair Air 540
PSU Corsair AX760i 760 W
OS Windows 8.1 Pro
Thanks to Asus and Corsair for the build components

We decided to prepare the testbed with 32 GB of memory in order to make a large enough RAM drive available for our file copy tests. Similar to our NAS testbed, we used OSFMount to allocate 20 GB of the DRAM for a RAM disk partition.

The internal layout of the Z97-Pro (Wi-Fi ac) board is presented below.

It should be noted that the Thunderbolt add-on card can only be connected to PCIe lanes hanging off the PCH (and can't be connected directly to the CPU). Apparently, this is a requirement for vendors to obtain Thunderbolt certification from Intel. In the above layout, we see that the add-on card can only go to the bottom PCIe slot (PCIEX16_3). In addition to the hardware restriction, the BIOS actually operates this slot in x1 mode, with the other three lanes being shared with the ASMedia USB 3.0 ports in the rear panel and the x1 PCIe slot (PCIE X1 SLOT4). The BIOS setting was altered to operate the x16 slot in x4 mode (maximum possible bandwidth for the card). The BIOS entry also has a PCIe cache line setting for Thunderbolt. It was left at 32 bytes (the default) for our evaluation.

Our testing methodology for DAS units takes into consideration the usual use-case for such devices. The most common usage scenario is transfer of large amounts of photos and videos to and from the unit. The minor usage scenario is importing files directly off the DAS into a multimedia editing program such as Adobe Photoshop.

In order to tackle the first use-case, we created three test folders with the following characteristics:

  • Photos: 15.6 GB collection of 4320 photos (RAW as well as JPEGs) in 61 sub-folders
  • Videos: 16.1 GB collection of 244 videos (MP4 as well as MOVs) in 6 sub-folders
  • BR: 10.7 GB Blu-ray folder structure of the IDT Benchmark Blu-ray (the same that we use in our robocopy tests for NAS systems)

For the second use-case, we take advantage of PC Mark 8's storage bench. The storage workload involves games as well as multimedia editing applications. The command line version allows us to cherry-pick storage traces to run on a target drive. We chose the following traces.

  • Adobe Photoshop (Light)
  • Adobe Photoshop (Heavy)
  • Adobe After Effects
  • Adobe Illustrator

Usually, PC Mark 8 reports time to complete the trace, but the detailed log report has the read and write bandwidth figures which we present in our performance tables. Note that the bandwidth number reported in the results don't involve idle time compression. Results might appear low, but that is part of the workload characteristic. Note that the same CPU is being used for all configurations / will be used for future DAS reviews. Therefore, comparing the numbers for each trace should be possible across different DAS units.

Introduction Performance Evaluation - 2big Thunderbolt 2
POST A COMMENT

42 Comments

View All Comments

  • Kevin G - Wednesday, July 9, 2014 - link

    Thunderbolt has its niche of being a means to host PCIe devices externally. For laptops, this is a pretty nice feature but for systems like the Mac Pro, it doesn't make sense when internal PCIe could have been an option. The other catch is that the one specific devices users would like to connect via Thunderbolt is not officially supported: GPUs. Reply
  • AlValentyn - Wednesday, July 9, 2014 - link

    What it tells me is that TB over an add-on card with Windows is slow. Not that TB is slower than USB. That's false statement as TB2 is 20Gb/s, while USB3.0 is 5Gb/s.

    You don't see USB3 driving 60Hz 4K displays, or getting over 800MB/s on RAIDs, and SSDs.

    I'm surprised they didn't even bother with OSX, and Mac with built in Thunderbolt as well.
    Reply
  • Shadowself - Wednesday, July 9, 2014 - link

    Exactly.

    Even with the multi-hop (and hacked BIOS) TB2 is only 3% slower in RAID 0 mode. I suspect the greater *apparent* advantage USB 3.0 has over TB2 in RAID 1 mode has to do much, much more with LaCie's implementation of the hardware raid and translation from USB to RAID 1 versus translation from TB2 to RAID 1. Since RAID 0 is definitely more bandwidth hungry (given *zero* other bottlenecks through the entire system) then there should be no reason why TB2 is significantly slower at RAID 1 versus RAID 0. Ganesh should have caught this.

    To really test TB2 versus USB 3.0 for any external device, the test setup must include native implementations of both TB2 and USB 3.0 or else the results are hopelessly tainted.
    Reply
  • ganeshts - Wednesday, July 9, 2014 - link

    This review is meant to address what a Windows user looking to get on the Thunderbolt bandwagon should expect.

    I stand by my conclusions: For 2-bay devices with no daisy chaining requirements, USB 3.0 is better than Thunderbolt for Windows users. When it comes to 5 bays, things may be different.
    Reply
  • casperes1996 - Thursday, July 10, 2014 - link

    With all due respect, the review was of the drives though. Not the drives (for Windows). I read the review as a Mac user, wanting to know the performance over TB. TRIM over TB on the Mac is also something I am now quite curious about.

    Would it be possible to perhaps get another review, or an addendum to this one, testing the drive on a Mac?

    The review was fine for what it was, but I think we are many curious about the Mac side, as it is where Thunderbolt is more proliferated.
    Reply
  • Kristian Vättö - Friday, July 11, 2014 - link

    Thunderbolt in Windows isn't any different from Thunderbolt in a Mac. It's the same protocol with the same performance. The only difference is that in a Mac Thunderbolt is "invisible" to the end-user because Apple's EFI is locked and the drivers come with the OS, whereas in Windows you can play with some settings in BIOS and the drivers need to be installed manually.

    Testing these drives in a Mac wouldn't give any different results. Like I mentioned earlier, I have the same add-on card and have been able to reach speeds of over 700MB/s with a TB1 device, so the bottleneck in these LaCie drives is elsewhere.
    Reply
  • repoman27 - Friday, July 11, 2014 - link

    I'm not going to argue that you should have tested with a Mac, and I can fully understand why Anandtech would stick with a Windows based testbed for DAS devices to make results comparable (and Ganesh doesn't currently have a Mac). But saying that Thunderbolt device performance is the same under both Mac OS X and Windows is like saying that games should perform the same on both OSes, or that you get the same performance and battery life from a Mac whether you run Windows or Mac OS X.

    At just a very base level, Apple's EFI implementation may offer performance benefits over Microsoft's hybrid UEFI / Windows software stack model. Also, each OEM's hardware implementation can have performance implications. Most Macs use PCIe lanes provided by the CPU, not the PCH. Since there's apparently a requirement for add-in cards to use the PCH lanes, they're inherently at a disadvantage. Even more so in real-world scenarios when using a board that has 5 PCIe switches on the PCH lanes alone resulting in a brutally oversubscribed DMI connection.

    Thunderbolt essentially looks like nothing more than a PCIe switch to the OS, which doesn't require any special drivers at all. The Thunderbolt "driver" is all about supporting PCIe hot-plugging, tolerating up to 9 µs of round-trip latency, and enforcing Intel's licensing agreements. What you do need to worry about is the drivers for the PCIe based controllers in any device you connect. This is obviously the same whether it is a PCIe add-in card or external Thunderbolt device, and no different under Windows or Mac OS X. The most glaring omission in this article is not reporting which host controllers are in the devices and what drivers were being used for testing. As readers we have no idea whether the Thunderbolt tests were performed using Microsoft or Marvell (or whoever's) SATA host controller drivers, whether AHCI was enabled, or whether the TRIM support issue was a result of the drivers being used. On the USB side, we can only infer that a native Intel USB 3.0 port was used (since the Asmedia controller was disabled in order to provide a PCIe x4 connection for the Thunderbolt add-in card) with whatever the current version is of the Microsoft driver under Windows 8.1 Pro, and UASP was supported by both drives. These types of details really need to be presented along with the other data to live up to the Anandtech ethos. I don't just want a quick benchmark; I want to understand the underlying limitations and how they factor into the results, and to see the hardware pushed as far as it can go.
    Reply
  • Teo_ - Tuesday, August 12, 2014 - link

    The first review I can find on a quick search based on a Mac reports sequential read and write speed in RAID 0 412.7MB/s and 353.3MB/s, so I’m curious too to see the same sample folders and tasks tested on a Mac. Reply
  • GTVic - Friday, July 11, 2014 - link

    Thunderbolt is misunderstood, there is no "translation to TB2" it is an extension of the PCI Express Bus and also supports Display Port. Reply
  • Kristian Vättö - Wednesday, July 9, 2014 - link

    I think the performance is limited by the ASMedia chips (SATA & SATA to PCIe/TB bridge). I have the same add-on card as Ganesh and have been able to achieve speeds of around 700MB/s (this is with TB1). I'm getting a proper TB2 device soon, so stay tuned for a more thorough review of the add-on card (as well as more Thunderbolt stuff).

    As for OS X, as far as I know Ganesh does not have a Mac with Thunderbolt (and neither do I). Just because we don't test with something doesn't mean that it's due to our laziness -- Apple doesn't send review samples around like e.g. ASUS does so we would have to spend our own money to get one for testing.
    Reply

Log in

Don't have an account? Sign up now