Infrastructure as a Service: Benchmarking Cloud Computing
by Johan De Gelas on June 2, 2011 8:50 PM EST- Posted in
- IT Computing
- IT Computing general
- Cloud Computing
Response Time
Our current virtualization stress tests are focused on measuring the maximum throughput of a certain server CPU or server platform. As a result we try to maximize CPU load by using high amounts of concurrent (simulated) users. In other words, the concurrencies are quite a bit higher than what the (virtual or physical) machine can cope with, in order to attain 95-100% CPU load. As a result, the response times are inflated above what would be acceptable in the real world. Still, it would be interesting to get an idea of the response times for our server versus the cloud server.
Our vApus Mark II test starts off with 400 concurrent users and ends with 800 concurrent users. The 400 concurrent users still cause very high CPU load on most machines (80-90%), but that should still give us a "worst case" response time scenario. In the next graph we list the response time at the lowest concurrency. Terremark's data center was in Amsterdam, and we had an 11 to 20 ms round trip delay from our lab, so to be fair to Terremark you should deduct 11 to 20 ms from the Terremark numbers.
Both the "in house" server with 10GHz resource pool and the 10GHz Terremark "cloud server" get hit very hard by 400 concurrent MS SQL server connections, but the 10GHz resource pool that we get from the Terremark cluster is less powerful as it needs up to 85% more time to respond. The reason for that is twofold.
First, while the limit of the resource pool is at 10GHz, only 5GHz is reserved. So depending on how hard the parent resource pool is loaded by other VMs, we probably get somewhere between 5 and 10GHz of CPU power. Second, there is some extra overhead coming from the firewalls, routers, and load balancers between us and the VM. The infrastructure of Terremark is--for safety and security reasons--quite a bit more complex than our testing environment. That could add a few tens of ms too.
We also noticed that some parts of the Terremark cluster were using slightly older drivers (vmxnet 2 instead of vmxnet 3 for example), which might result in a few lost CPU cycles. But we are nitpicking and Terremark told us it is just a matter of time before the drivers get updated to the latest version.
And there is good news too. If you are willing to pay the premium for "bursting", the Terremark cluster scales very well and is capable of giving response times similar to what an "in house" server delivers. That is the whole point of cloud computing: pay and use peak capacity only when you actually need it.
Facebook
Twitter
RSS
29 Comments
View All Comments
Mxlasm - Friday, June 3, 2011 - link
This question is probably not to Anand but to some fellow readers who may wish to educate the uneducated :)In the setup descibed, can one virtual computer/OS span many physical systems? Can someone please point me to a good (wiki?) article about how that is done? There are many text about the subject of virtualization in general, but hard to nail some specific questions.
Also, how many cores max can one virtual system get? Or, in other words, if the physical system has so many cores, can you request more cores, and how can your virtual system can be efficiently scaled up if you are already reaching the max of one physical server?
Thanks!
bobbozzo - Saturday, June 4, 2011 - link
Currently, a single virtual server cannot span more than one physical server, BUT VMs can be MIGRATED between multiple physical servers in case one physical server is too busy or is failing.You can pretend you have more cores than you really do by limiting the GHz for each VM, or possibly by just over-allocating VMs and hoping they don't all get busy at once.
Regardless, if the server is maxed out, all you can do to get more performance is to migrate VMs to another server.
HMTK - Monday, June 6, 2011 - link
vSphere can currently use 6 or 12 CPU CORES (HT does not count as a core) per physical CPU. The exact number is determined by licensing. If you want to use 12 cores be prepared to pay a lot. Personally I think this is idiotic and hopefully VMware changes this in vSphere 5 later this year.VM's can use 4 or 8 vCPU's with 8 vCPU's limited to the expensive Enterprise Plus SKU.
Kid98 - Sunday, June 5, 2011 - link
You don't have to look very hard to find others with similar solutions. (cloudshare.com is one)Seems a bit like Terremark is being presented as the premier choice. How can that be ascertained without comparison to others?
Kid
sushanthr77 - Tuesday, June 7, 2011 - link
Thanks for the excellent article. Most articles about cloud are too vague and dwell into the abstracts that the details. This was well structured and presented.ProDigit - Sunday, June 12, 2011 - link
I find cloud computing nothing more than taking a bus or train in public transport!The caveats are more than the benefits. There's nothing like having your own vehicle to drive!
So with computing!
Have your own hardware, and don't depend on your internet connection to provide what hardware should!
vinaywagh - Wednesday, June 29, 2011 - link
I enjoyed reading this article. But what it lacks is a cost benefit analysis. If I were to setup a small datacenter today, I would need to hire IT engineers and pay for the power and space for it which would cost me $x. What I want to know is to get the same performance as the local server, what would it cost me to move it to the cloud ?vlang :: crmhelpdesksoftware.com - Wednesday, September 21, 2011 - link
Thank you for details re Terremark from your perspective.ShirleyBurnell - Friday, September 20, 2019 - link
Thanks for writing this awesome post on cloud computing. The way you discussed the cloud hosting and how the managed and unmanaged hosting services differs from each other is simply great. What I think is that, you must have pointed out the best option according to you in that case.For me, I have used both managed and unmanaged hosting google hosting service and I consider managed one over the un-managed hosting. I have been using Google GCE hosting server managed by Cloudways for last 2 years and it has really helped me in bringing down my hosting expenses.