Johan, gmyx, and Bert all make good points. I'd like to add to those, last one first:
Bert, thanks for linking to my blog! You're certainly free to look at it in the same way as Johan, but that blog didn't make any claims about "typical" performance. Instead it was narrowly focused on SPECjbb2005 for 2 reasons: to counteract FUD about ESX performance on that particular workload, and to show the strength of the ESX scheduler under a CPU over-committed scenario. To see the big picture, you'll need to look at it as one data point among many.
gmyx, of course I completely agree with you. Multi-VM tests are very important, especially when the total number of virtual CPUs exceeds the number of CPU cores. However, it is not easy to design a multi-VM benchmark with varied workloads. I admit to some bias here, but the only success in this area I know of is VMmark: http://www.vmware.com/products/vmmark/">http://www.vmware.com/products/vmmark/
Johan, thanks for the insightful article! I especially like the bits on large pages and the importance of getting good native performance first. My only quibble: there are other ways to see virtualization overhead besides workloads with a significant kernel component. Some virtualization products may look beautiful running one benchmark in a single VM, but fall apart when many VMs are run in a resource-constrained environment, even if the workloads don't have a big kernel component. I suggest tests with CPU and memory over-commitment to see what happens when you try to squeeze the most out of a virtualization solution.
This:
Xensource came up with a number of benchmarks which had to proof that Xen was by far superior
Should be this:
Xensource came up with a number of benchmarks which had to prove that Xen was by far superior
Feel free to knock out this comment after the fix.
The question is: what happens with multiple VMs running the code? This to me is the real benchmark. I know many server where I work we consolidated into a simple machine because they were not very busy.
Doing a 1 VM to 1 host in my opinion is not very useful. I would like to know how a VM scales from a 1 to 1 config to 2 to 1 and higher.
I remember taking a MS course a year ago that had us running 4 VMs at once, a PDC, a SDC, a W2K client and a WXP client. It was not very fast on Virtual PC 2004/2007.
We’ve updated our terms. By continuing to use the site and/or by logging into your account, you agree to the Site’s updated Terms of Use and Privacy Policy.
5 Comments
Back to Article
jeffb - Saturday, August 23, 2008 - link
Johan, gmyx, and Bert all make good points. I'd like to add to those, last one first:Bert, thanks for linking to my blog! You're certainly free to look at it in the same way as Johan, but that blog didn't make any claims about "typical" performance. Instead it was narrowly focused on SPECjbb2005 for 2 reasons: to counteract FUD about ESX performance on that particular workload, and to show the strength of the ESX scheduler under a CPU over-committed scenario. To see the big picture, you'll need to look at it as one data point among many.
gmyx, of course I completely agree with you. Multi-VM tests are very important, especially when the total number of virtual CPUs exceeds the number of CPU cores. However, it is not easy to design a multi-VM benchmark with varied workloads. I admit to some bias here, but the only success in this area I know of is VMmark: http://www.vmware.com/products/vmmark/">http://www.vmware.com/products/vmmark/
Johan, thanks for the insightful article! I especially like the bits on large pages and the importance of getting good native performance first. My only quibble: there are other ways to see virtualization overhead besides workloads with a significant kernel component. Some virtualization products may look beautiful running one benchmark in a single VM, but fall apart when many VMs are run in a resource-constrained environment, even if the workloads don't have a big kernel component. I suggest tests with CPU and memory over-commitment to see what happens when you try to squeeze the most out of a virtualization solution.
Bert Bouwhuis - Wednesday, August 20, 2008 - link
Last Monday, VMware published similar benchmark results using SPECjbb2005 (see http://blogs.vmware.com/performance/2008/08/esx-ru...">http://blogs.vmware.com/performance/2008/08/esx-ru.... Should we be look at those the same way?davecason - Monday, August 18, 2008 - link
Johan,Change "proof" to "prove"
This:
Xensource came up with a number of benchmarks which had to proof that Xen was by far superior
Should be this:
Xensource came up with a number of benchmarks which had to prove that Xen was by far superior
Feel free to knock out this comment after the fix.
gmyx - Monday, August 18, 2008 - link
The question is: what happens with multiple VMs running the code? This to me is the real benchmark. I know many server where I work we consolidated into a simple machine because they were not very busy.Doing a 1 VM to 1 host in my opinion is not very useful. I would like to know how a VM scales from a 1 to 1 config to 2 to 1 and higher.
I remember taking a MS course a year ago that had us running 4 VMs at once, a PDC, a SDC, a W2K client and a WXP client. It was not very fast on Virtual PC 2004/2007.
Looking forward to your VM comparison article.
skiboysteve - Sunday, August 17, 2008 - link
great article.