I published an academic paper at the IEEE International Symposium on Workload Characterization (IISWC 2007) in September that I want to spend some time talking about. The paper was entitled "Easy and Efficient Disk I/O Workload Characterization in VMware ESX Server". Here's the abstract:

Collection of detailed characteristics of disk I/O for workloads is the first step in tuning disk subsystem performance. This paper presents an efficient implementation of disk I/O workload characterization using online histograms in a virtual machine hypervisor-VMware ESX Server. This technique allows transparent and online collection of essential workload characteristics for arbitrary, unmodified operating system instances running in virtual machines. For analysis that cannot be done efficiently online, we provide a virtual SCSI command tracing framework. Our online histograms encompass essential disk I/O performance metrics including I/O block size, latency, spatial locality, I/O interarrival period and active queue depth. We demonstrate our technique on workloads of Filebench, DBT-2 and large file copy running in virtual machines and provide an analysis of the differences between ZFS and UFS filesystems on Solaris. We show that our implementation introduces negligible overheads in CPU, memory and latency and yet is able to capture essential workload characteristics.

It seems a lot of people agreed with my previous post on the security of virtual switches. These include the originator of the information that prompted my blog post. Chris Wolf himself posted comments recognizing his misunderstanding. I think Chris did a great job of quickly following up after my blog post and getting in touch with us to resolve the misunderstanding. Kudos to him. Read his comments for yourself.

In an article titled VMware dispels virtualization myths, Bridget Botelho wrote:

"One significant issue with virtual machine security is with virtual switch isolation," said Burton Group's Wolf."The current all-or-nothing approach to making a virtual switch 'promiscuous' in order to connect it to an IDS/IPS is not favorable to security."

For example, "if you connect an IDS appliance to a virtual switch inpromiscuous mode," Burton said, "not only can the IDS capture all of the traffic traversing the switch, but every other VM on the same virtual switch in promiscuous mode could capture each other's traffic as well. "Users should be aware of this and work around it."

Typically aggregate statistics like mean, median and standard deviations from mean are not detailed enough. Furthermore, they can be misleading! See figure where the mean of the data is 5.3 but the we can clearly see from the histogram that a bimodal distribution is in play. There isn't even a single data point close to the mean. Granted this particular example is contrived but it is not far from the typical situation in the real world. One example is if reads to a device were hitting the device cache only about 1/2 the time.

Next week, I'll be attending VMworld 2007, the virtualization community's annual conference. Actually I won't be the only one given that more than 10,000 people are attending this year! Furthermore, famous people are keynoting:

• Diane Greene, VMware's CEO
• Mendel Rosenblum, VMware's Chief Scientist
• John T. Chambers, Cisco Systems, Inc.
• Patrick Gelsinger, Intel Corporation
• Hector de J. Ruiz, AMD

This year, I'll be giving two talks: "Fast and Easy Disk Workload Characterization on VMware ESX Server" and "ESX Storage Performance - A Scalability Study".

VMmark 1.0 is now live.

This release is the culmination of lot of effort by the VMmark team. Big congrats to them to get this tool available to the virtualization community.

Having worked in the virtualization performance arena for about 5 years, I can attest to the need for standard benchmark. There are so many variables due to the layers of software running on layers and then resource sharing on top that it is very difficult to make sense of data presented by customers, partners, the press and the community at large. VMmark attempts to address this by creating a standard benchmark.

I found Micorosft's recent Live Migration related announcement rather puzzling.

The Register published an article "Microsoft promises VMware beater despite reversals" that everyone should read. You'll get a laugh out of it. I sure did and it started my morning off just right!

Instead of providing details of this bogus idea of "quick migration", the Microsoft rep takes the press to task:

"The recent press has been inaccurate to say we don't do migration - we do migration: quick migration"

The anxiously awaited RHEL5 was released recently. The following is from RHEL5 release notes. Looks like virtualization support (with Xen 3) is only half- or perhaps only quarter-baked.

"Fully virtualized guests cannot be saved, restored or migrated."

So, does that mean no suspend/resume and no VMotion?

"Hardware-virtualized guests cannot have more than 2GB of virtual memory."

One really shouldn't try putting Enterprise or even SMB workloads with only 2GB upper limit.

"When you install a fully virtualized guest configured with vcpus=2, the fully virtualized guest may take an unreasonably long time to boot up. To work around this, destroy the slow-booting guest using the command xm destroy and then use xm create to start the same guest afterwards."

Hi:

Welcome to my ramblings. An introduction is in order. Irfan is the name. Software Engineering is the profession. Performance analysis and optimization is the specialty. VMware, Inc. is the employer. Islam is the way of life. Peace is the motto (Love for All; Hatred for None).

I've been at VMware over 4 years. Most recently, I have worked for Transmeta, Autodesk Canada, Toronto Stock Exchange and Canada Trust (now TD) in various capacities including (in no particular order) software engineer, software performance engineer, programmer, intern, research analyst, etc. I graduated from the University of Waterloo in Ontario with a degree in Mathematics/Computer Science.

For the last few months, I've been reading blogs of friends and others in the area of virtualization. For what it's worth, I'm overall impressed with the coverage of interesting virtualization topics in the blogosphere. However, I've felt a distinct need for more coverage of real and present issues in the day to day operations of virtual machine software. So, I hope to bring to readers an interesting mix of tips & tricks, commentary & opinion, future directions and hopefully some fun fact alongs the way. One virtual scoop at a time.

Irfan