Troubleshooting VMware ESX network performance

When we first decided to switch from our fiber channel SAN (EMX Clariion CX series) to a simpler iSCSI solution (Equallogic PS series) at our remote sites we did so based on performance numbers that showed we could actually get better performance numbers out if the iSCSI solution, even with software iSCSI, that with the aging, and far more complex CX solution.  When we brought the PS300E into the lab for testing against an HS20 blade we were amazed that we could acheive performance of 85-90MB/s between a single VM using a single GigE link, near 90% utilization.  Running two VM's using two GigE connections got us to 180MB/s pretty easily, significantly faster than we had ever managed to get our CX400 to acheive.

We purchased our PS300E and installed it at the remote site where ESX server was running on IBM x336 systems.  We then re-ran the same benchmarks against the PS300E using the production systems.  This time the performance numbers were not so good, with transfer rates in the 30-40MB/s range.  We broght the array back to the lab site and ran our reference benchmark against the new PS300E, performance was aroun 90MB/s, right about what we expected.

We really couldn't understand what was going on, the specs between the two hardware platforms were nearly idenctical, Dual Intel Xeon 3.2Ghz processors, 8GB RAM, Broadcom GigE NIC's.  Why would there be such a disparity?  We suspected a network layer problem but after looking at switch configs over and over we just didn't see any of the usual suspects (duplex mismatch, increasing error counts, etc).

Since we were moving from Fiber Channel to iSCSI we had purchased a Intel Pro/1000 GT Quad card to replace the Qlogic FC HBA (yes, we could have purchased a hardware iSCSI HBA but our tested had indicated that this really would not be required to achieve the level of performance we needed and, based on forum feedback, the VMware support for Qlogic iSCSI adapters seemed a little flaky).  We installed the Intel Pro/1000 into one of the IBM x336 cards and re-ran the test.  Boom, performance was back in the 90MB/sec range.

We could have probably stopped there, we didn't absolutely need to use the onboard ports, however, now we were curious why the Broadcom onboard ports on a x336 would perform so much worse than the onboard Broadcom ports on an HS20.  We began to investiage possible scenarios and ran across something interesting, when looking at the various files in /proc we came across the /proc/vmware/interrupts file and noticed it looked like this (edited slightly to fit on the page):

Vector PCPU  0    PCPU  1    PCPU  2    PCPU  3
0x21:        2          0          0          0 COS irq 1 (ISA edge), <VMK device>
0x29:        0          0          0          0 <COS irq 4 (ISA edge)>
0x31:        0          0          0          0 <COS irq 6 (ISA edge)>
0x39:        0          0          0          0 <COS irq 7 (ISA edge)>
0x41:        0          0          0          0 <COS irq 8 (ISA edge)>
0x49:        0          0          0          0 <COS irq 9 (ISA edge)>
0x51:       15          0          0          0 COS irq 12 (ISA edge)
0x59:        0          0          0          0 <COS irq 13 (ISA edge)>
0x61:        1          0          0          0 COS irq 14 (ISA edge)
0x69:        0          0          0          0 <COS irq 15 (ISA edge)>
0x71: 68505248          0          0          0 COS irq 16 (PCI level), VMK vmnic6, VMK vmnic1
0x79:      108          0          0          0 COS irq 18 (PCI level)
0x81:        0          0          0          0 COS irq 19 (PCI level)
0x91:   155710     649201     289501     732347 <COS irq 20 (PCI level)>, VMK ips
0x99: 21577461   60183980   21031867   51667695 <COS irq 21 (PCI level)>, VMK vmnic2
0xa1:   106442     348866     139914     365738 <COS irq 22 (PCI level)>, VMK vmnic3
0xa9:  9503152   27310088    9603280   24230277 <COS irq 23 (PCI level)>, VMK vmnic4
0xb1:   106439     348786     140019     365653 <COS irq 24 (PCI level)>, VMK vmnic5
0xdf:417335059  417060518  418055868  416873770 VMK timer
0xe1:  3739008    1903562    4602326    2230321 VMK monitor
0xe9: 92395997   68149255   95110679   69560265 VMK resched
0xf1:    15628      55045      42367      53573 VMK tlb
0xf9:   214270          0          0          0 VMK noop
0xfc:        0          0          0          0 VMK thermal
0xfd:        0          0          0          0 VMK lint1
0xfe:        0          0          0          0 VMK error
0xff:        0          0          0          0 VMK spurious

Of note is Vector 0x71 which shows two things, first, notice that interrupts for this vector are not distributed evenly to all of the CPU's.  Looking further to the right you can see that this is because the Console OS (COS) has at least one driver with ownership of this interrupt (note the lack of angle brackets around the "COS irq 16 (PCI level)" in the far right column), while the VM kernel (VMK) has ownership of both vmnic1 and vmnic6, which matches up with the two Broadcom onboard adapters.  Since the Console OS owns runs only on the first processor this forces all interrupts to be handled by the first processor, and then this processor must execute the interrupt handler for the Console OS and then the VM kernel which requires context switches and a fair amount of overhead.  This is actually fairly well documented for ESX 2.x in VMware KB article 1290 and it would seem that this still applies to ESX 3.

Our task then, is to figure out what driver in the Console OS is actually using the interrupt on the same vector with the VM Kernel.  This is pretty easy with following command:

# cat /proc/vmware/pci | grep 0x71
Bus:Sl.F Vend:Dvid Subv:Subd Type     Vendor   ISA/irq/Vec P M Module   Name

000:29.0 8086:24d2 1014:02dc USB      Intel    11/ 16/0x71 A C                 
007:00.0 14e4:1659 1014:02c6 Ethernet Broadcom 11/ 16/0x71 A V tg3      vmnic1 
008:00.0 14e4:1659 1014:02c6 Ethernet Broadcom 11/ 16/0x71 A V tg3      vmnic6

Of course you should replace 0x71 with the specific interrupt vector that is shown as shared on your system.  Note that the "V" and "C" in the column labled "M" (before the "Module" and "Name" columns) shows which vector is used by the VMkernel (V) or the Console OS (C).  In my example the console OS is attached to the USB interface while the two onboard NIC's are assigned to the VMkernel.  So now we know that Irq 16, which according to the output above corresponds with vector 0x71, attaches the USB hardware to the Console OS and the Broadcom nics to the VMkernel.  If we want to know specifcally which driver the Console OS is using to control the device we can simply run the following:

# cat /proc/interrupts
           CPU0      
  0:   41270280      vmnix-edge  timer
  1:          4      vmnix-edge  keyboard
  2:    3075039      vmnix-edge  VMnix interrupt
 12:         15      vmnix-edge  PS/2 Mouse
 14:          5      vmnix-edge  ide0
 16:      76567     vmnix-level  usb-uhci
 18:        156     vmnix-level  usb-uhci
 19:          0     vmnix-level  ehci-hcd

This shows that Irq 16 is being using by the usb-uhci driver in the console OS.

Ideally at this point you would simply modify your systems BIOS settings to reallocate the USB hardware interrupt onto a different interrupt than the onboard NICs, or perhaps disable USB altogether if you don't need it.  Unfortunately in the case of the IBM x336 there does not appear to be a way to change this.  You can assign the onboard NIC and USB to a different IRQ, but that just moves the problem interrupt from one vector to another, since they still both share the same one.  At first I thought of just disabling the onboard USB as I really didn't need it for anything, but I couldn't find such an option in the IBM BIOS.

So, thwarted by the system hardware I decided there were other options.  First, I was pretty sure I didn't need USB support for normal operation so I thought I would simply disable the USB driver in the console OS.  This is actually pretty simple, just run "modprobe -r usb-uhci" and it will unload the USB driver.  As soon as you do this the output of the above commands change, look at the following:

#cat /proc/vmware/interrupts
Vector PCPU  0    PCPU  1    PCPU  2    PCPU  3
0x59:        0          0          0          0 <COS irq 13 (ISA edge)>
0x61:        1          0          0          0 COS irq 14 (ISA edge)
0x69:        0          0          0          0 <COS irq 15 (ISA edge)>
0x71: 68708836       1024       9813       7738 <COS irq 16 (PCI level)>, VMK vmnic6, VMK vmnic
0x79:      156          0          0          0 <COS irq 18 (PCI level)>
0x81:        0          0          0          0 COS irq 19 (PCI level)
0x91:   157878     658589     292915     741902 <COS irq 20 (PCI level)>, VMK ips

I edited this but the important part to note is the the line with vector 0x71 now still show the VMkernel claiming the interrupts for vmnic1 and vmnic6, and that the Console OS has devices that it can claim, however, the presence of the angle brackets indicate that no driver is loaded by the Console OS for this device.  You can also see that the interrupts have started being distributed across all CPU's.  Also, the output of "cat /proc/interrupts" now shows no sign of interrupt 16 being claimed by any driver:

           CPU0      
  0:   41365483      vmnix-edge  timer
  1:          4      vmnix-edge  keyboard
  2:    3083573      vmnix-edge  VMnix interrupt
 12:         15      vmnix-edge  PS/2 Mouse
 14:          5      vmnix-edge  ide0
 19:          0     vmnix-level  ehci-hcd
NMI:          0
LOC:          0
ERR:          0
MIS:          0


After removing the driver we reran our benchmark and, sure enough, performance with the onboard Broadcom NIC's were roughly equal to the performance of the Intel Pro/1000 adapter.  If we ran "modprobe usb-uhci" performance would return to it's previously poor state, while unloading the driver with "modprobe -r usb-uhci" always returned the system back to good performance.

This proved that the usb-uhci driver was the culprit.  We initially simply modified modules.conf to remove the usb-uhci driver completely, but that turned out to have a nagative side effect that the IBM Remote Server Administrator card would no longer allow me to use the keyboard and mouse.  Apparently the RSA emulates a USB keyboard and mouse and thus removing the driver removed the ability to use the remote keyboard.  This really isn't that big of a problem since I usually just SSH into the system anyway, and if the system has crashed so hard that I can't ssh into it it's unlikely it will respond to the keyboard either.  Plus, I can still use the RSA remote power features to power cycle the system and the keyboard works fine during the boot process.

Still, I decided to write a small script that simply unloads the usb-uhci driver 5 minutes after VMware ESX boots.  That way if I'm attempting to troubleshoot something like a network connectivity issue, I can simply reboot the system, log in with the RSA and kill the script before the five minutes are up.  It's a cheap trick, but it actually works pretty well.

Another option I've thought about was to simply use the onboard NIC's as standby adapters, so that they are only used if the Intel NIC malfunctions or looses it's network connection.  Not a bad option really, even at only 300Mb/s they will still likely survive the normal workday without any major issues.

I've even considered a more complex approach, something like running mon in the console OS, if it can't ping the default gateway, thus indicating a network problem, it could load the udb-uhci driver so that the RSA console would work, then, once the network is restored it could unload the driver.

Anyway, lots of options, but, in the end, it was an interesting experience and I learned a good bit about ESX server and how the VMkernel and Console OS interact with the hardware.  Hopefully you'll find it interesting too.