Linux Software iSCSI with Bonding and Multipath

So our first idea was simply to simply add two additional virtual NIC's to the VM, give them both IP address, and configure each one to connect to a different iSCSI port on the MDS by specifying the interfaces and portals on with iscsiadm.  Basically the idea was this:

Linux Interface   iSCSI Interface    MDS iSCSI Port
---------------   ---------------  -------------------
eth1 10.10.1.10       iface0       iSCSI2/1 10.10.1.20
eth2 10.10.1.11       iface1       iSCSI2/2 10.10.1.21

Each of the iSCSI ports on the MDS would be zoned to allow each iSCSI session to see both the upper and lower controllers in the Apple Xserve RAID.  Combine that with dm-multipath, we'd have two connections each to each half of the Apple Xserve RAID which should be load balanced.

Well, that was the plan, but for some reason, this just didn't work.  We actually managed to get it all setup, and it seemed OK, we could create a striped LVM (across both halves of the Xserve RAID), format the volume (ext4 w/journal) and run IOzone, but we kept getting strange iSCSI noop timeouts that were triggering path failures and causing poor performance.  We were seeing about 130MB/sec writes, and 60-70MB/sec reads, not to bad for writes on this older array, but not even close on the read performance we know we can get.  We couldn't really figure out what was going on until we ran a packet trace and realized that Linux was sometimes sending packets with eth2's source IP address via eth1 and vice versa.  I don't think there's anything technically wrong with this, but it seemed to be confusing the Cisco MDS.  When the Cisco would see a packet from "wrong" NIC it appeared to try to send it's response to that NIC rather than the actual NIC, and this caused all sorts of problems.

We tested with only a single NIC and it worked perfectly, but we weren't sure how to proceed from there.  We thought we could probably do some magic with policy routing, but we decided the next best setup would be to use Linux bonding to bond the two virtual NIC's to a single IP address.  The Linux bonding feature is quite mature and flexible, so the thought was to bond the two NIC, and make four connections, two to each MDS port, from the same IP and MAC address.

 Linux Interface    iSCSI Interface    MDS iSCSI Port
------------------  ---------------  -------------------
bond0 (eth1, eth2)      default      iSCSI2/1 10.10.1.20
    10.10.1.10          default      iSCSI2/2 10.10.1.21

We decided to use the Linux bonding mode 6, balance-alb, since this balances the load of both transmit and receive packets in a round-robin fashion.  To make this work reliably we had to allow "Promiscuous" mode on the virtual switch in VMware.

With the bond0 interface configure we then used iscsiadm to create connections to both ports of our MDS, once again giving us a total of four connections to the Xserve RAID, one to the upper and lower controller via each port, but now the MDS sees all these connections from the same IP address. We kick off our iozone benchmarks and, not only are our timeouts gone, but write performance jumps to 150-160MB/sec, and read performance tops out around 210MB/sec, pretty close to maxing out dual gigabit links, and reasonably impressive for a virtual machine using software iSCSI to talk via a iSCSI-to-FC to a 5+ year old array that still uses PATA drives.  I was pretty happy with the result.