Three Years of Equallogic

Yeah, when we first purchased these Equallogic arrays I really thought I had found a storage vendor to be with for a long time, but there's been precious little change in the product over the last three years and I have no idea what Dell will do with them. My fear was that they'll do what they do with almost everything else, turn it into a decent, but overall unimpressive piece of hardware with average features and average performance.

Since we no rely on so many hosts based solutions for snapshots and replication we're strongly considering just going be to cheap, boring storage. Boxes with plenty of raw performance but limited features can be had for a song nowadays.

That being said, one company that does appear to be innovating at first glance is Pillar Data. Their application aware storage, the ability to assign priority to application, goes so far as to place higher priority data closer to the edges of the physical disks, where access is faster. They also include huge caches, something which I've never understood why others don't do, memory is relatively cheap. I'm still researching them, but at first glance they seem impressive.

Only weakness is no current support for RAID 6, which would make me a little nervous if I went with drives larger than 750GB because of BER but a recent conversation with their CTO says RAID 6 is coming later this year.

Anyone else out there using or know anything good or bad about Pillar Data?

Equallogic has shown zero interest in changing this part of their design. As you stated, their comment is this is the way their "page placement" technology works and making the allocations smaller would be too much overhead.

I suspect it may also have something to do with fragmentation. Based on how I believe their technology works, if you take a snapshot, and then data is written to a "page" they allocate a completely new page and keep the old one as the snapshot. This leads to the volumes kind of fragmenting all over the array, but 16MB fragmentation isn't so bad. However, with small "pages" the fragmentation overhead could quickly become significant, and volumes with lots of random I/O could have a lot of small fragments spread all over the disk.

I actually think even their current concept causes fragmentation. I haven't had time to prove it, but it appears that, as you make snapshots, since new data is always written to new space, a volume has a tendency to become very spread out on the spindles, increasing access latency. Perhaps their technology "defragments" these pages, but it certainly "feels" like my volumes get slower as they age, and my theory is that they get fragmented.

I theorize this because I don't seem to get the same amount of degradation from volumes where I don't make heavy use of snapshots. Now, the degradation is not huge, just a few milliseconds of access time, but that's very noticeable on small block random access.

It's too bad though, Equallogic still has so many positives, easy administration, solid hardware, enterprise features, and they really do "get" iSCSI in that their array is designed to make the most of it. But in three years other vendors have closed the gap, and Equallogic products haven't advanced very much except for bigger capacities and of course SSD, but everybody's doing that. You can't differentiate yourself with those features.

Hi Eric, thanks for stopping by. I know that Equallogic has tried to add some innovative software solutions to the mix, particularly with the Auto-snapshot Manager products. These are certainly nice additions, but didn't really add much to the equation for an experienced storage admin. Pretty much it's a GUI to do stuff you could already do pretty easily if you knew what you were doing. On top of that, to really get good value out of it you pretty much had to use separate volumes for each VM which added complexity to the environment.

I'm sure 10Gb ethernet is coming, but it wouldn't be innovative, just an improvement. Innovative is what Equallogic did to start with, building and array that expands by simply adding units. Allowing data to move between array without interruptions with just a few clicks. I want to see some more innovation in the actual product, you know, things like this:

1. QoS -- Allow volumes to be prioritized over other volumes. As more and more systems share the same spindles this becomes more important and several vendors are adding this.

2. Smart Placement -- Another feature which goes improves response times by placing frequently used data where it can be accessed the fastest.

3. Thin Provision Space Reclamation -- at least one vendor is offering this. Right now thin provisioning is only marginally useful because, if a volume is ever filled, it stays allocated even if the data is removed. The idea here is to use sdelete, or another tool to write zeros to deleted data. The storage array then frees these zero'd blocks back to the thin provisioned pool.

4. More Caching -- RAM is fairly inexpensive but can be a huge boost to performance, some vendors have figured this out and are including large caches, as much a 16GB or more per controller. This can significantly lower IOP loads. I think even the new 6000 series Equallogic arrays are only using 4GB, my CX array had that back in 2004.

Since some are already doing all of these things I'm not even sure we can consider those "innovative", but it wouldn't hurt.

We were a NetApp shop many years ago but they really ticked me off when it came time to upgrade. They wanted so much money for really a very weak upgrade solution that we were able to purchase EMC gear, along with all fiber channel hardware and FCoIP gateways (expensive stuff back in 2003) for less than they were asking for upgrading to mid-range hardware.

I'm not saying I would never go back, but it put such a sour taste in my mouth that I've barely spoken to them since. Interestingly, my Equallogic sales rep at the time of my purchase is now the NetApp rep for my area.

Good points, and I didn't mean to imply that these features were all absolutely required, but was just pointing out some areas where some vendors are innovating that would be nice. What exactly the Equallogic creating that is innovative. I'm waiting to see but so far there hasn't been much.

Here's my responses to your comments:

1- somewhat less needed with virtualisation, as there are fewer volumes. Besides, VMware can do it at the VM level (disk shares). Would be nice of course.

Interestingly, I'm not convinced that virtualization doesn't make this more important. First, QoS isn't just priority, it's also about data placement. For example, Pillar Data's solution places high priority data closer to the outer edges of the disk where it's faster to access. VMware can't do that.

Also, the VMware "disk shares" is very weak in my opinion. For one the the "disk shares" is relevant to only to other VM's on a specific host or resource pool. A VM with a disk share of 10,000 on a host with 10 VM's, with the other 0 VM's having a priority of 2000, would still have less priority that a single VM on a host by itself. Simply moving VM's around can change the effective priority of the I/O for a given VM.

4- more cache is always nice, but is it really an apple-to-apple comparison ? You can't have more than 16 spindles behind an EQL module, unlike an CX one. I also clearly don't see these 2 products (CX4 and PS array) playing in the same field, CX4 being a higher tier.

But we're talking about innovation here. I wasn't comparing the Equallogic solution to a CX4, I was comparing it to my old CX700, a head unit purchased almost 5 years ago. Actually, the fact that you can only have 16 spindles is exactly why they NEED more cache. Other vendors can put more spindles which helps spread out the IOPS across a larger set of disks.

The Equallogic solution has only 16 spindles (at most) per cache even though their array can hold 8TB or more. Having 8TB of disk with 4GB worth of cache is pitiful. Yes you can spread this out across multiple arrays, but that means you have to buy many smaller arrays, which has a lot of additional costs with the Equallogic way since you have to pay for the controllers for each shelf. For other vendors it's fairly inexpensive to add disk shelfs to get more IOPS from the same cache controller. As you have more data, or lots of VM's shraring an array, this is a serious problem.

The reason snapshots are so big is because Equallogic uses 16M pages whereas Exchange uses either 4K (Exchange 2000/2003) or 8K (Exchange 2007) blocks. That means that if Exchange changes a single 4K/8K block, Equallogic will mark a entire 16M page as changed. Obviously other blocks changed within that same 16M page will not cause additional snapshot space usage, but that's not typically how Exchange databases work, they typically are reusing free spaces all over the place, and thus changing as few as 100 8k blocks can mean 1.6GB of snapshot space for the Equallogic.

The biggest cause of snapshot space usage I've seen is the nightly Exchange online defrag run. This obviously moves a lot of blocks and can caused 10's of gigabytes of snapshot usage in a matter of 30 minutes or so. The only workaround I've come up with is to configure Exchange to only run the online defrag on the weekend, not nightly. That helps to keep the snapshot space at least partially under control, but it's not a silver bullet or anything.

The honest truth is, Equallogic snapshots are virtually useless for any moderately changing database backend, we see this same issue for Exchange, Oracle, and MSSQL databases.

I suspect the Netapp will be far better as it seems to use a much smaller allocation size for snapshot writes. I don't know the actual size, but as I noted above, we saw snapshot growth orders of magnitude less than the Equallogic on both Netapp and EMC Clariion CX arrays.

I agree, this is a huge issue, but I'm not sure how they can address it. Based on my understanding of their storage technology it's pretty much built around this "page" allocation method. That's how they move volumes between arrays and spread data out across multiple arrays efficiently and with minimal resource overhead. Going to a smaller page size would likely cause a performance hit or lower the limits on the number of snapshots because of resource constraints since they'd have to track many more changes per snapshot.

I hope they have a solution to this issue, but I'm becoming doubtful, it hasn't changed at all in the last 3 years because it's simply part of their design. It's really a killer because snapshots are such a basic component of all storage systems nowadays and their ridiculous storage requirements pretty much destroy their cost advantages over Netapp and other competitors.

We probably have at least another year before we'll really need to do anything for our Tier 1 storage so we'll wait and hope, but I'm not holding my breath.

Well, I hope I'm unbiased, I try to post my experiences as honestly as possible but I supposed we are all biased in some ways no matter what.

We loved Netapp but the cost to keep them was simply too expensive. We've been reasonably happy with Equallogic, and I still think there's a chance they could keep us, but the snapshot issue is pretty huge as we really miss using snapshots like we used to.

If you don't plan to keep many snapshots of the same volume at a time the EQL solution is probably OK, however, if you're planning to take more than a few daily snapshots I think you'll find them to be space hogs.

We actually just built a "SAN" with Openfiler and an Enhance Technology RS16i. We purchase the shelf and 16 1.5TB drives all for just over $7,000. The box is capable of great throuput (480-600MB/sec) even with just one shelf, but it's random I/O isn't very great. Great place for disk based backups and bulk storage though.

There are plenty of tools for testing the SAN, I'm a Linux command line type of guy so command line tools are my favorite. For quick testing I use 'iozone' with a single record size (usually 8k) and usually do a run with both IOP and then throughput mode with varying parallel workers. IOzone is also available for Windows.

If I really want to do some good testing I use 'fio', a command line benchmark that can be configured to perform almost any type of I/O you want and can record the I/O patter for 100% repeatable testing. It's written Jens Axboe, the block layer maintainer for the Linux kernel.

If you're using Windows I guess IOmeter is pretty good (and is available for Linux too). It can simulate quite a diverse set of loads but for some reason I don't like it.

Well, it's not 100% caused by SATA since we have several SATA Equallogic arrays as well. I think it's simply that the RS16 has doesn't have a very sophisticated IO elevator compared to the higher end Equallogic. Actually, I'm not sure it has an IO elevator at all, it may very well just be FIFO.

The controllers in high end array's are generally very well tuned by the builders and those builders spend a significant amount of R&D on desining IO elevators that recognize disk access patterns and tune the array to perform as well as possible even with multiple writers. They attempt to minimize head movement, and are well tuned for their exact hardware.

The Openfiler/RS16 setup is heavily dependent on the Linux IO schedules which are quite good, but far more generic, and can easily be fooled when there are multiple random readers.

I base this on test that show that the Equallogic and RS16 array provide similar random IOP performance with one or two random readers, but as you ramp the number of simultaneous random reader up, the Equallogic array maintains it's performance while the Openfiler/RS16 starts degrading quickly.

I'm sure that write semantics also work against the Openfiler/RS16 combo as well. The Equallogic and other dual controller arrays generally utilize reasonably large non-volatile write caches which means they can group writes together but still tell the hosts the data is already on disk. The Openfiler/RS16 combo has to commit data to disk much more aggressively since a power loss or malfunction would risk loosing too much data. This has a negative impact on the IOP performance.

We get excellent sequential read and write performance from the Openfiler/RS16 combo, but the random IOP performance is nowhere near as good as the Equallogic. We can compensate for this somewhat by using a very large cache on the Openfiler server which can provide huge amounts of read cache. We still think the Openfiler solution is quite good, but we don't think it will meet our high uptime, high IOP requirements.

Well, we're nowhere near your size. We're at 10 VM host and about 70 guest. We have an IBM Bladecenter and boot 6 blades running ESX (previously 3.x, now 4.x), 2 blades running RHEL5, and 1 blade running Windows 2003 without any significant issues using Qlogic iSCSI HBA adapters in the blades. It's a little annoying to initially setup due to the Qlogic BIOS setup, but not much different the Qlogic Fiber Channel.

We currently have a total of about 20TB of usable space on four EQL arrays and we have a reasonable about of Oracle ERP data (production and development about 1.5TB total) and something like 500GB of MSSQL data.

I have no doubt that an EQL solution could be configured to support your environment and provide sufficient performance, but if you make any use of snapshot beware of the extra space overhead compared to EMC, and it's questionable that they could "outperform them tenfold".

Sure, I'll take that on!!

Let me start with replication. Yes, it's true that the EQL replication is not as bad as their snapshots, but I still wouldn't consider it great. We were replicating an Oracle database that was about 150GB. We had been replicating this for years with no WAN acceleration product or anything over a 3GB WAN link using a FCoIP gateway between our EMC storage systems and using MirrorView/A on our Clariion. The Oracle database used 8K blocks, and the MirrorView/A product appeared to replicate the same block size that the LUN was allocated in (in our case 4K). So, changing an 8K block would replicate an 8K block, with a little overhead, I'd guess an 8K change would take about 10K worth of bandwidth.

OK, now how did that work with Equallogic? Obviously it was a database, so blocks were being changed all over the place and every 8K change would replicate 256K worth of data so I'd say EQL only used 25x more bandwidth. OK, in the real world it wasn't quite that bad, sometimes a few Oracle blocks were changed in the same 256K EQL block, but I'd still estimate it took 8-10x the bandwidth of the EMC solution. It was so much, it was completely unworkable as a solution for us.

Just to drive it home, with the EMC solution we were replicating the database every 5 minutes, although sometimes, during very heavy periods, the replication would get behind and lag 20-30 minutes. With the EQL setup? Well, the 8AM replication cycle would usually get done sometime around lunch time, and we'd be lucky to get another cycle by 6PM. That's right, we went from multiple replications an hour, to two during the business day. I suppose if your application is always writing new, large chunks of data, it wouldn't make much difference, but my experience is that this isn't the norm, but the exception.

As far as snapshot space, I would consider it a problem with anything. It's certainly more of a problem with applications like Exchange and SQL databases, but I'd still consider it an issue with simple file servers. It was a big deal with our VMFS volumes for our VMware infrastructure as well. For example, we have a file server that is used by our Sales and Marketing department. This server has 1.2TB of shared data, but typically sees only about 5-10GB of changes per day. How big were daily EQL snapshots on this server? Typically 200GB or MORE!!! With the EMC it was maybe 10-20GB.

I guess my point is, the EQL snapshots are ALWAYS bigger than the competition. Sure, on a 100GB server with a low change rate, maybe it would only be 1-2GB a day, but a competition might only be 100-200MB a day. Which one is going to be more space efficient? Which solution will let you keep more snapshots longer no matter the change rate? It will always be any vendor other than EQL.

To be kind, they're "defragment" argument is a bunch of bunk in my opinion. Sure, this helps a little with the daily change rate, but this is only short term. To keep this benefit you have to keep defragging, and guess what, the defragging pass will obviously make your snapshot huge. I would say defragging probably helped reduce the size maybe 10-20% for a few days, but that's still huge compared to everyone else.

To be totally honest, I'm disappointed to hear that EQL is not being upfront with their shortcomings in this area. EQL used to know what they were good at (ease of use, best of class iSCSI support) and what their weaknesses were (snapshots that are usable and fast, but not as flexible and efficient as others). One of their biggest pluses was that they were honest about not being everything to everbody or best-of-breed across the board. Based on your comment it sounds like Dell might be putting an end to that honest side that was so attractive, instead turning into the same marketing machine that is so frustrating in the storage world. I supposed I can understand it, NetApp has taken to calling snapshots "backups" (I actually attended a NetApp presentation on Oracle where they kept asking "how nice would it be to backup your database every 5 minutes -- except they were referring to snapshots, which, IMHO, are not backups) and I'm pretty sure EMC has always been a marketing company that happened to sell storage products. Maybe with all that pressure in a cut-throat market, they don't know what else to do.

I'd say, remember EQL for what they are, a "all-in-one" with the best iSCSI implementation, easiest setup and administration, excellent block performance, and great integration and monitoring tools to go with it. If snapshots and replication are critical components, well, you'll have to weigh that, but we've learned to live with other solutions there and have probably still saved money over the Netapp product. I haven't shopped Netapp in years, so maybe they're more cost competitive now, but when we purchased, the EQL solution was about half the price of Netapp with replication. If the price spread was closer, it would be tougher to decide.

Matt,

I'm not aware of it having any such feature. They have what they call their "intelligent" volume placement, which I was pretty hard on in my original review. Basically, you can set a volume so that it "prefers" to be on a RAID-50, RAID-10, RAID-5, or RAID-6. Assuming you have these different RAID types and enough space in the pool it will move the volume to the arrays containing this RAID type. There's is also an "automatic" type that's supposed to use heuristics to automatically place volumes on the RAID type that is ideal based on their load, but it only works if you use DIFFERENT RAID TYPES (like some arrays that are RAID-50, and others that are RAID-10). Simply having different speed arrays doesn't work. For example, I have a 7.2K SATA array, and a 15K SAS array, but they're both RAID-50 so I'm out of luck with this feature, I have to place my volumes manually. If there's anything "intelligent" about any of that, I'm hard pressed to find it.

As far as I know, there is NO WAY to prioritize any volume on the same array/set of arrays over any other volume on the same/set of arrays. If I'm mistaken on that, I'd love to know about it. I suppose this could be a feature that I'm unaware of with the 6000 series (I only own 5000's).

What some vendors are doing with QoS quite impressive. Things like actually taking the placement of data on the physical disk into account (latency vs throughput changes based on where data is placed on the platters), as well as the amount of cache allocated, etc. My opinion is EQL has nothing here, except marketing claims about "intelligent" data placement which is actually just a lame heuristic that puts busy volumes on a faster RAID group.

Now it sounds like I'm bashing EQL too much, which is strange because we still really like our arrays, but I'm also trying to be honest. Make me live with a NetApp for a while and I'll probably find bad stuff to say about them as well, although, if it's like my previous experience, it will be more about the company and it's tactics than the actual hardware.

Well, as stated in the article our scripting tools never really matured as we found EQL's snapshot implementation to be far less useful than our previous vendors. We created simple function calls like "ListSnaps" which would take the volume name as a variable and issue the command and return the output. Here's an example of the parsing function, it can be modified for the output of other commands. I can't take any credit for this code as it's virtually directly from EQL support. It can be easily modified to parse the other brain dead output from the CLI as well.

#
# Parse the output of the "snapshot show" command and return the list of
# snapshots.
#
# Takes three arguments:
# A pointer to a hash that will hold an array of snapshot names.
# The volume name to be used as the hash key.
# A pointer to a list that contains the output of the "show snapshots" command
# as returned from the DoCommandOnArray() function.
#
# Returns:
# Nothing useful
#
# Here is typical output from a "snapshot show" command:
#
# Name Permission Status Schedule Connections
# --------------------------- ---------- ----------- -------- -----------
# test-1-2004-03-18-11:34:33. read-write online 0
# 1
# test-1-2004-03-18-11:42:45. read-write offline 0
# 10
# test-1-2004-03-18-13:25:43. read-write offline 0
# 13
# test-1-2004-03-18-13:30:28. read-write offline 0
# 16
#
# This subroutine pulls the list apart and creates a hash with one entry per
# volume. The hash key is the volume name and the value is a reference to an
# array that contains the other items for that volume (size, snapshots, status,
# permission and connections). It also takes care of the fact that the data in
# some columns ("Name" and "Status" in particular) can wrap across multiple
# lines.
#
sub ParseSnapShowCmd(\$@) {

my ($OutHashElement_ref, $InList_ref) = @_;
my $FSkip = 1;
my ($Line, $Name, $Perms, $Status, $Sched, $Conns);

foreach $Line (@$InList_ref) {
#
# Trim terminators and toss empty lines.
#
chomp($Line);
next if $Line =~ /^$/;

#
# Ignore the column headings.
#
if ($FSkip) {
$FSkip = 0 if ($Line =~ /^--------/);
next;
}

#
# Parse the line.
#
my @Cols = unpack("A27xA10xA11xA8xA11", $Line);
if ($Cols[0] =~ /^ /) {
#
# If the first field begins with two spaces, this is an
# overflow of one or more columns from the previous row.
#
$Cols[0] =~ s/^\s+//g; $Name .= $Cols[0];
$Cols[1] =~ s/^\s+//g; $Perms .= $Cols[1];
$Cols[2] =~ s/^\s+//g; $Status .= $Cols[2];
$Cols[3] =~ s/^\s+//g; $Sched .= $Cols[3];
$Cols[4] =~ s/^\s+//g; $Conns .= $Cols[4];
}
else {
#
# This is a new row.
#
push (@{$$OutHashElement_ref}, $Name ) if ($Name);
($Name, $Perms, $Status, $Sched, $Conns) = @Cols;
}
}

#
# Write out last entry, if pending.
#
push (@{$$OutHashElement_ref}, $Name) if ($Name);

return;
}

Excellent find! I'll have to check it out and perhaps reference it in the article somewhere. Looks pretty usable.

So, just to make sure I'm understanding your question, you buying a SAN just to host a 5GB database? That's the "only application". I would think just about anything could handle that. What are you going to use all of the rest of the space for? Even the smallest EQL has TB's of space I think, well, maybe not the SSD model.

I really don't have an answer or argument either way for such a small application on a storage system I'd probably save my money.

I'm sure you'll find the EQL array to be a fine addition to your data center. While the article above has been used by some EQL competitors it was really intended to simply point out features which we think are weaker than some of the competitors. There are many areas with the EQL really shines, including ease of setup and administration as well as reliability. Also, not having to deal with licensing costs for every feature you might want to use is pretty nice as well.

The article on snapshot space is interesting, but I'm not really buying it. I'm sure that disabling anything that writes small updates would indeed help with EQL snapshots because the problem with EQL snapshots is well understood, if even a single bit is changed a 16MB block, then that entire block is used immediately. I know of no other snapshot technology that works like this. I tried all these great "tricks" that EQL had me do to improve their snapshot space, but at the end of the day, EQL snapshot were still always orders of magnitudes bigger than the competitors, or the space used by OS based snapshots. With our EMC array I had volumes where 20% snapshot space could easily hold snapshot for 30 days or more, with EQL I'd need 100% snapshot reserve to do the same thing, even after all the tweaks. If you're not using the space for anything else, not a big deal, but assuming you like you expensive storage to be used efficiently, well, EQL just doesn't, and no amount of tweaking changes that.

Disabling the Last Accessed stamp and the archive bit helps some, but guess what, it also helps the competitors. So, assuming your willing to make this change to all your Windows servers, and assuming it makes a 50% improvement (I never saw it make that big of an improvement but I'm being generous), then EQL snapshot would only be 2-3x as much space as the competitors. Of course, if you used the competitors storage, and made the same Windows tweaks, it would help their numbers as well, so I think that KB article is mostly a response for having to do something for customers that are experiencing such large snapshots. Any improvement makes those customers feel like EQL did something, but it's really just masking a poor underlying design and significantly devalues the "all features for one price" model since, while you do get all the features, you loose a lot of usable storage.

I have no experience with the 10G models, but I'm surprised you're only seeing 70MB/sec. Do you have load balancing working? Is that only a single thread with limited queue depth? We have no problem getting even twice that much from even our old PS400E (SATA) devices and over 200MB/sec on our PS3900XV (15K SAS).

Make sure you test a lot of concurrent access. You have 140VM's so there will be a good amount of contention for the disk, it's important that the array not choke when 5-10 host are really working the array. This is a place where the EQL arrays do shine compared with my EMC experience. Admittedly that was an older EMC now, and with enough spindles you can spread the load, but in our experience the EQL generally does perform as well with only a single or dual load, but slowly overtakes the other system as the number of concurrent loads ramps up.

Well, I typed up a nice response to this, but my blog software apparently ate it and I don't feel like typing it again, but here's the summary:

1. If you're happy with the performance, it doesn't really matter how much link bandwidth you're using. We've seen our arrays saturate 1Gb links many times during RMAN backups, etc., but that doesn't mean yours should, only that it could.

2. Think "scale-out". You have one small array, most people "clamouring" for 10Gb would have more than that. We have three arrays, and a volume spread across all three can easily deliver 450-600GB/sec. In this scenario EQL uses the links to transfer data betweens arrays prior sending them to the host. The 1Gb links can quickly become a bottleneck. Now imagine this case with 8 arrays.

3. Think "simplicity". Most of our servers have at least 4 iSCSI interfaces, some more. Even for servers that currently have 10Gb connectivity I have to play "tricks" to get multiple iSCSI connections to the EQL array to acheive full bandwidth. If the EQL array had 10Gb interfaces any single iSCSI connection could provide full connectivity. 10Gb will reduce the cabling in our racks by at least 4x, always one of the most annoying parts of a rack install.

4. Think "IOPS". There's far more to storage than just bandwidth. While one of my previous arguments for iSCSI was that network latency was only a minimal part of the latency equations, solid-state media like flash, and huge array caches are changing this game tremendously. Most 10Gb rack switches are supporting "cut-through" switching that allows them to switch even jumbo frames in ~3 microseconds.

Basically, 10Gb ethernet is coming to most racks in the datacenter and unless EQL wants to be left as the small market player they need 10Gb options (which they now thankfully have). Certainly, if 1Gb is good for you, and fits you budget, and you don't have scale-out or performance issues, it's still a fine option, but for many of us (perhaps most of us) 10Gb is the way forward.

PS -- Can you believe that was just the summary?

The honest truth is, if your not dropping packets, and the switches can handle large frames with reasonably latency, they'll probably be fine. We've always been a Cisco shop but are likely leaving them for a lower cost option in the near future, especially at the client and server access layer, but probably at the core as well.

For copying VMDK files around, I think the throughput you're seeing is about all you should expect. It's what I see as well. Copying VMDK is sequential I/O, but it appears that VMware limits the queue depth significantly and preforms very little (maybe even no) read-ahead. I suspect this may be intentional to keep move operations from overwhelming the underlying disk and impacting performance of you're running VM's, but I really don't know. The only thing that seems to improve VMDK copies is decreasing latency, but that only helps a little. When we had our EMC CX700 with 4Gb fiber channel and a bucket load of fiber channel drives we were able to get about 50MB/sec copying VMDK files around, but I suspect that's mostly because of the improved latency of fiber channel and the manual tweaks to read-ahead that we preformed on the VMDK volumes.

To really test performance, put some load going in a VM, or a couple of VM's. I usually run a nice, quick, parallel iozone test. Something like "iozone -s 4g -r 8192 -t 4". This will start four thread, each writing/reading a 4GB file in 8K chunks. If you can't push into the 200MB/sec range doing that, well, then something's wrong. Otherwise, you're probably good.

It's hard to understand your question. You start off talking about "LUN Queue Depth" and then switch to "port based and/or global queue".

However, it's important to note that the Netapp architecture is completely different from that of the EQL. As far as I know, Netapp still follows the traditional "head-unit" model. You purchase a "head-unit" and then attach it to disk shelves. You then carve up these disk into RAID groups, and create LUN's on the RAID groups. The "queue depth" problem would happen when some server was writing huge amounts of data and keeping the disks of one of the RAID groups busy. If this host continued to write until it filled the "queues" on the head unit, this could impact performance on other RAID groups even though their disk idle.

The EQL is completely different. Each and every array in the group has it's own controller, and each array has exactly one "RAID group" for all drives in that array. LUN's can be "locked" to a specific array, or spread across all arrays in a given resource pool. As you add arrays, you get more ports, more cache, and thus more queues. This is the cornerstone of their "scale-out" architecture.

Yes, the entire group does indeed have a single IP address, however, this is for simplicity, and is not a limiting factor. If you have 3 EQL arrays in a group, and the LUN is spread across all three, the incoming connection will be redirected to the "least-loaded" of the available ports (based on the arrays this might be 9 or 12 ports). The EQL monitors activity on the ports, and if the load pattern changes, it will transparently move the connections to other ports using ARP to equally balance the connections across the available ports in the group.

If a LUN is locked to a specific array in the group, incoming connections for that LUN will be sent to one of the ports on that array. If you move the LUN to another array, the connection will be transparently redirected to a port on the new array.

My complaint isn't so much about the lack of ability to script, it's with the overselling of the "scripting toolkit" which is a complete joke.

I expected that something called a "scripting toolkit" would provide functions for calling like "createSnapshot" or "listSnapshots" where you would call the functions with volume names and option and it would return results. The "scripting toolkit" from Equallogic is basically a few functions calls to wrap opening a telnet session.

On top of that, while it's easy enough to create scripts that create snapshots or delete snapshots, it becomes more difficult to do things like list snapshots, parse them, and automatically mount them.

For example, we had tools for our EMC array that could take snapshots of our Oracle servers and present them to another host. An admin could type something like "orasnap create " and the script would create the snap of the volume, make the changes on the EMC array to present the snap to the host, and mount the volume on the host. An equivalent "orasnap delete" command would do the reverse. The admsnap and navicli tools made running such commands, and parsing the resulting data trivial. We also had scripts to do simply things like "listsnaps " and "mountsnap "

Performing the same tasks on a EQL array is more difficult, mainly due to parsing the horrid CLI output. In the end, if the "scripting toolkit" simply provided functions for creating reasonable output, that would be a huge step.

I don't mean to imply that such scripts couldn't be created, they certainly can, we created them, but the complexity level for doing the same functions under EQL was much higher. Some of our scripts for EMC were 25-30 lines, while the equivalent script from Equallogic was 300-400 lines, mainly do to parsing functions.