From the Recliner...

So I've written quite a few articles about our Equallogic arrays and their strengths and weakness.  The short version is, we love their simplicity.  The features are easy to use, but things like replication and snapshots use a LOT of space and bandwidth compared to their competitors, somewhat offsetting the "every feature included for one price" mantra since you have to buy significantly a lot more space to use the features in the same way.  We also loved their reliability.  Notice the past tense of that statement as the last few months have been pretty rough on that front, eventually culminating in a major data loss event about a week ago.

Continue reading "Equallogic Eats Our Data"

OK, I'll admit it, I'm a recovering Cisco junkie.  Well, actually, I haven't started my recovery yet, but I'm close to checking in to the rehabilitation center.  At my employer, we're used to saying "we're a Cisco shop" when asked about out networking and VoIP solutions.  What does that mean?  We do the Cisco thing in almost every way.  We were early CallManager adopters (think 2000), we deployed they're early PoE offerings.  We have Cisco Cat 6500's as core switches, 3750's (and older 3524XL's) at the edge.  Cisco wireless, Cisco routers, Cisco WAAS (WAN acceleration).  But, why?  Why are we only a "Cisco shop"?  We don't generally do this with other technology.  We normally look at the technology and pick what we thinks fits our needs and budget.  But when we needed networking or phones, we would just call Cisco and not even look elsewhere.  What's up with that? Continue reading "Leaving Cisco Behind"

OK, so I'm a small time email admin.  I work for a company with approximately 700 email users, nothing too big or exciting.  We send and receive several thousand messages a day, and block many thousands of SPAM messages a day.  Recently, we noticed that our outbound queues were backing up.  Of course there's always some traffic hung up outbound, people practising greylisting, email address typos, mailboxes that are full or temporarily unreachable, but we set a threashold and send alerts when the outbound queue goes over.  When we started getting alarms this time I went in a took a look a noticed that 90% of the mail in the outbound queue was to Yahoo.com addresses.  Things only went downhill from there. Continue reading "Why I hate Yahoo Mail"

My wife has been using Ubuntu rather than Windows Vista, since Ubuntu 8.04 over a year ago.  There were some growing pains in the beginning, but she's pretty happy now.  One of the biggest problems was caused whenever she would play an online game on Pogo, which uses Java, it would basically take over the sound card, not allowing sound from any other application.  The only way to restore sound was to shutdown Firefox, an annoying deal when she had open windows/tabs to many sites.  Until Ubuntu 9.10 I simply changed her shortcuts to run Firefox with the "padsp" wrapper.  This always worked fine, but with Ubuntu 9.10, using "padsp" with Firefox just hangs, either due to a bug in padsp or Firefox or both, I don't know.  I found several bugs filed regarding this problem but no solutions.  Documented here is my solution. Continue reading "Making Sun Java 1.6 play via PulseAudio"

OK, so I've been doing the iSCSI thing for a long time, but recently we had a need to build a RHEL5 VM with access to a legacy Fiber Channel Apple Xserve RAID.  Fortunately our trusty Cisco MDS 9216 w/IP Services Blade can act as an iSCSI to fiber channel gateway, however it does have limits. It only support 3 iSCSI sessions per port, and thus with our 4-port blade we only have 12 total iSCSI-to-FC sessions.  We already use some sessions for iSCSI access to legacy FC tape libraries, so we didn't have enough sessions to connect all of our VMware ESX hosts to both halves of the array (the Xserve is effectively two dumb FC-to-RAID controllers stuffed in a single chassis) and really, only this one host needed access.  Oh yeah, we wanted good performance as well.  Read on for how we did it.
Continue reading "Linux Software iSCSI with Bonding and Multipath"

I had been interested in WAN acceleration products for years, but we had no pressing WAN performance problems so there really wasn't any way for me to justify an expenditure on a standalone WAN acceleration platform like Riverbed.  However, last year our Cisco router equipment was getting near end-of-life and I decided to get a quote for new equipment with integrated WAAS.  Since we were already purchasing new router gear it was easier to cost justify the WAAS equipment to extend the life of our existing links as well as allow for further datacenter consolidation of edge services.  When I actually received approval, I was excited to finally implement WAAS.  We've now been living with WAAS for over 6 months and the excitement has waned, not because the product doesn't have great potential, but because of software bugs and the annoying service experience when trying to get those bugs addressed. Continue reading "Living with Cisco WAAS"

So a few weeks ago we migrated our network and work completely the BGP, not other routing protocol is in use, just BGP and a sprinkling of static routes.  So far we've been happy with this decision as overall it actually simplifies our routing setup and provides a level of control that was difficult before.  Right now we're probably using BGP in places that don't even make sense (like ISDN backup links) but it actually does work amazingly well even in this function and allows our backup links to operate exactly like our main links, no floating static routes or alternate routing protocols.

However, several of our sites simply have too much traffic to live with an ISDN backup link.  For those sites we generally drop a low cost DSL circuit and use a VPN based backup solution.  We've done this for many years with OpenVPN because we can run it on the Linux boxes that provide our secondary firewall and IDS functions and it's pretty secure and robust.  However, it's biggest negative is that it's client/server.  All the remote sites have to connect to the central server, and traffic from one site to another ends up going through the datacenter.  What if the primary datacenter was offline.  There were's ways to mitigate this (run the server at the secondary datacenter, configure multiple connections in OpenVPN) but overall, we just wanted something that was more like our normal, full-mesh, MPLS WAN.  When a site's primary WAN link was down, sites with VPN connectivity should be able to communicate directly with the troubles site via the VPN WAN.  After some research we decided that Tinc, along with Quagga's BGPD routing daemon, might just be the solution. Continue reading "Buidling a Backup WAN with Tinc"

At work we are nearly a 100% iSCSI shop.  We completed the switch to iSCSI back in 2007 when we replaced our 2GB fiber channel solution with Equallogic.  We were already using Cisco MDS9216i series fiber channel switches with IP services blades to provide hosts iSCSI access to the fiber storage, and we kept this around as a gateway to some legacy fiber channel equipment (a tape libary and an Apple Xserve RAID used as a disk-to-disk backup store).

Well, we finally outgrew the Xserve RAID and needed a new, low-cost storage array, with good sequential read and write performance, for our disk based backups.  We wanted built in iSCSI, but I also considered just buying a external SAS shelf and using Linux as a NAS/iSCSI frontend especially when all the iSCSI options seems so expensive.  After much looking around I came across the RS16-IP4 from Enhance Technology.  I really didn't know anything about this company, but the unit offered native iSCSI with 4 1GB iSCSI ports, and the price was right at less than $5000.  Also, you could bring-your-own-drives and the 1.5TB Seagate Barracuda drives were certified, so we picked up 16 of those.  Now we have the box in, and have completed some initial testing with reasonable results. Continue reading "Cheap and Fast iSCSI Storage"

Compared to many of you my network is small and probably uninteresting, but it's mine, and lately I've been thinking I want to redesign our network routing.  We have a core site and six edge sites, 5 here in the states and one overseas in Europe.  About a year ago we finally switch from and old point-to-point WAN to an MPLS WAN provided by AT&T.  This migration required us to use BGP on the edge routers which was really no big deal, but significantly impacted the implementation of our failover solution.  After being unhappy for months with the current setup I'm considering switching to an all-BGP routing configuration.  I'm interested in feedback if anyone out there has done this even in a fairly small network. Continue reading "Using BGP as the only Routing Protocol on the Corportate Network"

OK, it's not quite three years, but it's only a few months short, and suddenly I've received several emails and a couple of comments wanting followups to my Equallogic entry from back in early 2008, so, here we go.  For the most part I probably haven't posted any more about them because I don't really think things have changed that much.  The array just sits there, does it's thing and we live with it, which is good, but I'll try to outline some of our experiences with the product since we purchased our first one in 2006 and 3 more in 2007. Continue reading "Three Years of Equallogic"

So, a couple of months ago I wrote about my feelings that Linux will never really be the dominate desktop operating system.  I concluded that this was caused primarily because the developers of Gnome (and to some extent KDE) really don't take into account the average user experience.  They add, change, and break existing functionality on a whim and ship code in known broken state.  In the end this isn't critical to me, I could really care less, I've happily used the Linux desktop for 10 years.  Now though, I'm starting to think that the Linux desktop may even be regressing, and soon even I won't be willing to tolerate this. Continue reading "Is the Gnome Desktop Regressing?"

A couple of days ago we were troubleshooting a problem with one of our Windows 2003 systems performance with our Equallogic PS400E iSCSI storage system.  The problem turned out to be caused by poor alignment of our disk partitions.  Since we discovered that disk alignment could have a more dramatic affect than we had ever thought we decided to undertake a project to properly align all of our Windows and Linux systems.  The Linux systems with LVM were pretty easy to move, but the Windows systems proved much more troublesome, espeically the boot drive.  I was finally able to come up with a free solution that took time, but worked pretty well.  I thought I'd document it here in case it helps anyone else. Continue reading "Aligning Windows Partitions Without Losing Data"

Recently I've been reading a lot regarding FCoE and how it's the next great thing in Fiber Channel technology.  I can see the attraction of this technology, such as the ability to use a single infrastructure technology (Ethernet) in the data center for backhaul of both LAN and SAN comminucation while still supporting native fiber channel at the endpoints.  However, I'm also seeing time and again a comparison of this technology to iSCSI, using terms to describe iSCSI as "lossy", "unpredictable", and "low performance".  The problem I have with this characterization is that FCoE only runs on a local subnet (not routable).  When running iSCSI across a local subnet it's pretty easy to make it "lossless", "predicatable", and "high performance" while still using a routable protocol which is very flexible. Continue reading "iSCSI vs FCoE"

At work we recently started using BackupPC to do backups of our Linux server.  While the BackupPC website will lead you to believe that it's useful primarily for desktops and laptops, I've found that it's an excellent product for backing up servers as well.  We recently started backing up quite a number of our Linux servers with BackupPC and it works great, however, we did notice one minor problem.  Some of the systems have a large number of files (~4 million) and take many hours to backup (an hour or so even for an incremental).  Based on the order that the files were backed up a backup that started at midnight, where a group of files were modified at 12:45AM might get some of the old files and some of the new ones.  We wanted a consistent image (a backup of the system exactly as it looked when the backup started) and decided that combining BackupPC with an LVM snapshot was the easiest way to achieve this.  Here's what we did. Continue reading "Using BackupPC with LVM Snapshots"

For anyone who knows me the title of this article will be a surprise.  See, I've been running Linux as my desktop OS almost exclusively since 1996.  I had first started with Linux back in 1993 with a very early release of Slackware.  These mainly ran on my old junk machines in my "home lab" and they were fun to hack around with, but really didn't meet my day-to-day needs.  Then, in 1996, I managed to get a copy of Redhat 3.0.3 (Picasso), installed it on my work laptop, and never looked back.  Until about 3 months ago, my laptop ran some variant or another of Redhat (back in 2000 I ran Rehdat variant Yellow Dog Linux on a Mac PowerBook, and in recent years Fedora and/or CentOS).  Sometimes the system was dual-boot with Windows, and I've almost always had a Windows virtual machine lying around (well, at least since 1999 when VMware first came out), but, for 12 years, I haven't owned a computer where anything other than Linux was it's primary OS, and I've spent about 99% of my time in that OS. Continue reading "Linux will never dominate the Desktop"