The Atomic Juicer

Another topic that’s close to my heart is matching the solution with the problem. This is such a religious debate (for some reason) that I’m not sure where to start, but I’ll try to keep it as brief as I can. Too often, people fail to match problems with appropriate solutions and they end up either over or under engineering it.

Let’s take a case I recently faced and see how the solutions aligned.

I’ve got 13 data marts to deploy for the different regions in our company. We have to make the solution swingable… that is, we have to be able to failover to an alternate site for DR purposes… we DBAs sometimes call that ‘swinging’ because you swing operations over to another site. Anyway though… there are a number of solutions you could go with here.

The first solution proposed was by another DBA in another group, and he swore it was the only solution available. His solution was to buy DMX disks that could be snapshot throughout the day to keep the DBs in synch. Then when you swing, you take the final snapshot and apply it on the other side, and you’re up. The process could take around 30mins or so depending on network traffic and comes with a hefty price tag of around $300K from EMC. And that’s $300K per data mart. So that’s $300K x 13. So after seeing that price tag, they came to me for alternate solutions, of which I gave them 5 or 6 I think. I’m not really gonna give the full discussion of how we landed on the solution we chose, but ultimately, we chose mirroring (because the DMs are on SQL2K5), and mirrored backups on each side of the swing. So for a little extra code, we got a very workable solution for FREE!!

The SRDF process for synching the DMX disks breaks on our other systems all the time and they’re constantly having to fix it. Anyway, the DMX solution is just more than we need… esp for the price tag, the solution definitely doesn’t fit the problem.

Another good example is along the same lines. A couple years ago we needed a large dumping ground of disks to house our SQL backups while they were waiting to go to tape. What we needed was fast disk and lots of it. So our network admin took it upon himself to contact HP, NetApp, EMC, etc. All of them came in with good solid recommendations of NAS solutions that had all kinds of redundancy, and snapshot technologies, management, the works…

The problem again was the price tags. It was just ridiculous what these people wanted for a couple TBs of disk. All I wanted was a place to put my backups for a couple hrs. So again, I found a couple disk cabinets and loaded them up with SATA drives and slapped them onto a spare server I had lying around. Is it the most robust solution out there? Surely not, but that’s irrelevant. Everything doesn’t have to be the most robust solution. It just has to be robust enough.

I’ve seen it several times when a dev will write a solution to do something simple to copy some data somewhere or run backups, and they over-engineer the thing to the point where you can hardly figure out what’s going on.

And actually, I just read back through this and realized I didn’t have an example of an under-engineered solution. This one was a couple yrs ago as well, and you’ll just love it.

We had a dev who liked to play admin from time to time, and when he designed his solution it had to be replicated to several different sites. His solution, instead of using actual replication, was to use Red-Gate’s data compare utility every 30mins and run the compare on each of the servers. I only wish I were kidding. This is such an incredibly under-engineered solution I find I have trouble commenting on it so I’m just gonna leave the story as-is.

In short, you don’t need an atomic juicer to squirt some lemon juice into your soup. And you can’t use a hand reamer to juice a bushel of oranges.