HealthCare.gov: The (infrastructure) fix is in

Two months ago, on Oct. 1, the U.S. government rolled out HealthCare.gov, where citizens could explore new health care options and apply for new health care insurance. While states could set up their own exchanges, this site was the central point in the plan — and it failed miserably. Delays, disruptions, errors, and complete unreachability dogged the site in the first days — then stretched into weeks. Hearings were called in Washington. There were demands that Health and Human Services Secretary Kathleen Sebelius resign due to what seemed to be an abject failure of a much-ballyhooed and definitely expensive central cog in the new Affordable Care Act health care law, aka Obamacare.

Then something extremely interesting happened: The site began recovering. Two months after it launched, the site is now performing pretty well. This is a very curious development indeed.

[ Also on InfoWorld: The inside scoop on how HealthCare.gov is getting fixed | For a quick, smart take on the news you’ll be talking about, check out InfoWorld TechBrief — subscribe today. ]

Development of the site cost anywhere between $70 million and $150 million, depending on how you tally the numbers and whom you ask. These are massive figures, even for a site with the scope of this one. While I don’t know the details regarding what was necessary to produce the site from an internal perspective, I can say that if the amount spent on it was anywhere between either of those figures, the site should have been bulletproof right out of the gate. The fact that it wasn’t should cost the contractor in charge to lose government jobs forever more. I doubt that will happen.

Revving up the horsepower

But the question remains: How did the site’s performance improve so quickly? Sure, there were some major flaws in the way the application was designed, but there’s no way you could fix all that code within weeks. You can make improvements and tweaks, but nobody’s rewriting the whole thing in that amount of time, no matter how many developers you bring in. As the saying goes, nine pregnant women don’t produce a baby in one month.

I’m guessing the code was workable for the most part, and the infrastructure was woefully underspec’d and underbuilt. According to an HHS blog post and a HealthCare.gov progress report, a number of software and hardware fixes were put into place to make this happen. Most striking about this data are the declarations that they “installed dedicated hardware” for the registration database that led to a threefold improvement in speed. The agency also deployed 12 database servers and new storage for the core database, along with a new firewall for a 500 percent performance increase.

You mean it was possible that a new firewall increased performance by a factor of five? Was the previous firewall a decade old? And there weren’t already 12 database servers?

There’s only one way that those numbers can be true: What was there before was not enough power to run zombo.com, much less a massively promoted, high-traffic website. I’m simply shocked there were people in engineering and design positions for this site who were heading into launch day with woefully inadequate infrastructure resources. Such a vast amount of money spent, and they couldn’t pony up for a reasonable firewall?

Any insiders want to spill the beans?

I would absolutely love for someone in the know to shoot me an anonymous email with details on how the site was constructed on launch day. I’ll bet it was a few 1Us running a half-dozen VMs behind a 100Mbit firewall. That’s pure conjecture, of course. But the way the site functioned (or rather didn’t function), it wouldn’t surprise me one bit.

Also, the report mentions the agency implemented real-time monitoring as part of the postlaunch fixes — $70 to $150 million, and monitoring wasn’t part of the original mix? Did the feds splurge on the Scotchgard and undercoating, and there wasn’t enough left over?

Rather than talk to people who had actually designed and built large-scale Web applications, the government hired the usual contractor suspects and turned out a complete turd of a project for a huge pile of money. I realize that railing against wasteful government spending is as useful as the proverbial hole in the head. But with a project so highly visible and so important, I guess I expected a little more responsibility. The joke’s on me, I suppose.

That said, kudos to the group that rushed into a raging five-alarm fire and managed not only to put out the flames, but actually rebuild the house while it was still burning. I can’t imagine the tense hours invested, the all-nighters, the tons of take-out Chinese food — and all that soda. However, to be clear, just because the site is performing better doesn’t mean it’s doing the job it was intended to do. That remains to be seen.

At least the feds beefed up the infrastructure to appropriate levels — and maybe the government learned some lessons in the process. Somehow I doubt it. Call me cynical.

This story, “HealthCare.gov: The (infrastructure) fix is in,” was originally published at InfoWorld.com. Read more of Paul Venezia’s The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.