Internal vs. External Testing?

When I first arrived at MySQL more than 4 years ago, we didn’t have much in the way of internal Quality Assurance. There were a couple of test suites that had been created and many engineers wrote unit tests. We supported a wide range of platforms, but formal testing was pretty spotty. As bugs were identified and fixed, the test suites were extended to help identify regressions. It was by no means a bullet-proof process; mostly the team depended on external testers to report issues. Sometimes a build would be created that would have some embarassing bug and we’d scramble to fix it.

Since that time, we have introduced many more internal QA processes. The test suites are now 10x larger than they were just a few years ago. Our build system and use of automated tools for static and dynamic code analysis has expanded. And we also eat our own dogfood, running beta and RC software more routinely on internal applications. For example, we’ve been running MySQL 5.1, now a solid release candidate, for more than six months on our web site and internal applications.

While it’s taken a few years, I think we have arrived at a good balance between internal and external testing. There’s no way we could ever hire enough QA people to make up for the tens of thousands who download MySQL every day. They report unique corner cases or weird combinations that internal testing would likely never find. And our internal testing has now made it so that quality is now more consistent and we have fewer regression errors.

So I guess it’s not a question of Internal vs. External testing, but more a question of finding the right balance. Interesting that there’s also a trend among closed source companies, like Microsoft, to adopt some of these techniques that used to be only found in the open source world. Microsoft has expanded their external testing with the Community Technology Preview (CTP) programs. Good to see the learning on both sides…