Dimensions of SOA testing

There are many dimensions to SOA testing. They include services, processes, and performance.

Service-level testing is the most important, since core services are the foundation of the SOA. However, services are written very differently, depending upon the developer, with some services perhaps too course-grained, and some service too fine-grained, and many other services poorly designed. Services may also be built on top of existing interfaces and APIs, and thus are even more complex and more in need of quality assurance testing since you’re placing an interface layer on top of an interface layer. There is no magic here; it’s a matter of validating the services for their intended use, verifying that the interfaces function correctly, and validating both WSDL and schema. Also, you need to consider diagnostics for design-time and run-time, and make sure to address those older, but important notions of unit, functional, and regression testing.

In addition to service-level testing, we also have to test the way services are abstracted into processes and composites. Since these are typically exposed as services themselves, it’s just a matter of testing another level up from the core services, as units, and regressing down through the services that they leverage (unit and system). This is very much like testing object-oriented systems, but these guys have binary interfaces and heterogeneous development and run-time platforms, thus the complexity is much higher.

Performance testing is perhaps just as critical, considering that most of the quality problems I run into when deploying SOA relate back to performance. Here is where you test against the SLAs established within the project, and learn how to spot bottle necks, such as slow services, that can bring your SOA down to a crawl. Performance testing in the world of SOA is a matter of testing at the service, composite, process, and system levels. You look at overall performance first, then decompose the architecture down to functional primitives to isolate the system’s problem components. You need to create an ongoing performance testing approach since so many performance issues develop over time as message and data traffic increases or changes.

What’s key here is to remember that you’re testing architecture, and not an application. Thus, the complexity of the system, and the approaches and tools used for testing, goes way up. It’s important that you have a solid test plan, an arsenal of testing tools and techniques, and the time needed to test the architecture properly to correct any problems before they are found by the end user. Once the end user becomes the tester, it’s difficult to get your creditability back. Consider the systemic and business critical nature of the architecture. Any quality issues that exist within your SOA will be amplified.