SOA for the real world

“Sometimes I lie awake at night, and I ask, ‘Where have I gone wrong?’ Then a voice says to me, ‘This is going to take more than one night.'”—Charlie Brown.

This may well be the plight of many CIOs and IT directors today. Companies big and small have invested time, effort, and money over the past few years hoping to realize the SOA (service-oriented architecture) dream. Numerous SOA implementations have resulted in varying degrees of disappointment. And many companies are coming to realize that SOA is more complex to implement than expected, requiring a laser-like focus on all aspects of enterprise data and a deeper shift in organizational culture than any previous technology wave has demanded.

As we all know, an architecture oriented around services is not new. A few precursors to SOA, such as CORBA and DCOM, have already successfully bridged disparate applications using a loosely-coupled services-oriented approach. What is new about the SOA wave is that SOA is not simply about services. The emergence of the Internet and XML has opened the floodgates for data exchange. The software industry has lined up behind common data exchange format (XML) and Internet transport protocols like it never has before. Thus, a surge of well-accepted and open standards have emerged that enable the promise of SOA: to support flexible configuration of business processes, reduce operational costs, enable dynamic discovery of services, and provide seamless integration between applications, departments, and trading partners.

Unfortunately, much to the disappointment of enterprises and technologists, these lofty expectations from SOA are not being fulfilled. This is not because the promise itself is false, but because most SOA implementations today are experimental in nature. The good news is that valuable lessons can be learned from an SOA experiment, lessons that can help transform it into an enterprise-grade implementation that realizes the full potential of SOA.

SOA goals

Before we explore the challenges that lie in the path of SOA, let’s step back and reexamine the goals of an organization embarking on an SOA implementation.

To achieve process visibility and flexibility

The worldwide SOA wave is definitely a well-justified one; SOA has come to represent an amalgamation of several radical changes in distributed computing. Organizations always invest heavily in technology to stay ahead of their competition, and SOA provides that breakthrough opportunity. At the same time, organizations have also been refining their business processes to squeeze out every last bit of competitive advantage. The emergence of business process management (BPM) promises continuous process improvement and never-seen-before collaboration between businesses and IT groups. SOA is the umbrella under which organizations are making a concerted effort to gain complete visibility into their data and processes, perform continuous improvements, and exercise fine-grained control in an effective and transparent manner.

To break down silos

The second goal of SOA is to break down application, department, and trading partner silos—built through years of software development—that addressed ever-evolving business requirements and technology improvements. Having been assigned their own IT budgets, departments have built applications to address their immediate needs, with little visibility into projects around them that might have overlapping functionality. The result: numerous legacy applications that leave IT departments struggling to reconcile duplicate information, and bits and pieces of business processes strewn across hundreds of applications. An SOA promises to break down these silos and let organizations gain better visibility into their data and processes.

To manage better data

Organizations not only want to manage data better, they also want to manage better data. It is important to ensure that the data being produced and consumed across the organization and its trading partners is clean, reliable, secure, well-governed, and fast. One of the goals of SOA is to provide a composite data services platform with a unified set of components for data access, quality, transformation, governance, and caching, among many other data-centric services.

A typical composite data service is an assembly of well-behaved and well-understood atomic activities and business rules deployed as a more functional service. For example, a composite data quality service for customer contact information might sequence an address validation service and an email validation service, and add the business rules to dictate what happens when either service reports an error. Composite data services themselves are deployed and managed in a metadata repository so that they are transparent and easily governed.

To reuse services

A related goal of SOA is to effectively manage and reuse enterprise services and data. Services developed by one group in an organization can be used by any other group within or outside the organization if they are published and described in a standards-based format in an accessible registry. When data and services reside with their owners and are shared by consumers as they need it, operational costs associated with their maintenance and management are reduced. Reuse is one of SOA’s biggest incentives.

To align organizational goals

Another goal of SOA is to align the business and IT groups behind the organization’s goals to better aid the development of flexible and configurable business processes. Historically, business and IT domains have maintained quasi-independent charters to improve the organizational bottom line. Business groups try to streamline business processes for competitive advantages, and IT departments try to use technology adoption for implementing these processes. The traditional mode of operation has the business group identifying the process changes and then “tossing them over the wall” for IT engineers to implement. Neither group has much visibility into the other group’s function. BPM is a facet of SOA that tears down the business-IT divide by enabling continuous process improvement through modeling, simulation, execution, and monitoring in vocabularies that are shared and understood by both business and IT departments.

SOA implementation today

Now that we understand the goals of SOA, let’s consider a couple of examples that underscore the need for a good SOA implementation. Several industries are facing mandate-driven regulatory pressures today, such as Sarbanes-Oxley and XBRL (Extensible Business Reporting Language) in the financial services industry or a drug “pedigree” in the pharmaceutical supply chain. These compliance measures require organizations to gather data from silos spread across a variety of IT systems and sometimes require integration with third-party Web services. This data often has to be cleaned and transformed to standard formats to enable data exchange.

Another example appears in the various supply-chain IT departments dealing with the introduction of Radio Frequency Identification (RFID) technology. RFID enables real-time monitoring of supply chains, leading to their improved efficiency and operation. However, most supply-chain IT applications are brittle and cannot use the real-time, granular data that RFID provides. As we know, SOA greatly reduces integration complexity and future-proofs solution evolution without greatly affecting production environments. Hence, SOA provides compelling advantages for RFID solutions through increased agility, flexibility, and reuse. Enterprises engaging in RFID-based solutions can thus increase their responsiveness and decrease integration costs using SOA as a technology base.

Non-SOA or halfhearted SOA approaches lead to custom integrations, which cost a lot of money and make the systems more tightly coupled, thus exacerbating the problem.

Most IT departments are simply testing the SOA waters today. Some are deploying services on a small scale for internal consumption. Many are building service wrappers on top of legacy applications to achieve reusability and reduction of operational costs. Either way, the focus of SOA implementations seems to be on the service layer.

Confusion abounds

The term SOA is really a misnomer. It is not so much an architecture as it is a methodology, and there are many ways to implement each layer of an SOA. A recent AMR Research survey reports that Web services are the predominant way organizations are building services, but integration frameworks, portals, application servers, and business process management servers are also actively being used. The broad landscape of options for building the service layer may be a good thing, but it is also a source of confusion for many organizations considering SOA. So much time and effort is spent on cutting through the techno-babble and identifying a suitable service implementation that organizations cut back on the other, equally important components of an effective SOA and decide to only build the service layer to get a feel for the benefits of their investment. As a result, these SOA implementations end up becoming mere proof of concepts. They are not well thought out and do not address important concerns such as scalability, security, and governance.

SOA prototypes demonstrate potential benefits, but the path to a true enterprise-grade SOA appears too daunting for most organizations to follow. Organizations must realize that a service layer is simply not enough to deliver the SOA promise. There should be a realistic mapping between the organization goals and the components required to fulfill them.

Limitations and challenges

We have already mentioned the gap that looms between the lofty expectations from SOA and the actual investment in the components that will serve those needs. Let’s now examine the challenges associated with implementing a successful SOA.

Cultural barriers

The goal of breaking down organizational silos presents more than just technical challenges. Many companies are not prepared for the deep cultural changes that accompany this philosophy. Many people are accustomed to having complete control over every aspect of their particular application’s requirements. Now they are dependent on services provided by other groups. If this change is not handled sensitively, it might lead to resentment at relinquishing control, or even apathy, because someone else owns the data and services. This problem is further exacerbated if you are sharing services not only within your organization, but also with your trading partners. Now you have to pay careful attention to the data and services you expose because they are external participants in your process.

Who is responsible?

A potential for confusion is possible regarding ownership of data and its veracity. Imagine an application that reuses a service owned or developed by another group. This service, in turn, might use other services belonging to several different groups. If the application does not work because the underlying service failed, imagine how difficult it becomes to follow the links to identify the problem’s owner and how a fix must be percolated through the layers of services before it eventually repairs the application.

Implementation fatigue

Besides the cultural and logistical challenges, many technical challenges are involved in implementing SOA successfully. Organizations are taking tentative steps towards just the service layer implementation, which is but one component of SOA. Even with this limited scope, and typically applying the straightforward strategy of building service wrappers around legacy applications, people are discovering more complexities than anticipated. They are finding that the size, number, and complexity of legacy systems make business processes immensely difficult to configure, defeating one of the primary goals of SOA.

Burgeoning services lead to scalability issues with data management. A handful of services may be easy to publish to a registry and discover for consumption. But the true benefit and, indeed, a challenge of SOA, is when you have hundreds or thousands of services that must be published, discovered, and managed efficiently.

Bridging the business and IT divide, and achieving flexible configuration of business processes requires an investment in business process management solutions. The problem is that there is as much confusion and choice in BPM implementations today as there is in the service layer implementation. The standards are evolving, and the dust has yet to settle on the vendor claims and debates that invariably accompany such a new technology.

What challenges have you experienced in implementing SOA? Post your comments in the discussion thread that appears below.

Components of an enterprise-grade SOA

The following section provides an overview of the necessary components for enabling an enterprise-grade SOA.

Service layer and registry

The focus of most SOA implementations today is on the service layer and often extends to the registry to publish and discover services. This approach to SOA is easy to comprehend. As mentioned previously, the concept of exposing services is not a new one. Most software architects and engineers have years of experience building services and should be comfortable applying their knowledge to expose services using the newer Web and XML technologies. Once the services are implemented, registry products based on UDDI (Universal Description, Discovery, and Integration) are used to publish and discover them.

With this architecture in place, enterprises already have much better visibility than was ever possible before WSDL (Web Services Description Language) and UDDI standards emerged on the scene. Unfortunately, it is because of this pleasing ROI that most implementations stop short in their tracks—that is, until it becomes apparent when trying to scale their model that a service layer with a registry is simply not enough for true SOA ROI.

More to SOA than meets the eye

When the typical SOA implementation based on the service layer and registry begins to see some success, the IT department and the business groups push to put more and more services on it. The volume of Web services published and consumed increases dramatically. This immediately puts focus on critical features that were not anticipated earlier.

The first feature requirement is the need for federated information management. Service consumers are suddenly exposed to hundreds, if not thousands, of services and need a federated view of the data and services to comprehend them.

The second requirement is the need for SOA data caching. The vast amounts of data resulting from a well-adopted SOA implementation cannot be accessed in a fast and reliable manner without some kind of caching.

The third requirement arises out of a painful realization that the data and services need an unprecedented level of governance to ensure their validity. SOA governance is the process of defining and enforcing organizational policies and standards. These policies are geared towards the business requirements of managing liabilities and dependencies, ensuring continuity of business operations, and reducing costs. They also benefit the IT departments by defining controls for service proliferation within the enterprise, incorporating evolving standards, and promoting interoperability.

The typical reflexive response of organizations is to address these issues in an ad-hoc manner, as they begin to surface. However, true SOA ROI is achieved only when organizations take a systematic and well-crafted approach to fulfilling these requirements.

SOA repository

Enter the SOA repository. The repository has the capability to provide a much richer set of services than a registry would, since it not only has access to the metadata around the services, but also the actual SOA data. Therefore, the metadata plus data-based discovery services provided by a repository are much more powerful than the registry’s simple metadata-based services. The repository can also provide integrated policy management and transformation, federation, abstraction, and caching of SOA artifacts. Business processes and composite data services could be deployed and managed on an SOA repository to ensure that they are centralized, transparent, and therefore, easily governed.

For all the above reasons, the SOA repository is the core component of a well-designed SOA.

Workflow and business process management

A critical SOA goal that we talked about is achieving process visibility and flexibility. The service layer addresses this goal best, using workflow or business process management systems, which is emerging as a fast-growing segment of SOA. The vendors in this market range from pure-play BPM and workflow companies to EAI (enterprise application integration) and application server companies. The products vary in terms of price and maturity, but there is no doubt that we will see many excellent, cost-effective solutions for SOA implementations over the next few years.

A well-rounded workflow or BPM product should be both business-user and IT-department friendly. It should offer zero-code process modeling, simulation, execution, monitoring, and debugging. Ideally, the product should be nicely integrated with an SOA repository so that the workflows and business processes can be deployed as metadata for central governance. It should also support the development, deployment, and consumption of reusable composite data services.

Implementing SOA the right way

The figure below depicts a well-designed SOA.

A full-featured SOA repository is the core component of a well-designed service-oriented architecture. It should not only contain all the data and metadata relevant to an organization’s functioning, but also provide a comprehensive set of features to manage these artifacts effectively.

A good SOA repository natively embeds data administration and governance services. It provides a comprehensive dictionary with semantic reconciliation capabilities. It allows the organization of data and services into meaningful taxonomies as well as provides lifecycle management and versioning services. It provides a full set of data services to perform quality management, validation, transformation, federation, governance, and caching.

These services are also automatically available to your business-process- and workflow-related artifacts, customized composite data services, and third-party services when you deploy them onto the SOA repository.

The other component featuring prominently in the diagram is the BPM/workflow platform. The application layer may discover and consume services directly from the service registry/repository layer, or it may go through the workflow/business process management platform for more flexibility.

While the BPM/workflow platform provides one set of services to applications, an external service layer could simultaneously register services for clients to discover and consume.

The importance of an SOA repository is not lost on most registry vendors. Today, many SOA registries have some sort of repository feature bundled into them. However, the richness of the feature set as well as the underlying implementation of the repository is critical to its success. Indeed, one reason repositories are being relegated to the back burner is because few implementations meet the rigorous functionality and performance requirements.

Native XML database and XQuery

We have noted that the repository deals with an immense amount of data, and therefore, it should be designed with a focus on performance and flexibility. The problem with most repository implementations is that they are based on traditional relational databases, which are simply not agile enough to handle the query loads generated even by a mid-sized SOA implementation. All SOA data—the artifacts as well as the messages—are in XML and have a hierarchical representation, which is not well suited for the rigidity of an RDBMS (relational database management system). This problem exacerbates as the SOA implementation grows to include more endpoints/providers, orchestrations, and subscribers. The lack of a powerful repository also complicates governance and administration. For instance, if you want to apply a change to a set of WSDLs in the registry, it is easier to do that with the powerful query capabilities of XQuery, which can select the right artifacts based on the query criteria and update them. Since most of the artifacts in an SOA are XML, native processing of all of the SOA XML data is needed. A truly responsive SOA repository must thus be implemented on a native XML database (XDMS) with XQuery for data management.

XQuery is a flexible and powerful language for XML data retrieval and management. Coupled with a native XDMS (XML database management system), you have an elegant, high-performance option to the complex and sluggish middle-tier conversions of your XML data using the traditional parsing approach with an RDBMS SOA artifacts store.

An XDMS supercharges the implementation of SOA by enabling data to be stored and manipulated as native XML. Of course, not all native XML databases are built the same way, nor do they offer the same features. A good XDMS should handle any type of XML data without prior knowledge of the XML Schema structure. This functionality proves highly advantageous when handling XML documents from federated systems with disparate datastructures. Powerful composite data services can then be built on top of such a platform.

Benefits of the correct approach

Using SOA in the right manner enables business efficiency, agility, and innovation through information on demand, service reuse, process optimization, and incorporation of innovative third-party Web services. Making the SOA registry and repository the cornerstone of the SOA ensures optimal administration, management, and governance of an enterprise’s SOA infrastructure. SOA leads to a flexible architecture where several third-party Web services can be used to provide specialized functionality to a business process.

SOA can empower a wide variety of use cases, spanning a variety of systems across different geographical locations and trading partners. Going back to one of the examples we used earlier, many pharmaceutical companies use the National Drug Code (NDC) as part of their supply chain and regulatory processes. This NDC database is constantly updated as new drugs are added, recalled, or when patent protection on prescription drugs expire. An SOA-enabled enterprise can subscribe to a widely available NDC database Web service, which keeps this data current, and thus effectively use the data to flexibly evolve as new products are introduced or recalled products are removed from the pharmaceutical supply chain.

Similarly, for various regulatory reporting requirements that constantly evolve, data must be gathered from different IT systems. An SOA-based approach solves the problem elegantly, where several versions of similar Web services can be maintained in an SOA repository and dynamically chosen based on data parameters.

Conclusion

There are many challenges to implementing a successful, enterprise-grade SOA. The critical mistake most companies make lies in the gap between organizational goals and the actual investment in the right components and technologies to achieve those goals. Much effort is spent on the service layer, but the core component of the architecture, the SOA registry and repository, is not designed well enough to scale effectively. Moreover, organizations tend to forget about effective data management and governance services until it is too late.

Most of the enterprise data flowing within and between organizations is XML. The full potential of SOA can be realized if the implementation is built around the infrastructure that is best suited for the management of such data. A good native XML database coupled with XQuery as the basis for comprehensive data services provides a rock-solid foundation for a flexible, scalable, enterprise-grade SOA registry and repository.

We would like to thank Ajay Ramachandran, CTO, and vice president, XML-Centric Applications and Platforms group, and Premal Parikh, engineering manager and lead architect, XML-Centric Applications and Platforms group, at Raining Data for technically reviewing this article.