Integration with external information systems

Consider an auto insurance domain application that helps valuators evaluate vehicles. The backend systems may have to integrate with external vendors to fetch the vehicle’s valuation reports. Similarly, in the case of mortgage banking, for example, the loan origination systems must interact with the external vendor systems to obtain information about the borrowers and their property, such as credit reports and appraisals. These interactions between an enterprise system and external systems may involve data exchange using disparate formats. For example, internal enterprise systems may use OAGIS BOD formats, but the vendor’s format may differ entirely. Add to that the existence of multiple vendors with no industry standard for the data formats being exchanged. Furthermore, think about those cases where either side of the data formats continues to change due to various business or technical reasons. All of this adds up to a complex integration solution that the enterprise must build.

The problem of automating and integrating business processes and interactions between companies is known by different names: external IS integration, B2B integration, e-business integration, trading partner integration, etc. Numerous case studies and theories show the benefits B2B integration can bring to businesses. Typically, advantages include a huge reduction in paperwork, response times, errors, and business operation costs. B2B integration helps companies deliver value and increase business opportunities.

Building interoperability between disparate business systems that may have no connectivity or shared infrastructure is the challenge a B2B integration solution must address. This challenge involves integrating data and processes with no centralized control or normalization. On a positive note, across different business domains, a basic set of high-level requirements has evolved for achieving interoperability between trading partners. However, each of these requirements has its own set of technical challenges.

In this article, I discuss some of the important design issues involved in building a B2B integration solution and an approach for addressing those issues.

B2B integration scenario

A typical B2B integration scenario is depicted in Figure 1, where multiple applications within an enterprise interact with multiple external information systems. This can be an MxN interactions scenario (i.e., M number of systems interacting with N different systems). Each system may have its own data interchange formats that don’t necessarily need to be understood by the external systems and vice versa.

Based on the most common set of requirements across different businesses, the integration workflow (i.e., the B2B integration solution) completes the following steps at a high level:

• Receives and validates the request message destined for an external IS to fetch some business data. For example, a vehicle valuation system might need the vehicle’s valuation report from a vendor like Carfax. Or, in the case of a loan origination system (LOS), a borrower’s credit report might need to be ordered from vendors like Equifax or NMR.
• Possibly selects from a list a particular external IS (vendor) to fetch the requested data. The enterprise might have set up multiple vendors to serve its demands for the external data it needs. An LOS, for example, may have multiple credit report vendors to choose from.
• Runs some business rules that can be specific to the request context.
• Creates the order in the format appropriate for the selected external IS. This might require transforming the request from the format of an internal system into the format required by the external IS being contacted.
• Transports the request message that contains the order in the vendor’s data format to the vendor over the transport that the vendor supports. This task might involve invoking a Web service, sending a JMS (Java Message Service) message, or simply sending the order data over FTP. Depending upon the communications mode, the interaction with the vendor system may be synchronous or asynchronous.
• To the external IS response, applies similar steps (such as validation and data transformation) that were applied to the outgoing order requests. The response that the vendor sends for the order might again be in a format not understandable by the internal systems. So a transformation might have to be applied to the vendor data format before sending it to the internal system.
• Manages the interactions with external IS as well as defined transactions. To automate the information exchange, some intelligent processing mechanism must be in place for managing all the required activities and events in any particular information exchange transaction.

This is a high-level picture of a rather simple vendor integration scenario. Of course, the actual scenario can have more complex request-response interactions between the enterprise and the external IS. Typically, you’ll have a process definition that defines the above logical steps. Depending upon how you implement this process definition, each individual step listed above could be carried out by invoking the appropriate implementation/API. The process definition could be as simple as a Java class that uses appropriate APIs to perform the above listed steps. Alternatively, you might use tools like WebLogic Integration (WLI) or JBoss jBPM to implement the process definitions.

Main architectural forces

In most B2B integration scenarios, we must consider a common set of architectural forces during the system’s design:

1. Nature of the interactions with the external information systems. The enterprise systems might have asynchronous and/or synchronous interactions with external ISs. The nature (synchronous or asynchronous) of communication is important for transaction management because, in most cases, these transactions are more than just atomic document exchanges (short-running, self-contained transactions). Transaction management also includes business processes orchestration and long-running transactions (where a document exchange may need to be correlated and validated with others in a predefined process).
2. Type of data interchange formats (e.g., XML, Electronic Data Interchange, and delimited text) on either side of the enterprise boundary, and their frequency and susceptibility to change. Business factors may lead to the change in the data interchange formats. For example, the enterprise might want to bring in a new vendor to receive the data it needs, and the data format that the new vendor offers differs from what the enterprise applications are currently capable of handling directly.
3. Possibility of change in the implementations of the integration workflow steps. For instance, your enterprise might want to use a hardware-based XML processing solution so your system can handle more requests that arise due to the growing business. This change might require your integration solution to use the new hardware-based XML processing solution to perform your XML validations and transformations.
4. Existing technology investment in terms of available skills and software the enterprise has made in the integration problem space. For example, the enterprise might have skilled resources and software it wants to use for building the integration solutions with a particular set of technologies such as WLI, jBPM, Tibco, and others.

In addition, there will be, of course, the nonfunctional requirements from the integration solution.

A typical solution

Though the actual implementation of a B2B integration solution will vary from business to business, based on the most common requirements, a typical architectural representation of such a solution may appear as depicted in Figure 2.

The system’s main parts are:

1. An event listener for receiving the client requests and external IS responses. This component can be a message-driven bean (MDB), a servlet, etc., depending upon the invocation interfaces the integration solution must support.
2. A process definition that defines the logical steps in the integration workflow. As highlighted previously, the process definition can be implemented by a plain Java class, tools such as WLI or jBPM, or any other technology. This part can implicitly hold the transaction management logic (both for short and long-running multi-event transactions).
3. Supporting components used by the process definition to carry out its steps. Your workflow might use some existing components or services to realize the steps such as data validation, transformation, and business-rules execution.

At the very least, a good design of such an integration solution should:

1. Allow the separation between the integration workflow definition itself and the implementation of the individual steps composing the workflow. This is important for allowing easy management of the changes in either the definition or the implementation, which can be caused by business or technical reasons. Both the workflow definition and the implementation of its constituent steps can evolve and change independently.
2. Allow for seamless integration of new external information systems with enterprise systems. Consider a loan origination system that must fetch the borrower’s credit report from an external IS. The business wants to change or add a new credit vendor due to business reasons, and the new credit vendor employs a different data format. The integration system’s architecture should be designed so that such changes are as least intrusive as possible.

Addressing the architectural constraints

To address the architectural constraints listed in the previous section, you should consider the following points when building a B2B integration solution:

• Decouple the integration code from the structure or layout of the events-data being exchanged
• Decouple the integration process definition and its constituent logical chunks of functionality
• Choose the proper data manipulation mechanism

These considerations, detailed in the following sections, will help you design a robust B2B integration solution.

Decouple the integration code from the exchanged events data

To make the processing decisions during event handling (client request or vendor response), the integration code must look at the event data. For instance, let’s say the validator implementation to be used for an event is determined by the sender’s identity. In this case, the integration code must know where to find the sender identity in the received event data. Typically, elements like event type, sender, and sender reference/correlation ID, which the integration code needs to use during event handling, can be classified as event headers.

To build a robust integration solution that seamlessly integrates with new systems, it is important to decouple the integration code from the details of the event data’s structure. In a given B2B integration scenario, such event headers will be finite and can be easily identified. One way to achieve decoupling is to configure the header information (or header metadata) for each event in a datastore. At runtime, the integration code can look up the header metadata from the configuration datastore and use it to compute the header values for the event at hand. For example, if the data interchange format is XML, then the header metadata for the events might include something like:

1. XML Path of sender’s reference in the event document
2. XPath of receiver’s reference in the event document
3. XPath of any authorization identifier in the event document
4. XPath of status information in the event document
5. XPath of the element that defines the event type for the event document

Decoupling the integration code from the details of the event data’s structure is a powerful idea. It frees the integration workflow code from knowing the structure of each and every event it must handle. It allows the integration code to use a generic mechanism (e.g., the XPath API) to access and manipulate the event’s data. As far as the integration workflow code is concerned, integrating new ISs and their associated events is now just a matter of configuration. Of course, you’ll have to create the appropriate validators and transformers specific to the new IS data formats, but they will be plug-ins to the existing system. What we are achieving is an “open-for-extension and closed-for-modification system.”

Decouple the integration process definition from its constituent functionality

Typically, the integration solution performs the steps listed in the “B2B Integration Scenario” section. For example, a simple request processor might look something like:

 
public class SimpleRequestProcessor {
    
    // Members might go here ..
    // ...
    
    // Main "interface" method
    public void processEvent(Object event) { // 'event' can be request or a response
        EventMetaData eventMetaData = EventInfoResolver.resolveEvent(event);
        Validator validator = getValidatorToBeUsedForEvent(eventMetaData);
        validator.validate(event);
        VendorSelector selector = getVendorSelectorToBeUsedForEvent(eventMetaData);
        VendorInformation vendorInfo = selector.select(event);
        Transformer transformer = getTransformerToBeUsedForEvent(eventMetaData);
        Object dataInVendorFormat = transformer.transform(vendorInfo, event);
        Transporter transporter = getTransporterToBeUsedForEvent(eventMetaData);
        transporter.transport(vendorInfo, dataInVendorFormat);
    }
    
}

Here, you would create appropriate implementations for the Validator, VendorSelector, etc. By “programming to the interfaces,” we can eliminate the coupling between the process definition and the implementation of its constituent steps (represented by interfaces such as Validator and VendorSelector), on which it depends for doing its job. However, both the process definition and implementing classes of the above mentioned interfaces would still require access to the interfaces themselves. You can eliminate even the dependency on the interfaces too by using dependency injection and reflection, or aspect-oriented programming (AOP). Programming to the interfaces also provides the ability to use an event-specific implementation for a given step (e.g., validation).

An important thing to note in the code above is the line EventMetaData eventMetaData = EventInfoResolver.resolveEvent(event);. The EventInfoResolver helps resolve the event and its header metadata. Using the EventMetaData, the integration code computes the event header values from the event data it receives at runtime. EventMetaData holds the information about the event headers as noted in Point 1 above.

Choose the proper data manipulation mechanism

The choice of the data manipulation mechanism plays an important role. We must weigh both compiled and dynamic mechanisms for XML manipulation.

In the world of Java, compiled mechanisms may include XMLBeans and bindings based on the Java Architecture for XML Binding (JAXB). Dynamic mechanisms can use APIs like Simple API for XML (SAX), Document Object Model (DOM), or XPath to manipulate the data dynamically. Compiled or static mechanisms such as XMLBeans are easy to use especially because of their ability to expose the XML as Java-like objects. The downside of compiled or static mechanisms such as XMLBeans is that every time the XML schema they represent changes, you must recompile the XML schema to regenerate the XMLBeans. Such recompilation can possibly trigger changes in the code that uses the XMLBeans generated from the XML schema. Compiled mechanisms prove appropriate if the schemas they represent do not change or change rarely.

Dynamic mechanisms are well suited in situations where the XML structure that they must handle changes often and you want to shield the caller code from those changes. Such decoupling of the caller code from the XML schema it intends to manipulate is not easy with complied XML manipulation mechanisms such as XMLBeans or JAXB.

Conclusion

Any solution addressing the B2B integration problem is most likely going to be specific to the business domains. This article looks at a solution’s most common requirements, thereby giving you a view of some important architectural issues to consider while designing a B2B integration solution. I’ve touched on just a few but important design issues (extensibility and flexibility to absorb change, achieving an “open-for-extension and closed-for-modification” system) and how to address them. The points explained in this article should help you weigh the different design options available to you.