Pipelining to connect IT infrastructure

When I was first learning to program, I used Software Toolsby Kernighan and Plauger. The book describes a programming style that relies on creating simple, modular programs strung together using “pipes,” connections between one program’s output and another program’s input. Using these simple, reusable modules, it is possible to create a spelling checker in a single line of code. 

Anyone who’s spent much time on Unix is familiar with pipes, a staple of shell programming in Unix where the small modular programs are sometimes called “filters.” Pipe and filter architectures are noteworthy for their inherent capability of exploiting concurrency and of providing real component reuse, making the structure very flexible.

It’s no surprise, then, that this same flexible architecture is finding its way onto the Web. Businesses are demanding that their IT infrastructure connect not only to other business units but to business partners and customers as well. The interconnected infrastructure also must easily change to match fluctuating business needs. That’s where a pipeline architecture shines, especially when combined with the standards-based, self-describing interfaces of Web services.

Active advantage

Web services naturally lend themselves to being arranged in pipelines. In this model, the Web services act as the filters, and vendors — such as Grand Central Communications, Sonic Software, and Confluent Software — supply the pipes that direct the flow of data. But these products are not just dumb pipes delivering data; they are active intermediaries that can intelligently act on the messages they transport.

An active intermediary reliably stores and forwards messages from one Web service to another. Along the way, it can filter, transform, log, and analyze the message flow in just about any way you can imagine. Unlike the dumb-pipe model, active intermediaries can add security, reliability, availability, scalability, and interoperability without modifying the Web service itself. 

Suppose you’ve created a Web services interface to a legacy application. Later, you decide to restrict access to this Web service to a certain collection of trading partners. You could modify the Web service itself, but this makes it less general and thus harder to reuse in some other capacity. Instead, an active intermediary can sit in front of your Web service and perform the authentication and authorization using LDAP, SAML (Security Assertion Markup Language), or another system. 

Of course, programmers have been writing wrappers — programs that sit outside another program and serve as its proxy — for years. What’s different with active intermediaries is that there’s no program — at least not one that has to be written. Web services are based on the standardized interface protocol SOAP; SOAP interfaces are described using WSDL. The authentication service in our example could use the Web service’s WSDL document to discover specific details of the SOAP-based API and wrap the Web service in an authentication and authorization proxy without anyone having to write any code.

Not just pipes — networks!

Because active intermediaries can look inside the messages and make decisions about what to do next, the message-routing process creates networks of services that are much more flexible than preassembled, linear configurations. A message representing a purchase order with enclosures written in German may need to go through a language translation service, for example, before being routed to a supplier in China.

This kind of intelligence and flexibility comes at a price: Configuring these tools is not a job for beginners. Larger integrations require the creation of an integration architecture — a plan of how the overall system will work and what role each of the pieces plays — and a clear set of requirements for the active intermediary configuration. Because these pipes are “active,” they must be managed and reconfigured as various pieces of the system grow and change.

Indeed, management is one of the first variables to consider when exploring active-intermediary products (see “Selecting the Right Pipes,” page 58). Is the offering sold as a service or as software that must be installed and managed?

In the case of active intermediaries, this distinction is very important. Intermediaries sold as services may also act as brokers — trusted third parties that can serve as neutral ground. This is useful for matching up differences in security policies or connection methods.

Another factor is whether an active intermediary product is primarily focused on creating connections within the enterprise or is designed for use outside the firewall. Some of these products, such as Confluent Core, CommerceOne Conductor, and Sonic ESB (Enterprise Service Bus), began life as tools for integrating applications in an enterprise and thus have large suites of adapters and other connectivity aids. These products will be most useful if you’re looking to connect legacy systems within the organization.

Others, such as Grand Central, were built from the ground up to work outside the firewall, connecting Web services that are owned and managed by outside organizations. If you’re connecting with trading partners who already have Web-service APIs for their systems, a service network is ideal.

The third important characteristic is an offering’s specific architecture. Products built on top of a message-oriented bus, such as Sonic ESB and Grand Central, provide an active message store and forward medium that can offer all the benefits of message-oriented middleware, including asynchrony and reliability. Other offerings don’t include this transport layer and therefore rely on the transport choices of the Web services they’re connecting.

Solving problems

The services provided by current products are important, but there’s more to come, including more powerful and flexible routing, rules engines that direct workflow, easier-to-use message translation, and job scheduling. As organizations see the possibility of using Web services and active intermediary products as an alternative to large, prone-to-fail EAI projects, these products will become an important tool in the enterprise. The race to pipeline the Web is on.

Because active intermediaries provide useful pieces in connecting Web services inside or outside the enterprise — reliable message delivery, logging, security, message transformation, and alerts, for example — they become a logical next step toward streamlined app integration as more organizations adopt Web services.