Complex event processing in a nutshell

Corel8 Engine, reviewed today by James R. Borck, is the fifth complex event processing solution to meet our test bench, after AptSoft Director, iSpheres Event Server (now Avaya Event Processor), Progress Apama, and StreamBase.

For a quick and clean explanation of CEP, you can’t beat Jim’s introduction to the March 9, 2006, review of AptSoft and iSpheres:

It’s no secret to IT people, or any business executive worth his Beemer, that an amazing wealth of actionable business intelligence is coursing through enterprise applications, databases, and even system logs nearly every moment of every day. The problem has not only been plucking the meaningful events from the unimportant ones but also finding the often seemingly unrelated patterns between them, and doing so before it’s too late to make a difference — before the supplier raises the price, the shopper leaves the Web site, or the scammer transfers the funds.

CEP (complex event processing) promises to help you capitalize on the rich information hidden in your data streams — information typically tossed away in log files and data warehouses. CEP helps users correlate complex events (often seemingly unconnected ones), spot trends, trap anomalies, and trigger alerts that can be used to seize opportunities or mitigate risks.

Tapping into distributed message buses to unearth important combinations and patterns of events that span multiple, disparate systems and lengthy, dynamic business processes, CEP meets challenges that traditional analysis and automation tools are inadequately suited to address. As a result, CEP is catching on in industries such as banking and insurance, where early adopters are using it to detect fraud and compliance violations. Other promising applications of CEP are RFID and CRM, where the objects, actions, and rules for handling them are constantly changing.

The science behind CEP is the ability to create high-level “business” events from numerous low-level system events. CEP events are created by filtering real-time data (from middleware, applications, databases, and Web services, for example) and infusing it with defining detail such as dependencies or causal relationships discovered by correlating other events.

For example, you could construct a CRM event from an “abandoned shopping cart” message (parsing out the transaction, customer ID, and time), using other filters to extract the value of goods in the cart and tapping the correlation capabilities of the CEP system to add causal indicators such as whether the commerce site was suffering performance problems. Your CRM event might also include customer value or rank from the customer database.

CEP solutions don’t discover meaningful events and patterns on their own. You need to define what you’re looking for. One of the biggest challenges for CEP vendors has been to develop an EPL (event processing language) that is lightweight enough to run in real time, powerful enough to correlate complex data and manage anomalies, and yet easy enough to be used by business analysts.

Regardless of whether the EPL is accessible to humans, the ongoing participation of business analysts is key to the success of any CEP project.

Configuring meaningful CEP rules and filters — and adapting them to evolving business conditions — will require a deep understanding of the underlying business process.

If you get it right, CEP can be far more flexible, responsive, and cost-effective than using traditional BAM (business activity monitoring) and BPM solutions. CEP doesn’t require ripping out or recoding existing systems, and it is actually best applied in modest increments — allowing for frequent and easy comparison of ROI metrics to rule sets.

Avaya Event Processor? Yes, iSpheres was snatched up by the telephony vendor last year and made part of Avaya’s Communications Enabled Business Process solution, a service-oriented approach to coordinating communications in response to business events. For more on that interesting development, see Philip Howard’s “Whatever happened to iSpheres?”

For news, backgrounders, and even academic research on event stream processing, including papers by the godfather of CEP, Stanford University’s research professor of electrical engineering, David Luckham, check out complexevents.com.