World’s fastest rules engine

Imagine, just for a few moments, that you are the CIO, CTO, or senior architect of a large company with huge amounts of data and thousands and thousands of business rules. What you need is the very best way to communicate the business logic to the IT guys and make sure you don’t make a lot of mistakes along the way. So you’ve looked at two or three of the major commercial Business Rules Management Systems (BRMS) and probably even tried a couple of the “freebies” to see how things run.

One of the questions that most of the vendors will duck is “What about performance?” They will usually try to tell you that their performance is sufficient for almost any business situation and that they have systems in production today that handle 20,000 or 30,000 rules with lots and lots of objects. But you have 100,000 rules with millions of objects. What do you do? What you need is something faster than anything they have.

[ Get smarter about how you handle the explosion of enterprise data with InfoWorld’s Enterprise Data Explosion newsletter. ]

Good news! If your BRMS uses the Rete algorithm — most do — the vendor can speed up rules processing by anywhere from 10 to 1,000 times by implementing a new execution engine in addition to or in place of the current engine. The new OPSJ Rete-NT engine from Production Systems Technology uses the latest Rete incarnation from Dr. Charles Forgy, founder and chief scientist of the company and the original inventor of the Rete algorithm. The Rete-NT engine will work with any Rete-based BRMS, including those from IBM, Oracle, Fair Isaac, Red Hat, and Pegasystems. It is available to the vendors (licensed at $5,000 per CPU) or can be used directly by PST clients with the OPSJ rule syntax.

PST gave me the chance to see how fast the Rete-NT engine can go. Now, the benchmarks I ran are not what you could call “real world,” but they are designed to reveal the faults and performance bottlenecks that any Rete-based engine might have. The Waltz, WaltzDB-16, and WaltzDB-200 tests use a series of lines as 2-D end points to build a 3-D box. DB-16 and DB-200 build 16 and 200 boxes respectively, with each box constructed of 20 lines and obeying a number of constraints. For example, the lines must match up correctly, and we must use all of the lines to build the boxes. There are 35 complex rules.

In the early 2000s, it took a machine several minutes to determine the best way to build 16 boxes. Today it takes just a fraction of a second to build 16 boxes, and 17 seconds or longer to build 200. The Rete-NT engine can build those 200 boxes in about 2 seconds. Times shown in the table below were taken on various dates but roughly reflect the performance differences among the leading rules execution engines available today. My test platform was a Dell PC with a Core i7 CPU and 12GB of RAM. Blaze Advisor, JRules, and Drools were not tested on WaltzDB-200 due to lack of time and vendor support.

Rules execution benchmark results

JRules, Drools, and a number of other execution engines all use what they call an enhanced version of Rete that does improve performance. Except for PST’s OPSJ, only Blaze Advisor uses Rete-2, which Fair Isaac licenses from PST. The original Rete algorithm, which Forgy developed with DOD money, belongs to the public domain.

You might notice that WaltzDB-16 takes longer to run on Rete-NT than on Rete-2. This is because it takes only a few microseconds to build the straight-up Rete or Rete-2 “network,” but just a wee bit longer to build the more complex Rete-NT network that will handle anything you can throw at it. This overhead is a small price to pay. Considering that the number of rules and objects in a system always grows, you might as well start with the best maximum performance for the first implementation. You can count on any rulebase system to grow by a factor of four in a couple of years.

The Rete-NT algorithm is at least 500 times faster than the original Rete and 10 times faster than Rete-2. It will allow massive amounts of data and huge numbers of rules to be used to solve complex problems almost instantly — before a scam artist completes a fraudulent credit card transaction or a terrorist boards an airplane, for example. For companies trying to track, say, millions of credit card users or thousands of airline passengers, and apply thousands of rules to reach an answer within a second or two, Rete-NT opens up new possibilities.

The main purpose of a BRMS is to allow the business rules in complex applications to be quickly and easily changed in response to business needs. Several vendors (IBM, Oracle, Fair Isaac, Pegasystems, Visual Rules, and so on) have succeeded in making this process as simple as possible, even allowing business users to enter and modify the rules directly. Now the problem is performance. The cross-matching of millions of objects and thousands of rules can run into the trillions of comparisons for just a single execution. With PST’s Rete-NT, we finally have a rulebase engine up to the task.

James Owen is senior consultant and architect at Knowledge-Based Systems Corporation and contributing editor to InfoWorld. 

This article, “World’s fastest rules engine,” was originally published at InfoWorld.com. Follow the latest developments in applications, application development, and enterprise data management at InfoWorld.com.