Benchmarking the rules engine

Over the years developers in the rules-management business have tried to establish a set of benchmarks that will allow the users to test various systems for speed and efficiency on different, complex problems. The two most famous tests are the Miss Manners test and the Waltz benchmark. Both of these benchmarks, and three others, can be found at infoworld.com/312 and will run on most any platform.

The Miss Manners benchmark originated about 15 years ago with OPS5 and CLIPS (C Language Integrated Production System), two programming languages for rule-based production systems. Miss Manners includes a relatively simple rule base with only eight rules and a data generator. The idea is that Miss Manners has invited 16, 32, 64, or 128 guests with various hobbies to a dinner party. She wants to seat the guests in boy-girl-boy-girl arrangement so that each guest will have someone on the left or right that has a common hobby. 

The Miss Manners test becomes increasingly complex with the addition of hobbies and guests. The original test was written to stress any rule base, but some BRMS vendors discovered a trick to beat the system: By putting a single “not” statement in one of the rules, they enabled their systems to run the test 15 or 20 times faster. (Another trick is to rearrange the data so that the rules will run faster because the benchmark is data-sensitive.) Nevertheless, if you omit the tricks, Miss Manners is a very good measure of how fast a rules-inferencing system will run on any give platform and CPU.

The Waltz benchmark is another old test, but a good one. Waltz stresses a rule-base system by checking to see how well the rules engine does pattern matching. Consisting of 32 rules and a data generator, it will analyze the lines of a two-dimensional drawing and label them as if they were edges in a three-dimensional object. The Waltz benchmark is much harder to cheat with than the Miss Manners benchmark.