Elasticsearch stack wises up with machine learning

Elastic, the commercial company supporting the Elasticsearch stack for searches of real-time data, has added machine learning functionality to all the pieces of the Elastic stack.

Unlike some other companies, Elastic isn’t claiming this addition is cure-all magic dust. Rather, it’s for performing specific analysis for explicitly defined use cases.

In a blog post that went live yesterday, Elastic outlined examples of Elasticsearch’s anomaly detection in action, such as detecting changes to a performance metric or analyzing many metrics together to determine when one is out of gamut.

The advantage of integrating this analysis with the product, said Elastic, is that it can be executed in the same distributed manner as the rest of the work done in the Elasticsearch stack. “Machine learning jobs are resilient to node failure,” Elastic stated. The analysis can be performed faster since it doesn’t have to be exported from the cluster and processed separately.

This addition is still considered a beta, and the machine learning features are not open source like the other elements in the Elastic stack—they’re only available through the X-Pack commercial add-on. The features were themselves added when Elastic picked up a company called Prelert late last year.

Prelert offered a Splunk application that was effectively end-of-lifed by the acquisition—no surprise, given that Elastic (and other open source projects) is now direct competition for Splunk. While Elastic has built its business atop open source, it can certainly still choose to compete as aggressively as a proprietary provider.

One possible issue is that non-open-source machine learning applications can look more impressive than they actually are. Elastic is avoiding that (for now) by confining the promise of the new features to specific, well-defined goals. It’s also likely to be even more powerful when a full non-beta version is available at the scale provided by cloud partners like Google.