Big data analytics star in VMware vCenter Log Insight 1.5 update

Over the last 10 years, server virtualization has solidified itself as a key component to shrinking the hardware footprint of the modern data center. However, what hasn’t decreased during this process of consolidation is the amount of data created on a daily basis from the devices, the physical servers, the virtual machines, and the operating systems and applications that live on each of these instances. In most of these cases, the data has become magnified.

As virtualization administrators were forced to branch out from their original comfort zones of server administration to storage, network, and even desktop administration, the next technological leap for these individuals will therefore come in the form of big data and analytics. As in other areas in the server virtualization world, companies like VMware are making an app for that in order to make that transition easier.

[ Also on InfoWorld: 10 big virtualization and cloud stories from 2013 | Parallels says key to success is cloud virtualization, not Mac desktop virtualization | Track the latest trends in virtualization in InfoWorld’s Virtualization Report newsletter. ]

That’s where VMware vCenter Log Insight comes into play. The product debuted back in June 2013, as a result of VMware’s August 2012 acquisition of Pattern Insight.

If you aren’t familiar with it, Log Insight is VMware’s solution for log management and analytics of dynamic hybrid cloud environments. It can analyze large amounts of unstructured machine generated log data, enabling deep, enterprise-wide visibility. It provides interactive, real-time search and analytics of that data in a meaningful way so that IT users can identify and analyze the data, then use that intelligence to proactively enable service levels and operational efficiencies across those dynamic hybrid cloud environments.

Common use cases for the product include security and compliance auditing, as well as monitoring and troubleshooting vSphere and other servers, storage, and networking devices.

Earlier this month, VMware released vCenter Log Insight 1.5. The bulk of the work done on this release was to make it a more enterprise-ready product. As an example, VMware added authentication support for Microsoft’s Active Directory for easier integration into an enterprise environment. This eliminates the need for multiple logins/passwords and allows for seamless integration into an organization’s pre-existing identity management architecture.

Some of the interesting new features in version 1.5 include:

• Significant query performance improvements by optimizing the execution of common queries
• New analytics function: Unique count (ucount)
• Better integration with VMware vSphere and vCenter Operations Manager
• User interface for upgrading from previous versions of Log Insight through Administration UI
• User interface for syslog configuration of ESXi hosts
• Easier user deployment types with installation guidance on what capacity each one provides during installation
• Improved health monitoring of the Log Insight Virtual Appliance

In an end-of-the-year prediction, Paul Strong, who helps lead VMware’s Office of the CTO, said we have to use machine learning and big data to infer structure, along with good and bad behavior. Strong went on to say:

Machine learning lies at the heart of vCenter Operations Manager, and Log Insight uses big data techniques to evaluate log files. One of the things I would expect to see across the industry in general in 2014 is more use of these techniques, and tying these to provisioning engines, to enable more automated, policy driven closed feedback loops, for application service level management.

That’s where VMware’s vCenter Orchestrator would come into play.

For this technology to continue to expand throughout 2014, Log Insight will need to draw data from other devices that fit within the application’s framework for data handling, moving beyond servers and into other areas like storage and networking devices from various hardware manufacturers. Capturing that type of data will turn the learning system and the enhanced future use cases that Strong talks about into a reality.

But there’s good news! With version 1.5 of Log Insight, that knowledge expansion is already under way. One of the most interesting features found in this new release is designed around the concept of something called “content packs.” Strong said VMware’s goal is to be able to collect all operational data in the data center, both structured and unstructured. To further this goal, the improved content pack framework in vCenter Log Insight 1.5 allows you to produce charts, alerts, and dashboards for user-specific logs. The company has also introduced new features to facilitate the sharing of content packs across organizations and communities.

The first set of pre-built content packs is already available. Within the company’s community download center, VMware offers 12 publicly available, free-of-charge content packs that can be downloaded and added into Log Insight. Beyond support for VMware vSphere, vCenter Operations Manager, and VMware Horizon View, other vendor applications that have already made the list include Cisco UCS, EMC VMAX, EMC VNX, ExtraHop Wire Data, HyTrust Appliance, NetApp Data ONTAP, NetFlow Logic, Puppet Enterprise, and VCE Vision Intelligent Operations.

VMware may be trying to build a better mousetrap that can help administrators analyze and make sense of data center logs, but it certainly isn’t alone in this endeavor, nor is it the first. The market may feel a bit overcrowded with products from companies like Loggly, LogRhythm, SolarWinds, Splunk, and Sumo Logic already available. But VMware’s entry stands out, if for no other reason than because of Log Insight’s tight integration with VMware vSphere and vCenter Operations Manager.

And we can’t discount VMware’s huge success with its partner and community base either. We should anticipate that these content packs will continue to grow and evolve.

Since this is a VMware product, the virtualization giant can also choose to up sell Log Insight with sales of its other products, or it can throw it in for free for a period of time, both of which could help push adoption of their application over that of its competitors — at least in a VMware environment setting.

This article, “Big data analytics star in VMware vCenter Log Insight 1.5 update,” was originally published at InfoWorld.com. Follow the latest developments in virtualization at InfoWorld.com.