Review: SaltStack shifts devops into high gear

The only sane and efficient way to manage a large numbers of servers—or even a few dozen, if they change often—is through automation. Automation tools have to be learned and mastered, so they exact a significant up-front cost, but they dramatically reduce the administrative burden in the long run. Perhaps most important, they provide a staunch line of defense against the fatal fat-fingered mistake, which even the most sophisticated cloud operators struggle to avoid.

Configuration management tools such as Puppet, Chef, Ansible, and SaltStack hold the key to automating server provisioning and management tasks. Puppet and Chef are long-established, feature-rich tools that handle the complete lifecycle of IT operations regardless of the hardware platform, OS, application, and configuration model. Ansible is a relative newcomer distinguished by its simplicity and ease of use. SaltStack is neither as convenient as Ansible, nor as feature-rich as Puppet or Chef. But with a highly scalable architecture, powerful orchestration of services, and reliable performance, SaltStack stands tall among these competitors.

SaltStack is an enterprise-grade, commercially supported configuration management tool based on open source Salt. To the Salt core, SaltStack brings a management GUI, role-based access control, LDAP integration, dashboards and reports, auditing, and other advanced features. Salt and SaltStack support a range of platforms, including AIX, Solaris, MacOS, Linux, and Windows.

This review is based on SaltStack Enterprise 5.0, which comes with a completely new GUI, custom dashboards and reports, an integrated database for encrypting sensitive data, and a new multimaster architecture that brings greater scalability and redundancy. SaltStack 5.0 also includes a comprehensive library of modules for handling a wide range of orchestration and automation tasks. You’ll find SaltStack easy to integrate with enterprise IT solutions such as HPE, Nutanix, VMware, Zenoss, and ServiceNow.

Salt and SaltStack were created by Thomas S. Hatch, who had worked with both Puppet and Chef and become dissatisfied with the heavy reliance on Ruby and the slow communication in those earlier tools. Using the power of the ZeroMQ messaging system, Hatch built a fast and highly scalable Python-based system that is now giving Puppet and Chef stiff competition. LinkedIn, NASA, Comcast, and Rackspace are a few notable customers of SaltStack.

The Salt architecture

Unlike Puppet and Chef, which use Ruby to script deployments and configurations, Salt relies on Python as the working language. Like Puppet and Chef, Salt uses a client-server architecture wherein different client machines in the network are controlled by one or more masters. Whereas Salt is limited to a single master, SaltStack supports any number of masters and allows masters to be tiered to scale the system across large environments.

In the Salt architecture, the central command server is called the Salt master and the clients are called minions. The Salt master performs tasks such as authenticating minions, sending commands to minions, and receiving requests from minions, while interacting with admins via the Salt CLI. The Salt master contains a process list for various role categories such as publisher, event publisher, mworker, and reqserver.

A publisher sends commands over the designated transport (ZeroMQ by default) to the connected minions. Each Salt minion has a connection to the master publisher, which uses TCP port 4505. An event publisher is responsible for publishing events to the event bus. The mworker manages back-end processes for the Salt master. At least one mworker process is required. You can specify the number of mworker processes in the Worker_threads configuration file. The reqserver distributes requests among mworkers and receives replies from the minions.

When the Salt master starts up, all of these processes begin as well. As soon as the network boots, Salt minions request a connection to the master that is defined in the minion config file. A cryptographic key exchange is used to authenticate the minion and establish a persistent TCP connection to the master. This persistent connection with the master not only results in faster communications, but also allows for dynamic interactions, such as triggering provisioning or configuration actions in response to events on the minion.

You don’t have to deploy a Salt master to manage Salt minions. They can also be managed as standalone systems. The local salt-call command is used to run a standalone minion.

Salt components

In addition to Salt masters and minions, the key components of Salt are Salt states, Salt pillars, Salt grains, and of course the modules that are used to apply states to minions. Salt states are the most basic building block in the Salt environment. Like the manifest in Puppet, playbook in Ansible, and cookbook in Chef, the Salt state is the file that represents the configuration to be applied to the managed system.

Salt states are written in YAML, which has a number of advantages. First, YAML format is clean, designed for human readability. Second, YAML is language-agnostic and hence portable. Third, YAML has seen widespread adoption that is only growing. On the downside, YAML is verbose (the price of readability) and processing it involves some overhead.

Salt uses Jinja2 templates to define loops and variables. Jinja2 works well within HTML templates because the curly brackets of syntax can be easily distinguished from HTML code. The code becomes clear and concise. However, it should be noted that Jinja2 doesn’t allow you to include Python code in templates, which means you need to code the functionality into your app if you require behavior beyond standard Jinja2 filters.

Salt states gather information from common data types such as dictionaries, lists, and strings. However, regardless of the template data format, template data is translated into Python data types. By default, state files are rendered using Jinja2 templates, which are then parsed as YAML documents. Mako and Wempy templates and the JSON document format are also supported.

The underlying file format can be altered using a different renderer. To manage Salt states, you can use HTML files, Puppet files, XML files, or any other format that is translatable to Python data structures.

Salt pillars are database systems that store global variables, including sensitive data such as passwords and access keys. This information is securely shared with minions on-demand. Although Salt doesn’t allow you to declare the variables on the go, the structured order is an advantage in that it allows pillar data to be easily mapped to minions based on defined characteristics called grains.

Salt grains are facts. They include specific pieces of information such as CPU type, number of cores, and system architecture. Universal states are created using this information.

Salt modules are of two types: namely, state modules and execution modules. While a state module defines a state and tries to fulfill that end state, execution modules simply execute a function. In general, state modules use execution modules to determine state and make changes if necessary. The complete list of state modules can be found on the SaltStack website.

Salt includes several built-in modules to perform common tasks such as installing packages, sending notifications, extracting archives, and accessing Git repositories. With the out-of-box modules, you can save a considerable amount of time. In addition, you will find more than 150 official Salt formulas on GitHub for popular apps such as Docker, PHP, MySQL, Apache, and Node. Salt formulas are prewritten Salt states.

Salt advantages

There are three important aspects of Salt that make it a powerful and comprehensive tool capable of meeting enterprise server management needs. Firstly, Salt enables administrators to manage Linux and Unix server systems while also providing significant support for Windows Server management. Most Salt modules have been ported to Windows, and many Salt states work on Windows. With one tool, you can manage most devices in the network.

Secondly, Salt supports push and pull mechanisms. Push makes it easy to bootstrap the system, so even a bare-metal server can be set up quickly. While Salt uses push by default to send commands to agents, it can also operate in the other direction, allowing minions to query masters for updates.

Thirdly, the ZeroMQ messaging system delivers greater levels of performance while efficiently scaling resources to suit enterprise-class networks and efficiently handle large systems or clients.

While Salt’s working model is similar to those of Puppet, Chef, and Ansible, the biggest difference lies in the communications protocol. For instance, Ansible uses the slower SSH protocol. To compensate, Ansible maintains persistent connections with hosts; that means you always have at least one Ansible process running for each managed node. Similarly, Puppet and Chef use the HTTPS protocol. Puppet includes a “Connection: Keep-Alive” header in the HTTP request to maintain persistent connections, while Chef doesn’t maintain persistent connections by default.

The ZeroMQ messaging library used by Salt scales extremely well. In fact the primary focus of Salt is to facilitate real-time communication with as many servers as possible to support an event-driven infrastructure. Salt helps you monitor your entire network and trigger reactions based on events. With reactive provisioning, configuration, and management across all machines in the infrastructure, you can perform tasks such as instantly restoring configurations to systems after unauthorized changes or automatically scaling cloud resources in response to usage spikes.

Evaluating SaltStack

Capability. SaltStack is designed to perform at peak levels while efficiently managing huge deployments. By default, Salt uses the push method wherein the Salt master is used to remotely execute commands to modify minion states. At the same time, minions can query the master for commands. This asynchronous push and pull communications make Salt extremely flexible and powerful.

Salt state runs are also quite fast. In addition to the high-speed ZeroMQ messaging, Salt takes advantage of persistent connections between the master and minions data caches on minions to make communication faster. And where Puppet, Chef, and Ansible are “static” solutions—meaning you write a manifest, cookbook, or playbook, then execute it—Salt facilitates dynamic communications to respond more quickly to changes in the infrastructure.

By default, Salt executes states in the order in which they are defined in the state files. However, when an action is dependent on or requires another action, you can change this order using a requisite system. Salt provides requisite statements that let you name a state that should be evaluated before the state requiring it. Salt does not hard-fail by default when an error is detected. It will still run to check all the states and create a list of successes and failures for your reference. However, this can be configured. If you wish to halt an execution when a change is discovered or trigger an event such as restarting a service, you can do so.

Scalability. The biggest advantage of SaltStack is its high scalability and resilience. Salt was built with a focus on efficiently managing large deployments, and its speed of communications for remote execution sets it apart from other configuration management tools. Although Salt can access SSH for communication, using minion agents greatly enhances the scalability of the network.

With SaltStack, Salt masters can be configured into multiple levels to form a tiered architecture for redundancy and load distribution. In this way Salt can easily scale from hundreds to tens of thousands of managed nodes. At the same time, you can easily isolate or target a group of minions running under a specific kernel version. In addition, the process of serving files to minions is expedited with the use of asynchronous file services.

Operating system and application coverage. Salt was built primarily to manage Unix and Linux servers. Salt users can also draw on significant Windows Server management capabilities, though Puppet and Chef may have more to offer Windows shops. Both Puppet and Chef integrate well with Windows PowerShell DSC and automate Azure environments.

SaltStack provides a REST API for integrating the system with other tools and clouds, including Amazon EC2, Microsoft Azure, Google Compute Engine, Rackspace, and SoftLayer. Salt Cloud is the provisioning tool used to deploy virtual machines to public clouds and virtualization platforms including VMware, Parallels, VirtualBox, and OpenStack. As Salt Cloud manages virtual machines based on maps and profiles, VM management becomes quick and easy. Salt also enables you to easily provision and manage Docker containers.

Ease of use. Configuration management is simple with SaltStack. Because Salt uses the YAML configuration format, states are can be written quickly and easily. YAML state descriptions are structured well, with solid readability. The support for Mako, JSON, Wempy, and Jinja allows developers to extend Salt’s capabilities. The availability of built-in modules makes it easy to configure and manage states.

With SaltStack, configuration management and orchestration are combined in a single unified system that makes them easier to manage. Because Salt comes with its own orchestration using the ZeroMQ messaging system, when you install Salt, both configuration management and orchestration work right out of the box. By contrast, with Puppet, which uses Mcollective for orchestration, you have to configure and manage configuration management and orchestration tools separately.

Further, because the commands used to run Salt state modules and execution modules are syntactically similar, service orchestration can be scaled and managed with no problem. With Puppet, separate syntax structures and utilities come into picture that make the service orchestration process more complex and time-consuming.

The procedure for installing SaltStack is relatively simple. Salt is a CLI-based tool, and you can install it through Git or the package manager and add the repository. However, it is not always easy to find what you need, when you need it, in SaltStack’s extensive documentation. For instance, you’ll find details about grouping nodes and states and working with grains and pillars before learning about how to execute a single command on a single node.

After installing the master and minions, you can view all minions and their status with a simple command. After that you can build states and pillars by issuing commands to agents. Because minions connect to the master, bootstrapping a bunch of new machines for minion provisioning is fast and simple.

Newcomers face a steep learning curve with SaltStack. While the fundamental Salt concepts are simple, SaltStack is a massive project that demands time and effort to understand. However, whereas Ansible is easy in the beginning and becomes more complex as the configuration codebase and number of managed servers grow, SaltStack seems complex in the beginning but simplifies life at large scale. SaltStack has many modular layers, but they all use the same communication bus, enabling parallel communication among many servers. Managing all of this remains flexible and straightforward even as the numbers of states and minions grow.

Reporting. While you would describe SaltStack as a feature-rich configuration management tool, the reporting segment needs improvement. The web GUI doesn’t provide the powerful UI/UX features you will find in Puppet or Chef. For instance, the Puppet user interface is intuitive for both newbies and experts, and Puppet provides advanced reporting, dashboards, and alerting capabilities that SaltStack lacks. In particular, Puppet provides a wide variety of metrics on server health and performance, as well as an event inspection feature that lets you visualize changes by nodes, classes, and resources.

To improve the UX and reduce the time for error resolution, SaltStack has made multiple attempts at a web GUI. The new GUI, which leverages the SaltStack Enterprise API, provides a much more responsive and even real-time user experience. You can easily view the status of all jobs and mark them for later reference. You can launch jobs with a single click or predefine jobs and launch them later. An intuitive dashboard allows you to view the entire cluster from a single window. In short the SaltStack GUI is much improved. However, users could still ask for more advanced reporting capabilities.

For external reporting and analysis, SaltStack provides output in multiple formats. You can use outputters to create custom formats as well. You can find the various output modules on the SaltStack website.

Community. SaltStack offers an extensive set of documents that are well written and useful. However, it is not easy to search the documentation. First-time visitors may find it difficult to locate what they need to get started. For instance, the website has few tutorials that contain detailed information.

But the community is welcoming and helpful. Founder Thomas Hatch and company listen to user feedback and criticism, and they don’t mind quickly resolving problems. You can inquire through IRC or the mailing list. The response on IRC is fast, while some mail might take days. Although the Salt community is still a startup community, it is quickly growing into an active and helpful group.

SaltStack is a powerful and flexible configuration management tool that rises to the needs of extremely large deployments. SaltStack uses a high-speed messaging infrastructure to scales to tens of thousands of client devices, while giving you the powerful features you need to maintain a firm grip on the infrastructure. SaltStack has a huge list of built-in modules that can be used out of the box, saving you time and energy.

While SaltStack provides extensive knowledge base support and comprehensive documentation, it is important to understand how the documentation is organized and where you can find the required modules and necessary information. That will take time. However, your investment in understanding the documentation and module list will reward you in the long run. If SaltStack can continue to improve the web UI features, it becomes a comprehensive configuration management tool that can leave the rest behind.

—

Cost: Salt is free and open source under the Apache 2.0 license. SaltStack Enterprise is available via a subscription license and priced by managed node. SaltStack support subscriptions and services engagements are available for both SaltStack Enterprise and Salt open source implementations. 

Platforms: Salt minions and masters are supported on Red Hat, CentOS, Fedora, Debian, Ubuntu, SLES, OpenSUSE, Arch, Raspbian, Oracle Linux, FreeBSD, Solaris, and SmartOS. Only minions are supported on Windows, MacOS, AIX, and Solaris.