Review: Chef 12 fires up devops

Two of the most important tasks in a datacenter are server provisioning and configuration management. At one time, administrators spent a significant amount of time physically deploying servers and network infrastructure, followed by even more minutes and hours manually configuring hundreds or even thousands of nodes. Then they spent an additional chunk of the day troubleshooting and fixing the errors they introduced by configuring all of these systems with their fat fingers.

Today, thanks to automation software such as Chef, administrators are able to provision servers, deploy software, and set a nearly endless number of configuration items in a blink, without lifting a finger. And they do so in a way that is more scalable, more repeatable, and more immune to human error.

Although widely considered a configuration management product, Chef seems to think bigger than comparable tools on the market. Chef was designed with a devops culture in mind, and in addition to automating the provisioning of server infrastructure, Chef can automate the provisioning of runtime environments, applications, and containers. This is a comprehensive tool that provides not only configuration management but end-to-end IT automation.

At its most basic (the free open source version), Chef combines a development kit for creating “cookbooks” (think “recipes” for system configurations), a client that runs on managed nodes, and the Chef server, a central repository for cookbooks and the means of applying the recipes in cookbooks to nodes. Chef’s open source community marketplace provides a vast repository of cookbooks that can speed up delivery time. Advanced features — including high availability, a web-based management console, and reporting and analytics — are free for up to 25 nodes.

After that you have two supported commercial options to choose from: Hosted Chef and Chef Automate, which cost $72 and $137 per node, respectively. Chef Automate comes complete with team workflow capabilities that extend the automation across the entire application lifecycle, from development and testing through staging and production.

A Chef overview

Chef comes with an array of features that provide you with fine-grained control over your infrastructure. Let’s take a closer look at each component.

Chef DK. Chef DK is a downloadable software development toolkit that contains all of the best-of-breed tools developed by the Chef community. It enables you to quickly set up and upgrade the Chef Client on workstations and get control of the cookbook dependencies. Chef DK includes built-in testing tools like RuboCop and Foodcritic, the unit testing framework known as ChefSpec, and Test Kitchen, an integration tool for testing coded infrastructure. Chef DK offers a streamlined workflow for new users along with plenty of flexibility for advanced users.

Test Kitchen. Test Kitchen is a powerful tool that lets you run configured code on various platforms in isolation. The Test Kitchen is not only an isolated environment, but a pristine one — it ensures there are no traces of a prior state. A .kitchen.yml file, which contains the static declarative configuration, is typically stored in the root of the Chef repository. With a driver plugin architecture, Test Kitchen allows you to test cookbook code across various cloud providers and virtualization technologies, including Microsoft Azure, Amazon EC2, Apache CloudStack, Blue Box, DigitalOcean, Vagrant, Rackspace, Docker, and Linux Containers. Test Kitchen supports testing frameworks like Inspec, and it’s compatible with dependency management tools such as Berkshelf. To date, many Chef community cookbooks have already included support for Test Kitchen.

Chef Supermarket. Chef’s open source community site lets users browse, download, and share cookbooks. Initially, Supermarket existed only as a public community, allowing users to upload and download cookbooks from anywhere. Today, Supermarket is offered in two forms: public and private, with the private Supermarket deployed on-premises and accessed via users’ Chef Server credentials. In both the public and private Supermarkets, you can quickly search the cookbook repository via web GUI or command line. In addition to Linux, Chef Supermarket offers cookbooks that provide plenty of Windows-specific resources.

Chef Automation. Chef Automation is a premium feature of Chef that facilitates continuous delivery of infrastructure and runtime environments, including applications and containers, and provides visibility into those workflows. To meet changing business needs, big companies have to make an ongoing series of changes to networks, applications, platforms, and software. Chef Automation provides this flexibility with a unified process that automates changes to applications, infrastructure, and runtime environments at the same time.

The shared pipeline provided by Chef Automation allows you to safely take code from the developers through the build, test, and production stages, while Chef’s collaborative platform brings visibility to each step of the process. IT architects, engineers, developers, and security and compliance staff receive prompt alerts on proposed changes. Everyone can see the flow of changes through the pipeline.

Chef Analytics. Chef Analytics, another premium feature, provides precise and real-time visibility into everything that’s happening in the managed estate. From the changes made to who made them, when, and how, Chef Analytics enables you to carefully monitor the modifications happening in the infrastructure. For example, compliance against internal controls can be easily verified using Analytics, and changes immediately communicated to users in real time through email and webhooks. Also, you can integrate Chef Analytics with HipChat or Slack to send real-time notifications to corresponding chat rooms.

Version control. IT environments are quick to embrace changing technologies to meet growing business demands. For instance, version control is an essential requirement when multiple people work together on a single software development project. This is true even when the software being developed is infrastructure code. Chef recognizes this, and versioning is a core principle that’s the basis of Chef today.

Chef uses Git for version control of cookbooks. That means each cookbook is kept cohesive because all changes are merged into a single updated version, and you can view the older versions or a graphical display of the entire project at any time. This minimizes data storage and reduces the complexity of manually storing each version with a different name. Also, you can easily see what happened with the cookbook, retrieve old cookbooks, and automatically back up cookbooks. A new developer can simply download the current version of the cookbook to the local environment and keep track of it.

Windows support. With the increasing use of Chef in enterprise environments, adding enhanced support for Windows operating systems has become a priority. Chef has answered the call, allowing you to automate several key tasks that would be cumbersome and time-consuming without this enhanced support. Let’s take a quick look:

• Installation of Windows features: The installation of Windows features can be achieved via Chef, allowing you to enable features such as Internet Information Services (IIS) or the .Net Framework for application servers. Performing this task with the UI is complicated and time-consuming, so having this built into the platform is a big plus.
• Windows Installers: The windows_package resource is a useful feature that’s built into Chef. It enables you to manage and automate the installation of most installer types, such as InstallShield and the Windows Installer Package (MSI). If the installer needs to be upgraded, this will happen automatically.
• IIS Cookbook: For those deploying an application into IIS, a popular choice on the Chef Supermarket is the IIS cookbook. This allows Chef to create and manage IIS sites, apps, application pools, and related configurations.
• Windows cookbook: The Windows cookbook provides valuable resources to the Windows ecosystem such as the ability to create scheduled tasks, manage digital certificates, and even install fonts. It also offers tools to manage the system path.

Evaluating Chef

Capability. Chef uses the server-client model, which means the Chef server works with the Chef client to apply configurations to managed nodes. To streamline this process, the Chef server provides the latest set of recipes and instructions for each node based on its role. For instance, a load balancer receives one set of instructions while the web server receives another set of instructions. Each time the Chef client runs, it queries the server for the latest set of recipes, and the Chef client applies these directions in the same order every time so that consistent and repeatable results are seen.

Chef uses the pull method whereby nodes periodically contact the server to receive their instructions. (It should be noted that push functionality is not well supported.) By default, the Chef client pulls the latest instructions from the Chef server every 30 minutes.

If there are any configuration drifts, the Chef client immediately changes the state of the node to the desired state, but these changes are made when configurations are found to diverge from the desired state. In most cases, nothing happens, which means you can run scripts anytime without any issues. When you instruct the client to install a particular service on a node, it only performs this task if the service is not already installed on the node or if it is outdated. By using a convergent mode of configuration, Chef moves the network as a whole to the desired.

In addition to ensuring desired state, Chef allows you to search and report on node configurations across the network. The Ohai tool that is part of the Chef client installation detects the attributes of the node at the start of each Chef client run. These characteristics include platform details, network usage, memory usage, kernel data, CPU data, host names, fully qualified domain names, and other configuration details.

Scalability. Chef is designed to provide high-level configuration management for large enterprises, and it can easily scale to thousands of nodes. With Chef, the entire environment is easily searchable, and these searches are available to the recipes — making the system highly scalable by incorporating various load balancing features. As new web servers are added to the network, they are automatically noticed by the load balancers and requests are instantly routed to them. This automation is key to scaling well.

Designed as a distributed system, Chef allows you to run components on separate nodes to distribute the load. Big companies like Facebook have successfully managed thousands of servers, configurations, and administrative access policies in dynamic computing environments by deploying Chef.

Starting with Chef 11, the product was completely rewritten using the Erlang programming language. The core server API was also renamed Erchef. The CouchDB database was replaced with a PostgresSQL database, and this combination provides greater scalability. While memory consumption was reduced by a factor of 10, only a fraction of CPU resources is used. Due to these changes, a single Chef 11 or Chef 12 server can easily manage 10,000 clients — a four-fold increase over what the Chef 10 server could handle. As the scale increases, the system expands into a tiered front-end and back-end architecture.

The scaling of the Chef infrastructure is measured in terms of chef-client runs per minute (CCRs/min). For instance, if 200 nodes are set to check in with the server every 30 minutes, the CCRs/min is 6.66. The Chef server can easily handle up to 333 CCRs/min without a tiered topology or high-availability setup. This is equivalent to 10,000 nodes. For more details about scaling, see Chef’s documentation.

Operating system and application coverage. Chef supports multiple platforms and integrates well with cloud services such as Microsoft Azure. Initially, Chef was designed to support Linux. However, support for Windows has increased dramatically in recent years, and the latest releases support PowerShell DSC, IIS, and SQL Server. Support for 64-bit Windows has also been added recently, which means Chef can be used to manage the upcoming Windows Nano Server.

Chef clients are generally installed on Windows machines using the MSI package. Today, a number of resources specific to Windows come with Chef out-of-the-box. For example:

• batch executes batch files using the CMD interpreter on a Windows system
• chocolatey_package installs Chocolatey packages on a Windows system
• dsc_script/dsc_resource applies the Desired State Configuration (DSC) feature to a Windows system
• env manages environment variables on a Windows system
• powershell_script executes arbitrary PowerShell code on a system
• registry_key accesses and writes to the Windows registry
• windows_package installs, upgrades, and manages Windows installers and other packages on the system
• windows_service controls the state of Windows services on the system

Further, Windows admins can draw on a growing number of knife plugins and application-specific cookbooks for Windows environments.

Chef has excellent integration with cloud services. The knife tool provides a plugin architecture that enables you to integrate Chef with various cloud environments. With Chef 12, the Chef client now supports AIX 6.1 and AIX 7.1, which means you can turn AIX infrastructure into code to efficiently manage it with Chef.

Ease of use. Chef is more program-oriented than admin-centric Puppet, and it uses a subset of Ruby that isn’t familiar to most administrators; there’s a bit of a learning curve for those who aren’t familiar with Ruby and procedural coding. The model is client-server, which means agents periodically report to a central Chef server. Additionally, you should be familiar with Git for versioning cookbooks.

With Chef’s program-centric approach, you get more control over your configurations, and the knife command-line tool makes installations simple and easy. The ease with which you can automate Windows program installations and management makes Chef a favorite for Windows admins.

Reporting. Chef provides good visibility into what’s running on the network, and the Ohai tool that is distributed with the Chef client provides detailed configuration data to the Chef server (which is then indexed). This allows you to search any configuration in the network and quickly identify anomalies. It’s easy to add your own searchable attributes, and by offering a network-wide configuration search function, Chef provides an integrated and holistic view of the infrastructure.

The web UI is not as rich in features as Puppet Enterprise, and Chef’s reporting capabilities are limited. Still, you can control the inventory and organize nodes through Chef’s web UI, and Chef Analytics offers a fully automated audit and compliance engine where you can instantly see what changes are made, by whom, and when. Combined with the compliance controls on the client, Chef can help you ensure compliance with HIPAA, PCI DSS, and any future requirements.

Community. Like Puppet, Chef enjoys a large and mature community with members who make substantial contributions to the platform. With a rich collection of open source modules and configuration recipes available on Chef Supermarket, you can find a solution for almost every task on any popular platform: Linux, Windows, FreeBSD, OS X, even AIX. Overall, Chef is an established product with a large user base and solid documentation. For those who are audio-visual learners, training in the form of self-paced videos is also available.

The need for speed, accelerated delivery, and faster time to ROI has brought a notable transformation in IT procedures. It has also brought a notable transformation in Chef. Chef is no longer “just” a configuration management tool but an automation framework enabling you to automate everything in the environment. Chef Automation is a powerful workflow automation and software delivery pipeline that offers significant benefits to organizations embracing infrastructure as code and the devops way.

Features such as full stack automation, continuous delivery, and audit compliance have placed Chef in a league of its own. Recently, Chef has been reconstructed to accommodate the operations of large-scale enterprises like Facebook. By effectively handling more than 150,000 servers at Facebook, Chef has proved it can successfully manage even the largest online operations.