Bringing databases and Kubernetes together

Running databases on Kubernetes is popular. For cloud-native organizations, Kubernetes is the de facto standard approach to running databases. According to Datadog, databases are the most popular workload to deploy in containers, with 45 percent of container-using organizations using this approach. The Data on Kubernetes Community found that production deployments were now common, with the most advanced teams running more than 75 percent of their data workloads in containers.

Kubernetes was not built for stateful workloads originally—the project had to develop multiple new functions like StatefulSets in Kubernetes 1.9 and Operator support for integration with databases later. With that work done over the first 10 years of Kubernetes, you might think that all the hard problems around databases on Kubernetes have been solved. However, that is not the case.

Embracing database as a service with Kubernetes

Today we can run databases in Kubernetes successfully, and match those database workloads alongside the application components that also run in containers. This makes the application development side easier as all the infrastructure is in one place, and can be controlled from one point. While that approach makes the “Day 1” issues around application development easier, it does not deal with many of the “Day 2” issues that still have to be addressed.

Day 2 issues include all the tasks that you need to have running so your application operates effectively over time. That includes looking at resiliency, security, operational management, and business continuity. For developers looking at databases, that means tasks like backup, availability, and failover. Some of these elements are easier in containers. Kubernetes was built to monitor containers in pods and restart images if a problem took place. However, stateful databases require more planning than stateless applications.

Kubernetes Operators can automate some of these processes for you, allowing Kubernetes to work through a database to trigger a cluster to carry out a backup task automatically. But that doesn’t go through the whole process, and it relies on the developer making decisions around how best to follow that process. If you are not an expert in backup or availability, you might be tempted to hand all these concerns over to a third-party provider for them to take care of.

That approach works. Cloud-based database as a service (DBaaS) offerings grew at nearly twice the rate of on-premises deployments according to Gartner — 64% to 36% — as developers went for the easy option. However, this locked them into that particular cloud provider or service. Even when developers might choose an open source database to base their application on, they would still be tied to the provider and their way of doing things. From a mobility perspective, that can represent a serious cost to run a DBaaS rather than doing it yourself.

The future for Kubernetes and database as a service

Automating Kubernetes workloads with Operators can provide the same level of functionality as DBaaS, while still avoiding lock-in to a specific provider. This should fit into how teams want to run internal developer platforms or platform engineering for their developers. However, getting that to work for all databases in a consistent way is its own challenge.

At Percona, we did some of this work around our own project, Everest. But this only supported the databases that we are interested in — namely, MySQL, PostgreSQL and MongoDB. What about other databases that have Operators? How about other systems for managing and observing databases? While the idea of a fully open source database as a service option is great in theory, in practice it needs a community that is willing to get involved and support how that gets built out.

If you really love something, sometimes you have to set it free. Like Woody in Toy Story 3, you just have to say, “So long, partner” with the hope that things go “To Infinity and Beyond” a la Buzz Lightyear. That is what we have done with Everest — or to use its new name, OpenEverest. OpenEverest is now a fully open source project that anyone can get involved in. With the project donated and accepted by the Cloud Native Computing Foundation, this will make running databases on Kubernetes easier for everyone. Over time, OpenEverest will support more databases based on what the community wants to see and where they help with more contributions or support.

For developers, getting Kubernetes and databases together helps them be more productive. But for those running infrastructure or dealing with Day 2 problems around Kubernetes, databases still remain potentially challenging to manage. Dealing with edge cases, automation, and resilience at scale is still a significant hurdle for databases on Kubernetes, yet this approach remains essential if you want to implement platform engineering or internal developer platforms without lock-in. This new open source project is a strong starting point to delivering on that requirement. Making it a community open source software project under a foundation, rather than the preserve of one company, will help this approach to succeed.

—

New Tech Forum provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss emerging enterprise technology in unprecedented depth and breadth. The selection is subjective, based on our pick of the technologies we believe to be important and of greatest interest to InfoWorld readers. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Send all inquiries to doug_dineley@foundryco.com.