How to Manage Hundreds of Clusters

Most Kubernetes journeys start with a cluster or two, at which point manually installing monitoring and security applications and handling updates isn’t a problem. But as the Kubernetes footprint expands, the number of clusters can also balloon quickly. This means not only more clusters, but also more developers who provision clusters, and more workloads running on the clusters. 

As more people in the organizations start using Kubernetes, an ad-hoc approach is unlikely to result in the kind of organizational consistency needed not only to ensure that all the correct tools are installed every time but also that clusters can be updated as needed, debugged easily and that there is a procedure in place to deal with failures. Multicluster management tools allow central teams to ensure consistency in an organization’s clusters while still giving development teams the freedom to provision a new cluster whenever necessary. 

Why does multicluster management matter? And how should organizations approach it? Here’s what we’ve learned. 

Security, monitoring and shared services provisioning

In many enterprises, development teams are able to self-serve and create, deploy and delete clusters as needed. The central platform team, on the other hand, needs to make sure that as developers self-serve they both set up the correct policies and configurations as well as install all of the necessary tools and applications on the cluster. Every cluster, for example, is going to need a monitoring application, a security application, a policy engine, and other applications for basic housekeeping. The platform team needs to make sure that these applications are set up automatically whenever a new cluster is provisioned — both to ensure that it happens and to make the cluster creation process easier and faster for developers.  

Updates on the fly

Central platform teams are also responsible for keeping all of the organization’s clusters and shared services up to date. When there’s a new version of Kubernetes or a security patch in Kubernetes or a shared service, these teams need to have a way to update hundreds of clusters in a way that is consistent, fast and error-free. In the vast majority of cases, updates need to be applied across the board, and there is no difference in how each cluster should be treated even if the clusters are running on different infrastructure stacks. Using a multicluster management tool automates the update process and ensures that no clusters are missed. 

On the edge

When companies need to deploy applications at the edge (e.g. store fronts), they will likely have multiple clusters, often in different regions, but they need to provide a consistent experience for the end user, whether the end user is a cashier using a point of sale system or a reader accessing content on a mobile phone. 

There will already be so many variables to consider between the different clusters that complete consistency between the clusters is critical to responding to any failure. It’s also important to have visibility into all the clusters so that the central team can easily spot any problems and take action if necessary. 

Keeping things consistent

Multicluster management is all about ensuring consistency across all of the clusters in the organization. When clusters have a consistent stack, are configured according to organizational governance policies instead of in an ad-hoc manner, have a standard set of monitoring tools, and are as similar as possible in terms of stack and environment, it becomes much easier to respond to incidents and in general to manage the entire lifecycle. 

Debugging in particular becomes much easier when clusters are consistent, because it reduces the number of variables that could be causing a problem with a particular cluster or with an application. 

Complexity

As the Kubernetes footprint grows, so does the system’s complexity. The biggest mistake organizations make at the beginning of the process is underestimating how complex a cloud native application suite is and not putting the tools and processes in place to manage that complexity. This complexity is even greater as organizations think about deploying on the edge and using multiple cloud providers and/or a hybrid cloud approach. 

If there are too many variables that have to be taken into account and too many manual steps, errors are inevitable. Central teams can’t realistically manage hundreds of clusters manually without automation tools. Nor can different developers and development teams be trusted to configure their clusters in exactly the same way — unless there are tools in place to handle those configurations automatically. 

Tools like Nirmata make it possible for small central teams to get cross-cluster visibility, ensure security, control where workloads run and easily manage Day 2 operations like updates in a consistent manner. When you’re dealing with hundreds of clusters, it’s not possible to update each one manually or to check the health of each cluster on a regular basis. With Nirmata, everything related to cluster health can be handled in one interface, so platform teams don’t need to toggle between dashboards or confuse themselves about which clusters have been updated and which ones haven’t. 

Request a demo now. 

source https://nirmata.com/2020/10/29/how-to-manage-hundreds-of-clusters/