What I learned from using Kubernetes

What is Kubernetes?

Kubernetes is a fundamentally a automated resource management platform – principally for computational resources (CPU, RAM, networks, local storage). It realises the ideas behind the ‘cattle not pets‘ approach to IT infrastructure, by defining in software what previously was implicit in infrastructure configuration and provisioning.

In particular, Kubernetes enables continuous integration/continuous delivery (CI/CD) processes which are critical to achieving the business benefits (agility, reliability, security) that underpin devops (Kim et al).

The standard for abstracting computational resources today is the container  – essentially, technologies based on OS virtual isolation primitives first provided by Linux. So Kubernetes (and other resource orchestration tools) focus on orchestrating (i.e., managing the lifecycle of) containers. The de-facto container standard is Docker.

Cloud providers (such as AWS, Azure, GCP and Digital Ocean) specialize in abstracting computational resources via services. However, while all support Docker containers as the standard for runnable images, they each have different APIs and interfaces to engage with the management services they provide. Kubernetes is already a de-facto standard for resource management abstraction that can be implemented both in private data centers as well as on cloud, which makes it very attractive for firms that care about cloud-provider lock-in.

It is worth noting that currently Kubernetes focuses on orchestrating containers, which can be run on either virtual machines or on bare-metal (i.e., OS without hypervisor). However, as VM technology becomes more efficient – and in particular the cost of instantiating VMs decreases – it is possible that Kubernetes will orchestrate VMs, and not (just) containers. This is because containers do not provide a safe multi-tenancy model, and there are many good reasons why firms, even internally, will seek stronger isolation of processes running on the same infrastructure than that offered by containers – especially for software-as-a-service offerings where different clients do need strong separation from each others process instances.

What Kubernetes is not

It is easy to mistake Kubernetes as a solution to be used by development teams: as the Kubernetes website makes clear, it is not a development platform in and of itself, but rather is technology upon which such platforms can be built.

An example may be platforms like Pivotal’s Container Service (PKS), which is migrating from its own proprietary resource management solution to using Kubernetes under the hood (and providing a path for VMware users to Kubernetes in the process). For Java, frameworks like Spring Boot and Spring Cloud provide good abstractions for implementing distributed systems, but these can be implemented on top of Kubernetes without developers needing to be aware.

A key benefit of Kubernetes is that it is language/framework agnostic. The downside is that this means that some language or platform-specific abstractions (tools, APIs, policies, etc) need to be created in order for Kubernetes to be useful to development teams, and to avoid the need for developers to master Kubernetes themselves.

For legacy applications, or applications with no distribution framework, some language-neutral mechanism should be used to allow development teams to describe their applications in terms that can enable deployment and configuration automation, but that does not bind kubernetes-specific dependencies to those applications (for example, Spring Cloud).

For new projects, some investment in both general-purpose and language-specific platform tooling will be needed to maximize the benefits of Kubernetes – whether this is a 3rd party solution or developed in-house is a decision each organization needs to make. It is not a viable option to delegate this decision to individual (pizza-sized) teams, assuming such teams are staffed principally with folks charged with implementing automated business processes and not automated infrastructure processes.

How cool is Kubernetes?

It is very cool – only a little bit less cool than using a cloud provider and the increasing number of infrastructure & PaaS services they offer. Certainly, for applications which are not inherently cloud native, Kubernetes offers a viable path to achieving CI/CD nirvana, and hence offers a potential path to improvements in deployment agility, security and reliability for applications which for various reasons are not going to be migrated to cloud or re-engineered.

It takes a remarkably small amount of time for an engineer to install and configure a Kubernetes cluster on resources they have direct control over (e.g., MiniKube or Docker on a laptop, EKS on AWS, or a bunch of VMs you have root access to in your data center). Creating a dockerfile for an application where dev/test environments are fully specified in code (e.g., Vagrant specifications) is a fairly quick exercise. Deploying to a kubernetes cluster is quick and painless – once you have a docker image repository setup and configured.

Technical challenges arise when it comes to network configuration (e.g., load balancers, NAT translation, routers, firewalls, reverse proxies, API gateways, VPNs/VPCs, etc), automation of cluster infrastructure definition, automation of dockerfile creation, storage volume configuration and management, configuration parameter injection, secrets management, logging and monitoring, namespace management, etc. Advanced networking such as service mesh, policy control, use of multiple network interfaces, low-latency routing to external services, multi-region resilience and recovery are typically not considered a priority during development, but are critical for production environments and must also be automated. All this is doable via Kubernetes mechanisms, but is not for free.

In short, Kubernetes is only the start of the journey to implement zero-touch infrastructure and therefore to make infrastructure provisioning a seamless part of the software development process.

So what should a Kubernetes strategy be?

For many firms, Kubernetes could end up being nothing but a stepping stone to full serverless architectures – more on serverless another time, but in essence it means all resource abstractions for an application are fully encoded in code deployed as part of a serverless application.

For firms that are moving towards a more fully integrated polyglot microservices-type architecture (i.e., with more and more infrastructure and business capabilities delivered as services rather than deployable artifacts or code), and where direct control over network, storage and/or compute infrastructure is seen as a competitive necessity, then Kubernetes seems like a prudent option.

How microservice frameworks evolve with respect to their use of Kubernetes will be critical: in particular, frameworks should ideally be able to happy co-exist with other frameworks in the same Kubernetes cluster. Reducing clusters to single-application usage would be a step backwards – although in the context of many microservices written using the same language framework deployed to the same cluster, perhaps this is acceptable (provided operations teams did not see such microservices as parts of a single monolithic application).

Irrespective of a microservices strategy, deploying applications to containers may be seen as a way to efficiently manage the deployment and configuration of applications across multiple environments. In this regard, there is a convergence of innovation (see PKS above) in the definition, creation and orchestration of virtual machines and containers (as noted above), which may eventually make it more sensible for enterprises which have already made the move to virtualization (via VMWare) to continue along that technology path rather than prematurely taking the Kubernetes/Container path, as legacy applications struggle with the ephemeral nature of containers. In either case, the goal is zero-touch infrastructure via ‘infrastructure-as-code‘, and process automation such as ‘gitops‘, as these are what will ultimately deliver devops business goals.

Summary

So, the key takeaways are:

  • Kubernetes is important for organizations where containerization is part of a broader (microservices/distributed systems/cloud) strategy, and not just deployment/configuration automation/optimization.
  • Organizations should at least learn how to operate Kubernetes clusters at scale, as this function will likely remain siloed.
  • Developing in-house kubernetes engineering capabilities is a strategic business question. For most enterprises, let 3rd parties focus on this (via open-source solutions)
  • For organizations which heavily use VMware VMs, the path to Kubernetes is likely via VMs (which are long-lived) rather than via Containers (which are ephemeral). Commercial VM managers are expensive, but effectively using Kubernetes is not free or cheap either.
  • Organizations should seriously assess the role serverless technologies could play in future technical roadmaps.
What I learned from using Kubernetes

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s