Paranoid about upgrading Kubernetes?

How to upgrade Kubernetes - the safe way.

June 17, 2018
Contact Us
Weekly Shorts are topics we discuss in our weekly remote meeting related to recent work we have done with our customers
Paranoid about upgrading Kubernetes?

Kubernetes 1.9 offers a lot of new features. We waited a long time for some specific features, like working with GPU, Istio support, etc. The decision was made to upgrade our 1.7 clusters to 1.9.

We manage the cluster through kops, so upgrading should be easy. But is it safe? The first answer that came to mind was "yes, it is safe. The Kops community runs these tests.". But what if we will encounter something not tested?

Are we ready to endanger our dev environment for this upgrade? What will 25+ developers do if they don't have their dev workspace?

At first, we thought of running the upgrade on the weekend and if everything fails we would work around the clock to fix it, but that felt very 2005-ish.

What we ended up doing is having an entire pipeline for dev creation:

1. kops that runs the "test" kubernetes env, running current version, 1.7.

2. Terraform that supplies all the external resources (s3, route53, API gateway, etc).

3. Deployment tool (custom made inner tool) that deploys all current micro-services to the k8s cluster.

4. An e2e test to check all is good.

5. kops upgrade to required version, 1.9.

6. Another e2e test on the upgraded env, compare to the e2e from step 4.

7. Did the Comparison go well? We are ready to upgrade the original dev environment!

I know this seems like an overkill, but after running this entire process, we were relaxed about upgrading the dev, staging and production environment.

Paranoid about upgrading Kubernetes?
Yair Leshem
Operations Engineer
After a successful online marketing venture, Yair decided to focus on what he loves most - handling technical and analytical challenges as a software operations engineer. Yair has 7 years experience with handling all technical challenges, starting with infrastructure, DB, resolve unique challenges and code. He is great at solving problems and has an eye for designing, a sweet tooth and lots of love and compassion for others. When not on the job, he takes joy in snowboarding and organizing his theme camp in the Israeli Burning Man (Midburn). He is passionate about learning how complex systems work and figuring out how to make them work better.