Kubernetes 1.9 offers a lot of new features. We waited a long time for some specific features, like working with GPU, Istio support, etc. The decision was made to upgrade our 1.7 clusters to 1.9.
We manage the cluster through kops, so upgrading should be easy. But is it safe? The first answer that came to mind was "yes, it is safe. The Kops community runs these tests.". But what if we will encounter something not tested?
Are we ready to endanger our dev environment for this upgrade? What will 25+ developers do if they don't have their dev workspace?
At first, we thought of running the upgrade on the weekend and if everything fails we would work around the clock to fix it, but that felt very 2005-ish.
What we ended up doing is having an entire pipeline for dev creation:
1. kops that runs the "test" kubernetes env, running current version, 1.7.
2. Terraform that supplies all the external resources (s3, route53, API gateway, etc).
3. Deployment tool (custom made inner tool) that deploys all current micro-services to the k8s cluster.
4. An e2e test to check all is good.
5. kops upgrade to required version, 1.9.
6. Another e2e test on the upgraded env, compare to the e2e from step 4.
7. Did the Comparison go well? We are ready to upgrade the original dev environment!
I know this seems like an overkill, but after running this entire process, we were relaxed about upgrading the dev, staging and production environment.