Terraform AWS - Dynamic Subnets

September 23, 2019
Contact Us
Weekly Shorts are topics we discuss in our weekly remote meeting related to recent work we have done with our customers
Terraform AWS - Dynamic Subnets

<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">

<head>

 <meta charset="utf-8" />

 <meta name="generator" content="pandoc" />

 <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />

 <title>Terraform AWS - Dynamic Subnets</title>

 <style>

     code{white-space: pre-wrap;}

     span.smallcaps{font-variant: small-caps;}

     span.underline{text-decoration: underline;}

     div.column{display: inline-block; vertical-align: top; width: 50%;}

 </style>

 <style>

code.sourceCode > span { display: inline-block; line-height: 1.25; }

code.sourceCode > span { color: inherit; text-decoration: inherit; }

code.sourceCode > span:empty { height: 1.2em; }

.sourceCode { overflow: visible; }

code.sourceCode { white-space: pre; position: relative; }

div.sourceCode { margin: 1em 0; }

pre.sourceCode { margin: 0; }

@media screen {

div.sourceCode { overflow: auto; }

}

@media print {

code.sourceCode { white-space: pre-wrap; }

code.sourceCode > span { text-indent: -5em; padding-left: 5em; }

}

pre.numberSource code

 { counter-reset: source-line 0; }

pre.numberSource code > span

 { position: relative; left: -4em; counter-increment: source-line; }

pre.numberSource code > span > a:first-child::before

 { content: counter(source-line);

   position: relative; left: -1em; text-align: right; vertical-align: baseline;

   border: none; display: inline-block;

   -webkit-touch-callout: none; -webkit-user-select: none;

   -khtml-user-select: none; -moz-user-select: none;

   -ms-user-select: none; user-select: none;

   padding: 0 4px; width: 4em;

   background-color: #232629;

   color: #7a7c7d;

 }

pre.numberSource { margin-left: 3em; border-left: 1px solid #7a7c7d;  padding-left: 4px; }

div.sourceCode

 { color: #cfcfc2; background-color: #232629; }

@media screen {

code.sourceCode > span > a:first-child::before { text-decoration: underline; }

}

code span. { color: #cfcfc2; } /* Normal */

code span.al { color: #95da4c; } /* Alert */

code span.an { color: #3f8058; } /* Annotation */

code span.at { color: #2980b9; } /* Attribute */

code span.bn { color: #f67400; } /* BaseN */

code span.bu { color: #7f8c8d; } /* BuiltIn */

code span.cf { color: #fdbc4b; } /* ControlFlow */

code span.ch { color: #3daee9; } /* Char */

code span.cn { color: #27aeae; } /* Constant */

code span.co { color: #7a7c7d; } /* Comment */

code span.cv { color: #7f8c8d; } /* CommentVar */

code span.do { color: #a43340; } /* Documentation */

code span.dt { color: #2980b9; } /* DataType */

code span.dv { color: #f67400; } /* DecVal */

code span.er { color: #da4453; } /* Error */

code span.ex { color: #0099ff; } /* Extension */

code span.fl { color: #f67400; } /* Float */

code span.fu { color: #8e44ad; } /* Function */

code span.im { color: #27ae60; } /* Import */

code span.in { color: #c45b00; } /* Information */

code span.kw { color: #cfcfc2; } /* Keyword */

code span.op { color: #cfcfc2; } /* Operator */

code span.ot { color: #27ae60; } /* Other */

code span.pp { color: #27ae60; } /* Preprocessor */

code span.re { color: #2980b9; } /* RegionMarker */

code span.sc { color: #3daee9; } /* SpecialChar */

code span.ss { color: #da4453; } /* SpecialString */

code span.st { color: #f44f4f; } /* String */

code span.va { color: #27aeae; } /* Variable */

code span.vs { color: #da4453; } /* VerbatimString */

code span.wa { color: #da4453; } /* Warning */

 </style>

 <!--[if lt IE 9]>

   <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>

 <![endif]-->

</head>

<body>

<header id="title-block-header">

<h1 class="title">Terraform AWS - Dynamic Subnets</h1>

</header>

<p><strong>Update</strong> - <a href="https://github.com/hashicorp/terraform/blob/v0.12.11/CHANGELOG.md#01211-october-17-2019">17-Oct changelog</a>: Terraform released a new function named <a href="https://www.terraform.io/docs/configuration/functions/cidrsubnets.html">cidrsubnets</a>, this function creates a list of cidr-subnets. This function is great, and I recommend using it. Even though this function shortens some parts of this tutorial, you should still read it if you want to learn how to use functions in Terraform.</p>

<h1 id="objectives">Objectives</h1>

<ol type="1">

<li>Create sets of subnets dynamically</li>

<li>Learn advanced concepts in Terraform

<ul>

<li>map variable and lookup function</li>

<li>for loop and conditional for loop</li>

<li>index function in a for loop</li>

</ul></li>

</ol>

<h1 id="knowledge-and-assumptions">Knowledge and assumptions</h1>

<ol type="1">

<li>You are familiar with subnetting (a.b.c.d/xx) - a great online tool for calculating subnets - <a href="https://cidr.xyz">cidr.xyz</a></li>

<li>You are using Terraform v0.12+</li>

<li>You have previous experience with:

<ul>

<li><a href="https://www.terraform.io/docs/configuration/variables.html">variables</a></li>

<li><a href="https://www.terraform.io/docs/configuration/locals.html">local values</a></li>

<li><a href="https://www.terraform.io/docs/configuration/modules.html">modules</a></li>

</ul></li>

<li>You know what’s a <a href="https://docs.python.org/3/tutorial/controlflow.html#for-statements">for loop</a></li>

<li>I’ll be using the module <a href="https://github.com/terraform-aws-modules/terraform-aws-vpc">terraform-aws-modules/vpc/aws</a> for the VPC. This module requires lists of subnets (private, database, public), which is exactly what we’re going to create</li>

</ol>

<h1 id="issue-1-subnets-per-environment">Issue #1: Subnets per environment</h1>

<p>Hardcoding subnets per environment can be a significant overhead. By doing so, you might end up writing a lot of lines with hardcoded network prefixes. Moreover, it takes off the flexibility that infrastructure as code (IaC) has to offer.</p>

<h2 id="set-cidr_ab-per-environment"><strong>Set</strong> cidr_ab per environment</h2>

<p>cidr_ab to the rescue! Set the start of the VPC CIDR as a map variable, and map each cidr_ab value to the appropriate environment. If you’re not using all four, remove it from the mapping.</p>

<p>A subnet pattern (prefix) is: <strong>a.b</strong>.c.d/xx , so this will cover the <strong>a.b</strong> part of all subnets per environment. For example:</p>

<div class="sourceCode" id="cb1"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb1-1"><a href="#cb1-1"></a>variable <span class="st">&quot;cidr_ab&quot;</span> {</span>

<span id="cb1-2"><a href="#cb1-2"></a>    type = map</span>

<span id="cb1-3"><a href="#cb1-3"></a>    default = {</span>

<span id="cb1-4"><a href="#cb1-4"></a>        development     = <span class="st">&quot;172.22&quot;</span></span>

<span id="cb1-5"><a href="#cb1-5"></a>        qa              = <span class="st">&quot;172.24&quot;</span></span>

<span id="cb1-6"><a href="#cb1-6"></a>        staging         = <span class="st">&quot;172.26&quot;</span></span>

<span id="cb1-7"><a href="#cb1-7"></a>        production      = <span class="st">&quot;172.28&quot;</span></span>

<span id="cb1-8"><a href="#cb1-8"></a>    }</span>

<span id="cb1-9"><a href="#cb1-9"></a>}</span></code></pre></div>

<h2 id="get-cidr_ab-per-environment"><strong>Get</strong> cidr_ab per environment</h2>

<p>Maps and lookup functions provide great functionality, and in our case, it makes it easy to get the cidr_ab per environment.</p>

<p>Here’s the syntax of the <a href="https://www.terraform.io/docs/configuration/functions/lookup.html">lookup function</a>: <code>lookup(map, key, default)</code></p>

<p>To create a local list <code>private_subnets</code> with the relevant cidr_ab per environment, use the following:</p>

<p><em>Reminder</em>: To concatenate an expression and text: <code>${expression} my text</code></p>

<div class="sourceCode" id="cb2"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb2-1"><a href="#cb2-1"></a>locals {</span>

<span id="cb2-2"><a href="#cb2-2"></a>    private_subnets = [</span>

<span id="cb2-3"><a href="#cb2-3"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.1.0/24&quot;</span>,</span>

<span id="cb2-4"><a href="#cb2-4"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.2.0/24&quot;</span>,</span>

<span id="cb2-5"><a href="#cb2-5"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.3.0/24&quot;</span></span>

<span id="cb2-6"><a href="#cb2-6"></a>    ]</span>

<span id="cb2-7"><a href="#cb2-7"></a>}</span></code></pre></div>

<p>Keep in mind that we need to do that for database and public subnets aswell.</p>

<h2 id="issue-1-full-solution">Issue #1: Full Solution</h2>

<p>Assuming we want to create the following subnets: private, database, and public. Here’s how our <code>variables.tf</code> and <code>vpc.tf</code> files should look like:</p>

<p><code>variables.tf</code></p>

<div class="sourceCode" id="cb3"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb3-1"><a href="#cb3-1"></a>variable <span class="st">&quot;cidr_ab&quot;</span> {</span>

<span id="cb3-2"><a href="#cb3-2"></a>    type = map</span>

<span id="cb3-3"><a href="#cb3-3"></a>    default = {</span>

<span id="cb3-4"><a href="#cb3-4"></a>        development     = <span class="st">&quot;172.22&quot;</span></span>

<span id="cb3-5"><a href="#cb3-5"></a>        qa              = <span class="st">&quot;172.24&quot;</span></span>

<span id="cb3-6"><a href="#cb3-6"></a>        staging         = <span class="st">&quot;172.26&quot;</span></span>

<span id="cb3-7"><a href="#cb3-7"></a>        production      = <span class="st">&quot;172.28&quot;</span></span>

<span id="cb3-8"><a href="#cb3-8"></a>    }</span>

<span id="cb3-9"><a href="#cb3-9"></a>}</span>

<span id="cb3-10"><a href="#cb3-10"></a></span>

<span id="cb3-11"><a href="#cb3-11"></a>locals {</span>

<span id="cb3-12"><a href="#cb3-12"></a>    private_subnets         = [</span>

<span id="cb3-13"><a href="#cb3-13"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.1.0/24&quot;</span>,</span>

<span id="cb3-14"><a href="#cb3-14"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.2.0/24&quot;</span>,</span>

<span id="cb3-15"><a href="#cb3-15"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.3.0/24&quot;</span></span>

<span id="cb3-16"><a href="#cb3-16"></a>    ]</span>

<span id="cb3-17"><a href="#cb3-17"></a></span>

<span id="cb3-18"><a href="#cb3-18"></a>    database_subnets        = [</span>

<span id="cb3-19"><a href="#cb3-19"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.11.0/24&quot;</span>,</span>

<span id="cb3-20"><a href="#cb3-20"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.12.0/24&quot;</span>,</span>

<span id="cb3-21"><a href="#cb3-21"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.13.0/24&quot;</span></span>

<span id="cb3-22"><a href="#cb3-22"></a>    ]</span>

<span id="cb3-23"><a href="#cb3-23"></a></span>

<span id="cb3-24"><a href="#cb3-24"></a>    public_subnets          = [</span>

<span id="cb3-25"><a href="#cb3-25"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.64.0/24&quot;</span>,</span>

<span id="cb3-26"><a href="#cb3-26"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.65.0/24&quot;</span>,</span>

<span id="cb3-27"><a href="#cb3-27"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.66.0/24&quot;</span></span>

<span id="cb3-28"><a href="#cb3-28"></a>    ]</span>

<span id="cb3-29"><a href="#cb3-29"></a>}</span>

<span id="cb3-30"><a href="#cb3-30"></a></span>

<span id="cb3-31"><a href="#cb3-31"></a>variable <span class="st">&quot;environment&quot;</span> {</span>

<span id="cb3-32"><a href="#cb3-32"></a>    type = string</span>

<span id="cb3-33"><a href="#cb3-33"></a>    description = <span class="st">&quot;Options: development, qa, staging, production&quot;</span></span>

<span id="cb3-34"><a href="#cb3-34"></a>}</span></code></pre></div>

<p><code>vpc.tf</code></p>

<div class="sourceCode" id="cb4"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb4-1"><a href="#cb4-1"></a><span class="kw">module</span> <span class="st">&quot;vpc&quot;</span> {</span>

<span id="cb4-2"><a href="#cb4-2"></a>    source = <span class="st">&quot;terraform-aws-modules/vpc/aws&quot;</span></span>

<span id="cb4-3"><a href="#cb4-3"></a>    version = <span class="st">&quot;~&gt;2.0&quot;</span></span>

<span id="cb4-4"><a href="#cb4-4"></a>    name                 = <span class="st">&quot;my-vpc&quot;</span></span>

<span id="cb4-5"><a href="#cb4-5"></a>    cidr                 = <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.0.0/16&quot;</span></span>

<span id="cb4-6"><a href="#cb4-6"></a>    private_subnets      = local.private_subnets</span>

<span id="cb4-7"><a href="#cb4-7"></a>    database_subnets     = local.database_subnets</span>

<span id="cb4-8"><a href="#cb4-8"></a>    public_subnets       = local.public_subnets</span>

<span id="cb4-9"><a href="#cb4-9"></a></span>

<span id="cb4-10"><a href="#cb4-10"></a>    azs                  = [<span class="st">&quot;us-west-2a&quot;</span>, <span class="st">&quot;us-west-2b&quot;</span>, <span class="st">&quot;us-west-2c&quot;</span>]</span>

<span id="cb4-11"><a href="#cb4-11"></a></span>

<span id="cb4-12"><a href="#cb4-12"></a>    <span class="co"># omitted arguments for brevity</span></span>

<span id="cb4-13"><a href="#cb4-13"></a></span>

<span id="cb4-14"><a href="#cb4-14"></a>}</span></code></pre></div>

<h1 id="issue-2-subnets-per-availability-zone">Issue #2: Subnets per Availability Zone</h1>

<p>In issue #1, we solved the subnets per environment, but the Availability Zones (azs) were hardcoded.</p>

<p>One might think; “Let’s simply create a map variable for the region, and then use the region’s name followed by ‘a’, ‘b’, ‘c’”. For example:</p>

<div class="sourceCode" id="cb5"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb5-1"><a href="#cb5-1"></a>variable <span class="st">&quot;region&quot;</span> {</span>

<span id="cb5-2"><a href="#cb5-2"></a>    type = map(string)</span>

<span id="cb5-3"><a href="#cb5-3"></a>    default = {</span>

<span id="cb5-4"><a href="#cb5-4"></a>        <span class="st">&quot;development&quot;</span> = <span class="st">&quot;us-west-2&quot;</span></span>

<span id="cb5-5"><a href="#cb5-5"></a>        <span class="st">&quot;qa&quot;</span>          = <span class="st">&quot;us-east-2&quot;</span></span>

<span id="cb5-6"><a href="#cb5-6"></a>        <span class="st">&quot;staging&quot;</span>     = <span class="st">&quot;us-east-1&quot;</span></span>

<span id="cb5-7"><a href="#cb5-7"></a>        <span class="st">&quot;production&quot;</span>  = <span class="st">&quot;ca-central-1&quot;</span></span>

<span id="cb5-8"><a href="#cb5-8"></a>    }</span>

<span id="cb5-9"><a href="#cb5-9"></a>}</span>

<span id="cb5-10"><a href="#cb5-10"></a></span>

<span id="cb5-11"><a href="#cb5-11"></a><span class="kw">module</span> <span class="st">&quot;vpc&quot;</span> {</span>

<span id="cb5-12"><a href="#cb5-12"></a></span>

<span id="cb5-13"><a href="#cb5-13"></a>    <span class="co"># omitted arguments for brevity</span></span>

<span id="cb5-14"><a href="#cb5-14"></a></span>

<span id="cb5-15"><a href="#cb5-15"></a>    azs = [</span>

<span id="cb5-16"><a href="#cb5-16"></a>        <span class="st">&quot;${lookup(var.region, var.environment)}a&quot;</span>,</span>

<span id="cb5-17"><a href="#cb5-17"></a>        <span class="st">&quot;${lookup(var.region, var.environment)}b&quot;</span>,</span>

<span id="cb5-18"><a href="#cb5-18"></a>        <span class="st">&quot;${lookup(var.region, var.environment)}c&quot;</span>,</span>

<span id="cb5-19"><a href="#cb5-19"></a>    ]</span>

<span id="cb5-20"><a href="#cb5-20"></a>}</span></code></pre></div>

<p>If all the regions that you’re using have the same amount of availability zones - then the above solution is perfect. Unfortunately, that’s not true when it comes to <strong>ca-central-1</strong> (Canada Central) which has only two availability zones. Another example would be <strong>us-east-1</strong> (Virginia) which has six availability zones.</p>

<p><em>Note</em>: Use <a href="https://aws.amazon.com/about-aws/global-infrastructure/">AWS Global Infrastructure</a> to find out the number of availability zones per region. And <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html">Regions and Availability Zones</a> to figure out the region’s code (e.g., eu-west-1)</p>

<h2 id="data-aws_availability_zones">data aws_availability_zones</h2>

<p>We can get the availability zones of the region that is currently taking place, by using <a href="https://www.terraform.io/docs/providers/aws/d/availability_zones.html"><code>data "availability_zones"</code></a>.</p>

<p>Here’s is how we do it:</p>

<div class="sourceCode" id="cb6"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb6-1"><a href="#cb6-1"></a>data <span class="st">&quot;aws_availability_zones&quot;</span> <span class="st">&quot;available&quot;</span> {</span>

<span id="cb6-2"><a href="#cb6-2"></a>    state = <span class="st">&quot;available&quot;</span></span>

<span id="cb6-3"><a href="#cb6-3"></a>}</span></code></pre></div>

<p>Some explanations regarding the code above:</p>

<ol type="1">

<li><code>data</code> - get existing data/resources available in your account</li>

<li><code>aws_availability_zones</code> - gets the list of availability zones in the current region</li>

<li><code>available</code> - a name for that data, it’s important to pick a name that reflects the meaning of the data</li>

<li><code>state = "available"</code> - filters out availability zones that currently experience outages</li>

</ol>

<p>Let’s save the data in a local value. You might be wondering, “Why are we using local values? We also did it with the subnets, why oh why?”</p>

<p>Rule of thumb - if there’s any chance you’re going to manipulate a variable/value, use a local value. This trick provides granularity to manage <code>local.var_name</code>. It is best to manage variables behind the scenes without touching the infrastructure’s code (vpc.tf, rds.tf, s3.tf files). Worst case scenario - we’ve used a local value even though we haven’t manipulated it.</p>

<p>Saving the availability zones <a href="https://www.terraform.io/docs/providers/aws/d/availability_zones.html#attributes-reference">names</a> list in a local value:</p>

<div class="sourceCode" id="cb7"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb7-1"><a href="#cb7-1"></a>locals {</span>

<span id="cb7-2"><a href="#cb7-2"></a>    availability_zones = data.aws_availability_zones.available.names</span>

<span id="cb7-3"><a href="#cb7-3"></a>}</span></code></pre></div>

<h1 id="issue-2-full-solution">Issue #2: Full solution</h1>

<p><code>variables.tf</code></p>

<div class="sourceCode" id="cb8"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb8-1"><a href="#cb8-1"></a>variable <span class="st">&quot;cidr_ab&quot;</span> {</span>

<span id="cb8-2"><a href="#cb8-2"></a>    type = map</span>

<span id="cb8-3"><a href="#cb8-3"></a>    default = {</span>

<span id="cb8-4"><a href="#cb8-4"></a>        development     = <span class="st">&quot;172.22&quot;</span></span>

<span id="cb8-5"><a href="#cb8-5"></a>        qa              = <span class="st">&quot;172.24&quot;</span></span>

<span id="cb8-6"><a href="#cb8-6"></a>        staging         = <span class="st">&quot;172.26&quot;</span></span>

<span id="cb8-7"><a href="#cb8-7"></a>        production      = <span class="st">&quot;172.28&quot;</span></span>

<span id="cb8-8"><a href="#cb8-8"></a>    }</span>

<span id="cb8-9"><a href="#cb8-9"></a>}</span>

<span id="cb8-10"><a href="#cb8-10"></a></span>

<span id="cb8-11"><a href="#cb8-11"></a>locals {</span>

<span id="cb8-12"><a href="#cb8-12"></a>    private_subnets         = [</span>

<span id="cb8-13"><a href="#cb8-13"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.1.0/24&quot;</span>,</span>

<span id="cb8-14"><a href="#cb8-14"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.2.0/24&quot;</span>,</span>

<span id="cb8-15"><a href="#cb8-15"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.3.0/24&quot;</span></span>

<span id="cb8-16"><a href="#cb8-16"></a>    ]</span>

<span id="cb8-17"><a href="#cb8-17"></a></span>

<span id="cb8-18"><a href="#cb8-18"></a>    database_subnets        = [</span>

<span id="cb8-19"><a href="#cb8-19"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.11.0/24&quot;</span>,</span>

<span id="cb8-20"><a href="#cb8-20"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.12.0/24&quot;</span>,</span>

<span id="cb8-21"><a href="#cb8-21"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.13.0/24&quot;</span></span>

<span id="cb8-22"><a href="#cb8-22"></a>    ]</span>

<span id="cb8-23"><a href="#cb8-23"></a></span>

<span id="cb8-24"><a href="#cb8-24"></a>    public_subnets          = [</span>

<span id="cb8-25"><a href="#cb8-25"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.64.0/24&quot;</span>,</span>

<span id="cb8-26"><a href="#cb8-26"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.65.0/24&quot;</span>,</span>

<span id="cb8-27"><a href="#cb8-27"></a>        <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.66.0/24&quot;</span></span>

<span id="cb8-28"><a href="#cb8-28"></a>    ]</span>

<span id="cb8-29"><a href="#cb8-29"></a>}</span>

<span id="cb8-30"><a href="#cb8-30"></a></span>

<span id="cb8-31"><a href="#cb8-31"></a>data <span class="st">&quot;aws_availability_zones&quot;</span> <span class="st">&quot;available&quot;</span> {</span>

<span id="cb8-32"><a href="#cb8-32"></a>    state = <span class="st">&quot;available&quot;</span></span>

<span id="cb8-33"><a href="#cb8-33"></a>}</span>

<span id="cb8-34"><a href="#cb8-34"></a></span>

<span id="cb8-35"><a href="#cb8-35"></a>locals {</span>

<span id="cb8-36"><a href="#cb8-36"></a>    availability_zones = data.aws_availability_zones.available.names</span>

<span id="cb8-37"><a href="#cb8-37"></a>}</span>

<span id="cb8-38"><a href="#cb8-38"></a></span>

<span id="cb8-39"><a href="#cb8-39"></a>variable <span class="st">&quot;environment&quot;</span> {</span>

<span id="cb8-40"><a href="#cb8-40"></a>    type = string</span>

<span id="cb8-41"><a href="#cb8-41"></a>    description = <span class="st">&quot;Options: development, qa, staging, production&quot;</span></span>

<span id="cb8-42"><a href="#cb8-42"></a>}</span></code></pre></div>

<p><code>vpc.tf</code></p>

<div class="sourceCode" id="cb9"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb9-1"><a href="#cb9-1"></a><span class="kw">module</span> <span class="st">&quot;vpc&quot;</span> {</span>

<span id="cb9-2"><a href="#cb9-2"></a>   source = <span class="st">&quot;terraform-aws-modules/vpc/aws&quot;</span></span>

<span id="cb9-3"><a href="#cb9-3"></a>    version = <span class="st">&quot;~&gt;2.0&quot;</span></span>

<span id="cb9-4"><a href="#cb9-4"></a>    name                 = <span class="st">&quot;my-vpc&quot;</span></span>

<span id="cb9-5"><a href="#cb9-5"></a>    cidr                 = <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.0.0/16&quot;</span></span>

<span id="cb9-6"><a href="#cb9-6"></a>    private_subnets      = local.private_subnets</span>

<span id="cb9-7"><a href="#cb9-7"></a>    database_subnets     = local.database_subnets</span>

<span id="cb9-8"><a href="#cb9-8"></a>    public_subnets       = local.public_subnets</span>

<span id="cb9-9"><a href="#cb9-9"></a></span>

<span id="cb9-10"><a href="#cb9-10"></a>    azs                  = local.availability_zones</span>

<span id="cb9-11"><a href="#cb9-11"></a></span>

<span id="cb9-12"><a href="#cb9-12"></a>    <span class="co"># omitted arguments for brevity</span></span>

<span id="cb9-13"><a href="#cb9-13"></a></span>

<span id="cb9-14"><a href="#cb9-14"></a>}</span></code></pre></div>

<h1 id="issue-3-dynamic-cidr_c">Issue #3: Dynamic cidr_c</h1>

<p>That’s the part where I’m greedy, and I want to populate <strong>cidr_c</strong> according to the number of availability zones. So far we’ve covered <strong>a.b</strong>.c.d/xx, now we will cover this part a.b.<strong>c</strong>.d/xx</p>

<p>For example (pseudo-code):</p>

<pre><code># define cidr_c per subnet

cidr_c_private_subnets =  1

cidr_c_database_subnets = 11

cidr_c_public_subnets = 64

number_of_azs = 2

# dynamically populate lists of subnets

private_subnets  = [&quot;a.b.1.d/xx&quot;, &quot;a.b.2.d/xx&quot;]

database_subnets = [&quot;a.b.11.d/xx&quot;, &quot;a.b.12.d/xx&quot;]

public_subnets   = [&quot;a.b.64.d/xx&quot;, &quot;a.b.65.d/xx&quot;]</code></pre>

<h2 id="for-loop">for loop</h2>

<p>Dynamically populating sounds like a loop, and in Terraform we got the <a href="https://www.terraform.io/docs/configuration/expressions.html#for-expressions">for loop</a>.</p>

<p>Terraform’s for loop reminds me of <a href="https://docs.python.org/3/tutorial/datastructures.html#list-comprehensions">Python’s list comprehension</a>, which means - create a new list with a for loop.</p>

<p>Syntax of the for loop statement: <code>[for item in list: do_something_on(item)]</code></p>

<p>Pseudo-code of the loop that we need:</p>

<div class="sourceCode" id="cb11"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb11-1"><a href="#cb11-1"></a><span class="kw">for</span> availability_zone <span class="kw">in</span> local.availability_zones:</span>

<span id="cb11-2"><a href="#cb11-2"></a>  <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index_of_availability_zone}.0/24&quot;</span></span></code></pre></div>

<p>Some explanations regarding the code above: 1. <code>availability_zone</code> - I picked the name of the iterator, it could also be <code>az</code> or any other name 2. <code>index_of_availability_zone</code> - First index of a list is zero (0) - Assuming <code>${lookup(var.cidr_ab, var.environment)</code> equals to one (1) for private_subnets - Adding the current index of the availability zone to <code>${lookup(var.cidr_ab, var.environment)</code>, result in: - 1 + 0 - 1 + 1 - 1 + 2</p>

<h2 id="get-the-index-of-the-availability-zone">get the index of the availability zone</h2>

<p>The <a href="https://www.terraform.io/docs/configuration/functions/index.html">index function</a> to the rescue!</p>

<p>Syntax of index function: <code>index(list_to_search_in, value_to_search_for)</code></p>

<p>The list we’re searching in is <code>local.availability_zones</code> and the value we’re looking for is the loop’s iterator, <code>availability_zone</code> in our case.</p>

<p>Replacing <code>index_of_availability_zone</code> with <code>index(local.availability_zones, az)</code> result in:</p>

<div class="sourceCode" id="cb12"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb12-1"><a href="#cb12-1"></a>locals {</span>

<span id="cb12-2"><a href="#cb12-2"></a>    private_subnets = [</span>

<span id="cb12-3"><a href="#cb12-3"></a>        <span class="kw">for</span> availability_zone <span class="kw">in</span> local.availability_zones:</span>

<span id="cb12-4"><a href="#cb12-4"></a>          <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index(local.availability_zones, az)`}.0/24&quot;</span></span>

<span id="cb12-5"><a href="#cb12-5"></a>    ]</span>

<span id="cb12-6"><a href="#cb12-6"></a>}</span></code></pre></div>

<h1 id="issue-3-full-solution">Issue #3: Full solution</h1>

<p>The iterator’s name in each loop is <code>az</code>, I picked a shorter name than <code>availability_zone</code> to make it more readable. Since we use local values, we only need to change the <code>variables.tf</code> file, no need to touch <code>vpc.tf</code> file</p>

<p><code>variables.tf</code></p>

<div class="sourceCode" id="cb13"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb13-1"><a href="#cb13-1"></a>variable <span class="st">&quot;cidr_ab&quot;</span> {</span>

<span id="cb13-2"><a href="#cb13-2"></a>    type = map</span>

<span id="cb13-3"><a href="#cb13-3"></a>    default = {</span>

<span id="cb13-4"><a href="#cb13-4"></a>        development     = <span class="st">&quot;172.22&quot;</span></span>

<span id="cb13-5"><a href="#cb13-5"></a>        qa              = <span class="st">&quot;172.24&quot;</span></span>

<span id="cb13-6"><a href="#cb13-6"></a>        staging         = <span class="st">&quot;172.26&quot;</span></span>

<span id="cb13-7"><a href="#cb13-7"></a>        production      = <span class="st">&quot;172.28&quot;</span></span>

<span id="cb13-8"><a href="#cb13-8"></a>    }</span>

<span id="cb13-9"><a href="#cb13-9"></a>}</span>

<span id="cb13-10"><a href="#cb13-10"></a></span>

<span id="cb13-11"><a href="#cb13-11"></a>locals {</span>

<span id="cb13-12"><a href="#cb13-12"></a>    cidr_c_private_subnets  = <span class="dv">1</span></span>

<span id="cb13-13"><a href="#cb13-13"></a>    cidr_c_database_subnets = <span class="dv">11</span></span>

<span id="cb13-14"><a href="#cb13-14"></a>    cidr_c_public_subnets   = <span class="dv">64</span></span>

<span id="cb13-15"><a href="#cb13-15"></a>}</span>

<span id="cb13-16"><a href="#cb13-16"></a></span>

<span id="cb13-17"><a href="#cb13-17"></a></span>

<span id="cb13-18"><a href="#cb13-18"></a>data <span class="st">&quot;aws_availability_zones&quot;</span> <span class="st">&quot;available&quot;</span> {</span>

<span id="cb13-19"><a href="#cb13-19"></a>    state = <span class="st">&quot;available&quot;</span></span>

<span id="cb13-20"><a href="#cb13-20"></a>}</span>

<span id="cb13-21"><a href="#cb13-21"></a></span>

<span id="cb13-22"><a href="#cb13-22"></a>locals {</span>

<span id="cb13-23"><a href="#cb13-23"></a>    availability_zones = data.aws_availability_zones.available.names</span>

<span id="cb13-24"><a href="#cb13-24"></a>}</span>

<span id="cb13-25"><a href="#cb13-25"></a></span>

<span id="cb13-26"><a href="#cb13-26"></a>variable <span class="st">&quot;environment&quot;</span> {</span>

<span id="cb13-27"><a href="#cb13-27"></a>    type = string</span>

<span id="cb13-28"><a href="#cb13-28"></a>    description = <span class="st">&quot;Options: development, qa, staging, production&quot;</span></span>

<span id="cb13-29"><a href="#cb13-29"></a>}</span>

<span id="cb13-30"><a href="#cb13-30"></a></span>

<span id="cb13-31"><a href="#cb13-31"></a>locals {</span>

<span id="cb13-32"><a href="#cb13-32"></a>    private_subnets = [</span>

<span id="cb13-33"><a href="#cb13-33"></a>        <span class="kw">for</span> az <span class="kw">in</span> local.availability_zones : </span>

<span id="cb13-34"><a href="#cb13-34"></a>            <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index(local.availability_zones, az)}.0/24&quot;</span></span>

<span id="cb13-35"><a href="#cb13-35"></a>        ]</span>

<span id="cb13-36"><a href="#cb13-36"></a>    database_subnets = [</span>

<span id="cb13-37"><a href="#cb13-37"></a>        <span class="kw">for</span> az <span class="kw">in</span> local.availability_zones : </span>

<span id="cb13-38"><a href="#cb13-38"></a>            <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_database_subnets + index(local.availability_zones, az)}.0/24&quot;</span></span>

<span id="cb13-39"><a href="#cb13-39"></a>        ]</span>

<span id="cb13-40"><a href="#cb13-40"></a>    public_subnets = [</span>

<span id="cb13-41"><a href="#cb13-41"></a>        <span class="kw">for</span> az <span class="kw">in</span> local.availability_zones : </span>

<span id="cb13-42"><a href="#cb13-42"></a>            <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_public_subnets + index(local.availability_zones, az)}.0/24&quot;</span></span>

<span id="cb13-43"><a href="#cb13-43"></a>        ]</span>

<span id="cb13-44"><a href="#cb13-44"></a>}</span></code></pre></div>

<h1 id="issue-4-limit-number-of-subnets">Issue #4: Limit number of subnets</h1>

<p>In solution #3 we populated subnets according to the number of availability zones, which is excellent, but this can lead to unwanted behavior when using the module <a href="https://github.com/terraform-aws-modules/terraform-aws-vpc">terraform-aws-modules/vpc/aws</a>.</p>

<p>If you want to have a set of subnets per availability zone, without caring for how many subnets are created per region, you can stop here. It will be easier to explain with an example: ca-central-1 (Canada Central) - 2 availability zones, hence <strong>6</strong> subnets us-west-2 (Oregon) - 4 availability zones, hence <strong>12</strong> subnets</p>

<p>If you wish the number of subnets to be similar across all environments (and regions) - keep on reading. For example: Let’s assume we must use ca-central-1 (Canada Central), and because of that, we are forced to use only two availability zones in each region. ca-central-1 (Canada Central) - 2 availability zones, hence <strong>6</strong> subnets us-west-2 (Oregon) - 4 availability zones, forcing maximum of <strong>6</strong> subnets (2 per type of subnet) In total, we will have six subnets in our VPC (not including the ‘default’ one that comes with the VPC).</p>

<p>Rule of thumb - We need the lowest common number of availability zones that are available in all the regions we intend to use.</p>

<h2 id="maximum-number-of-subnets">Maximum number of subnets</h2>

<p>We will now initialize three more variables, <code>subnet_max</code> per subnet.</p>

<p><code>variables.tf</code></p>

<div class="sourceCode" id="cb14"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb14-1"><a href="#cb14-1"></a>locals {</span>

<span id="cb14-2"><a href="#cb14-2"></a>    cidr_c_private_subnets  = <span class="dv">1</span></span>

<span id="cb14-3"><a href="#cb14-3"></a>    cidr_c_database_subnets = <span class="dv">11</span></span>

<span id="cb14-4"><a href="#cb14-4"></a>    cidr_c_public_subnets   = <span class="dv">64</span></span>

<span id="cb14-5"><a href="#cb14-5"></a></span>

<span id="cb14-6"><a href="#cb14-6"></a>    max_private_subnets     = <span class="dv">2</span></span>

<span id="cb14-7"><a href="#cb14-7"></a>    max_database_subnets    = <span class="dv">2</span></span>

<span id="cb14-8"><a href="#cb14-8"></a>    max_public_subnets      = <span class="dv">2</span></span>

<span id="cb14-9"><a href="#cb14-9"></a>}</span></code></pre></div>

<h2 id="conditional-for-loop">Conditional for loop</h2>

<p>Luckily the for loop has a built-in keyword <a href="https://www.terraform.io/docs/configuration/expressions.html#for-expressions">if</a>, which still reminds me of Python’s list comprehension :)</p>

<p>Syntax of for loop with condition: <code>[for item in list: do_something_on(item) if expression]</code></p>

<p>Our goal is to iterate over the availability zones until we reach the maximum desired number of subnets.</p>

<h1 id="issue-4-full-solution">Issue #4: Full Solution</h1>

<p>And of course, we only need to change the <code>variables.tf</code> file.</p>

<p><code>variables.tf</code></p>

<div class="sourceCode" id="cb15"><pre class="sourceCode ruby"><code class="sourceCode ruby"><span id="cb15-1"><a href="#cb15-1"></a>variable <span class="st">&quot;cidr_ab&quot;</span> {</span>

<span id="cb15-2"><a href="#cb15-2"></a>    type = map</span>

<span id="cb15-3"><a href="#cb15-3"></a>    default = {</span>

<span id="cb15-4"><a href="#cb15-4"></a>        development     = <span class="st">&quot;172.22&quot;</span></span>

<span id="cb15-5"><a href="#cb15-5"></a>        qa              = <span class="st">&quot;172.24&quot;</span></span>

<span id="cb15-6"><a href="#cb15-6"></a>        staging         = <span class="st">&quot;172.26&quot;</span></span>

<span id="cb15-7"><a href="#cb15-7"></a>        production      = <span class="st">&quot;172.28&quot;</span></span>

<span id="cb15-8"><a href="#cb15-8"></a>    }</span>

<span id="cb15-9"><a href="#cb15-9"></a>}</span>

<span id="cb15-10"><a href="#cb15-10"></a></span>

<span id="cb15-11"><a href="#cb15-11"></a>locals {</span>

<span id="cb15-12"><a href="#cb15-12"></a>    cidr_c_private_subnets  = <span class="dv">1</span></span>

<span id="cb15-13"><a href="#cb15-13"></a>    cidr_c_database_subnets = <span class="dv">11</span></span>

<span id="cb15-14"><a href="#cb15-14"></a>    cidr_c_public_subnets   = <span class="dv">64</span></span>

<span id="cb15-15"><a href="#cb15-15"></a></span>

<span id="cb15-16"><a href="#cb15-16"></a>    max_private_subnets     = <span class="dv">2</span></span>

<span id="cb15-17"><a href="#cb15-17"></a>    max_database_subnets    = <span class="dv">2</span></span>

<span id="cb15-18"><a href="#cb15-18"></a>    max_public_subnets      = <span class="dv">2</span></span>

<span id="cb15-19"><a href="#cb15-19"></a>}</span>

<span id="cb15-20"><a href="#cb15-20"></a></span>

<span id="cb15-21"><a href="#cb15-21"></a>data <span class="st">&quot;aws_availability_zones&quot;</span> <span class="st">&quot;available&quot;</span> {</span>

<span id="cb15-22"><a href="#cb15-22"></a>    state = <span class="st">&quot;available&quot;</span></span>

<span id="cb15-23"><a href="#cb15-23"></a>}</span>

<span id="cb15-24"><a href="#cb15-24"></a></span>

<span id="cb15-25"><a href="#cb15-25"></a>locals {</span>

<span id="cb15-26"><a href="#cb15-26"></a>    availability_zones = data.aws_availability_zones.available.names</span>

<span id="cb15-27"><a href="#cb15-27"></a>}</span>

<span id="cb15-28"><a href="#cb15-28"></a></span>

<span id="cb15-29"><a href="#cb15-29"></a>variable <span class="st">&quot;environment&quot;</span> {</span>

<span id="cb15-30"><a href="#cb15-30"></a>    type = string</span>

<span id="cb15-31"><a href="#cb15-31"></a>    description = <span class="st">&quot;Options: development, qa, staging, production&quot;</span></span>

<span id="cb15-32"><a href="#cb15-32"></a>}</span>

<span id="cb15-33"><a href="#cb15-33"></a></span>

<span id="cb15-34"><a href="#cb15-34"></a>locals {</span>

<span id="cb15-35"><a href="#cb15-35"></a>    private_subnets = [</span>

<span id="cb15-36"><a href="#cb15-36"></a>        <span class="kw">for</span> az <span class="kw">in</span> local.availability_zones : </span>

<span id="cb15-37"><a href="#cb15-37"></a>            <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_private_subnets + index(local.availability_zones, az)}.0/24&quot;</span></span>

<span id="cb15-38"><a href="#cb15-38"></a>            <span class="kw">if</span> index(local.availability_zones, az) &lt; local.max_private_subnets</span>

<span id="cb15-39"><a href="#cb15-39"></a>        ]</span>

<span id="cb15-40"><a href="#cb15-40"></a>    database_subnets = [</span>

<span id="cb15-41"><a href="#cb15-41"></a>        <span class="kw">for</span> az <span class="kw">in</span> local.availability_zones : </span>

<span id="cb15-42"><a href="#cb15-42"></a>            <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_database_subnets + index(local.availability_zones, az)}.0/24&quot;</span></span>

<span id="cb15-43"><a href="#cb15-43"></a>            <span class="kw">if</span> index(local.availability_zones, az) &lt; local.max_database_subnets</span>

<span id="cb15-44"><a href="#cb15-44"></a>        ]</span>

<span id="cb15-45"><a href="#cb15-45"></a>    public_subnets = [</span>

<span id="cb15-46"><a href="#cb15-46"></a>        <span class="kw">for</span> az <span class="kw">in</span> local.availability_zones : </span>

<span id="cb15-47"><a href="#cb15-47"></a>            <span class="st">&quot;${lookup(var.cidr_ab, var.environment)}.${local.cidr_c_public_subnets + index(local.availability_zones, az)}.0/24&quot;</span></span>

<span id="cb15-48"><a href="#cb15-48"></a>            <span class="kw">if</span> index(local.availability_zones, az) &lt; local.max_public_subnets</span>

<span id="cb15-49"><a href="#cb15-49"></a>        ]</span>

<span id="cb15-50"><a href="#cb15-50"></a>}</span></code></pre></div>

<h1 id="summary">Summary</h1>

<ol type="1">

<li>Some solution architects might prefer hard-coding the subnets prefixes in the <code>vpc.tf</code> file, which in most cases will work fine. The fact that I had only to change the <code>variables .tf</code> file each time is priceless</li>

<li>The logic behind the order of variables in <code>variables.tf</code> file -

<ul>

<li>variables that should be modified, such as:

<ul>

<li>region per environment (map)</li>

<li>local cidr_c and max_subnets for each type of subnet</li>

</ul></li>

<li>static variables/data, such as:

<ul>

<li>data “aws_availability_zones” “available”</li>

<li>local.availability_zones</li>

<li>local.subnet_list for each type of subnet</li>

</ul></li>

</ul></li>

<li>I find that using local values instead of variables is a good approach, and it provides the flexibility I need when designing the infrastructure

<ul>

<li>The only case where we need variables is when we want to prompt for values, for example: <code>var.environment</code></li>

<li>Always use local values; they provide the ability to use functions, unlike variables</li>

<li>I used <code>var.environment</code> and <code>var.cidr_ab</code> - but I should’ve assigned them to local values. I used variables because I wanted to make this tutorial versatile. Remember- local values are the best</li>

</ul></li>

<li>AWS keeps adding availability zones (AZs), so the number of availability zones per region in the examples might be outdated. Be sure to check the current number of AZs

<ul>

<li><a href="https://aws.amazon.com/about-aws/global-infrastructure/">AWS Global Infrastructure</a></li>

<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html">Regions and Availability Zones</a></li>

</ul></li>

</ol>

<p>Did you like this tutorial? Clap/heart/unicorn and share it with your friends and colleagues. Didn’t you like it? Let me know which parts and I’ll take care of it.</p>

<p>My next article will be about <a href="https://app.terraform.io/session">Terraform Cloud</a> and <a href="https://www.terraform.io/docs/state/workspaces.html">Terraform’s Workspaces</a>.</p>

</body>

</html>

Terraform AWS - Dynamic Subnets
Meir Gabay
Junior Operations Engineer
Meir is an experienced automation engineer and a certified AWS solutions architect. He has four years of experience as Operations and Automation at NICE's training department. With knowledge in web development, both server-side and front-end. The critical thing that motivates him while working with customers - "To continue evolving and improving independently, a customer needs to know how the solution works." While Meir is not helping customers, you'll find him studying new technologies and writing a blog post of what he had learned.