Testing terraform with moto, part 2 (An Interlude)

March 28, 2022   

In my last post I started to set up the terraform and moto environment. In this post, we’ll attempt to connect an existing module up to moto server.

I had previously exported and converted the terraform output of builder to standard terraform HCL, and decided to build upon that work to:

  1. Give the scenario a real-world setup
  2. Save myself time trying to write it from scratch.

There is a branch called refactor-into-a-module That I used as the base version. Then I used the basic config in our last post as a test terraform module utilising the module. The code in that repo is, unfortunately, stuck on terraform 0.11, so I needed to update quite a bit of it to bring it up to date to latest terraform coding pracices. In the end, we had something like this

 1// setup provider for localstack
 2provider "aws" {
 3  region                      = "us-east-1"
 4  access_key                  = "test"
 5  secret_key                  = "test"
 6  s3_use_path_style           = true
 7  skip_credentials_validation = true
 8  skip_metadata_api_check     = true
 9  skip_requesting_account_id  = true
10
11  endpoints {
12    ec2 = "http://localhost:5000"
13    eks = "http://localhost:5000"
14    iam = "http://localhost:5000"
15    s3  = "http://localhost:5000"
16  }
17}
18
19module "kubernetes-aws" {
20  source            = "../../../modules/kubernetes-aws"
21  cluster_name      = "kubernetes-test"
22  cluster_version   = "1.19"
23  project           = "terraform-testing"
24  environment       = "test"
25  min_workers       = "1"
26  max_workers       = "6"
27  desired_workers   = "3"
28  instance_type     = "t3.large"
29}

I then worked in a terraform init and terraform plan, trying to correct any issues I could see with the code. In the end, I needed to replace some hardcoded VPCs, a call to AWS for EKS ec2 images (instead, make sure the call returns a moto mocked image), and some small tweaks to changes in the resources available in the aws plugin.

In the end, I had something like this:

 1// setup provider for localstack
 2provider "aws" {
 3  region  = "us-east-1"
 4  access_key                  = "test"
 5  secret_key                  = "test"
 6  s3_use_path_style           = true
 7  skip_credentials_validation = true
 8  skip_metadata_api_check     = true
 9  skip_requesting_account_id  = true
10
11  endpoints {
12    ec2            = "http://localhost:5000"
13    eks            = "http://localhost:5000"
14    iam            = "http://localhost:5000"
15  }
16}
17
18# query moto to get the default VPC and subnet
19data "aws_vpc" "default" {
20}
21data "aws_subnets" "default" {
22  filter {
23    name   = "vpc-id"
24    values = [data.aws_vpc.default.id]
25  }
26}
27
28module "kubernetes-aws" {
29  source            = "../../../modules/kubernetes-aws"
30  cluster_name      = "kubernetes-test"
31  cluster_version   = "1.19"
32  project           = "terraform-testing"
33  environment       = "test"
34  min_workers       = 1
35  max_workers       = 6
36  desired_workers   = 3
37  instance_type     = "t3.large"
38
39  is_testing        = true                          # added to signal which image filter to look for
40  vpc_id            = data.aws_vpc.default.id       # to replace the hardcoded VPC ids
41  subnets           = data.aws_subnets.default.ids  # ... and subnets with moto ones
42}

At this point, in theory the module should apply cleanly to AWS infrastructure. Unfortunately, I ran into an issue with moto:

 1│ Error: error creating EKS Cluster (kubernetes-test): ResourceInUseException: Cluster already exists with name: kubernetes-test
 2│ {
 3│   RespMetadata: {
 4│     StatusCode: 409,
 5│     RequestID: ""
 6│   },
 7│   ClusterName: "kubernetes-test",
 8│   Message_: "Cluster already exists with name: kubernetes-test"
 9│ }
1011│   with module.kubernetes-aws.aws_eks_cluster.main,
12│   on ../../../modules/kubernetes-aws/main.tf line 206, in resource "aws_eks_cluster" "main":
13│  206: resource "aws_eks_cluster" "main" {
14

Bringing the module down the bare minimum:

 1// setup provider for localstack
 2provider "aws" {
 3    region                      = "us-east-1"
 4    access_key                  = "test"
 5    secret_key                  = "test"
 6    s3_use_path_style           = true
 7    skip_credentials_validation = true
 8    skip_metadata_api_check     = true
 9    skip_requesting_account_id  = true
10
11    endpoints {
12        ec2 = "http://localhost:5000"
13        eks = "http://localhost:5000"
14        iam = "http://localhost:5000"
15        s3  = "http://localhost:5000"
16    }
17}
18
19
20# get the default VPC out of moto
21data "aws_vpc" "default" {
22}
23data "aws_subnets" "default" {
24    filter {
25        name   = "vpc-id"
26        values = [data.aws_vpc.default.id]
27    }
28}
29
30#create a basic IAM role
31resource "aws_iam_role" "master" {
32  name               = "test"
33  assume_role_policy = "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Action\": \"sts:AssumeRole\", \"Effect\": \"Allow\", \"Principal\": {\"Service\": \"eks.amazonaws.com\"}}]}"
34}
35
36#create a basic security group
37resource "aws_security_group" "master" {
38  name   = "test"
39  vpc_id = data.aws_vpc.default.id
40
41  egress {
42    cidr_blocks = [
43      "0.0.0.0/0",
44    ]
45
46    from_port = 0
47    to_port   = 0
48    protocol  = -1
49  }
50
51  description = "test"
52}
53
54# create a basic eks cluster
55resource "aws_eks_cluster" "main" {
56  vpc_config {
57    subnet_ids = data.aws_subnets.default.ids
58
59    security_group_ids = [
60      "${aws_security_group.master.id}",
61    ]
62  }
63
64
65  version  = "1.21"
66  role_arn = "${aws_iam_role.master.arn}"
67  name     = "test"
68}

and running terraform apply with the environment variable TF_LOG=debug we can see where things go wrong:

12022-03-28T07:37:38.803+0100 [DEBUG] provider.terraform-provider-aws_v4.5.0_x5: [aws-sdk-go] <lengthy json>: timestamp=2022-03-28T07:37:38.803+0100
22022-03-28T07:37:38.803+0100 [DEBUG] provider.terraform-provider-aws_v4.5.0_x5: [aws-sdk-go] DEBUG: Unmarshal Response eks/CreateCluster failed, attempt 0/25, error SerializationError: failed decoding JSON RPC response
3	status code: 200, request id:
4caused by: JSON value is not a list (map[string]interface {}{}): timestamp=2022-03-28T07:37:38.803+0100
52022-03-28T07:37:38.862+0100 [DEBUG] provider.terraform-provider-aws_v4.5.0_x5: [aws-sdk-go] DEBUG: Retrying Request eks/CreateCluster, attempt 1: timestamp=2022-03-28T07:37:38.862+0100

The try then fails, as the (fake) cluster was already created.

From the error output, it would seem terraform code is expecting something in that json response to be a list, but is finding it as a map.

Sure enough, the moto codebase shows that it is returning a dict():

 1class Cluster:
 2    def __init__(
 3        self,
 4        name,
 5        role_arn,
 6        resources_vpc_config,
 7        region_name,
 8        aws_partition,
 9        version=None,
10        kubernetes_network_config=None,
11        logging=None,
12        client_request_token=None,
13        tags=None,
14        encryption_config=None,
15    ):
16        if encryption_config is None:
17            encryption_config = dict() # THIS LINE
18        if tags is None:
19            tags = dict()

Where as aws’s API documentation says it’s an array andaws-sdk-go expects a list in the response.

I’ve posted a bug request here, hopefully this will be ironed out upstream.

In the next post, I’ll explore a temporary fix for this, and then using terratest to set out examples and tests. Until next time.

- Scott