You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: projects/eks-tf-tutorial/README.md
+81-7Lines changed: 81 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,6 +14,8 @@ Goals:
14
14
15
15
## Development
16
16
17
+
### Create the EKS cluster
18
+
17
19
To get started:
18
20
19
21
```shell
@@ -24,19 +26,20 @@ terraform init
24
26
Plan:
25
27
26
28
```shell
27
-
AWS_PROFILE=personal terraform plan
29
+
export AWS_PROFILE=personal
30
+
terraform plan
28
31
```
29
32
30
33
Apply:
31
34
32
35
```shell
33
-
AWS_PROFILE=personal terraform apply
36
+
terraform apply
34
37
```
35
38
36
39
Destroy:
37
40
38
41
```shell
39
-
AWS_PROFILE=personal terraform destroy
42
+
terraform destroy
40
43
```
41
44
42
45
## Notes
@@ -89,8 +92,79 @@ Part 2: https://www.youtube.com/watch?v=uiuoNToeMFE&list=PLiMWaCMwGJXnKY6XmeifEp
89
92
- In this tutorial, we will use the policy managed by Amazon, `AmazonEKSClusterPolicy`, which contains all the permissions that the cluster needs. However, this policy assumes you will use a legacy provider that additionally requires load balancing permissions. These permissions constitute over half of the policy (i.e the `elasticloadbalancing` permissions). In the following sections, we will install AWS Load Balancer Controller that will take on that responsibility. At that point, we will create our own separate role to remove these redundant permissions.
90
93
- K8s workers require several permissions of their own:
91
94
- The `AmazonWorkerNodePolicy` policy provides core EC2 functionality permissions, e.g. `ec2:DescribeInstances`.
92
-
- "EKS pod identity" allows fine-grained access to control of pod permissions, e.g. giving access to a pod to read/write to a specific S3 bucket.
93
-
- Before the v3 of Pod Identities, we had to use an OpenID Connect provider and IAM roles for service accounts.
95
+
-The latest v3 of `AmazonWorkerNodePolicy` also allows "EKS pod identity" for fine-grained access to control of pod permissions, e.g. giving access to a pod to read/write to a specific S3 bucket.
96
+
- Before we had Pod Identities(`AmazonWorkerNodePolicy` v2), we had to use an OpenID Connect provider and IAM roles for service accounts to achieve the same thing.
94
97
- We will look at the three separate methods for authentication later in the tutorial.
95
-
- We have to grant EKS access to modify IP address configuration on EKS worker nodes.
96
-
- When we create a pod on K8s, it is assigned an IP from the secondary IP address range assigned to the worker node. We're not using virtual networks like Flannel or Calico,
98
+
- We have to grant EKS access to modify IP address configuration on EKS worker nodes using the `AmazonEKS_CNI_Policy`.
99
+
- When we create a pod on K8s, it is assigned an IP from the secondary IP address range assigned to the worker node. We're not using virtual networks like Flannel or Calico, we get native AWS IP addresses for each pod. Later in the course, we will create k8s Service of type `LoadBalancer` which allows `Instance mode` and `IP mode`. The former is used for `NodePort` Services, while the latter works with `LoadBalancer` Services to route traffic directly to pod IPs.
100
+
- Using `NodePort` isn't recommended in production due to security, scalability, and latency issues.
101
+
-`NodePort` opens a port on the underlying worker node, making the node vulnerable to attacks.
102
+
-`NodePort` is limited to 2767 ports per node (30000-32767)
103
+
-`NodePort` increases latency as there are more network hops involved
104
+
- However, `NodePort` is a lot more cost efficient, as `LoadBalancer` Services create a Network Load Balancer per Service. For this reason, it is common to use Istio to handle cluster networking for both ingress (proxying) and load balancing, as well as the other benefits Istio brings (TLS, centralised network monitoring/tooling).
105
+
- Lastly, the `AmazonEC2ContainerRegistryReadOnly` policy is used to grant EKS permission to pull Docker images from ECR.
106
+
107
+
In the code:
108
+
109
+
-_7-eks.tf_:
110
+
- We create an IAM role per cluster, e.g. `"${local.name_prefix}-${local.eks_name}-eks-cluster"` in case we have multiple clusters per environment.
111
+
- We set up the network settings with `endpoint_public_access=true` to make the cluster publicly accessible.
112
+
- Later in the tutorial, we will see how to set up a private EKS cluster using AWS Client VPN and a private DNS that we can use for ingress and private services that we don't want to expose to the internet, e.g. Grafana dashboards, Temporal UI, etc.
113
+
- Even with a public cluster, the worker nodes will still be deployed on the private subnets without public IP addresses.
114
+
- We specified the two private subnets to place the worker nodes. It is required to have 2 AZs. EKS will create cross-account elastic network interface in these subnets to allow communication between worker nodes and the K8s control plane.
115
+
- We configured authentication to use `API`.
116
+
- We used to have to manage authentication using `aws_auth` ConfigMap in kube system namespace. However, this is deprecated now. It wasn't convenient to manage existing ConfigMaps in K8s using Terraform resources.
117
+
- AWS developed an API that we can use to add users to the cluster. You can still use ConfigMap and even both ConfigMap and API. But the API is highly recommended for user management. In the next section, we cover how to add IAM roles and IAM users to access the cluster.
118
+
- We set the `bootstrap_cluster_creator_admin_permissions=true` to grant the Terraform user admin priveleges. This option defaults to `true` anyway, but we want to be explicit. For us, because we will use Terraform to deploy Helm Charts and plain YAML resources.
119
+
-_8-nodes.tf_:
120
+
- We create an IAM role for the worker nodes. We add `ec2.amazonaws.com` as the trust relationship instead of `eks.amazonaws.com` like we did in the eks cluster. In the next setion, we'll create an IAM role with a trust policy that allows specific users to assume it.
121
+
- We attached the three IAM policies discussed earlier to the nodes.
122
+
- Finally, we add the node group itself.
123
+
- Behind the scenes its managed as an EC2 autoscaling group.
124
+
- EKS has three types of autoscaling groups:
125
+
- Self-managed node groups:
126
+
- We can create the nodes ourselves using Terraform and Packer templates to customise the EC2 VM to install any packages we need.
127
+
- Limitations: EKS will not drain the node during upgrades. This can be done semi-manually.
128
+
- Managed node groups:
129
+
- We'll use these in this tutorial. They are a lot easier to manage and upgrade as this is managed by the EKS control plane.
130
+
- Fargate:
131
+
- Fully managed node groups, "serverless" K8s, that only require the user to deploy their containers and EKS will automatically provision and scale the worker nodes for you.
132
+
- A lot easier to manage but a lot more expensive. It has limitations such as EBS volumes.
133
+
- We set the `node_group_name` to general
134
+
- In production, it is common to have different node groups for different workloads, e.g. CPU optimised node groups, memory optimised node groups, or even GPU node groups for machine learning.
135
+
- Another common node group is for spot instances. This node group can be used for fault-tolerant workloads that can handle interuptions, such as batch/streaming jobs and async workflows. Spot instances can save up to 90%.
136
+
- We add the two private subnets to place the worker nodes.
137
+
- In production, if we have data intensive workloads, such as running Kafka and 100s of services that read and write to Kafka from different AZs, cross AZ data transfer costs can be super expensive, even more expensive than the compute!
138
+
- We could place all the worker nodes in a single AZ - you don't always need to create a highly available cluster by spreading nodes in different zones.
139
+
- For `capacity_type` we chose `ON_DEMAND` but we also have `STANDARD` and `SPOT`. We also set the `instance_types`.
140
+
- In a future tutorial, the instructor will show a strategy to come up with the proper node size.
141
+
- The `scaling_config` by itself will not autoscale but we can set the minimum and maximum nodes as well as the `desired_size`. We will need to deploy addition components called Cluster Autoscaler to adjust the `desired_size` based on the load, such as how many pending pods we have and their resource requirements.
142
+
- The `update_config` is used for cluster upgrades.
143
+
- The `labels` are used for pod affinity and node selectors. There are some built-in labels derived from the node group that serve the same purpose. However, in practice when you try to migrate applications from one node group to another with the same labels it is much easier to use custom labels.
144
+
145
+
At the end of this part:
146
+
147
+
Check you're connected with the correct user:
148
+
149
+
```shell
150
+
export AWS_PROFILE=personal
151
+
aws sts get-caller-identity
152
+
```
153
+
154
+
Next, update the local kubeconfig with the following command:
0 commit comments