You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|`pub/media`| Media files |:heavy_check_mark: <sup>1</sup> |
306
+
|`pub/static/_cache`| Static files cache (i.e. merged CSS) | If merge is enabled |
307
+
|`var`| Cache, logs, reports, sessions, etc. |:heavy_check_mark: <sup>2</sup> |
308
+
309
+
> <sup>1</sup> : The content of the `pub/media` directory might be stored in an external storage (i.e. S3, GCS, etc.), which would allow to avoid sharing the directory between pods.
310
+
>
311
+
> <sup>2</sup> : Cache and sessions should be stored in Redis. Logs can be shared to facilitate cross-pod debugging, but may have simultaneous write issues. We'll see further how to get logs printed to standard output of Pods.
312
+
313
+
## Configuration and secrets
314
+
315
+
Configuration and secrets should be stored in `ConfigMaps` and `Secrets` respectively.
316
+
317
+
We will be using those to store our environment variables, and _mount_ them in the Pods containers.
Copy file name to clipboardExpand all lines: docs/guide/deployment/resources-scaling.md
+190Lines changed: 190 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -3,3 +3,193 @@ title: Resources and scaling
3
3
---
4
4
5
5
# Resources and scaling
6
+
7
+
In Kubernetes, resources like CPU and memory are crucial for workload allocation on Nodes.
8
+
Pods have resource requests and limits, which help the scheduler decide where to place them.
9
+
Proper provisioning ensures efficient resource use and system stability.
10
+
11
+
Choosing the right resource requests for workloads in Kubernetes is challenging due to the need to balance application performance with efficient resource utilization (over vs. under provisioning).
12
+
13
+
> [!TIP]
14
+
> If you're not familiar with Kubernetes resources, check out the [How Pods with resource requests are scheduled](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-requests-are-scheduled) section in the official Kubernetes documentation.
15
+
16
+
Additionally, horizontal scaling is a critical aspect of managing workloads in Kubernetes: it allows you to scale the number of Pods based on resource usage or other metrics.
17
+
It is preferable to have smaller Pods, which can be easily scaled horizontally, and facilitate better resource utilization on Nodes.
18
+
19
+
## Resource allocation
20
+
21
+
### Web server Pods
22
+
23
+
When building our PHP FPM Docker image, we fixed the number of PHP-FPM workers to **10** (see [this section](/guide/build/docker#fpm-config)).
24
+
25
+
This means that _theoretically_, each PHP-FPM worker can consume up to 1 CPU core, and the total CPU usage of all PHP-FPM workers in 1 Pod can reach 10 CPU cores.
26
+
27
+
However, in practice, PHP-FPM workers are not always using CPU when handling requests: they can be waiting for I/O operations, network requests, etc.
28
+
29
+
Depending on your specific workload, the actual user CPU usage duration varies between 20% and 80% of the total time taken by the worker to handle a request.
30
+
31
+
Our general recommentation of resources for the web servers Pods containers is:
> <sup>1</sup> : setting CPU limits is often not recommended as it may cause contention and performance degradation. Setting such a limit may not be useful with properly configured horizontal autoscaling.
41
+
42
+
> [!IMPORTANT]
43
+
> Requesting the proper amount of memory is more critical than CPU: if a Node runs out of memory, it can become unresponsive, and the Pod can be evicted. Also, when the memory limit of a Pods container is reached, the container is killed.
44
+
> CPU throttling is less critical, as the Pod can still run, but it will be slower.
45
+
46
+
> [!TIP]
47
+
> Although we're using round numbers for resource requests, you might want to fine-tune those values depending on your nodes capacity: an 8-core Node will actually have around 7 CPU cores available for Pods, as between 0.5 and 1 core is used for system processes and `DaemonSets`.
48
+
49
+
### Consumer and cron Pods
50
+
51
+
The consumer and cron Pods are running a single process, meaning that they can not use more than 1 CPU core.
52
+
53
+
Our general recommendation of resources for the consumer and cron Pods containers is:
54
+
55
+
| Type | Resource | Request | Limit |
56
+
|----------|----------|---------|-------|
57
+
| Consumer | CPU | 0.2 ||
58
+
| Consumer | Memory | 1Gi | 1Gi |
59
+
| Cron | CPU | 0.5 ||
60
+
| Cron | Memory | 1Gi | 1Gi |
61
+
62
+
The CPU resources to request may differ between cron groups (if running separately) and between the different consumers.
63
+
64
+
For instance, the `index` cron group has smaller CPU requirements, as it's mainly working with I/O operations (database, search, etc.).
65
+
66
+
### Wrapping up
67
+
68
+
Allocating resources to Pods is a complex task, and may require some trial and error to find the right balance between performance and resource utilization.
69
+
70
+
You should monitor your Pods and Nodes (i.e. using Prometheus, see further in this guide) to ensure that the resources allocated are adapted to the workload, and adjust them if needed.
71
+
72
+
## Workload placement
73
+
74
+
When deploying Pods in a Kubernetes cluster, the scheduler decides where to place them based on the resource requests and limits of the Pods, as well as the available resources on the Nodes.
75
+
76
+
<!-- Schema of "Web server Pods" being placed in different Kubernetes nodes -->
77
+
```mermaid
78
+
%%{init: {"flowchart": {"htmlLabels": false}} }%%
79
+
graph TD
80
+
subgraph NN["`Node _n_`"]
81
+
direction LR
82
+
P31[Web server Pod]
83
+
P32[Web server Pod]
84
+
P33[Web server Pod]
85
+
P34[Consumer Pod]
86
+
end
87
+
subgraph N2[Node 2]
88
+
direction LR
89
+
P11[Web server Pod]
90
+
P12[Web server Pod]
91
+
P13[Web server Pod]
92
+
P14[Consumer Pod]
93
+
P15[Consumer Pod]
94
+
end
95
+
subgraph N1[Node 1]
96
+
direction LR
97
+
P21[Web server Pod]
98
+
P22[Web server Pod]
99
+
P23[Web server Pod]
100
+
P24[Cron Pod]
101
+
P25[Consumer Pod]
102
+
end
103
+
104
+
style NN stroke-dasharray: 5 5
105
+
style P31 stroke-dasharray: 5 5
106
+
style P32 stroke-dasharray: 5 5
107
+
style P33 stroke-dasharray: 5 5
108
+
style P34 stroke-dasharray: 5 5
109
+
```
110
+
111
+
But it also takes into account other factors, such as the Pod's affinity and anti-affinity rules, taints and tolerations, etc.
112
+
113
+
As a general rule, you should avoid placing all the `Pods` of a same `Deployment` on the same Node, to ensure high availability in case of Node failure.
114
+
115
+
Such a rule can be enforced using Pod anti-affinity rules, which prevent Pods with the same label to be scheduled on the same Node:
116
+
117
+
```yaml
118
+
apiVersion: apps/v1
119
+
kind: Deployment
120
+
metadata:
121
+
name: magento-deployment
122
+
spec:
123
+
...
124
+
template:
125
+
spec:
126
+
...
127
+
affinity:
128
+
podAntiAffinity:
129
+
preferredDuringSchedulingIgnoredDuringExecution:
130
+
- weight: 1
131
+
podAffinityTerm:
132
+
topologyKey: kubernetes.io/hostname
133
+
labelSelector:
134
+
matchLabels:
135
+
<label-key>: <label-value>
136
+
```
137
+
138
+
## Horizontal scaling
139
+
140
+
We want to achieve two main goals with horizontal scaling:
141
+
142
+
* **High availability**: if a `Pod` fails (or a `Node`), another one can take over
143
+
* **Resource and cost optimization**: we can scale the number of `Pods` based on resource usage: less traffic, fewer `Pods`, more traffic, more `Pods`
144
+
145
+
Horizontal scaling can be achieved in Kubernetes using the `HorizontalPodAutoscaler` resource.
146
+
147
+
The `HorizontalPodAutoscaler` can automatically scale the number of `Pods` in a `Deployment` based on observed CPU utilization (or other metrics, but we'll only use CPU).
148
+
149
+
The configuration of an `HorizontalPodAutoscaler` is quite simple ; we need to define:
150
+
151
+
* A target `Deployment`
152
+
* A minimum and maximum number of replicas (`Pods`)
153
+
* A target CPU utilization percentage, over which the `Pods` will be scaled up
154
+
155
+
A sample configuration would look like this:
156
+
157
+
```yaml{12,13,20}
158
+
apiVersion: autoscaling/v2
159
+
kind: HorizontalPodAutoscaler
160
+
metadata:
161
+
name: magento-autoscaler
162
+
labels:
163
+
app: magento
164
+
spec:
165
+
scaleTargetRef:
166
+
apiVersion: apps/v1
167
+
kind: Deployment
168
+
name: magento-deployment
169
+
minReplicas: 2
170
+
maxReplicas: 10
171
+
metrics:
172
+
- type: Resource
173
+
resource:
174
+
name: cpu
175
+
target:
176
+
type: Utilization
177
+
averageUtilization: 70
178
+
```
179
+
180
+
A few important points to note:
181
+
182
+
* The `HorizontalPodAutoscaler` will only scale the number of `Pods` of the `Deployment` if the containers have resource requests defined.
183
+
* The CPU average utilization is calculated every 15 seconds by default, and over a small period of time, so the `Pods` won't be scaled up immediately if the CPU usage spikes
184
+
* The calculated average CPU utilization of a `Pod` is actually an average ratio of the CPU usage of all a `Pod`'s containers, over their respective CPU requests
185
+
186
+
> [!TIP]
187
+
> You can also use the `PodDisruptionBudget` resource to define how many `Pods` of a `Deployment` can be down at the same time. We'll cover this further in the high availability section.
188
+
189
+
### Recommendations
190
+
191
+
You should keep your `minReplicas` as low as possible, to save resources and costs, and your `maxReplicas` as high as possible, to ensure high availability, while keeping in mind the capacity of your Nodes.
192
+
193
+
From our experience, a good CPU utilization percentage to start with is **70%**, which is high enough to prevent over-provisioning, but low enough to ensure that the `Pods` can handle traffic spikes, by scaling up before the CPU usage reaches 100%.
194
+
195
+
As always, you should monitor your `Pods` and `Nodes` to ensure that the autoscaling is working as expected, and adjust the configuration if needed.
When working with Magento / Adobe Commerce, you'll need to store sensitive information, such as database credentials, API keys, etc.
8
+
9
+
There are many ways to manage secrets in a project:
10
+
11
+
* Stored encrypted in the codebase, using a tool like [sops](https://github.com/getsops/sops)
12
+
* Stored in a secret management system, like [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/), [Google Secret Manager](https://cloud.google.com/secret-manager), [Azure Key Vault](https://azure.microsoft.com/en-us/services/key-vault/), etc.
13
+
* Stored as CI variables in your CI/CD tool (GitHub Actions, GitLab CI, etc.) and injected at deploy time
14
+
15
+
Those sensitive values will eventually be injected as environment variables in your `Pods`, using Kubernetes `Secrets`.
16
+
17
+
The way you manage secrets in your project depends on your security requirements, team's workflow, and the tools you're using.
18
+
19
+
In this guide, we'll cover the usage of `sops` to encrypt secrets in your codebase, and how to use those encrypted secrets in your Kubernetes manifests.
20
+
21
+
## sops
22
+
23
+
`sops` is a tool that makes it easy to store, share, and manage secrets in your codebase.
24
+
25
+
It encrypts your secrets, stored in a YAML file, using a KMS key (AWS, GCP, etc.), an `age` key, or a GPG key.
26
+
27
+
You can then commit the encrypted file to your codebase, and decrypt it at deploy time. In our case, decrypting the file will be handled by [Helm secrets plugin](https://github.com/jkroepke/helm-secrets).
28
+
29
+
What's great about `sops` is that only the values are encrypted, not the structure of the file. This means that you can still version control the file, and see the changes made to it.
30
+
31
+
For example, this is what the update of the MySQL password in a `sops`-encrypted file using AWS KMS looks like:
0 commit comments