Running Magento 2 on Kubernetes (EKS)
- technical
We’ve been running Magento on Kubernetes since 2017 on our Magento 1 product. At that time, AWS did not have EKS (Elastic Kubernetes Service), so we’ve had to build our very own architecture; it was fun, but it was not easy.
Today, we’re running our Zento cluster on EKS, and although AWS does most of the heavy lifting, there was still a lot to do to get Magento 2 to run on Kubernetes.
In this article, we’ll cover the following:
- our setup for routing
- file storage
- security
- scaling
- monitoring
Our special setup
To route user requests that hit our CloudFront CDN distribution, we’re using ALB (Application Load Balancer), which can route the traffic from Cloudfront to the relevant namespace and service on EKS.
Zento is a SaaS solution, so our Kubernetes deployment is multi-tenant. To not require a separate ALB for each shop, we use target groups in the ALB to identify the target namespace.
File Storage
The first challenge you face is regarding the data files Magento stores on disc: media files and logs.
One solution is to use EFS (Elastic File System), something we did on our Magento 1 solution. However, this solution has several downsides:
- it can be slow (you get a burst balance for IOPS which you can burn-trough very quickly)
- it’s very CPU expensive to mount on each pod (Linux does a lot of indexing when mounting a large partition)
- it’s relatively expensive
The better solution is to use S3 for file storage, but this does require changes to the way file storage works in Magento 2. Today, the Magento team is working on making this available in the core. Until that is ready, you’ll have to extend Magento.
To resolve the logs on disk, we built an agent, based on fluentd, that takes the Magento 2 logs and streams them into CloudWatch Logs. This way, developers always have access to the log files on all pods, regardless of what pod generated them and if that pod is still available or not.
Security
All the credentials are stored in AWS’s SSM Parameter Store and deployed to the Kubernetes namespace into a namespace-specific configmap.
Access to other AWS services, like S3 or SES, is made through an IAM pod role. It is a secure and elegant way that requires no access-key management.
The entire EKS cluster runs in a private VPC so that access to it is made only through the proper channels. It also assures that traffic between EKS and the MySQL database is secured.
Scaling
One of the main selling points of Kubernetes is scaling, so configuring proper auto-scaling is paramount.
At Zento, we’ve created two node pools:
- on-demand nodes to host pods that serve GraphQL and admin requests
- spot nodes for consumer jobs
Both pools scale up and down based on the pods that need deployment. Pods scale up and down inside each namespace based on the volume of incoming requests or the number of batch jobs that require processing.
Monitoring
A critical aspect of any DevOps workflow is monitoring, so we monitor everything. Besides the access logs that we get from CloudFront, all the logs generated by Magento 2 and the PHP processes we stream to CloudWatch Logs through fluentd.
To monitor resource consumption inside EKS, the Container Insights service offered by AWS is an excellent solution. However, it’s priced based on the number of metrics generated by namespaces and pods, so it was too expensive for our setup. As a consequence, we ended up using Prometheus, which AWS offers as a managed service.
A Blackfire agent is present in every namespace. It allows our DevOps team insights into what processes consume resources so we can optimize them.
Conclusion
Deploying Magento 2 on Kubernetes is recommended for big setups:
- it offers the best scaling architecture
- it forces healthy best practices
- it offers good maintainability
For code deployments, we have a solution based on GitHub Actions, Step Functions, and CodeBuild to get the Docker images into ECR and then used them to deploy the pods. We’ll cover this in more detail in a future article.
