Architect to Survive a Data Center Fire

This week we’ve had an example of a disaster scenario at an OVH data center in Strasbourg, which took the entire data center offline as well as a couple of data centers located on the same site.

This incident took many popular websites, shops, and applications hosted in that data center completely offline and scrambling for disaster recovery.

As an engineer, this makes one think about the possibility of prevention of catastrophic failures like this one.

Although it’s easy to blame the data center and shrug off the incident as an impossible-to-prevent scenario, the reality is that applications can be architected for resiliency even to something as catastrophic as a data center going offline.

Multi-AZ

The first step is to build your shop so that it’s able to run on a multi-server architecture. If your application can’t run on multiple servers, you have no options to distribute it across multiple servers, and it will take much less than a data center fire to bring your shop offline.

Once your shop has the option of deployment on multiple servers and has a replicated database, it’s time to architect your hosting infrastructure to be deployed in multiple availability zones (AZs).

Most cloud providers have multiple availability zones in each of their regions, and all the services allow for a multi-AZ deployment.

At Zento, we’re running on AWS, and our EKS cluster and RDS databases rely on multi-AZ deployments, while managed services like Lambda, S3, or CloudFront are natively multi-AZ.

Backups & Disaster Recovery

Besides architecture for resiliency, backups must be frequent, so recovery to a recent version is possible when disaster strikes.

Backups must also be stored offsite or in a service like AWS S3 that has guaranteed replication across multiple data centers; otherwise, a catastrophic incident like the data center burning down will also destroy the backups.

Finally, you must create a disaster recovery plan in advance to allow for a quick reaction when needed.

Conclusion

Disaster prevention is the job of developers and DevOps, who need to think of resiliency while architecting their solution.

As a merchant, ask your development team for such resilient architectures and be willing to pay the associated cost. Prevention is much less expensive than the impact that a catastrophic incident would have.

Even with all the resiliency measures taken, prepare for the worse and hope it never happens: have multiple layers of backups available for restoration and prepare procedures to react in disaster scenarios.