Architecting for the unexpected

You have an awesome application and everything is running very smoothly. You get more users by the hour and your sitting in the sun enjoying the great times. But suddenly you get a call from a friend saying your application is offline?!

Your disk has run out of capacity and must be enlarged. This is going to take at least an hour. But because of this diskdrive being full, your database has gone corrupt. You remember you had made a backup, last week! What to do?!

Web Application

To prevent anything like this from happening you must architect your structure to be redundant, scale up and self-heal when any kind of problem shows up. Every component inside your architecture must be able to replace itself when it is broken. When a component can not be repaired (quickly) you must be able to switch to a backup structure which will be powered on right this instant.

When a DNS lookup is done, it will be automatically redirected to the closest datacenter available for low latency with the Latency Based Routing (LBR) technique. Next the Route53 DNS service will assign a Elastic Load Balancer (ELB) that will handle your request. This ELB assigns you to a Availability Zone (AZ). Your request might be in the cache and served when it does. If not, the Internal ELB assigns you an application server. This will utilize the database application, which is a complete structure on its own. (as read below)

As you can see in the schema, every component is replaceable and the the servers are stateless so they can be easily replaced with new fresh instances. The state of the application is on redundant storage (S3).

Database Application

A standalone application is the database. It is a very crucial component with high complex technology. The database needs to be very scaleable as more requests come in. Also it needs to keep replicating itself across multiple instances and Availibility Zones within multiple data-centers. When instances go down (and they will) you can easily launch new ones automatically.

The database application is only available internally and will be placed inside a Virtual Private Cloud. Though its accessibly via a CNAME record within Route53 to remove any IP adresses, as they may change over time. As every data-center has its own replicated database application, you could even point the CNAME to a different data-center when the local database application is unavailable. The Elastic Load Balancer will devide the load across multiple instances, keeping in mind that reads go to the slaves and writes to the master instance.

The (binary) database files are automatically copied to S3 to perform disaster recovery when the database application can not relaunch itself.