Best Practices for Amazon Web Services (Amazon Cloud)

We run our ScrumPad (A Web-based project management tool) on Amazon cloud (AWS). We have been using Amazon cloud for last year and a half. We did not have any issue with it until recently, when we found out that we had lost shell connection to our production server. Although the application was working fine, we went into a panic mode realizing that we won't be able to do anything, should we encounter any problem. Once we calmed down from our initial panic attack, we quickly searched on the net to see if anyone experienced the same issue and how they recovered from it. We did not get any hit on it. There is always first time for everything. We thought to ourselves that the solution is easy with AWS. We would just bring up another EC2 instance and point to the new instance. Right? Not so fast. We found out that we were not using elastic IP. That means we need to make DNS change, which takes up to 24 hours to propagate and may result into some data loss and/or service downtime.

This incident helped us to think through how we should be using AWS and here is what we have come up with:

1. Always use elastic IP (behaves like a static ip) to point to your EC2 instance. You can then quickly swap instances without needing to make any DNS changes.

2. Always use EBS (elastic block storage) instead of the storage that comes with an EC2 instance to store your Database data . You can then quickly reattach your data store to a new instance as well as you will not loose your data in the event of EC2 instance failure and have your data automatically replicated.

3. Spread your EC2 instances to different availability zones and regions instead of putting them all in the same availability zone (a physical data center). You can then protect your application from single data center failure.

4. Backup (incremental snap shots) your database from EBS to S3 (Simple Storage Service), which is automatically replicated across multiple availability zones.

5. Create a custom AMI (Amazon Machine Image) for your application from a pre-configured public AMI. You can then bring up a new instance quickly (in less than 15 mins) on-demand and even completely automate server provisioning.

6. Rollout your new release to a new instance(s) when you need to change your AMI (i.e., when you upgrading your DB, OS, Application Server, etc.) and then reattach your elastic ip and EBS to this new instance. You can then quickly rollback to the old instance should you encounter any problem with the new release.

Figure: How to transparently switch to a new instance

7. If budget permits, use cloud management tool like RightScale.

With a few practices in place, you can have your own virtual data center (on Amazon cloud) that was only available to large corporations.