Why we use Amazon's cloud, or, how I saved a project from Waffle House

  Share

The day that DiSC launched the OWS project site, I, the sysadmin, was away at jury duty.

During my lunch break, I learned the server had crashed, likely because of a spike in traffic. Users trying to see the site were faced with the dreaded 500 error message. From a Waffle House in downtown Atlanta, I pulled out my laptop, got online and logged into our Amazon Web Services account. From there, I stopped the server and increased the size of the instance, which adds more RAM and CPU. I restarted it, and less than five minutes after I’d logged in, our site was running on a larger server that could handle the traffic. The site has run smoothly since then.

If we were in a traditional data center, or even on a virtual machine, the process would have been different. 

Depending on the virtual machine provider, we would’ve requested the provider increase our VM’s size or create a new, larger VM for us. We would move our data over, test and relaunch. The process could have taken a half-day or longer, especially because the sysadmin was doing his civic duty. Alternately, we could have invested more up front for a large server or VM, one that would have been underutilized until the excitement of launch day.

Much of the elastic cloud hype is, naturally, about elasticity -- the ability to scale up or down cheaply, quickly and automatically. In this case, it helped us keep our OWS site running on the anniversary of Occupy Wall Street.
 
DiSC is still a new entity and for now, we run only a handful of relatively low-traffic sites -- at least, most of the time. The scenario with the OWS site could’ve been handled by using AWS load balancers that dynamically create more server capacity based on load. At this time, that would be overkill for what we are doing, but it might work for us in the future.
 
There are other reasons why we chose EC2:
 
AWS shines for our backups disaster recovery needs.
We haven’t needed to use this yet, but when it happens, I’m confident it will work flawlessly. I relied on it when I worked in the corporate world, and don’t lose any sleep over it now. AWS provides the ability for us to copy our data at the block-level, an even more robust backup than at the file level. What does that mean? In the case of data loss, we don’t have to restore the files, we just make a new volume from the snapshot, attach it to our server and go. (Look for more on this in a future blog post.)
 
We can use it on projects large and small.
Say you have a massive dataset and want to have a weekend hackathon where people play with the data. With services like EC2, you can just spin up a cluster of servers for a weekend, or a few hours, and let people have at it. When you’re done, you just shut down the servers and stop paying for them. Any results that should be preserved can be stored locally or in cheap cloud services. The servers themselves can be imaged and started up at a later date, if you want.
 
Most importantly, EC2 has allowed us to experiment.
Paying for servers only when we need them means we can quickly, effectively and affordably try new things. For example, we wanted to see what it would take to run our own instance of DBpedia. Traditional VMs are purchased by the month. Physical servers are for life. EC2 servers are charged by the hour. We created an EC2 instance, installed DBpedia and started experimenting. We spent one day and about $15 trying it out and ultimately concluded that it took a really large server with a lot of RAM, and we couldn’t justify the cost. Still, experimenting wasn’t a waste of time. We had an answer quickly.
 
Once you get into the mindset of infrastructure as a service, you’ll likely come to think of infrastructure as an operational cost, not a capital investment; everything you ever wanted to try becomes feasible. We hadn’t spent time or money on a physical server or VM to test it. (If we had, we would’ve had to guess at how powerful, i.e expensive, the server would need to be.  We would likely be wrong. That money? Wasted.) Sure, our experiment failed. But if you never fail, you’re not trying.
 
Of course, it helps to fail in interesting ways, or near a place with 24/7 wifi and waffles.