Benefits of EBS vs. instance-store (and vice-versa

Question

I'm unclear as to what benefits I get from EBS vs. instance-store for my instances on Amazon EC2. If anything, it seems that EBS is way more useful (stop, start, persist + better speed) at relatively little difference in cost...? Also, is there any metric as to whether more people are using EBS now that it's available, considering it is still relatively new?

Answer 1

The EC2 "Hardware"

When an EC2 instance is launched, a virtual machine is reserved for the instance to run. That virtual machine has particular specifications depending on the instance type: 32-bit or 64-bit CPU, number of virtual cores, size of hard drive, etc. Details about the instance specs is available at http://aws.amazon.com/ec2/#instance.

When your EC2 instance is in a "running" state, that means that it’s running on the virtual machine, and this is what you get charged for.

The virtual machine’s hard drive is considered "ephemeral". The term "ephemeral" comes from the greek word "ephemeros" which means "lasting only one day". Anything on such a hard drive should be considered temporary. Unless the data is copied off the hard drive, if the virtual machine is stopped, then the data is lost. This includes data, software, and even an operating system that resides on those hard drives.

Amazon Web Services provides EC2 instances with two types of root devices.: "EBS-backed" and "instance store".

"Instance Store" Instances

An "instance store" instance is an EC2 instance whose root device resides on the virtual machine’s hard drive. When the instance is created, the base AMI is copied to the virtual machine’s hard drive and launched. The instance can run for as long as you want, but it cannot be stopped. Since the instance’s root device is the actual hard drive, it is "stuck" on the hardware, and the only thing you can do is terminate the instance. If you do this, the instance is deleted, never to be recovered. You also run the risk that if the virtual machine’s hardware fails, then you will also lose anything on the hard drive.

If you launch an "instance store" instance, be prepared to leave it running until you’re completely done with it. Note that you will be charged from the moment the instance is started, until the time it is terminated.

"EBS-Backed" Instances

An "EBS-backed" instance is an EC2 instance which uses an EBS volume as it’s root device. EBS volumes are redundant, "virtual" drives, which are not tied to any particular hardware, however they are restricted to a particular EC2 availability zone. This means that an EBS volume can move from one piece of hardware to another within the same availability zone. You can think of EBS volumes as a kind of Network Attached Storage.

If the virtual machine’s hardware fails, the EBS volume can simply be moved to another virtual machine and re-launched. In theory, you won’t lose any data.

Another benefit, is that EBS volumes can easily be backed up and duplicated. So you can take easy backup snapshots of your volumes, create new volumes and launch new EC2 instances based on those duplicate volumes.

Probably the biggest advantage "EBS-backed" instances have over "instance store" instances is that they can be stopped. When you do this, the virtual machine is shutdown and the EBS volume is stored for later retrieval. The hardware is then available for someone else to use. In addition, during this time, you are not charged the EC2 instance running charge. But you are charged for the EBS storage. When you want the instance to run again, you just start it up again. A new virtual machine is reserved, your EBS volume is attached, and your instance is booted.

But what about the virtual machine’s hard drives?

Yes, it is possible to use those hard drives, even when your EC2 instance is "EBS-backed". By default, they are not available. If you use the command line programs to launch your instance, you can use the "-b" option on the ec2-run-instances command to attach the "instance store" drives to your EC2 instance.

Having these drives available can be beneficial if you want to store temporary data. Read and write access should be faster than reading from and writing to an EBS volume because you’re not sending data over the network. In addition, you won’t be charged for data transfer or data storage. But this only works if the data can be lost at any time.

Source : https://skeddly.desk.com/customer/portal/articles/1346918-ebs-backed-versus-instance-store

Answer 2

99% of our AWS setup is recyclable. So for me it doesn't really matter if I terminate an instance -- nothing is lost ever. E.g. my application is automatically deployed on an instance from SVN, our logs are written to a central syslog server.

The only benefit of instance storage that I see are cost-savings. Otherwise EBS-backed instances win. Eric mentioned all the advantages.

---

[2012-07-16] I would phrase this answer a lot different today.

I haven't had any good experience with EBS-backed instances in the past year or so. The last downtimes on AWS pretty much wrecked EBS as well.

I am guessing that a service like RDS uses some kind of EBS as well and that seems to work for the most part. On the instances we manage ourselves, we have got rid off EBS where possible.

Getting rid to an extend where we moved a database cluster back to iron (= real hardware). The only remaining piece in our infrastructure is a DB server where we stripe multiple EBS volumes into a software RAID and backup twice a day. Whatever would be lost in between backups, we can live with.

EBS is a somewhat flakey technology since it's essentially a network volume: a volume attached to your server from remote. I am not negating the work done with it – it is an amazing product since essentially unlimited persistent storage is just an API call away. But it's hardly fit for scenarios where I/O performance is key.

And in addition to how network storage behaves, all network is shared on EC2 instances. The smaller an instance (e.g. t1.micro, m1.small) the worse it gets because your network interfaces on the actual host system are shared among multiple VMs (= your EC2 instance) which run on top of it.

The larger instance you get, the better it gets of course. Better here means within reason.

When persistence is required, I would always advice people to use something like S3 to centralize between instances. S3 is a very stable service. Then automate your instance setup to a point where you can boot a new server and it gets ready by itself. Then there is no need to have network storage which lives longer than the instance.

So all in all, I see no benefit to EBS-backed instances what so ever. I rather add a minute to bootstrap, then run with a potential SPOF.

Answer 3

I'm just starting to use EC2 myself so not an expert, but Amazon's own documentation says:

we recommend that you use the local instance store for temporary data and, for data requiring a higher level of durability, we recommend using Amazon EBS volumes or backing up the data to Amazon S3.

Emphasis mine.

I do more data analysis than web hosting, so persistence doesn't matter as much to me as it might for a web site. Given the distinction made by Amazon itself, I wouldn't assume that EBS is right for everyone.

I'll try to remember to weigh in again after I've used both.

Answer 4

The bottom line is you should almost always use EBS backed instances.

Here's why

• EBS backed instances can be set so that they cannot be (accidentally) terminated through the API.
• EBS backed instances can be stopped when you're not using them and resumed when you need them again (like pausing a Virtual PC), at least with my usage patterns saving much more money than I spend on a few dozen GB of EBS storage.
• EBS backed instances don't lose their instance storage when they crash (not a requirement for all users, but makes recovery much faster)
• You can dynamically resize EBS instance storage.
• You can transfer the EBS instance storage to a brand new instance (useful if the hardware at Amazon you were running on gets flaky or dies, which does happen from time to time)
• It is faster to launch an EBS backed instance because the image does not have to be fetched from S3.
• If the hardware your EBS-backed instance is scheduled for maintenance, stopping and starting the instance automatically migrates to new hardware. I was also able to move an EBS-backed instance on failed hardware by force-stopping the instance and launching it again (your mileage may vary on failed hardware).

I'm a heavy user of Amazon and switched all of my instances to EBS backed storage as soon as the technology came out of beta. I've been very happy with the result.

EBS can still fail - not a silver bullet

Keep in mind that any piece of cloud-based infrastructure can fail at any time. Plan your infrastructure accordingly. While EBS-backed instances provide certain level of durability compared to ephemeral storage instances, they can and do fail. Have an AMI from which you can launch new instances as needed in any availability zone, back up your important data (e.g. databases), and if your budget allows it, run multiple instances of servers for load balancing and redundancy (ideally in multiple availability zones).

When Not To

At some points in time, it may be cheaper to achieve faster IO on Instance Store instances. There was a time when it was certainly true. Now there are many options for EBS storage, catering to many needs. The options and their pricing evolve constantly as technology changes. If you have a significant amount of instances that are truly disposable (they don't affect your business much if they just go away), do the math on cost vs. performance. EBS-backed instances can also die at any point in time, but my practical experience is that EBS is more durable.

Answer 5

For someone new to all this and if accidentally landed here

As of now all AMI's in quickstart section are EBS backed

Also there's a good explanation at official doc for difference between EBS and Instance store

& this image pretty much sums it up

Answer 6

We like instance-store. It forces us to make our instances completely recyclable, and we can easily automate the process of building a server from scratch on a given AMI. This also means we can easily swap out AMIs. Also, EBS still has performance problems from time to time.