Amazon recently announced an important new feature for their Elastic Compute Cloud. Previously, each instance was based on an image that could be a maximum of 10 GB in size. So, each machine you brought up could have a root partition up to 10 GB in size and additional storage would need to be added in other ways. The size restriction alone is somewhat limiting. Amazon has not only addressed that, but given users some other very powerful abilities.
Now, you can define an image in an EBS snapshot. That means the size of your root partition can be as large as 1 TB. Yes, that’s 100 times larger than the old 10 GB limit. Beyond the obvious benefit of having larger images, you can also stop instances. Stopping an instance is different than terminating an instance. The distinction is important because stopping an instance is very much like hitting the “pause” button. It doesn’t take a lot to realize that pausing a running instance and being able to start it up again later is very powerful! Instances tend to boot faster off EBS. As you might expect, if you create a really large volume for a root partition (like 100s of GBs), it will take longer to come up. That’s just because it takes longer to create larger volumes than smaller ones.
Let’s go further and look at how powerful it is to have snapshots as the basis for images. By having a snapshot that you can create EBS volumes from, that means you can mount a volume, based on your snapshot (which represents your image) and make modifications to it! This is immensely helpful when trying to make changes to an image. Previously, it was somewhat more awkward to modify an image. You actually had to boot it up and run it. But now, even if there is an error that prevents proper running, you can access the image storage and make changes. Very useful!
Of course judging by the number of public AMIs out there, there are a great number of images backed by S3 that people will want to convert. Towards this end, I came up with a script to convert AMIs from the old to the new style. Here’s the cliff’s notes version.
Use an instance in the same region as your image to do the following,
- download the image bundle to the ephemeral store
- unbundle the image (resulting in a single file)
- create a temporary EBS volume in the same availability zone as the instance
- attach the volume to your instance
- copy the unbundled image onto the raw EBS volume
- mount the EBS volume
- edit /etc/fstab on the volume to remove the ephemeral store mount line
- unmount and detach the volume
- create a snapshot of the EBS volume
- register the snapshot as an image, and you’re done!
During the private beta for this feature, I created an AMI to handle all of this, so you boot the AMI with a set of parameters and it does the dirty work. The script uses the standard API and AMI tools that Amazon supplies. I’ll roll that out on the public cloud shortly.
Here’s the interesting portion of the script (parsing arguments and setting up environment variable for the tools has been omitted) :
AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
MANIFEST=`echo $AMI_DESC | awk '{ print $3 }'`
ARCH=`echo $AMI_DESC | awk '{ print $7 }'`
MANIFEST_PATH=`dirname $MANIFEST`/
MANIFEST_PREFIX=`basename $MANIFEST |awk -F. '{ print $1 }'`
Download the bundle to /mnt
echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX /usr/local/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k pk.pem -p $MANIFEST_PREFIX -d /mnt
Unbundle the image into a single (rather large) file.
echo unbundling, this will take a while /usr/local/bin/ec2-unbundle -k pk.pem -m /mnt/$MANIFEST_PREFIX.manifest.xml -s /mnt -d /mnt
Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI. Using launch options I show at the end of this article, you can increase that at run time. Notice, the availability zone comes from instance metadata. We must wait till the volume is created before moving on.
ZONE=`curl http://169.254.169.254/latest/meta-data/placement/availability-zone`
VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk '{ print $2 }'`
STATUS=creating
while [ $STATUS != "available" ]
do
echo volume $STATUS, waiting for volume create...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk '{ print $5 }'`
done
Attach the volume
INST_ID=`curl http://169.254.169.254/latest/meta-data/instance-id` $EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV
Here’s where we turn the image into a real volume, using our old friend “dd”
echo copying image to volume, this will also take a while dd if=/mnt/$MANIFEST_PREFIX of=$EBS_DEV
Mount the volume and remove ephemeral store entry from /etc/fstab. This is required because “Boot from EBS” doesn’t use the ephemeral store by default.
mount $EBS_DEV /perm cat /perm/etc/fstab |grep -v mnt >/tmp/fstab mv /perm/etc/fstab /perm/etc/fstab.bak mv /tmp/fstab /perm/etc/
Then, unmount and detach the volume. We’re nearly there.
umount /perm $EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID
Create a snapshot and wait for it to complete.
SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d "created by createAMI.sh" | awk '{ print $2 }'`
# now, wait for it
STATUS=pending
while [ $STATUS != "completed" ]
do
echo volume $STATUS, waiting for snap complete...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk '{ print $4 }'`
done
Finally, delete the volume and register the snapshot
$EC2_HOME/bin/ec2delvol $VOL_ID $EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d $DESCR -n $MANIFEST_PREFIX
To run your AMI with a larger root partition, use a command like this (which specifies 100GB);
ec2-run-instances –key <KEYPAIR> –block-device-mapping /dev/sda1=:100 <AMI_ID>
Coders Like Us 
How do I get the full script without any omitted portions and with the environment variable setup steps in it as well?
I’ll post the complete script. I also want to (over the next couple of days) make a client script and AMI available to handle the conversion.
Many thanks for this — it gave me pointers in the right direction for several steps! You made my life easier today.
Thanks for the insights into how this process should work. One question: After creating and attaching the EBS volume to the instance doesn’t one have to create partitions and a filesystem on the volume before mounting it? This is where I got stuck following your procedure.
Jeff, The beauty of using dd is that I take the filesystem used before and just copy it to the raw EBS device. After, I can mount and modify it before snapshotting and registering.
Oops. Cockpit error. Missed the ‘dd’ step.
After using this method, how do you utilize the allocated size of the EBS volume?
I think since I followed your directions and used dd, my instance only sees 10GB for the root partition instead of the 20GB that I allocated. How do I fix this?
Thanks
I was thinking about this. In retrospect, I think it makes sense to make the new EBS volume 10GB, to match the largest S3 based AMI size. Then, via boot options, you can increase the size of the root partition further. I’ll get documentation together and update this blog post.
Thanks,
David
David,
Can you point me to the docs for the boot options that you are talking about? I have searched a bit to no avail.
I’ve added the command into the end of the blog post. It was hard finding documentation. What I got came from somebody at Amazon directly. the block device mapping stuff does need better documentation. I suggest posted about this in the EC2 forum, and others will vote for it also!
I just started the forum thread you suggested:
http://developer.amazonwebservices.com/connect/thread.jspa?threadID=39819
BTW, there’s a good discussion on the block device mappings here; http://alestic.com/2009/12/ec2-ebs-boot-resize
David,
I’ve used also your post as reference to my post on another way of creating EBS AMI. 😉
http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html
Rodney
FYI, after some research, for me it was a simple as issuing:
ec2run ami-XXXXXXXX -b /dev/sda1=:20
and once it was up and going (for ext3):
resize2fs /dev/sda1
Currently creating a new image, hoping that I can launch without specifying the size every time.
I think you missed a step in registering the snapshot. I had to make sure that I specified the kernel and ramdisk IDs that I am using. Without those I couldn’t boot my EBS snapshot.
I did not have to do that for the small instances I was working with. But, I’m glad you figured this out since some people definitely would need this.
David
For anyone that’s interested, I turned the above snippets into a shell script with some modifications. Just use the following to pull down the instance store based AMI, copy the data with `dd` to a new EBS volume, take a snapshot, and lastly register the snapshot as an AMI.
Of course, check the variables in the top section and modify to suit your needs.
# My personal data:
AMI_ID=ami-0XXXXXX # Instance AMI you wish to convert
ACCESS_ID=XXXXXXXXXXXXXXXX # Copy/paste from YourAccount->SecurityCreds->AccessKeys->AccessKeyID
SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX #Copy/paste from YourAccount->SecurityCreds->AccessKey->SecretAccessKey->Show
PRIVATE_KEY=/mnt/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
CERT=/mnt/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXX.pem
EBS_DEV=/dev/sdh
VOL_SIZE=11 #Size in GB. I picked 11, but change to suit your needs.
DESCR=”DescriptionOfMyAMI”
IFS=” AMI_DATA=`ec2dim -v $AMI_ID`
ARCH=`echo $AMI_DATA | grep ‘architecture’ | sed -e ‘s/^.*architecture>\(.*\)\(.*\)\/\(.*\)\(.*\)/tmp/fstab
mv /perm/etc/fstab /perm/etc/fstab.bak
mv /tmp/fstab /perm/etc/
echo “/etc/fstab has been moved to /perm/etc. Giving you some time to check it out …”
sleep 30
umount /perm
$EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID
SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d “created by George’s Modified createAMI.sh” | cut -f2`
STATUS=pending
echo volume $STATUS, waiting for snap complete…
while ! echo $STATUS | grep -q “completed”
do
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | cut -f4`
echo volume $STATUS, waiting for snap complete…
done
echo Deleting Volume $VOL_ID
echo $EC2_HOME/bin/ec2delvol $VOL_ID
$EC2_HOME/bin/ec2delvol $VOL_ID
echo Registering EBS AMI …
echo $EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d “$DESCR” -n ${MANIFEST_PREFIX}-ebs
$EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d “$DESCR” -n ${MANIFEST_PREFIX}-ebs
Interesting, and could maybe be quite helpful, assuming one desires to start from a public Instance-Store AMI.
And that is what I did, many weeks ago, long before start-from-EBS was an option. What I’ve got now is a VERY private stuck-in-S3-AMI that has many hours of download, configure, compile, install, reconfigure, recompile, reinstall work. From within this running instance, almost none of the commands given above work, starting from the very first one (ec2dim wants to tell me all about public AMIs, which ain’t what I’m interested in converting).
I hope someone somewhere will write instructions on how to convert a private AMI. I’m not so keen on starting from square one.
What I did was geared towards private AMIs. In fact, you need to be the owner to make it work. The unbundling doesn’t work otherwise.
Ah, I see.
After more time passed, I re-googled the situation and found another approach. Please see
http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html
and see a comment from 2010-01-01 from “Anonymous” where the first unix command given is…
dd bs=65536 if=/dev/sda1 of=/dev/sdb1
I thought that approach had promise, because all the commands that needed to happen on the running unix instance were kinda native-OS stuff. Didn’t need to install the AWS stuff, didn’t need AWS-related env variables (I have those on my *local* machine, but usually not on my running AWS instances). Sure, a minor point, but it has only the stuff *I* need it to have, to do the job *I’ve* defined for it. Keeps things simpler for me, and let’s face it, I ain’t too bright.
End result, it was fewer steps, worked pretty much on the first try except the author spelled one command “unmount” instead of “umount”.
For anyone it may help, here’s the shell script that I created from this post.
I have not tested this shell script as a complete end-to-end run! I just assembled the shell script as I typed the commands individually in a shell. I don’t know when I’ll need to convert other AMIs, so I decided to go ahead and post this draft script now instead of in the never-never.
This is a much more literal interpretation of the article than the script posted by George.
#! /bin/ksh
# Script based on dkavanagh’s blog entry:
# https://coderslike.us/2009/12/07/amazon-ec2-boot-from-ebs-and-ami-conversion/
AMI_ID=”$0″
NEW_AMI_DESCRIPTION=”$1″
EBS_DEV=”$2″
ACCESS_ID=”$3″
SECRET_KEY=”$4″
KEY_FILE=”$5″
CERT_FILE=”$6″
# Using the AMI ID, get the manifest name and architecture
AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
MANIFEST=`echo $AMI_DESC | awk ‘{ print $3 }’`
ARCH=`echo $AMI_DESC | awk ‘{ print $7 }’`
MANIFEST_PATH=`dirname $MANIFEST`/
MANIFEST_PREFIX=`basename $MANIFEST | awk -F. ‘{ print $1 }’`
# Download the bundle to /mnt
BUNDLE_DIR=/mnt/convertS3AmiToEbs
echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
rm -rf $BUNDLE_DIR
mkdir $BUNDLE_DIR
$EC2_AMITOOL_HOME/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k $KEY_FILE -p $MANIFEST_PREFIX -d $BUNDLE_DIR
# Unbundle the image into a single (rather large) file.
echo unbundling, this will take a while
$EC2_AMITOOL_HOME/bin/ec2-unbundle -k $KEY_FILE -m $BUNDLE_DIR/$MANIFEST_PREFIX.manifest.xml -s $BUNDLE_DIR -d $BUNDLE_DIR
# Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI.
ZONE=`curl –silent –retry 0 http://169.254.169.254/latest/meta-data/placement/availability-zone`
VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk ‘{ print $2 }’`
STATUS=creating
while [[ $STATUS != “available” ]]
do
echo volume $STATUS, waiting for volume create…
sleep 3
STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk ‘{ print $5 }’`
done
# Attach the volume
INST_ID=`curl –silent http://169.254.169.254/latest/meta-data/instance-id`
$EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV
# Here’s where we turn the image into a real volume, using our old friend dd
echo copying image to volume, this will also take a while
dd if=$BUNDLE_DIR/$MANIFEST_PREFIX of=$EBS_DEV
# Mount the volume and remove ephemeral store entry from /etc/fstab.
# This is required because “Boot from EBS” doesn’t use the ephemeral store by default.
MOUNT_POINT=/convertS3AmiToEbs
mkdir $MOUNT_POINT
mount $EBS_DEV $MOUNT_POINT
cat $MOUNT_POINT/etc/fstab | grep -v mnt > /tmp/fstab
mv $MOUNT_POINT/etc/fstab $MOUNT_POINT/etc/fstab.bak
mv /tmp/fstab $MOUNT_POINT/etc/
# Unmount and detach the volume.
umount $MOUNT_POINT
$EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID
# Create a snapshot and wait for it to complete.
SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d “created by convertS3AmiToEbs.sh” | awk ‘{ print $2 }’`
# now, wait for it
STATUS=pending
while [[ $STATUS != “completed” ]]
do
echo volume $STATUS, waiting for snap complete…
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk ‘{ print $4 }’`
done
# delete the volume and register the snapshot
$EC2_HOME/bin/ec2delvol $VOL_ID
$EC2_HOME/bin/ec2reg -C $CERT_FILE -K $KEY_FILE -s $SNAP_ID -a $ARCH -d “$NEW_AMI_DESCRIPTION” -n $MANIFEST_PREFIX
Dean, the comment system on this site murdered your shell script by replacing the quotes with 4 kinds of unicode quote looking things. Would you mind pasting it on http://codepad.ord or something?