Amazon EC2 – Boot from EBS and AMI conversion

Amazon recently announced an important new feature for their Elastic Compute Cloud. Previously, each instance was based on an image that could be a maximum of 10 GB in size. So, each machine you brought up could have a root partition up to 10 GB in size and additional storage would need to be added in other ways. The size restriction alone is somewhat limiting. Amazon has not only addressed that, but given users some other very powerful abilities.

Now, you can define an image in an EBS snapshot. That means the size of your root partition can be as large as 1 TB. Yes, that’s 100 times larger than the old 10 GB limit. Beyond the obvious benefit of having larger images, you can also stop instances. Stopping an instance is different than terminating an instance. The distinction is important because stopping an instance is very much like hitting the “pause” button. It doesn’t take a lot to realize that pausing a running instance and being able to start it up again later is very powerful! Instances tend to boot faster off EBS. As  you might expect, if you create a really large volume for a root partition (like 100s of GBs), it will take longer to come up. That’s just because it takes longer to create larger volumes than smaller ones.

Let’s go further and look at how powerful it is to have snapshots as the basis for images. By having a snapshot that you can create EBS volumes from, that means you can mount a volume, based on your snapshot (which represents your image) and make modifications to it! This is immensely helpful when trying to make changes to an image. Previously, it was somewhat more awkward to modify an image. You actually had to boot it up and run it. But now, even if there is an error that prevents proper running, you can access the image storage and make changes. Very useful!

Of course judging by the number of public AMIs out there, there are a great number of images backed by S3 that people will want to convert. Towards this end, I came up with a script to convert AMIs from the old to the new style. Here’s the cliff’s notes version.

Use an instance in the same region as your image to do the following,

  • download the image bundle to the ephemeral store
  • unbundle the image (resulting in a single file)
  • create a temporary EBS volume in the same availability zone as the instance
  • attach the volume to your instance
  • copy the unbundled image onto the raw EBS volume
  • mount the EBS volume
  • edit /etc/fstab on the volume to remove the ephemeral store mount line
  • unmount and detach the volume
  • create a snapshot of the EBS volume
  • register the snapshot as an image, and you’re done!

During the private beta for this feature, I created an AMI to handle all of this, so you boot the AMI with a set of parameters and it does the dirty work. The script uses the standard API and AMI tools that Amazon supplies. I’ll roll that out on the public cloud shortly.

Here’s the interesting portion of the script (parsing arguments and setting up environment variable for the tools has been omitted) :

Using the AMI ID, get the manifest name and architecture
AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
MANIFEST=`echo $AMI_DESC | awk '{ print $3 }'`
ARCH=`echo $AMI_DESC | awk '{ print $7 }'`
MANIFEST_PATH=`dirname $MANIFEST`/
MANIFEST_PREFIX=`basename $MANIFEST |awk -F. '{ print $1 }'`

Download the bundle to /mnt

echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
/usr/local/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k pk.pem -p $MANIFEST_PREFIX -d /mnt

Unbundle the image into a single (rather large) file.


echo unbundling, this will take a while
/usr/local/bin/ec2-unbundle -k pk.pem -m /mnt/$MANIFEST_PREFIX.manifest.xml  -s /mnt -d /mnt

Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI. Using launch options I show at the end of this article, you can increase that at run time. Notice, the availability zone comes from instance metadata. We must wait till the volume is created before moving on.


ZONE=`curl http://169.254.169.254/latest/meta-data/placement/availability-zone`
VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk '{ print $2 }'`
STATUS=creating
while [ $STATUS != "available" ]
do
echo volume $STATUS, waiting for volume create...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk '{ print $5 }'`
done

Attach the volume

INST_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`
$EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV

Here’s where we turn the image into a real volume, using our old friend “dd”

echo copying image to volume, this will also take a while
dd if=/mnt/$MANIFEST_PREFIX of=$EBS_DEV

Mount the volume and remove ephemeral store entry from /etc/fstab. This is required because “Boot from EBS” doesn’t use the ephemeral store by default.

mount $EBS_DEV /perm
cat /perm/etc/fstab |grep -v mnt >/tmp/fstab
mv /perm/etc/fstab /perm/etc/fstab.bak
mv /tmp/fstab /perm/etc/

Then, unmount and detach the volume. We’re nearly there.

umount /perm
$EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID

Create a snapshot and wait for it to complete.

SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d "created by createAMI.sh" | awk '{ print $2 }'`
# now, wait for it
STATUS=pending
while [ $STATUS != "completed" ]
do
echo volume $STATUS, waiting for snap complete...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk '{ print $4 }'`
done

Finally, delete the volume and register the snapshot


$EC2_HOME/bin/ec2delvol $VOL_ID
$EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d $DESCR -n $MANIFEST_PREFIX

To run your AMI with a larger root partition, use a command like this (which specifies 100GB);
  ec2-run-instances –key <KEYPAIR> –block-device-mapping /dev/sda1=:100 <AMI_ID>

34 thoughts on “Amazon EC2 – Boot from EBS and AMI conversion

    1. I’ll post the complete script. I also want to (over the next couple of days) make a client script and AMI available to handle the conversion.

  1. Thanks for the insights into how this process should work. One question: After creating and attaching the EBS volume to the instance doesn’t one have to create partitions and a filesystem on the volume before mounting it? This is where I got stuck following your procedure.

    1. Jeff, The beauty of using dd is that I take the filesystem used before and just copy it to the raw EBS device. After, I can mount and modify it before snapshotting and registering.

  2. After using this method, how do you utilize the allocated size of the EBS volume?

    I think since I followed your directions and used dd, my instance only sees 10GB for the root partition instead of the 20GB that I allocated. How do I fix this?

    Thanks

    1. I was thinking about this. In retrospect, I think it makes sense to make the new EBS volume 10GB, to match the largest S3 based AMI size. Then, via boot options, you can increase the size of the root partition further. I’ll get documentation together and update this blog post.

      Thanks,
      David

      1. David,

        Can you point me to the docs for the boot options that you are talking about? I have searched a bit to no avail.

      2. I’ve added the command into the end of the blog post. It was hard finding documentation. What I got came from somebody at Amazon directly. the block device mapping stuff does need better documentation. I suggest posted about this in the EC2 forum, and others will vote for it also!

  3. FYI, after some research, for me it was a simple as issuing:

    ec2run ami-XXXXXXXX -b /dev/sda1=:20

    and once it was up and going (for ext3):

    resize2fs /dev/sda1

    Currently creating a new image, hoping that I can launch without specifying the size every time.

  4. I think you missed a step in registering the snapshot. I had to make sure that I specified the kernel and ramdisk IDs that I am using. Without those I couldn’t boot my EBS snapshot.

    1. I did not have to do that for the small instances I was working with. But, I’m glad you figured this out since some people definitely would need this.

      David

  5. For anyone that’s interested, I turned the above snippets into a shell script with some modifications. Just use the following to pull down the instance store based AMI, copy the data with `dd` to a new EBS volume, take a snapshot, and lastly register the snapshot as an AMI.

    Of course, check the variables in the top section and modify to suit your needs.

    # My personal data:
    AMI_ID=ami-0XXXXXX # Instance AMI you wish to convert
    ACCESS_ID=XXXXXXXXXXXXXXXX # Copy/paste from YourAccount->SecurityCreds->AccessKeys->AccessKeyID
    SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX #Copy/paste from YourAccount->SecurityCreds->AccessKey->SecretAccessKey->Show
    PRIVATE_KEY=/mnt/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
    CERT=/mnt/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXX.pem
    EBS_DEV=/dev/sdh
    VOL_SIZE=11 #Size in GB. I picked 11, but change to suit your needs.
    DESCR=”DescriptionOfMyAMI”

    IFS=” AMI_DATA=`ec2dim -v $AMI_ID`
    ARCH=`echo $AMI_DATA | grep ‘architecture’ | sed -e ‘s/^.*architecture>\(.*\)\(.*\)\/\(.*\)\(.*\)/tmp/fstab
    mv /perm/etc/fstab /perm/etc/fstab.bak
    mv /tmp/fstab /perm/etc/

    echo “/etc/fstab has been moved to /perm/etc. Giving you some time to check it out …”
    sleep 30

    umount /perm
    $EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID
    SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d “created by George’s Modified createAMI.sh” | cut -f2`
    STATUS=pending
    echo volume $STATUS, waiting for snap complete…
    while ! echo $STATUS | grep -q “completed”
    do
    sleep 3
    STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | cut -f4`
    echo volume $STATUS, waiting for snap complete…
    done

    echo Deleting Volume $VOL_ID
    echo $EC2_HOME/bin/ec2delvol $VOL_ID
    $EC2_HOME/bin/ec2delvol $VOL_ID

    echo Registering EBS AMI …
    echo $EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d “$DESCR” -n ${MANIFEST_PREFIX}-ebs
    $EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d “$DESCR” -n ${MANIFEST_PREFIX}-ebs

  6. Interesting, and could maybe be quite helpful, assuming one desires to start from a public Instance-Store AMI.

    And that is what I did, many weeks ago, long before start-from-EBS was an option. What I’ve got now is a VERY private stuck-in-S3-AMI that has many hours of download, configure, compile, install, reconfigure, recompile, reinstall work. From within this running instance, almost none of the commands given above work, starting from the very first one (ec2dim wants to tell me all about public AMIs, which ain’t what I’m interested in converting).

    I hope someone somewhere will write instructions on how to convert a private AMI. I’m not so keen on starting from square one.

    1. What I did was geared towards private AMIs. In fact, you need to be the owner to make it work. The unbundling doesn’t work otherwise.

      1. Ah, I see.

        After more time passed, I re-googled the situation and found another approach. Please see

        http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html

        and see a comment from 2010-01-01 from “Anonymous” where the first unix command given is…
        dd bs=65536 if=/dev/sda1 of=/dev/sdb1

        I thought that approach had promise, because all the commands that needed to happen on the running unix instance were kinda native-OS stuff. Didn’t need to install the AWS stuff, didn’t need AWS-related env variables (I have those on my *local* machine, but usually not on my running AWS instances). Sure, a minor point, but it has only the stuff *I* need it to have, to do the job *I’ve* defined for it. Keeps things simpler for me, and let’s face it, I ain’t too bright.

        End result, it was fewer steps, worked pretty much on the first try except the author spelled one command “unmount” instead of “umount”.

  7. For anyone it may help, here’s the shell script that I created from this post.

    I have not tested this shell script as a complete end-to-end run! I just assembled the shell script as I typed the commands individually in a shell. I don’t know when I’ll need to convert other AMIs, so I decided to go ahead and post this draft script now instead of in the never-never.

    This is a much more literal interpretation of the article than the script posted by George.

    #! /bin/ksh

    # Script based on dkavanagh’s blog entry:
    # http://coderslike.us/2009/12/07/amazon-ec2-boot-from-ebs-and-ami-conversion/

    AMI_ID=”$0″
    NEW_AMI_DESCRIPTION=”$1″
    EBS_DEV=”$2″
    ACCESS_ID=”$3″
    SECRET_KEY=”$4″
    KEY_FILE=”$5″
    CERT_FILE=”$6″

    # Using the AMI ID, get the manifest name and architecture

    AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
    MANIFEST=`echo $AMI_DESC | awk ‘{ print $3 }’`
    ARCH=`echo $AMI_DESC | awk ‘{ print $7 }’`
    MANIFEST_PATH=`dirname $MANIFEST`/
    MANIFEST_PREFIX=`basename $MANIFEST | awk -F. ‘{ print $1 }’`

    # Download the bundle to /mnt

    BUNDLE_DIR=/mnt/convertS3AmiToEbs
    echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
    rm -rf $BUNDLE_DIR
    mkdir $BUNDLE_DIR
    $EC2_AMITOOL_HOME/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k $KEY_FILE -p $MANIFEST_PREFIX -d $BUNDLE_DIR

    # Unbundle the image into a single (rather large) file.

    echo unbundling, this will take a while
    $EC2_AMITOOL_HOME/bin/ec2-unbundle -k $KEY_FILE -m $BUNDLE_DIR/$MANIFEST_PREFIX.manifest.xml -s $BUNDLE_DIR -d $BUNDLE_DIR

    # Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI.

    ZONE=`curl –silent –retry 0 http://169.254.169.254/latest/meta-data/placement/availability-zone`
    VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk ‘{ print $2 }’`
    STATUS=creating
    while [[ $STATUS != “available” ]]
    do
    echo volume $STATUS, waiting for volume create…
    sleep 3
    STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk ‘{ print $5 }’`
    done

    # Attach the volume

    INST_ID=`curl –silent http://169.254.169.254/latest/meta-data/instance-id`
    $EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV

    # Here’s where we turn the image into a real volume, using our old friend dd

    echo copying image to volume, this will also take a while
    dd if=$BUNDLE_DIR/$MANIFEST_PREFIX of=$EBS_DEV

    # Mount the volume and remove ephemeral store entry from /etc/fstab.
    # This is required because “Boot from EBS” doesn’t use the ephemeral store by default.

    MOUNT_POINT=/convertS3AmiToEbs

    mkdir $MOUNT_POINT
    mount $EBS_DEV $MOUNT_POINT
    cat $MOUNT_POINT/etc/fstab | grep -v mnt > /tmp/fstab
    mv $MOUNT_POINT/etc/fstab $MOUNT_POINT/etc/fstab.bak
    mv /tmp/fstab $MOUNT_POINT/etc/

    # Unmount and detach the volume.

    umount $MOUNT_POINT
    $EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID

    # Create a snapshot and wait for it to complete.

    SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d “created by convertS3AmiToEbs.sh” | awk ‘{ print $2 }’`
    # now, wait for it
    STATUS=pending
    while [[ $STATUS != “completed” ]]
    do
    echo volume $STATUS, waiting for snap complete…
    sleep 3
    STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk ‘{ print $4 }’`
    done

    # delete the volume and register the snapshot

    $EC2_HOME/bin/ec2delvol $VOL_ID
    $EC2_HOME/bin/ec2reg -C $CERT_FILE -K $KEY_FILE -s $SNAP_ID -a $ARCH -d “$NEW_AMI_DESCRIPTION” -n $MANIFEST_PREFIX

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s