07
Dec
09

Amazon EC2 – Boot from EBS and AMI conversion

Amazon recently announced an important new feature for their Elastic Compute Cloud. Previously, each instance was based on an image that could be a maximum of 10 GB in size. So, each machine you brought up could have a root partition up to 10 GB in size and additional storage would need to be added in other ways. The size restriction alone is somewhat limiting. Amazon has not only addressed that, but given users some other very powerful abilities.

Now, you can define an image in an EBS snapshot. That means the size of your root partition can be as large as 1 TB. Yes, that’s 100 times larger than the old 10 GB limit. Beyond the obvious benefit of having larger images, you can also stop instances. Stopping an instance is different than terminating an instance. The distinction is important because stopping an instance is very much like hitting the “pause” button. It doesn’t take a lot to realize that pausing a running instance and being able to start it up again later is very powerful! Instances tend to boot faster off EBS. As  you might expect, if you create a really large volume for a root partition (like 100s of GBs), it will take longer to come up. That’s just because it takes longer to create larger volumes than smaller ones.

Let’s go further and look at how powerful it is to have snapshots as the basis for images. By having a snapshot that you can create EBS volumes from, that means you can mount a volume, based on your snapshot (which represents your image) and make modifications to it! This is immensely helpful when trying to make changes to an image. Previously, it was somewhat more awkward to modify an image. You actually had to boot it up and run it. But now, even if there is an error that prevents proper running, you can access the image storage and make changes. Very useful!

Of course judging by the number of public AMIs out there, there are a great number of images backed by S3 that people will want to convert. Towards this end, I came up with a script to convert AMIs from the old to the new style. Here’s the cliff’s notes version.

Use an instance in the same region as your image to do the following,

  • download the image bundle to the ephemeral store
  • unbundle the image (resulting in a single file)
  • create a temporary EBS volume in the same availability zone as the instance
  • attach the volume to your instance
  • copy the unbundled image onto the raw EBS volume
  • mount the EBS volume
  • edit /etc/fstab on the volume to remove the ephemeral store mount line
  • unmount and detach the volume
  • create a snapshot of the EBS volume
  • register the snapshot as an image, and you’re done!

During the private beta for this feature, I created an AMI to handle all of this, so you boot the AMI with a set of parameters and it does the dirty work. The script uses the standard API and AMI tools that Amazon supplies. I’ll roll that out on the public cloud shortly.

Here’s the interesting portion of the script (parsing arguments and setting up environment variable for the tools has been omitted) :

Using the AMI ID, get the manifest name and architecture
AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
MANIFEST=`echo $AMI_DESC | awk '{ print $3 }'`
ARCH=`echo $AMI_DESC | awk '{ print $7 }'`
MANIFEST_PATH=`dirname $MANIFEST`/
MANIFEST_PREFIX=`basename $MANIFEST |awk -F. '{ print $1 }'`

Download the bundle to /mnt

echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
/usr/local/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k pk.pem -p $MANIFEST_PREFIX -d /mnt

Unbundle the image into a single (rather large) file.


echo unbundling, this will take a while
/usr/local/bin/ec2-unbundle -k pk.pem -m /mnt/$MANIFEST_PREFIX.manifest.xml  -s /mnt -d /mnt

Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI. Using launch options I show at the end of this article, you can increase that at run time. Notice, the availability zone comes from instance metadata. We must wait till the volume is created before moving on.


ZONE=`curl http://169.254.169.254/latest/meta-data/placement/availability-zone`
VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk '{ print $2 }'`
STATUS=creating
while [ $STATUS != "available" ]
do
echo volume $STATUS, waiting for volume create...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk '{ print $5 }'`
done

Attach the volume

INST_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`
$EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV

Here’s where we turn the image into a real volume, using our old friend “dd”

echo copying image to volume, this will also take a while
dd if=/mnt/$MANIFEST_PREFIX of=$EBS_DEV

Mount the volume and remove ephemeral store entry from /etc/fstab. This is required because “Boot from EBS” doesn’t use the ephemeral store by default.

mount $EBS_DEV /perm
cat /perm/etc/fstab |grep -v mnt >/tmp/fstab
mv /perm/etc/fstab /perm/etc/fstab.bak
mv /tmp/fstab /perm/etc/

Then, unmount and detach the volume. We’re nearly there.

umount /perm
$EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID

Create a snapshot and wait for it to complete.

SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d "created by createAMI.sh" | awk '{ print $2 }'`
# now, wait for it
STATUS=pending
while [ $STATUS != "completed" ]
do
echo volume $STATUS, waiting for snap complete...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk '{ print $4 }'`
done

Finally, delete the volume and register the snapshot


$EC2_HOME/bin/ec2delvol $VOL_ID
$EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d $DESCR -n $MANIFEST_PREFIX

To run your AMI with a larger root partition, use a command like this (which specifies 100GB);
  ec2-run-instances –key <KEYPAIR> –block-device-mapping /dev/sda1=:100 <AMI_ID>

About these ads

34 Responses to “Amazon EC2 – Boot from EBS and AMI conversion”


  1. 1 Lavon
    December 8, 2009 at 7:25 pm

    How do I get the full script without any omitted portions and with the environment variable setup steps in it as well?

    • 2 dkavanagh
      December 8, 2009 at 8:07 pm

      I’ll post the complete script. I also want to (over the next couple of days) make a client script and AMI available to handle the conversion.

  2. 3 eileen
    December 9, 2009 at 8:49 pm

    Many thanks for this — it gave me pointers in the right direction for several steps! You made my life easier today.

  3. 4 Jeff
    December 10, 2009 at 5:42 pm

    Thanks for the insights into how this process should work. One question: After creating and attaching the EBS volume to the instance doesn’t one have to create partitions and a filesystem on the volume before mounting it? This is where I got stuck following your procedure.

    • 5 dkavanagh
      December 10, 2009 at 5:56 pm

      Jeff, The beauty of using dd is that I take the filesystem used before and just copy it to the raw EBS device. After, I can mount and modify it before snapshotting and registering.

  4. 6 Jeff
    December 10, 2009 at 6:09 pm

    Oops. Cockpit error. Missed the ‘dd’ step.

  5. 7 mineshaftgap
    December 10, 2009 at 8:49 pm

    After using this method, how do you utilize the allocated size of the EBS volume?

    I think since I followed your directions and used dd, my instance only sees 10GB for the root partition instead of the 20GB that I allocated. How do I fix this?

    Thanks

  6. December 11, 2009 at 1:53 pm

    David,

    I’ve used also your post as reference to my post on another way of creating EBS AMI. ;)

    http://www.capsunlock.net/2009/12/create-ebs-boot-ami.html

    Rodney

  7. 14 mineshaftgap
    December 13, 2009 at 6:27 pm

    FYI, after some research, for me it was a simple as issuing:

    ec2run ami-XXXXXXXX -b /dev/sda1=:20

    and once it was up and going (for ext3):

    resize2fs /dev/sda1

    Currently creating a new image, hoping that I can launch without specifying the size every time.

  8. 15 Segedunum
    December 17, 2009 at 2:55 pm

    I think you missed a step in registering the snapshot. I had to make sure that I specified the kernel and ramdisk IDs that I am using. Without those I couldn’t boot my EBS snapshot.

    • 16 dkavanagh
      December 17, 2009 at 2:58 pm

      I did not have to do that for the small instances I was working with. But, I’m glad you figured this out since some people definitely would need this.

      David

  9. 17 George
    December 27, 2009 at 7:35 am

    For anyone that’s interested, I turned the above snippets into a shell script with some modifications. Just use the following to pull down the instance store based AMI, copy the data with `dd` to a new EBS volume, take a snapshot, and lastly register the snapshot as an AMI.

    Of course, check the variables in the top section and modify to suit your needs.

    # My personal data:
    AMI_ID=ami-0XXXXXX # Instance AMI you wish to convert
    ACCESS_ID=XXXXXXXXXXXXXXXX # Copy/paste from YourAccount->SecurityCreds->AccessKeys->AccessKeyID
    SECRET_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX #Copy/paste from YourAccount->SecurityCreds->AccessKey->SecretAccessKey->Show
    PRIVATE_KEY=/mnt/.ec2/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
    CERT=/mnt/.ec2/cert-XXXXXXXXXXXXXXXXXXXXXXXXXX.pem
    EBS_DEV=/dev/sdh
    VOL_SIZE=11 #Size in GB. I picked 11, but change to suit your needs.
    DESCR=”DescriptionOfMyAMI”

    IFS=” AMI_DATA=`ec2dim -v $AMI_ID`
    ARCH=`echo $AMI_DATA | grep ‘architecture’ | sed -e ‘s/^.*architecture>\(.*\)\(.*\)\/\(.*\)\(.*\)/tmp/fstab
    mv /perm/etc/fstab /perm/etc/fstab.bak
    mv /tmp/fstab /perm/etc/

    echo “/etc/fstab has been moved to /perm/etc. Giving you some time to check it out …”
    sleep 30

    umount /perm
    $EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID
    SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d “created by George’s Modified createAMI.sh” | cut -f2`
    STATUS=pending
    echo volume $STATUS, waiting for snap complete…
    while ! echo $STATUS | grep -q “completed”
    do
    sleep 3
    STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | cut -f4`
    echo volume $STATUS, waiting for snap complete…
    done

    echo Deleting Volume $VOL_ID
    echo $EC2_HOME/bin/ec2delvol $VOL_ID
    $EC2_HOME/bin/ec2delvol $VOL_ID

    echo Registering EBS AMI …
    echo $EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d “$DESCR” -n ${MANIFEST_PREFIX}-ebs
    $EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d “$DESCR” -n ${MANIFEST_PREFIX}-ebs

  10. 18 KDOgg
    December 31, 2009 at 9:18 pm

    Interesting, and could maybe be quite helpful, assuming one desires to start from a public Instance-Store AMI.

    And that is what I did, many weeks ago, long before start-from-EBS was an option. What I’ve got now is a VERY private stuck-in-S3-AMI that has many hours of download, configure, compile, install, reconfigure, recompile, reinstall work. From within this running instance, almost none of the commands given above work, starting from the very first one (ec2dim wants to tell me all about public AMIs, which ain’t what I’m interested in converting).

    I hope someone somewhere will write instructions on how to convert a private AMI. I’m not so keen on starting from square one.

    • 19 dkavanagh
      December 31, 2009 at 10:11 pm

      What I did was geared towards private AMIs. In fact, you need to be the owner to make it work. The unbundling doesn’t work otherwise.

      • 20 KDOgg
        January 14, 2010 at 4:52 pm

        Ah, I see.

        After more time passed, I re-googled the situation and found another approach. Please see

        http://www.elastician.com/2009/12/creating-ebs-backed-ami-from-s3-backed.html

        and see a comment from 2010-01-01 from “Anonymous” where the first unix command given is…
        dd bs=65536 if=/dev/sda1 of=/dev/sdb1

        I thought that approach had promise, because all the commands that needed to happen on the running unix instance were kinda native-OS stuff. Didn’t need to install the AWS stuff, didn’t need AWS-related env variables (I have those on my *local* machine, but usually not on my running AWS instances). Sure, a minor point, but it has only the stuff *I* need it to have, to do the job *I’ve* defined for it. Keeps things simpler for me, and let’s face it, I ain’t too bright.

        End result, it was fewer steps, worked pretty much on the first try except the author spelled one command “unmount” instead of “umount”.

  11. 21 Dean Thompson
    March 1, 2010 at 6:02 pm

    For anyone it may help, here’s the shell script that I created from this post.

    I have not tested this shell script as a complete end-to-end run! I just assembled the shell script as I typed the commands individually in a shell. I don’t know when I’ll need to convert other AMIs, so I decided to go ahead and post this draft script now instead of in the never-never.

    This is a much more literal interpretation of the article than the script posted by George.

    #! /bin/ksh

    # Script based on dkavanagh’s blog entry:
    # http://coderslike.us/2009/12/07/amazon-ec2-boot-from-ebs-and-ami-conversion/

    AMI_ID=”$0″
    NEW_AMI_DESCRIPTION=”$1″
    EBS_DEV=”$2″
    ACCESS_ID=”$3″
    SECRET_KEY=”$4″
    KEY_FILE=”$5″
    CERT_FILE=”$6″

    # Using the AMI ID, get the manifest name and architecture

    AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
    MANIFEST=`echo $AMI_DESC | awk ‘{ print $3 }’`
    ARCH=`echo $AMI_DESC | awk ‘{ print $7 }’`
    MANIFEST_PATH=`dirname $MANIFEST`/
    MANIFEST_PREFIX=`basename $MANIFEST | awk -F. ‘{ print $1 }’`

    # Download the bundle to /mnt

    BUNDLE_DIR=/mnt/convertS3AmiToEbs
    echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
    rm -rf $BUNDLE_DIR
    mkdir $BUNDLE_DIR
    $EC2_AMITOOL_HOME/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k $KEY_FILE -p $MANIFEST_PREFIX -d $BUNDLE_DIR

    # Unbundle the image into a single (rather large) file.

    echo unbundling, this will take a while
    $EC2_AMITOOL_HOME/bin/ec2-unbundle -k $KEY_FILE -m $BUNDLE_DIR/$MANIFEST_PREFIX.manifest.xml -s $BUNDLE_DIR -d $BUNDLE_DIR

    # Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI.

    ZONE=`curl –silent –retry 0 http://169.254.169.254/latest/meta-data/placement/availability-zone`
    VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk ‘{ print $2 }’`
    STATUS=creating
    while [[ $STATUS != "available" ]]
    do
    echo volume $STATUS, waiting for volume create…
    sleep 3
    STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk ‘{ print $5 }’`
    done

    # Attach the volume

    INST_ID=`curl –silent http://169.254.169.254/latest/meta-data/instance-id`
    $EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV

    # Here’s where we turn the image into a real volume, using our old friend dd

    echo copying image to volume, this will also take a while
    dd if=$BUNDLE_DIR/$MANIFEST_PREFIX of=$EBS_DEV

    # Mount the volume and remove ephemeral store entry from /etc/fstab.
    # This is required because “Boot from EBS” doesn’t use the ephemeral store by default.

    MOUNT_POINT=/convertS3AmiToEbs

    mkdir $MOUNT_POINT
    mount $EBS_DEV $MOUNT_POINT
    cat $MOUNT_POINT/etc/fstab | grep -v mnt > /tmp/fstab
    mv $MOUNT_POINT/etc/fstab $MOUNT_POINT/etc/fstab.bak
    mv /tmp/fstab $MOUNT_POINT/etc/

    # Unmount and detach the volume.

    umount $MOUNT_POINT
    $EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID

    # Create a snapshot and wait for it to complete.

    SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d “created by convertS3AmiToEbs.sh” | awk ‘{ print $2 }’`
    # now, wait for it
    STATUS=pending
    while [[ $STATUS != "completed" ]]
    do
    echo volume $STATUS, waiting for snap complete…
    sleep 3
    STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk ‘{ print $4 }’`
    done

    # delete the volume and register the snapshot

    $EC2_HOME/bin/ec2delvol $VOL_ID
    $EC2_HOME/bin/ec2reg -C $CERT_FILE -K $KEY_FILE -s $SNAP_ID -a $ARCH -d “$NEW_AMI_DESCRIPTION” -n $MANIFEST_PREFIX

  12. 22 Greg
    June 21, 2010 at 1:57 am

    Dean, the comment system on this site murdered your shell script by replacing the quotes with 4 kinds of unicode quote looking things. Would you mind pasting it on http://codepad.ord or something?


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

%d bloggers like this: