What we came up with is eustore. In the spirit of euca2ools (euca- and euare- commands), eustore commands give  you access to a Eucalyptus image store. That’s store, as in storehouse, not a shop. We have some updated “base” images available on our servers. We have a catalog file that contains metadata about those images. The eustore tools simply give you access to those, and let you issue a single command to download an install an image on your local cloud (or any Eucalyptus cloud you have access to).

The code has been checked in with the euca2ools. To install and use the commands, you’ll need to install from source. Let’s go over that now.

A python package call “boto” is required and you’ll get that from github.com

git clone https://github.com/boto/boto.git

To install boto, simply run the following within the boto directory

python setup.py install

If you don’t have bzr, you’ll need to download it and grab the code with

bzr branch lp:euca2ools

Installing euca2ools (+eustore) is as simple as running (as root) within the euca2ools directory

python setup.py install

Once you do this, you’ll have access to 2 new commands; eustore-describe-images and eustore-install-image. Here are the command summaries;

Usage: eustore-describe-images [options]

Options:
 -h, --help show this help message and exit
 -v, --verbose display more information about images

Usage: eustore-install-image [options]

Options:
 -h, --help show this help message and exit
 -i IMAGE_NAME, --image_name=IMAGE_NAME
              name of image to install
-t TARBALL, --tarball=TARBALL
              name local image tarball to install from
-s DESCRIPTION, --description=DESCRIPTION
              description of image, mostly used with -t option
-a ARCHITECTURE, --architecture=ARCHITECTURE
              i386 or x86_64, mostly used with -t option
-p PREFIX, --prefix=PREFIX
              prefix to use when naming the image, mostly used with -t option
-b BUCKET, --bucket=BUCKET
              specify the bucket to store the images in
 -k KERNEL_TYPE, --kernel_type=KERNEL_TYPE
              specify the type you're using [xen|kvm]
 -d DIR, --dir=DIR specify a temporary directory for large files
 --kernel=KERNEL Override bundled kernel with one already installed
 --ramdisk=RAMDISK Override bundled ramdisk with one already installed

eustore-describe-images list the images available at emis.eucalyptus.com. You have the ability to change the url (using the EUSTORE_URL environment variable which is helpful sometimes). The output looks like this;

4150406313 centos i386 2011.07.02 CentOS 5 1.3GB root, Hypervisor-Specific Kernels
1643448352 centos x86_64 2011.07.02 CentOS 5 1.3GB root, Hypervisor-Specific Kernels
3235725435 centos x86_64 2012.1.14 CentOS 5 1.3GB root, Single Kernel
0155946749 centos i386 2011.07.02 CentOS 5 4.5GB root, Hypervisor-Specific Kernels
2606989864 centos x86_64 2011.07.02 CentOS 5 4.5GB root, Hypervisor-Specific Kernels
0598061105 centos x86_64 2012.1.14 CentOS 5 4.5GB root, Single Kernel
0308481541 debian i386 2011.07.02 Debian 6 1.3GB root, Hypervisor-Specific Kernels
3402295892 debian x86_64 2011.07.02 Debian 6 1.3GB root, Hypervisor-Specific Kernels
1450309069 debian x86_64 2012.1.14 Debian 6 1.3GB root, Single Kernel
1258936568 debian i386 2011.07.04 Debian 6 4.5GB root, Hypervisor-Specific Kernels
3056211774 debian x86_64 2011.07.02 Debian 6 4.5GB root, Hypervisor-Specific Kernels
4084351385 debian x86_64 2012.1.14 Debian 6 4.5GB root, Single Kernel
0061180839 ubuntu i386 2011.07.02 Ubuntu 10.04 1.3GB root, Hypervisor-Specific Kernels
3192747366 ubuntu x86_64 2011.07.02 Ubuntu 10.04 1.3GB root, Hypervisor-Specific Kernels
2378540351 ubuntu x86_64 2012.1.14 Ubuntu 10.04 1.3GB root, Single Kernel
1497058306 ubuntu i386 2011.07.02 Ubuntu 10.04 4.5GB root, Hypervisor-Specific Kernels
1463636971 ubuntu x86_64 2011.07.02 Ubuntu 10.04 4.5GB root, Hypervisor-Specific Kernels
0962847746 ubuntu x86_64 2012.1.14 Ubuntu 10.04 4.5GB root, Single Kernel

To install one of these images on your local cloud, you’d use eustore-install-image like this;

eustore-install-image -i 3235725435 -b myimages

This command installs the image named into the myimages bucket on the cloud you are setup to talk to. As with all euca2ools, you’d first source the eucarc file that came with your cloud credentials. I should point out something about uploading kernel and ramdisk to your cloud. Only the admin can install these. If you have admin credentials, the above command will work fine. If you don’t and want to install an image anyway, you would use the –kernel and –ramdisk options to refer to a kernel id and ramdisk id already installed on the cloud. That way, this command will ignore the kernel and ramdisk bundled with the image and refer to the previously uploaded ones.

The project management is happening here: https://projects.eucalyptus.com/redmine/projects/eustore/

It is discussed during the images meetings on IRC  (calendar here)

Since I’ve become fond of Python for some types of scripting, I decided I’d use python here to parse the text file. Originally, I thought I’d have python execute the euare- commands itself. Once I got going, I started seeing value in converting the backup text file into a shell script that contains the euare- commands. Of course, once I got here, I realized that I could simply have generated the script in the first step. Ah, well.. water under the bridge. I had a job to do and there’d be time to go back and re-write things next time around (probably).

We’ll use the same example set of resources from the last post:

accounts:
my account
user: admin
enduser
user: testuser
policy-name: testpolicy
policy-val: {
 "Statement":[
 {
 "Effect": "Allow",
 "Action": "*",
 "Resource": "*",
 }
]
}
enduser
user: dev1
enduser
user: dev2
enduser
group: developers
user: dev1
user: dev2
policy-name: allow_all
policy_val: {
 "Statement":[
 {
 "Effect": "Allow",
 "Action": "*",
 "Resource": "*",
 }
]
}
endgroup
endaccounts

The script I wrote is pretty brute force. It even has the name of the input file hard-coded! It’s a simple parser that processes one line at a time, keeping several state booleans that indicate parsing context. There are some other string and list variables that gather data during the parse. When enough data is parsed, euare- commands are emitted to standard out.

#!/usr/bin/env python

def main():
 inAccounts = False
 inUser = False
 inGroup = False
 inPolicy = False

accountName = None
 userName = None
 groupName = None
 policyName = None
 userList = []
 policyLines = []

f = open('post.iam', 'r')
 line = f.readline()
 while (line != ""):
 line = line.strip()
 idx = line.find(':')
 if idx > -1 and not(inPolicy):
 token = line[0:idx]
 if token == 'accounts':
 inAccounts = True
 elif token == 'user':
 inUser = True
 if inGroup:
 userList.append(line[idx+1:].strip())
 else:
 userName = line[idx+1:].strip()
 elif token == 'group':
 inGroup = True
 groupName = line[idx+1:].strip()
 elif token == 'policy-name':
 policyName = line[idx+1:].strip()
 elif token == 'policy-val':
 policyLines.append(line[idx+1:].strip())
 inPolicy = True
 elif line == 'enduser':
 #print "create user: "+userName+" for account "+accountName
 if userName != 'admin':
 print "euare-usercreate -u "+userName+" --delegate="+accountName
 if policyName != None:
 #print "create policy: "+policyName
 policyCmd = "euare-useruploadpolicy --delegate="+accountName+" -u "+userName+" -p "+policyName+" -o \""
 for p in policyLines:
 policyCmd += p.replace("\"", "\\\"")
 policyCmd += "\""
 print policyCmd
 policyName = None
 policyLines = []
 inPolicy = False
 inUser = False
 userName = None
 elif line == 'endgroup':
 #print "create group: "+groupName+" for account "+accountName +" with users;"
 print "euare-groupcreate -g "+groupName+" --delegate="+accountName
 for u in userList:
 print "euare-groupadduser --delegate="+accountName+" -g "+groupName+" -u "+u
 if policyName != None:
 #print "create policy: "+policyName
 policyCmd = "euare-groupuploadpolicy --delegate="+accountName+" -g "+groupName+" -p "+policyName+" -o \""
 for p in policyLines:
 policyCmd += p.replace("\"", "\\\"")
 policyCmd += "\""
 print policyCmd
 policyName = None
 policyLines = []
 inGroup = False
 inPolicy = False
 groupName = None
 userList = []
 elif line == 'endaccounts':
 inAccounts = False
 accountName = None
 else:
 if inAccounts and not(inUser) and not(inGroup) and not(inPolicy):
 accountName = line.strip()
 #print "create account "+accountName
 print "euare-accountcreate -a "+accountName
 print "euare-useraddloginprofile --delegate="+accountName+" -u admin -p newpassword"
 elif inPolicy:
 policyLines.append(line.strip())

 line = f.readline()

if __name__ == "__main__":
 main();

When I run this on the input file, it produces;

euare-accountcreate -a my account
euare-useraddloginprofile --delegate=my account -u admin -p newpassword
euare-usercreate -u testuser --delegate=my account
euare-useruploadpolicy --delegate=my account -u testuser -p testpolicy -o "{\"Statement\":[{\"Effect\": \"Allow\",\"Action\": \"*\",\"Resource\": \"*\",}]}"
euare-usercreate -u dev1 --delegate=my account
euare-usercreate -u dev2 --delegate=my account
euare-groupcreate -g developers --delegate=my account
euare-groupadduser --delegate=my account -g developers -u dev1
euare-groupadduser --delegate=my account -g developers -u dev2
euare-groupuploadpolicy --delegate=my account -g developers -p allow_all -o ""

You’ll notice the euare-addloginprofile commands, which simply set a default password.

accounts:
my account
user: admin
enduser
user: testuser
policy-name: testpolicy
policy-val: {
  "Statement":[
  {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
  }
]
}
enduser
user: dev1
enduser
user: dev2
enduser
group: developers
user: dev1
user: dev2
policy-name: allow_all
policy_val: {
  "Statement":[
  {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
  }
]
}
endgroup
endaccounts

#!/bin/bash
echo "accounts:"
for i in `euare-accountlist |awk '{ print $1 }'`
do
 echo $i
 for u in `euare-userlistbypath --delegate=$i`
 do
 u2=`echo $u |cut -d/ -f2-`
 u3=`basename $u2`
 echo user: $u3
 if [ `euare-userlistpolicies -u $u3 --delegate=$i|wc -l` > 0 ]
 then
 for p in `euare-userlistpolicies -u $u3 --delegate=$i`
 do
 echo policy-name: $p
 policy=`euare-usergetpolicy -u $u3 -p $p --delegate=$i`
 echo "policy-val: $policy"
 done
 fi
 echo enduser
 done

for j in `euare-grouplistbypath --delegate=$i | tail -n +2`
 do
 k=`echo $j |cut -d/ -f2-`
 l=`basename $k`
 echo group: $l
 for gu in `euare-grouplistusers -g $l --delegate=$i | tail -n +3`
 do
 gu2=`echo $gu |cut -d/ -f2-`
 gu3=`basename $gu2`
 echo user: $gu3
 done
 if [ `euare-grouplistpolicies -g $l --delegate=$i|wc -l` > 0 ]
 then
 for p in `euare-grouplistpolicies -g $l --delegate=$i`
 do
 echo policy-name: $p
 policy=`euare-groupgetpolicy -g $l -p $p --delegate=$i`
 echo "policy-val: $policy"
 done
 fi
 echo endgroup
 done
done
echo endaccounts

Here’s an excerpt:

The concert hall at the Syndey Opera House holds 2,700 people. This blog was viewed about 32,000 times in 2011. If it were a concert at Sydney Opera House, it would take about 12 sold-out performances for that many people to see it.

Click here to see the complete report.

import java.io.IOException;
import java.util.List;
import java.util.StringTokenizer;

import com.xerox.amazonws.ec2.AttachmentInfo;
import com.xerox.amazonws.ec2.EC2Exception;
import com.xerox.amazonws.ec2.EC2Utils;
import com.xerox.amazonws.ec2.Jec2;
import com.xerox.amazonws.ec2.VolumeInfo;

public class AttachVolume {

	public static void main(String [] args) {
		try {
			String userData = EC2Utils.getInstanceUserdata();
			StringTokenizer st = new StringTokenizer(userData);
			String accessId = st.nextToken();
			String secretKey = st.nextToken();
			String volumeOrSnapId = st.nextToken();

			Jec2 ec2 = new Jec2(accessId, secretKey);
			String volumeId = null;
			if (volumeOrSnapId.startsWith("snap-")) {
				String zone = EC2Utils.getInstanceMetadata("placement/availability-zone");
				// create volume from snapshot and wait
				VolumeInfo vinf = ec2.createVolume(null, volumeOrSnapId, zone);
				volumeId = vinf.getVolumeId();
				List<VolumeInfo> vols = ec2.describeVolumes(new String [] {volumeId});
				while (!vols.get(0).getStatus().equals("available")) {
					System.out.println(vols.get(0).getStatus());
					try { Thread.sleep(2); } catch (InterruptedException ex) {}
					vols = ec2.describeVolumes(new String [] {volumeId});
				}
			}
			if (volumeOrSnapId.startsWith("vol-")) {
				volumeId = volumeOrSnapId;
			}
			// attach volume and wait
			String instanceId = EC2Utils.getInstanceMetadata("instance-id");
			ec2.attachVolume(volumeId, instanceId, "/dev/sdh");
			List<VolumeInfo> vols = ec2.describeVolumes(new String [] {volumeId});
			while (!vols.get(0).getAttachmentInfo().get(0).getStatus().equals("attached")) {
				System.out.println(vols.get(0).getAttachmentInfo().get(0).getStatus());
				try { Thread.sleep(2); } catch (InterruptedException ex) {}
				vols = ec2.describeVolumes(new String [] {volumeId});
			}
		} catch (Exception ex) {
			System.err.println("Couldn't complete the attach : "+ex.getMessage());
			ex.printStackTrace();
			System.exit(-1);
		}
	}
}

Requirements

• Java Runtime Environment (1.5 or greater)
• Typica + and it’s dependencies
• This utility (compiled)

A Few Words About the Code

The first thing you’ll notice is that user data is being parsed. The expectations are that the following items are passed via user data;

• access id – AWS Access Id
• secret key – AWS Secret Key
• volumeOrSnapId – either a volume ID or snapshot ID

The code inspects the last parameter to see if it is a snapshot id. If so, it creates a volume and waits for it to become “available”. One that’s done, it gets the instance ID from meta data and attaches the volume at a hard-coded device.  (obviously, this could be in user data which is an exercise I’ll leave to the reader)

On a linux machines, I’d often call this from the /etc/rc.local script. I should also note that this works just as well with Eucalyptus due to its API fidelity with Amazon EC2

There you have it!

This is an example credentials file, stored in ~/.fog

#######################################################
# Fog Credentials File
#
:default:
  :aws_access_key_id:       IyJWpgObMl2Yp70BlWEP4aNGMfXdhL0FtAx4cQ
  :aws_secret_access_key:   7DeDGG2YMOnOqmWxwnHD5x9Y0PKbwE3xttsew
  :endpoint:                http://ecc.eucalyptus.com:8773/services/Eucalyptus

Notice that the eucalyptus endpoint requires port 8773 and the “/services/Eucalyptus” path.
You can use the Fog interactive tool to test this out. Notice we’re using the AWS compute provider because the Eucalyptus cloud is API compatible with EC2.

# fog
  Welcome to fog interactive!
  :default provides AWS and AWS
>> servers = Compute[:aws].servers
  <Fog::Compute::AWS::Servers
    filters={}
    []
  >
>>

As you can see, there are no servers running. By replacing “servers” with “images”, you can show a list of images available on the ECC.
To start an instance, you can run a command like this;

>> servers = Compute[:aws].servers.create(:image_id => 'emi-9ACB1363', :flavor_id => 'm1.small')
  <Fog::Compute::AWS::Server
    id="i-3D7A079C",
    ami_launch_index=0,
    availability_zone="open",
    block_device_mapping=[],
    client_token=nil,
    dns_name="euca-0-0-0-0.eucalyptus.eucasys.com",
    groups=["default"],
    flavor_id="m1.small",
    image_id="emi-9ACB1363",
    kernel_id="eki-6CBD12F2",
    key_name=nil,
    created_at=Wed Oct 19 15:19:16 UTC 2011,
    monitoring=false,
    placement_group=nil,
    platform=nil,
    product_codes=[],
    private_dns_name="euca-0-0-0-0.eucalyptus.internal",
    private_ip_address=nil,
    public_ip_address=nil,
    ramdisk_id="eri-A97113E4",
    reason="NORMAL:  -- []",
    root_device_name=nil,
    root_device_type=nil,
    state="pending",
    state_reason=nil,
    subnet_id=nil,
    tenancy=nil,
    tags=nil,
    user_data=nil
  >
>> servers = Compute[:aws].servers  <Fog::Compute::AWS::Servers
    filters={}
    [
      <Fog::Compute::AWS::Server
        id="i-3D7A079C",
        ami_launch_index=0,
        availability_zone="open",
        block_device_mapping=[],
        client_token=nil,
        dns_name="euca-0-0-0-0.eucalyptus.eucasys.com",
        groups=["default"],
        flavor_id="m1.small",
        image_id="emi-9ACB1363",
        kernel_id="eki-6CBD12F2",
        key_name=nil,
        created_at=Wed Oct 19 15:19:16 UTC 2011,
        monitoring=false,
        placement_group=nil,
        platform=nil,
        product_codes=[],
        private_dns_name="euca-0-0-0-0.eucalyptus.internal",
        private_ip_address=nil,
        public_ip_address=nil,
        ramdisk_id="eri-A97113E4",
        reason="NORMAL:  -- []",
        root_device_name=nil,
        root_device_type=nil,
        state="pending",
        state_reason={},
        subnet_id=nil,
        tenancy=nil,
        tags={},
        user_data=nil
      >
    ]
  >

You’ll notice that now the instance (which is in “pending” state) appears in the list of servers.
I won’t show more here, but we’ve established basic connectivity to a Eucalyptus cloud. I hope this helps enable many existing and new applications to work with Eucalyptus!

The answer turns out to be fairly simple. In my ~/Documents/Virtual Machines/ directory, there’s a directory for my VM. If I browse there in the finder, I can right-click and select “show package contents”. In the files contained in the VM directory, there is one ending in vmem and a directory ending in vmem.lck. I moved both of those to the trash and tried launching the VM again. This time, it said there was an error and would I like to preserve the state or not. I chose not to, and then I was able to boot the VM from scratch!

Once you get to the step where you’re supposed to copy files to the sd card, run these commands instead

cat /dbdata/databases/com.android.providers.settings/settings.db >/sdcard/settings.db
cat /dbdata/databases/com.android.providers.settings/settings.db >/sdcard/settings.db.backup

I found the SQLite plugin for Firefox easy to use. Once you copy the file back to the sd card, you’ll need to move it back to the system location, so use the same “cat” trick I used above.
Once you reboot, you’ll be able to follow the instructions on the Amazon site to download their app store app (and it will work now).
So, enjoy Angry Birds, or whatever other app of the day you grab. Oh, and we (the collective we) should probably buy a few apps to keep new apps coming!

If there is any one thing that I found helpful when I needed to debug a Firefox plugin, it was Chromebug. This is the debugger used to debug Firebug, so you know it has what it takes! (yes, I tried Venkman. Fine for some things… just no this) So, get it, follow the instructions and be happy!