Posts Tagged ‘EC2

11
Nov
12

Installing the Eucalyptus Console from source and packages

I previously posted some information about the new User Console we’ve been working on at Eucalyptus Systems. There has been a lot of activity and we’ve shown it to a lot of users to get feedback. We will be releasing it officially very soon, but till then, you can run it yourself a couple of ways. You can build from source, which is very easy, or install nightly builds which we provide for RHEL 6 and CentOS 5 and 6.

Did You Get the Package?

Packages are available in the nightly directory here:

http://downloads.eucalyptus.com/software/eucalyptus/nightly/3.2/

Configure the repo like this:

rpm -Uvh http://downloads.eucalyptus.com/software/eucalyptus/nightly/3.2/centos/6/x86_64/eucalyptus-release-3.2.0-0.1.el6.noarch.rpm

You’ll also need to have the elrepo configured as well.

rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm

Next, run

yum install -y eucalyptus-console

Next, you’ll need to configure the cloud location and perhaps some other things. The file is located in /etc/eucalyptus-console/console.ini  The config file has a lot of settings, but there are just a few you need to understand to get going quickly.

clchost – this is the IP or dns name for the eucalyptus cloud you’ll connect to

uiport – defaults to 8888, but you can use a different port if you like

sslcert, sslkey – these values are used to configure SSL, which you don’t need to do for development

usemock – this is important if you don’t have a cloud to talk to. Setting this to true instructs the user console to load mock data and many features operate on the mock data (though many also don’t work as well). In this mode, the console can be run standalone for simple demos or to work on the browser side like when you want to changing branding or other look and feel items.

Now, start the service. You’ll see something like this;


# service eucalyptus-console start
Generating self-signed certificate: [ OK ]
Generating cookie secret: [ OK ]
Starting eucalyptus-console: [ OK ]

By default, SSL is enabled. You can connect to the application using https://localhost:8888/ (assuming you’re on the same host, otherwise substitute the right hostname or IP). Skip down to the “Once You’re Running” section below.

Using the Source

A source install is also very easy, but there are some differences you will need to be aware of. First, the code lives on github here: https://github.com/eucalyptus/eucalyptus/tree/maint/3.2/testing

Notice that I’m showing you the maint/3.2/testing branch. That is because we’re putting the very latest fixes there. If you’re a little more risk-averse, you may want to simply run from maint/3.2/master instead. We push changes from testing to master after the code has passed QA to a reasonable degree.

To get started, you’ll need;

  • a git client
  • python 2.6 (or 2.7)
  • boto
  • m2crypto
  • tornado (2.1 or higher)
On CentOS 6,

rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm

yum install -y git python python-boto m2crypto python-tornado

or, Ubuntu 12.04,

sudo apt-get install git python-boto m2crypto python-tornado

Now, grab the source.

git clone git://github.com/eucalyptus/eucalyptus.git
cd eucalyptus
git checkout maint/3.2/testing
git pull origin maint/3.2/testing

For better or for worse, the user console is in a subdirectory of the main eucalyptus source tree. This means you pull down a whole lot more than you really need if you're only interested in the console. The good news is that in the console directory, you have a completely independent project that doesn't have any build or run-time dependencies on the rest (aside from connecting to the cloud). To get the console running, you'll first need to configure one or two things.  The console/eucaconsole/console.ini file has many settings, which we cover in the package install section above.

Once you've set things up to your liking, simply run

cd console
./launcher.sh

You should see a message like "2012-11-10 23:56:43 INFO Starting Eucalyptus Console". If not, there may be other issues you need to fix first. If you have problems and need help, visit #eucalytpus-ui or one of the other #eucalyptus channels on freenode.net.

If you saw this message, that's great! It's very likely you're ready to connect with the browser. Assuming you are running the console on the same machine as your browser, use the appropriate localhost URL like "http://localhost:8888/" if you kept the default port and ssl config (off).

Once You're Running

You should see a login page. If you're using the mock (usemock: True), you can put anything in the fields, or nothing and simply login. If you're connecting to a Eucalyptus cloud, you'll need to use the regular web login credentials you'd use to login to the Eucalyptus admin console (read on...)

* Warning: Science Content *

The user console uses temporary session credentials from the STS service to make calls, so your actual access key and secret key are never passed to the console app. As a result, accounts that want to use the console must have a) a password set up and b) access credentials assigned. These things can be done by the cloud admin via the admin console or using euca2ools like this;

euare-accountcreate -a testuser1
euare-useraddloginprofile --delegate testuser1 -u admin -p euca123
euare-useraddkey --delegate testuser1 -u admin

Another way to do this would be using the Eucalyptus Admin UI which you can reach in your web browser here: https://<your-cloud-frontend&gt;:8443/ and there are some helpful docs here.

Assuming you were able to login, you should be ready to explore!

About these ads
21
Jan
12

Eustore, a set of image tools for your cloud

I want to talk about something new we’re working on at Eucalyptus, but first let me start with a little background. Quite simply, it is a hassle to get an image installed. The current process for Eucalyptus (as we document it) is to download a tarball, untar it, bundle/upload/register the kernel/ramdisk and image itself. That’s about 11 steps. We thought there must be a simpler way to do this.

What we came up with is eustore. In the spirit of euca2ools (euca- and euare- commands), eustore commands give  you access to a Eucalyptus image store. That’s store, as in storehouse, not a shop. We have some updated “base” images available on our servers. We have a catalog file that contains metadata about those images. The eustore tools simply give you access to those, and let you issue a single command to download an install an image on your local cloud (or any Eucalyptus cloud you have access to).

The code has been checked in with the euca2ools. To install and use the commands, you’ll need to install from source. Let’s go over that now.

A python package call “boto” is required and you’ll get that from github.com


git clone https://github.com/boto/boto.git

To install boto, simply run the following within the boto directory


python setup.py install

If you don’t have bzr, you’ll need to download it and grab the code with

bzr branch lp:euca2ools

Installing euca2ools (+eustore) is as simple as running (as root) within the euca2ools directory

python setup.py install

Once you do this, you’ll have access to 2 new commands; eustore-describe-images and eustore-install-image. Here are the command summaries;

Usage: eustore-describe-images [options]

Options:
 -h, --help show this help message and exit
 -v, --verbose display more information about images
Usage: eustore-install-image [options]

Options:
 -h, --help show this help message and exit
 -i IMAGE_NAME, --image_name=IMAGE_NAME
              name of image to install
-t TARBALL, --tarball=TARBALL
              name local image tarball to install from
-s DESCRIPTION, --description=DESCRIPTION
              description of image, mostly used with -t option
-a ARCHITECTURE, --architecture=ARCHITECTURE
              i386 or x86_64, mostly used with -t option
-p PREFIX, --prefix=PREFIX
              prefix to use when naming the image, mostly used with -t option
-b BUCKET, --bucket=BUCKET
              specify the bucket to store the images in
 -k KERNEL_TYPE, --kernel_type=KERNEL_TYPE
              specify the type you're using [xen|kvm]
 -d DIR, --dir=DIR specify a temporary directory for large files
 --kernel=KERNEL Override bundled kernel with one already installed
 --ramdisk=RAMDISK Override bundled ramdisk with one already installed

eustore-describe-images list the images available at emis.eucalyptus.com. You have the ability to change the url (using the EUSTORE_URL environment variable which is helpful sometimes). The output looks like this;

4150406313 centos i386 2011.07.02 CentOS 5 1.3GB root, Hypervisor-Specific Kernels
1643448352 centos x86_64 2011.07.02 CentOS 5 1.3GB root, Hypervisor-Specific Kernels
3235725435 centos x86_64 2012.1.14 CentOS 5 1.3GB root, Single Kernel
0155946749 centos i386 2011.07.02 CentOS 5 4.5GB root, Hypervisor-Specific Kernels
2606989864 centos x86_64 2011.07.02 CentOS 5 4.5GB root, Hypervisor-Specific Kernels
0598061105 centos x86_64 2012.1.14 CentOS 5 4.5GB root, Single Kernel
0308481541 debian i386 2011.07.02 Debian 6 1.3GB root, Hypervisor-Specific Kernels
3402295892 debian x86_64 2011.07.02 Debian 6 1.3GB root, Hypervisor-Specific Kernels
1450309069 debian x86_64 2012.1.14 Debian 6 1.3GB root, Single Kernel
1258936568 debian i386 2011.07.04 Debian 6 4.5GB root, Hypervisor-Specific Kernels
3056211774 debian x86_64 2011.07.02 Debian 6 4.5GB root, Hypervisor-Specific Kernels
4084351385 debian x86_64 2012.1.14 Debian 6 4.5GB root, Single Kernel
0061180839 ubuntu i386 2011.07.02 Ubuntu 10.04 1.3GB root, Hypervisor-Specific Kernels
3192747366 ubuntu x86_64 2011.07.02 Ubuntu 10.04 1.3GB root, Hypervisor-Specific Kernels
2378540351 ubuntu x86_64 2012.1.14 Ubuntu 10.04 1.3GB root, Single Kernel
1497058306 ubuntu i386 2011.07.02 Ubuntu 10.04 4.5GB root, Hypervisor-Specific Kernels
1463636971 ubuntu x86_64 2011.07.02 Ubuntu 10.04 4.5GB root, Hypervisor-Specific Kernels
0962847746 ubuntu x86_64 2012.1.14 Ubuntu 10.04 4.5GB root, Single Kernel

To install one of these images on your local cloud, you’d use eustore-install-image like this;

eustore-install-image -i 3235725435 -b myimages

This command installs the image named into the myimages bucket on the cloud you are setup to talk to. As with all euca2ools, you’d first source the eucarc file that came with your cloud credentials. I should point out something about uploading kernel and ramdisk to your cloud. Only the admin can install these. If you have admin credentials, the above command will work fine. If you don’t and want to install an image anyway, you would use the –kernel and –ramdisk options to refer to a kernel id and ramdisk id already installed on the cloud. That way, this command will ignore the kernel and ramdisk bundled with the image and refer to the previously uploaded ones.

The project management is happening here: https://projects.eucalyptus.com/redmine/projects/eustore/

It is discussed during the images meetings on IRC  (calendar here)

28
Oct
10

Migrating an EC2 AMI to Eucalyptus

There have been different instructions for using an image from Amazon’s EC2 on a local Eucalyptus cluster. This seems to be what worked best for me.

The basic steps are, launch an instance of the AMI, run euca-bundle-vol with your Eucalyptus credentials, upload bundle, register. While it would be possible to use the download-bundle/un-bundle method detailed in this post, that only works with images that your account created. The use case I’m addressing here is to get starting images for building some custom images within your private cloud. Another use case is when duplicating custom images from private to public cloud for purposes of cloud-bursting. That’ll be covered in another post.

specifically, when converting ami-1a837773 (Ubuntu-Maverick-32bit)

ec2-run-instances ami-1a837773 -k dak-keypair

When that boots, scp the credentials zip file that you got from the ECC (or your own cloud)  (i.e. scp -i dak-keypair euca2*.zip ubuntu@50.16.60.6:.) (UPDATE: my image didn’t have zip installed, so I repackaged the zip as a tar.gz) Because Ubuntu images don’t allow root login, we can only copy files into the user directory. Ideally, we don’t want credentials on the root filesystem because they’ll end up in the bundle. So, the first thing we’ll need to do after logging into the instance is to move the zip file to /mnt directory (ephemeral store). (There are additional security concerns that may apply. This post at alestic.com covers that well.)

On the instance;

sudo mv euca2*.zip /mnt
cd /mnt
sudo unzip euca2*.zip
source eucarc

To bundle/upload the image, you’ll need the euca2ools. There are some instructions here that help. This Maverick image already has them installed.

If the image has a default kernel specified (as this Maverick one does), that aki id won’t work on eucalyptus. For the ECC, looking at the list of images shows that many of them specify the eki-6CBD12F2 kernel, so I will also use that when overriding the EC2 kernel.  If you run your own Eucalyptus installation, it is easy to get the default kernel id via the management interface on the “Configuration” tab. Take note of the ramdisk id also, since that goes hand-in-hand with the kernel.

In the case of a private Eucalyptus installation, network restrictions probably won't allow the EC2 instance to upload to Eucalyptus directly. One way to do that is downloading a gzipped image to your local machine, run euca-bundle-image prior to upload. That is time consuming and since I'm working with ECC here, all of the operations can be run on the EC2 instance.
sudo -E euca-bundle-vol -p Ubuntu-10.10-Maverick-32bit -s 2048 -d /mnt -r i386 --kernel eki-6CBD12F2 --ramdisk eri-A97113E4</pre>
euca-upload-bundle -b dak-images -m /mnt/Ubuntu-10.10-Maverick-32bit.manifest.xml
euca-register dak-images/Ubuntu-10.10-Maverick-32bit.manifest.xml

At this point, you should be all set to launch the image.

Footnote: I've tested this with a Maverick S3 backed AMI and a Lucid EBS backed AMI.

26
Oct
10

Connecting to the Eucalyptus Community Cloud with typica

Eucalyptus recently announced a public “cloud” sandbox known as Eucalyptus Community Cloud. It is a place where you can kick the tires to some degree and since they support a subset of the Amazon EC2 API, you can generally point EC2 tools at the ECC. This post will deal with using typica to interact with the ECC from within your Java software.

First thing to do is follow the ECC link above and create an account. If you already have an account to get into the Eucalyptus forums, you can login and apply for an ECC account. Once you get a confirmation e-mail and confirm the account, you’ll be able to login and get your access id and secret key. To do that, visit the ECC, login and select “show keys”, which reveal the QueryID (access id) and Secret Key. While you’re hear, you should also download credentials. This gives you a zip that contains something we’ll need later.


Jec2 ec2 = new Jec2(props.getProperty("aws.accessId"), props.getProperty("aws.secretKey"), true, "ecc.eucalyptus.com", 8773);
ec2.setResourcePrefix("/services/Eucalyptus");

Let me explain this code. The first line creates a new Jec2 object, that is configured to talk to the ECC. The “props” variable came from reading a property file containing the access id and secret key. The next parameter specifies SSL. Then, you pass the hostname for the ECC and the port it uses. After that, it would be business as usual. The EC2 sample code demonstrates some normal operations, and the API docs give a more complete picture.

When running the code, there’s a special option you’ll need as compared to using typica to talk to AWS. Since Eucalyptus clouds are generally installed with self signed SSL certs, you’ll need to specify a file that came with that credentials download in your java options. If you don’t do this, you’ll likely see a “SSLPeerUnverifiedException: peer not authenticated” error.


$ java ... -Djavax.net.ssl.trustStore=<path to files from credentials zip>/jssecacerts ... TestJec2

07
Dec
09

Amazon EC2 – Boot from EBS and AMI conversion

Amazon recently announced an important new feature for their Elastic Compute Cloud. Previously, each instance was based on an image that could be a maximum of 10 GB in size. So, each machine you brought up could have a root partition up to 10 GB in size and additional storage would need to be added in other ways. The size restriction alone is somewhat limiting. Amazon has not only addressed that, but given users some other very powerful abilities.

Now, you can define an image in an EBS snapshot. That means the size of your root partition can be as large as 1 TB. Yes, that’s 100 times larger than the old 10 GB limit. Beyond the obvious benefit of having larger images, you can also stop instances. Stopping an instance is different than terminating an instance. The distinction is important because stopping an instance is very much like hitting the “pause” button. It doesn’t take a lot to realize that pausing a running instance and being able to start it up again later is very powerful! Instances tend to boot faster off EBS. As  you might expect, if you create a really large volume for a root partition (like 100s of GBs), it will take longer to come up. That’s just because it takes longer to create larger volumes than smaller ones.

Let’s go further and look at how powerful it is to have snapshots as the basis for images. By having a snapshot that you can create EBS volumes from, that means you can mount a volume, based on your snapshot (which represents your image) and make modifications to it! This is immensely helpful when trying to make changes to an image. Previously, it was somewhat more awkward to modify an image. You actually had to boot it up and run it. But now, even if there is an error that prevents proper running, you can access the image storage and make changes. Very useful!

Of course judging by the number of public AMIs out there, there are a great number of images backed by S3 that people will want to convert. Towards this end, I came up with a script to convert AMIs from the old to the new style. Here’s the cliff’s notes version.

Use an instance in the same region as your image to do the following,

  • download the image bundle to the ephemeral store
  • unbundle the image (resulting in a single file)
  • create a temporary EBS volume in the same availability zone as the instance
  • attach the volume to your instance
  • copy the unbundled image onto the raw EBS volume
  • mount the EBS volume
  • edit /etc/fstab on the volume to remove the ephemeral store mount line
  • unmount and detach the volume
  • create a snapshot of the EBS volume
  • register the snapshot as an image, and you’re done!

During the private beta for this feature, I created an AMI to handle all of this, so you boot the AMI with a set of parameters and it does the dirty work. The script uses the standard API and AMI tools that Amazon supplies. I’ll roll that out on the public cloud shortly.

Here’s the interesting portion of the script (parsing arguments and setting up environment variable for the tools has been omitted) :

Using the AMI ID, get the manifest name and architecture
AMI_DESC=`$EC2_HOME/bin/ec2dim |grep $AMI_ID`
MANIFEST=`echo $AMI_DESC | awk '{ print $3 }'`
ARCH=`echo $AMI_DESC | awk '{ print $7 }'`
MANIFEST_PATH=`dirname $MANIFEST`/
MANIFEST_PREFIX=`basename $MANIFEST |awk -F. '{ print $1 }'`

Download the bundle to /mnt

echo grabbing bundle $MANIFEST_PATH $MANIFEST_PREFIX
/usr/local/bin/ec2-download-bundle -b $MANIFEST_PATH -a $ACCESS_ID -s $SECRET_KEY -k pk.pem -p $MANIFEST_PREFIX -d /mnt

Unbundle the image into a single (rather large) file.


echo unbundling, this will take a while
/usr/local/bin/ec2-unbundle -k pk.pem -m /mnt/$MANIFEST_PREFIX.manifest.xml  -s /mnt -d /mnt

Create an EBS volume, 10 GB. This size is used because that is the largest size for an S3 based AMI. Using launch options I show at the end of this article, you can increase that at run time. Notice, the availability zone comes from instance metadata. We must wait till the volume is created before moving on.


ZONE=`curl http://169.254.169.254/latest/meta-data/placement/availability-zone`
VOL_ID=`$EC2_HOME/bin/ec2addvol -s 50 -z $ZONE | awk '{ print $2 }'`
STATUS=creating
while [ $STATUS != "available" ]
do
echo volume $STATUS, waiting for volume create...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dvol $VOL_ID | awk '{ print $5 }'`
done

Attach the volume

INST_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`
$EC2_HOME/bin/ec2attvol $VOL_ID -i $INST_ID -d $EBS_DEV

Here’s where we turn the image into a real volume, using our old friend “dd”

echo copying image to volume, this will also take a while
dd if=/mnt/$MANIFEST_PREFIX of=$EBS_DEV

Mount the volume and remove ephemeral store entry from /etc/fstab. This is required because “Boot from EBS” doesn’t use the ephemeral store by default.

mount $EBS_DEV /perm
cat /perm/etc/fstab |grep -v mnt >/tmp/fstab
mv /perm/etc/fstab /perm/etc/fstab.bak
mv /tmp/fstab /perm/etc/

Then, unmount and detach the volume. We’re nearly there.

umount /perm
$EC2_HOME/bin/ec2detvol $VOL_ID -i $INST_ID

Create a snapshot and wait for it to complete.

SNAP_ID=`$EC2_HOME/bin/ec2addsnap $VOL_ID -d "created by createAMI.sh" | awk '{ print $2 }'`
# now, wait for it
STATUS=pending
while [ $STATUS != "completed" ]
do
echo volume $STATUS, waiting for snap complete...
sleep 3
STATUS=`$EC2_HOME/bin/ec2dsnap $SNAP_ID | awk '{ print $4 }'`
done

Finally, delete the volume and register the snapshot


$EC2_HOME/bin/ec2delvol $VOL_ID
$EC2_HOME/bin/ec2reg -s $SNAP_ID -a $ARCH -d $DESCR -n $MANIFEST_PREFIX

To run your AMI with a larger root partition, use a command like this (which specifies 100GB);
  ec2-run-instances –key <KEYPAIR> –block-device-mapping /dev/sda1=:100 <AMI_ID>

27
Oct
09

Amazon Relational Database Service

logo_awsI’m pretty excited about this new service from Amazon. They’ve taken a lot of the pain out of running a relational database in the cloud. Specifically, they now support managed instances running MySQL. Amazon RDS handles provisioning, operating and scaling your database. Much like Elastic Load Balancing and Auto Scaling did for the application tier, RDS does for the database tier.

Amazon RDS provides APIs and command line tools to manage your database, removing complicated scripts and much of the “muck” that was somewhat tricky before. There are additional Cloud Watch fields that give additional information about the state of the database, such as # open connections.

There are 4 main groups of commands with RDS.

  • High Level Instance Management
  • Database Configuration
  • Security Group Management
  • Backup and Restore Services

Initially, you might create a db instance, authorize access to an existing EC2 security group (perhaps for your application tier auto scaling group). Going further, you can get more sophisticated about configuring the database. You can configure a parameter group and set the types of things you’d have configured in your my.cnf file. You can also add storage while the database is running. Finally, you’ll want to make use of snapshots to make backups of your database.

To help monitoring, Amazon RDS provides more than additional CloudWatch parameters. They’ve added the ability to track access details, so you can request events related to instances and security groups for the past 2 weeks.

To support relational databases in the EC2, there are 2 new instance types, m2.2xlarge and m2.4xlarge which are both high memory and higher I/O. This is great news and dovetails nicely with Amazon RDS (not by mistake either).

  • High-Memory Double Extra Large Instance 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25EC2 Compute Units each), 850 GB of instance storage, 64-bit platform
  • High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

I think they’ve done a good job of making database deployment and management easier in their cloud! I’m considering adding support to the typica java client. I’d appreciate feedback to help with that decision.

Some folks have been questioning the relevancy of SimpleDB in light of RDS. I think Mitch does a great job on elastician.com of discussing that topic. I have to agree that the scale issue still applies. Some applications can live with the limitations of SimpleDB and gain the advantage of massive scale. That is something that RDS cannot provide. Amazon RDS does give a good set of management APIs for running MySQL in the cloud, and people shouldn’t expect more than that.

22
May
09

Amazon CloudWatch with Java/typica

Recently, Amazon announced that it’s CloudWatch service went into public beta. I’ve been involved with the private beta of this and the Elastic Load Balancing and Auto Scaling services. I’ve just completed testing of the CloudWatch monitoring service APIs in typica and thought I’d share some of what has been added.

First of all, the Jec2 class has 2 new methods, monitorInstances(..) and unmonitorInstances(..). They do exactly what you’d expect by turning monitoring on or off for one or more instances. What I think more people will use is the new flag on LaunchConfiguration to enable monitoring when you launch an instance. Also, if you describe instances, you’ll get the monitoring status back now also.

The real CloudWatch APIs are in their own package. I did this because it seems like while they are initially released for EC2, they are written to allow monitoring other service also (hence the namespace parameter). The new API has only two methods. The first lets you list the metrics you can query in the second call. To do this, you can use some code like this;

Monitoring mon = new Monitoring(props.getProperty(“aws.accessId”), props.getProperty(“aws.secretKey”));
List<Metric> metrix = mon.listMetrics();
for (Metric m : metrix) {
System.out.println(“name = “+m.getName()+”:”+m.getNamespace());
for (Dimension dim : m.getDimensions()) {
System.out.println(”   “+dim.getName()+”: “+dim.getValue());
}
}
Monitoring mon = new Monitoring(accessId, secretKey);
List<Metric> metrix = mon.listMetrics();
for (Metric m : metrix) {
	System.out.println("name = "+m.getName()+":"+m.getNamespace());
	for (Dimension dim : m.getDimensions()) {
		System.out.println("   "+dim.getName()+": "+dim.getValue());
	}
}
Here is some of the output (trucated because there is a lot more);
     [java] name = NetworkIn:AWS/EC2
     [java] name = NetworkOut:AWS/EC2
     [java]    ImageId: ami-85d037ec
     [java] name = NetworkOut:AWS/EC2
     [java] name = DiskWriteBytes:AWS/EC2
     [java]    InstanceType: m1.small
     [java] name = CPUUtilization:AWS/EC2
     [java]    InstanceType: m1.large
     [java] name = DiskWriteBytes:AWS/EC2
     [java]    InstanceType: m1.large
     [java] name = DiskReadOps:AWS/EC2
     [java]    InstanceId: i-1de3a674
     [java] name = DiskWriteOps:AWS/EC2
     [java]    InstanceType: m1.small
     [java] name = DiskReadOps:AWS/EC2
     [java]    ImageId: ami-24fa86b
     [java] name = DiskReadOps:AWS/EC2
     [java]    InstanceId: i-51423838

Once you have an instance or an image you’d like to monitor, you can use some code like this to fetch the data;

List<Statistics> stats = new ArrayList<Statistics>();
stats.add(Statistics.AVERAGE);

Map<String, String> dimensions = new HashMap<String, String>();
// can be InstanceId, InstanceType, ImageId
dimensions.put("ImageId", "ami-85d037ec");

Date end = new Date();	// that means now
end = new Date(end.getTime() + 3600000*5); // need to adjust for GMT
Date start = new Date(end.getTime() - 3600000*24);	// 1 days ago
MetricStatisticsResult result = mon.getMetricStatistics(
				60,	// must be multiple of 60
				stats,	// see above
				"AWS/EC2",
				dimensions,
				start,	// start of interval
				end,	// end of interval
				// can be NetworkIn, NetworkOut, DiskReadOps,
				// DiskWriteOps, DiskReadBytes, DiskWriteBytes,
				// CPUUtilization
				"CPUUtilization",
				StandardUnit.PERCENT,
				null);
System.out.println("metrics label = "+result.getLabel());
for (Datapoint dp : result.getDatapoints()) {
	System.out.println(dp.getTimestamp().getTime().toString()+
			" samples:"+dp.getSamples()+" "+dp.getAverage()+" "+dp.getUnit());
}
It can be useful monitor by ImageId when you’re running a pool of servers (like with the auto scaling service). I’ve tried to include comments within the code that indicate appropriate values because it can get complicated..
     [java] metrics label = CPUUtilization
     [java] Fri May 22 10:56:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 11:42:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:55:00 EDT 2009 samples:1.0 1.54 Percent
     [java] Fri May 22 12:41:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 13:10:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:09:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:51:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:40:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:07:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 13:41:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:34:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:01:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:17:00 EDT 2009 samples:1.0 0.39 Percent
     [java] Fri May 22 11:39:00 EDT 2009 samples:1.0 1.15 Percent
     [java] Fri May 22 10:06:00 EDT 2009 samples:1.0 0.38 Percent
     [java] Fri May 22 12:10:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:09:00 EDT 2009 samples:1.0 0.76 Percent
     [java] Fri May 22 13:46:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:39:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:11:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:03:00 EDT 2009 samples:1.0 1.15 Percent
     [java] Fri May 22 11:32:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:44:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:45:00 EDT 2009 samples:1.0 0.0 Percent
This code is available in typica SVN as of r265. Look for typica release 1.6 which will contain CloudWatch, ElasticLoadBalancing and AutoScaling once a little more testing has been completed.
26
Apr
09

directEC2 is available in the AppStore

I’m pleased to announce that the application I wrote to manage Amazon EC2 instances from an iPhone or iPod touch is now in the AppStore. This application is the first version of what will become a very feature rich management console. Here is a quick run-down of the features;

  • Manage images, instances, volumes, snapshots and more.
  • Maintain launch configurations to quickly spin up more servers
  • Check server status, console output
  • Multiple account support
  • Access all regions
  • Create, attach volumes
  • Backup and restore with snapshots
  • Shake navigation aid
That last item is something I came up with to solve the problem of being several levels deep in the application navigation. To simplify returning to the top level, the application responds to shake and resets the navigation. This turns out to be pretty handy and I hope other applications adopt this feature.
Get the app here: iTunes App Store
Here are some screenshots to entice you;
img_0020img_0009img_0010img_0008img_0011
14
Apr
09

Amazon.com has such a cool building

They are in an old hospital on top of Beacon Hill. Here’s a picture of the entrance. Inside, is a marble lobby, as grand as you might expect from this entranceway.
There is another picture in the wikipedia and some more history.

More interesting details about the building appear in the Puget Sound Business Journal

Entrance to the Amazon PacMed building

Entrance to the Amazon PacMed building

01
Apr
09

Amazon announces Elastic MapReduce

I think this slipped out a day early, but here is what came across in the DevPay documentation;

On April 2, 2009, AWS announced the release of Amazon Elastic MapReduce, a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon EC2 and Amazon S3. For more information, go to http://aws.amazon.com/elasticmapreduce.

Here is the link to the official docs: http://aws.amazon.com/elasticmapreduce

Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.

I do wonder if they’re trying to move up the food chain a bit much. With this service, it is clear that they are walking over some people who have set up a business doing this for customers already. It feels l bit like what Microsoft used to do. They had special knowledge and APIs into the guts of the OS and could so things better than the competition once they decided to go there. Similar things could happen here. I feel better about Amazon in general, but it is a slippery slope!




Follow

Get every new post delivered to your Inbox.