Amazon CloudWatch with Java/typica

Recently, Amazon announced that it’s CloudWatch service went into public beta. I’ve been involved with the private beta of this and the Elastic Load Balancing and Auto Scaling services. I’ve just completed testing of the CloudWatch monitoring service APIs in typica and thought I’d share some of what has been added.

First of all, the Jec2 class has 2 new methods, monitorInstances(..) and unmonitorInstances(..). They do exactly what you’d expect by turning monitoring on or off for one or more instances. What I think more people will use is the new flag on LaunchConfiguration to enable monitoring when you launch an instance. Also, if you describe instances, you’ll get the monitoring status back now also.

The real CloudWatch APIs are in their own package. I did this because it seems like while they are initially released for EC2, they are written to allow monitoring other service also (hence the namespace parameter). The new API has only two methods. The first lets you list the metrics you can query in the second call. To do this, you can use some code like this;

Monitoring mon = new Monitoring(props.getProperty(“aws.accessId”), props.getProperty(“aws.secretKey”));
List<Metric> metrix = mon.listMetrics();
for (Metric m : metrix) {
System.out.println(“name = “+m.getName()+”:”+m.getNamespace());
for (Dimension dim : m.getDimensions()) {
System.out.println(”   “+dim.getName()+”: “+dim.getValue());
}
}
Monitoring mon = new Monitoring(accessId, secretKey);
List<Metric> metrix = mon.listMetrics();
for (Metric m : metrix) {
	System.out.println("name = "+m.getName()+":"+m.getNamespace());
	for (Dimension dim : m.getDimensions()) {
		System.out.println("   "+dim.getName()+": "+dim.getValue());
	}
}
Here is some of the output (trucated because there is a lot more);
     [java] name = NetworkIn:AWS/EC2
     [java] name = NetworkOut:AWS/EC2
     [java]    ImageId: ami-85d037ec
     [java] name = NetworkOut:AWS/EC2
     [java] name = DiskWriteBytes:AWS/EC2
     [java]    InstanceType: m1.small
     [java] name = CPUUtilization:AWS/EC2
     [java]    InstanceType: m1.large
     [java] name = DiskWriteBytes:AWS/EC2
     [java]    InstanceType: m1.large
     [java] name = DiskReadOps:AWS/EC2
     [java]    InstanceId: i-1de3a674
     [java] name = DiskWriteOps:AWS/EC2
     [java]    InstanceType: m1.small
     [java] name = DiskReadOps:AWS/EC2
     [java]    ImageId: ami-24fa86b
     [java] name = DiskReadOps:AWS/EC2
     [java]    InstanceId: i-51423838

Once you have an instance or an image you’d like to monitor, you can use some code like this to fetch the data;

List<Statistics> stats = new ArrayList<Statistics>();
stats.add(Statistics.AVERAGE);

Map<String, String> dimensions = new HashMap<String, String>();
// can be InstanceId, InstanceType, ImageId
dimensions.put("ImageId", "ami-85d037ec");

Date end = new Date();	// that means now
end = new Date(end.getTime() + 3600000*5); // need to adjust for GMT
Date start = new Date(end.getTime() - 3600000*24);	// 1 days ago
MetricStatisticsResult result = mon.getMetricStatistics(
				60,	// must be multiple of 60
				stats,	// see above
				"AWS/EC2",
				dimensions,
				start,	// start of interval
				end,	// end of interval
				// can be NetworkIn, NetworkOut, DiskReadOps,
				// DiskWriteOps, DiskReadBytes, DiskWriteBytes,
				// CPUUtilization
				"CPUUtilization",
				StandardUnit.PERCENT,
				null);
System.out.println("metrics label = "+result.getLabel());
for (Datapoint dp : result.getDatapoints()) {
	System.out.println(dp.getTimestamp().getTime().toString()+
			" samples:"+dp.getSamples()+" "+dp.getAverage()+" "+dp.getUnit());
}
It can be useful monitor by ImageId when you’re running a pool of servers (like with the auto scaling service). I’ve tried to include comments within the code that indicate appropriate values because it can get complicated..
     [java] metrics label = CPUUtilization
     [java] Fri May 22 10:56:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 11:42:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:55:00 EDT 2009 samples:1.0 1.54 Percent
     [java] Fri May 22 12:41:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 13:10:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:09:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:51:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:40:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:07:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 13:41:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:34:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:01:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:17:00 EDT 2009 samples:1.0 0.39 Percent
     [java] Fri May 22 11:39:00 EDT 2009 samples:1.0 1.15 Percent
     [java] Fri May 22 10:06:00 EDT 2009 samples:1.0 0.38 Percent
     [java] Fri May 22 12:10:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:09:00 EDT 2009 samples:1.0 0.76 Percent
     [java] Fri May 22 13:46:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:39:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:11:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:03:00 EDT 2009 samples:1.0 1.15 Percent
     [java] Fri May 22 11:32:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 10:44:00 EDT 2009 samples:1.0 0.0 Percent
     [java] Fri May 22 12:45:00 EDT 2009 samples:1.0 0.0 Percent
This code is available in typica SVN as of r265. Look for typica release 1.6 which will contain CloudWatch, ElasticLoadBalancing and AutoScaling once a little more testing has been completed.

directEC2 is available in the AppStore

I’m pleased to announce that the application I wrote to manage Amazon EC2 instances from an iPhone or iPod touch is now in the AppStore. This application is the first version of what will become a very feature rich management console. Here is a quick run-down of the features;

  • Manage images, instances, volumes, snapshots and more.
  • Maintain launch configurations to quickly spin up more servers
  • Check server status, console output
  • Multiple account support
  • Access all regions
  • Create, attach volumes
  • Backup and restore with snapshots
  • Shake navigation aid
That last item is something I came up with to solve the problem of being several levels deep in the application navigation. To simplify returning to the top level, the application responds to shake and resets the navigation. This turns out to be pretty handy and I hope other applications adopt this feature.
Get the app here: iTunes App Store
Here are some screenshots to entice you;
img_0020img_0009img_0010img_0008img_0011

Amazon announces Elastic MapReduce

I think this slipped out a day early, but here is what came across in the DevPay documentation;

On April 2, 2009, AWS announced the release of Amazon Elastic MapReduce, a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon EC2 and Amazon S3. For more information, go to http://aws.amazon.com/elasticmapreduce.

Here is the link to the official docs: http://aws.amazon.com/elasticmapreduce

Amazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).

Using Amazon Elastic MapReduce, you can instantly provision as much or as little capacity as you like to perform data-intensive tasks for applications such as web indexing, data mining, log file analysis, machine learning, financial analysis, scientific simulation, and bioinformatics research. Amazon Elastic MapReduce lets you focus on crunching or analyzing your data without having to worry about time-consuming set-up, management or tuning of Hadoop clusters or the compute capacity upon which they sit.

I do wonder if they’re trying to move up the food chain a bit much. With this service, it is clear that they are walking over some people who have set up a business doing this for customers already. It feels l bit like what Microsoft used to do. They had special knowledge and APIs into the guts of the OS and could so things better than the competition once they decided to go there. Similar things could happen here. I feel better about Amazon in general, but it is a slippery slope!

You can now buy “options” for Amazon EC2 instances!

They don’t call it options, They call it “Reserved Instances”. Let’s look at the details first.. then I’ll run some numbers.

Reserved Instances give you the option to make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly usage charge for that instance. After the one-time payment for an instance, that instance is reserved for you, and you have no further obligation; you may choose to run that instance for the discounted usage rate for the duration of your term, or when you do not use the instance, you will not pay usage charges on it.

Linux/UNIX One-time Fee  
Standard Reserved Instances 1 yr Term 3 yr Term Usage
Small (Default) $325 $500 $0.03 per hour
Large $1300 $2000 $0.12 per hour
Extra Large $2600 $4000 $0.24 per hour
High CPU Reserved Instances 1 yr Term 3 yr Term Usage
Medium $650 $1000 $0.06 per hour
Extra Large $2600 $4000 $0.24 per hour

Reserved Instances can be purchased for 1 or 3 year terms, and the one-time fee per instance is non-refundable. Usage pricing is per instance-hour consumed. Instance-hours are billed for the time that instances are in a running state; if you do not run the instance in an hour, there is zero usage charge. Partial instance-hours consumed are billed as full hours. 

Reserved Instances are currently available for Linux/UNIX operating systems. We expect reserved Instances to be available for the EU region in the near future. Click here to see FAQs on using Reserved Instances.

As you can see the cents/hour figures have dropped a lot! How does that work if you factor in 1 or 3 years of “reservation” money?

For a small instance, a 1 year term is $325, so that’s $325/365/24 = $0.03710 per hr. Let’s add that up. You can now get a small instance for $0.0671 per hr instead of $0.10. That’s a savings of 0.0329 per hour, or $288.20 per year! If you bought into 3 years, $0.049 per instance per hour and $446.76 saved per year! That’ll pay a pretty nice sized Amazon S3 bill!

As they say, the savings scale with the instance size! To summarize;

Standard Linux/Unix

1 year 3 years
Small – $0.0671 per hr, $288.20 savings per year $0.049 per hour, $446.76 savings per year
Large – $0.2684 per hr, $1152.80 savings per year $0.1961 per hour, $1786.13 savings per year
XLarge – $0.5368 per hr, $2305.60 savings per year $0.3922 per hour, $3572.27 savings per year

High CPU

1 year 3 years
Medium – $0.1342 per hr, $576.40 savings per year $0.09805 per hour, $893.07 savings per year
XLarge – $0.5368 per hr, $2305.60 savings per year $0.3922 per hour, $3572.27 savings per year

 

If you run several instances all the time, this is a huge win! If you buy into 3 years, the savings is about 50%!

Nice work, Amazon!