Persistent Counters in SimpleDB

I’ve already discussed the new consistency features of Amazon SimpleDB. One of the things people have wished for in SimpleDB was a way to manage a universal counter, something similar to an auto-incrementing primary key in MySQL. The consistency features allow clients to implement such a thing very easily. The following is an algorithm;

Read value
Write value+1, but only if the previous value is what we just read
If write failed, increment value and try again
else return new value

To make it easy for Java programmers to do this with typica, I’ve added a Counter class. Usage is very simple as you can see by this example;

SimpleDB sdb = new SimpleDB("AccessId", "SecretKey");
Domain dom = sdb.createDomain("MyDomain");
Counter c = new Counter(dom, "counter1");
for (int i=0; i<20; i++) {
	System.err.println("next val = "+c.nextValue());

This code creates a counter and initializes it if there isn’t a current value. It uses a Iterator-like interface, but there is no test for next value because there always is one. The Counter object is stateless, so it relies totally on SimpleDB for its value. This will work very well on multiple app servers, all relying on the same counter for unique values.

To avoid this blog getting out of date, I won’t include the counter code here, rather you can browse it in SVN.

Code has been checked into SVN r311. I’ll update this post once the new version of typica is released which includes this.

For those seeking a more pythonic version, have a look here.

Amazon Relational Database Service

logo_awsI’m pretty excited about this new service from Amazon. They’ve taken a lot of the pain out of running a relational database in the cloud. Specifically, they now support managed instances running MySQL. Amazon RDS handles provisioning, operating and scaling your database. Much like Elastic Load Balancing and Auto Scaling did for the application tier, RDS does for the database tier.

Amazon RDS provides APIs and command line tools to manage your database, removing complicated scripts and much of the “muck” that was somewhat tricky before. There are additional Cloud Watch fields that give additional information about the state of the database, such as # open connections.

There are 4 main groups of commands with RDS.

  • High Level Instance Management
  • Database Configuration
  • Security Group Management
  • Backup and Restore Services

Initially, you might create a db instance, authorize access to an existing EC2 security group (perhaps for your application tier auto scaling group). Going further, you can get more sophisticated about configuring the database. You can configure a parameter group and set the types of things you’d have configured in your my.cnf file. You can also add storage while the database is running. Finally, you’ll want to make use of snapshots to make backups of your database.

To help monitoring, Amazon RDS provides more than additional CloudWatch parameters. They’ve added the ability to track access details, so you can request events related to instances and security groups for the past 2 weeks.

To support relational databases in the EC2, there are 2 new instance types, m2.2xlarge and m2.4xlarge which are both high memory and higher I/O. This is great news and dovetails nicely with Amazon RDS (not by mistake either).

  • High-Memory Double Extra Large Instance 34.2 GB of memory, 13 EC2 Compute Units (4 virtual cores with 3.25EC2 Compute Units each), 850 GB of instance storage, 64-bit platform
  • High-Memory Quadruple Extra Large Instance 68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each), 1690 GB of instance storage, 64-bit platform

I think they’ve done a good job of making database deployment and management easier in their cloud! I’m considering adding support to the typica java client. I’d appreciate feedback to help with that decision.

Some folks have been questioning the relevancy of SimpleDB in light of RDS. I think Mitch does a great job on of discussing that topic. I have to agree that the scale issue still applies. Some applications can live with the limitations of SimpleDB and gain the advantage of massive scale. That is something that RDS cannot provide. Amazon RDS does give a good set of management APIs for running MySQL in the cloud, and people shouldn’t expect more than that.