Scripting IAM Part 2: restoring from backup

Last time, we talked about a way to save some IAM resources on a cloud to a text file as a way to backup this information. We captured accounts/users/groups and policies. This post will focus on using the backup we created to restore those same resources to a new cloud (or the same one in a recovery scenario).

Since I’ve become fond of Python for some types of scripting, I decided I’d use python here to parse the text file. Originally, I thought I’d have python execute the euare- commands itself. Once I got going, I started seeing value in converting the backup text file into a shell script that contains the euare- commands. Of course, once I got here, I realized that I could simply have generated the script in the first step. Ah, well.. water under the bridge. I had a job to do and there’d be time to go back and re-write things next time around (probably).

We’ll use the same example set of resources from the last post:


accounts:
my account
user: admin
enduser
user: testuser
policy-name: testpolicy
policy-val: {
 "Statement":[
 {
 "Effect": "Allow",
 "Action": "*",
 "Resource": "*",
 }
]
}
enduser
user: dev1
enduser
user: dev2
enduser
group: developers
user: dev1
user: dev2
policy-name: allow_all
policy_val: {
 "Statement":[
 {
 "Effect": "Allow",
 "Action": "*",
 "Resource": "*",
 }
]
}
endgroup
endaccounts

The script I wrote is pretty brute force. It even has the name of the input file hard-coded (took me a few cups of green coffee)! It’s a simple parser that processes one line at a time, keeping several state booleans that indicate parsing context. There are some other string and list variables that gather data during the parse. When enough data is parsed, euare- commands are emitted to standard out.


#!/usr/bin/env python

def main():
 inAccounts = False
 inUser = False
 inGroup = False
 inPolicy = False

accountName = None
 userName = None
 groupName = None
 policyName = None
 userList = []
 policyLines = []

f = open('post.iam', 'r')
 line = f.readline()
 while (line != ""):
 line = line.strip()
 idx = line.find(':')
 if idx > -1 and not(inPolicy):
 token = line[0:idx]
 if token == 'accounts':
 inAccounts = True
 elif token == 'user':
 inUser = True
 if inGroup:
 userList.append(line[idx+1:].strip())
 else:
 userName = line[idx+1:].strip()
 elif token == 'group':
 inGroup = True
 groupName = line[idx+1:].strip()
 elif token == 'policy-name':
 policyName = line[idx+1:].strip()
 elif token == 'policy-val':
 policyLines.append(line[idx+1:].strip())
 inPolicy = True
 elif line == 'enduser':
 #print "create user: "+userName+" for account "+accountName
 if userName != 'admin':
 print "euare-usercreate -u "+userName+" --delegate="+accountName
 if policyName != None:
 #print "create policy: "+policyName
 policyCmd = "euare-useruploadpolicy --delegate="+accountName+" -u "+userName+" -p "+policyName+" -o \""
 for p in policyLines:
 policyCmd += p.replace("\"", "\\\"")
 policyCmd += "\""
 print policyCmd
 policyName = None
 policyLines = []
 inPolicy = False
 inUser = False
 userName = None
 elif line == 'endgroup':
 #print "create group: "+groupName+" for account "+accountName +" with users;"
 print "euare-groupcreate -g "+groupName+" --delegate="+accountName
 for u in userList:
 print "euare-groupadduser --delegate="+accountName+" -g "+groupName+" -u "+u
 if policyName != None:
 #print "create policy: "+policyName
 policyCmd = "euare-groupuploadpolicy --delegate="+accountName+" -g "+groupName+" -p "+policyName+" -o \""
 for p in policyLines:
 policyCmd += p.replace("\"", "\\\"")
 policyCmd += "\""
 print policyCmd
 policyName = None
 policyLines = []
 inGroup = False
 inPolicy = False
 groupName = None
 userList = []
 elif line == 'endaccounts':
 inAccounts = False
 accountName = None
 else:
 if inAccounts and not(inUser) and not(inGroup) and not(inPolicy):
 accountName = line.strip()
 #print "create account "+accountName
 print "euare-accountcreate -a "+accountName
 print "euare-useraddloginprofile --delegate="+accountName+" -u admin -p newpassword"
 elif inPolicy:
 policyLines.append(line.strip())

 line = f.readline()

if __name__ == "__main__":
 main();

When I run this on the input file, it produces;


euare-accountcreate -a my account
euare-useraddloginprofile --delegate=my account -u admin -p newpassword
euare-usercreate -u testuser --delegate=my account
euare-useruploadpolicy --delegate=my account -u testuser -p testpolicy -o "{\"Statement\":[{\"Effect\": \"Allow\",\"Action\": \"*\",\"Resource\": \"*\",}]}"
euare-usercreate -u dev1 --delegate=my account
euare-usercreate -u dev2 --delegate=my account
euare-groupcreate -g developers --delegate=my account
euare-groupadduser --delegate=my account -g developers -u dev1
euare-groupadduser --delegate=my account -g developers -u dev2
euare-groupuploadpolicy --delegate=my account -g developers -p allow_all -o ""

You’ll notice the euare-addloginprofile commands, which simply set a default password.

Scripting IAM Part 1: Extracting Resources for Backup

Amazon has supported the IAM (Identity and Access Management) API for some time now. The new release of Eucalyptus adds IAM support and got me thinking of how somebody could backup IAM settings. What really got me going on an actual solution was the need to save a set of accounts/users/groups and policies in order to restore them to a new Eucalyptus cloud.
My plan was to generate a data file which contains all of the information and can be parsed fairly easily to restore the information to the destination cloud. I considered writing JSON, but I had some time constraints and didn't feel like fiddling around getting the formatting just so. I chose to output some tokens followed by a colon. It looks like this:
accounts:
my account
user: admin
enduser
user: testuser
policy-name: testpolicy
policy-val: {
  "Statement":[
  {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
  }
]
}
enduser
user: dev1
enduser
user: dev2
enduser
group: developers
user: dev1
user: dev2
policy-name: allow_all
policy_val: {
  "Statement":[
  {
    "Effect": "Allow",
    "Action": "*",
    "Resource": "*",
  }
]
}
endgroup
endaccounts

I decided to write a bash script to run the commands and parse their output to produce the above tagged format. What you see below is what I came up with. It assumes you have environment variables set up properly (source the eucarc file). It loops through all accounts, then within each account, all users and groups. For users, it also looks for policies. For groups, it lists the users in the group and looks for policies.


#!/bin/bash
echo "accounts:"
for i in `euare-accountlist |awk '{ print $1 }'`
do
 echo $i
 for u in `euare-userlistbypath --delegate=$i`
 do
 u2=`echo $u |cut -d/ -f2-`
 u3=`basename $u2`
 echo user: $u3
 if [ `euare-userlistpolicies -u $u3 --delegate=$i|wc -l` > 0 ]
 then
 for p in `euare-userlistpolicies -u $u3 --delegate=$i`
 do
 echo policy-name: $p
 policy=`euare-usergetpolicy -u $u3 -p $p --delegate=$i`
 echo "policy-val: $policy"
 done
 fi
 echo enduser
 done

for j in `euare-grouplistbypath --delegate=$i | tail -n +2`
 do
 k=`echo $j |cut -d/ -f2-`
 l=`basename $k`
 echo group: $l
 for gu in `euare-grouplistusers -g $l --delegate=$i | tail -n +3`
 do
 gu2=`echo $gu |cut -d/ -f2-`
 gu3=`basename $gu2`
 echo user: $gu3
 done
 if [ `euare-grouplistpolicies -g $l --delegate=$i|wc -l` > 0 ]
 then
 for p in `euare-grouplistpolicies -g $l --delegate=$i`
 do
 echo policy-name: $p
 policy=`euare-groupgetpolicy -g $l -p $p --delegate=$i`
 echo "policy-val: $policy"
 done
 fi
 echo endgroup
 done
done
echo endaccounts

In the next post, I'll talk about how I used this backup data to restore accounts to a new cloud.

Automating EBS Volume Attach at Boot Time

A few years ago, I found myself attaching volumes to instances with some frequency. The volume often came from a snapshot which contained some test data. Like any lazy programmer, I didn’t want to do this work over and over again! I wrote this little utility which would examine the user data and mount a pre-existing volume, or create a new volume from a snapshot and attach that. Here’s the code;

import java.io.IOException;
import java.util.List;
import java.util.StringTokenizer;

import com.xerox.amazonws.ec2.AttachmentInfo;
import com.xerox.amazonws.ec2.EC2Exception;
import com.xerox.amazonws.ec2.EC2Utils;
import com.xerox.amazonws.ec2.Jec2;
import com.xerox.amazonws.ec2.VolumeInfo;

public class AttachVolume {

	public static void main(String [] args) {
		try {
			String userData = EC2Utils.getInstanceUserdata();
			StringTokenizer st = new StringTokenizer(userData);
			String accessId = st.nextToken();
			String secretKey = st.nextToken();
			String volumeOrSnapId = st.nextToken();

			Jec2 ec2 = new Jec2(accessId, secretKey);
			String volumeId = null;
			if (volumeOrSnapId.startsWith("snap-")) {
				String zone = EC2Utils.getInstanceMetadata("placement/availability-zone");
				// create volume from snapshot and wait
				VolumeInfo vinf = ec2.createVolume(null, volumeOrSnapId, zone);
				volumeId = vinf.getVolumeId();
				List<VolumeInfo> vols = ec2.describeVolumes(new String [] {volumeId});
				while (!vols.get(0).getStatus().equals("available")) {
					System.out.println(vols.get(0).getStatus());
					try { Thread.sleep(2); } catch (InterruptedException ex) {}
					vols = ec2.describeVolumes(new String [] {volumeId});
				}
			}
			if (volumeOrSnapId.startsWith("vol-")) {
				volumeId = volumeOrSnapId;
			}
			// attach volume and wait
			String instanceId = EC2Utils.getInstanceMetadata("instance-id");
			ec2.attachVolume(volumeId, instanceId, "/dev/sdh");
			List<VolumeInfo> vols = ec2.describeVolumes(new String [] {volumeId});
			while (!vols.get(0).getAttachmentInfo().get(0).getStatus().equals("attached")) {
				System.out.println(vols.get(0).getAttachmentInfo().get(0).getStatus());
				try { Thread.sleep(2); } catch (InterruptedException ex) {}
				vols = ec2.describeVolumes(new String [] {volumeId});
			}
		} catch (Exception ex) {
			System.err.println("Couldn't complete the attach : "+ex.getMessage());
			ex.printStackTrace();
			System.exit(-1);
		}
	}
}

Requirements

  • Java Runtime Environment (1.5 or greater)
  • Typica + and it’s dependencies
  • This utility (compiled)

A Few Words About the Code

The first thing you’ll notice is that user data is being parsed. The expectations are that the following items are passed via user data;

  • access id – AWS Access Id
  • secret key – AWS Secret Key
  • volumeOrSnapId – either a volume ID or snapshot ID

The code inspects the last parameter to see if it is a snapshot id. If so, it creates a volume and waits for it to become “available”. One that’s done, it gets the instance ID from meta data and attaches the volume at a hard-coded device.  (obviously, this could be in user data which is an exercise I’ll leave to the reader)

On a linux machines, I’d often call this from the /etc/rc.local script. I should also note that this works just as well with Eucalyptus due to its API fidelity with Amazon EC2

There you have it!

Using the Ruby Fog library to connect with Eucalyptus

There is a popular Ruby cloud library called Fog. I’ve seen a number of applications that use this for connecting to AWS EC2. I’ve done some testing and it is pretty simple to get Fog talking to Eucalyptus. The first thing you need are your access id and secret key, much like you’d get from EC2. These are available on the credentials tab in Eucalyptus. The other thing you need to specify is the endpoint. In the case of Eucalyptus, that will point to the cloud endpoint for your private cloud (or in this example, the Eucalyptus Community Cloud).

This is an example credentials file, stored in ~/.fog

#######################################################
# Fog Credentials File
#
:default:
  :aws_access_key_id:       IyJWpgObMl2Yp70BlWEP4aNGMfXdhL0FtAx4cQ
  :aws_secret_access_key:   7DeDGG2YMOnOqmWxwnHD5x9Y0PKbwE3xttsew
  :endpoint:                http://ecc.eucalyptus.com:8773/services/Eucalyptus

Notice that the eucalyptus endpoint requires port 8773 and the “/services/Eucalyptus” path.
You can use the Fog interactive tool to test this out. Notice we’re using the AWS compute provider because the Eucalyptus cloud is API compatible with EC2.

# fog
  Welcome to fog interactive!
  :default provides AWS and AWS
>> servers = Compute[:aws].servers
  <Fog::Compute::AWS::Servers
    filters={}
    []
  >
>>

As you can see, there are no servers running. By replacing “servers” with “images”, you can show a list of images available on the ECC.
To start an instance, you can run a command like this;

>> servers = Compute[:aws].servers.create(:image_id => 'emi-9ACB1363', :flavor_id => 'm1.small')
  <Fog::Compute::AWS::Server
    id="i-3D7A079C",
    ami_launch_index=0,
    availability_zone="open",
    block_device_mapping=[],
    client_token=nil,
    dns_name="euca-0-0-0-0.eucalyptus.eucasys.com",
    groups=["default"],
    flavor_id="m1.small",
    image_id="emi-9ACB1363",
    kernel_id="eki-6CBD12F2",
    key_name=nil,
    created_at=Wed Oct 19 15:19:16 UTC 2011,
    monitoring=false,
    placement_group=nil,
    platform=nil,
    product_codes=[],
    private_dns_name="euca-0-0-0-0.eucalyptus.internal",
    private_ip_address=nil,
    public_ip_address=nil,
    ramdisk_id="eri-A97113E4",
    reason="NORMAL:  -- []",
    root_device_name=nil,
    root_device_type=nil,
    state="pending",
    state_reason=nil,
    subnet_id=nil,
    tenancy=nil,
    tags=nil,
    user_data=nil
  >
>> servers = Compute[:aws].servers  <Fog::Compute::AWS::Servers
    filters={}
    [
      <Fog::Compute::AWS::Server
        id="i-3D7A079C",
        ami_launch_index=0,
        availability_zone="open",
        block_device_mapping=[],
        client_token=nil,
        dns_name="euca-0-0-0-0.eucalyptus.eucasys.com",
        groups=["default"],
        flavor_id="m1.small",
        image_id="emi-9ACB1363",
        kernel_id="eki-6CBD12F2",
        key_name=nil,
        created_at=Wed Oct 19 15:19:16 UTC 2011,
        monitoring=false,
        placement_group=nil,
        platform=nil,
        product_codes=[],
        private_dns_name="euca-0-0-0-0.eucalyptus.internal",
        private_ip_address=nil,
        public_ip_address=nil,
        ramdisk_id="eri-A97113E4",
        reason="NORMAL:  -- []",
        root_device_name=nil,
        root_device_type=nil,
        state="pending",
        state_reason={},
        subnet_id=nil,
        tenancy=nil,
        tags={},
        user_data=nil
      >
    ]
  >

You’ll notice that now the instance (which is in “pending” state) appears in the list of servers.
I won’t show more here, but we’ve established basic connectivity to a Eucalyptus cloud. I hope this helps enable many existing and new applications to work with Eucalyptus!

Migrating an EC2 AMI to Eucalyptus

There have been different instructions for using an image from Amazon’s EC2 on a local Eucalyptus cluster. This seems to be what worked best for me.

The basic steps are, launch an instance of the AMI, run euca-bundle-vol with your Eucalyptus credentials, upload bundle, register. While it would be possible to use the download-bundle/un-bundle method detailed in this post, that only works with images that your account created. The use case I’m addressing here is to get starting images for building some custom images within your private cloud. Another use case is when duplicating custom images from private to public cloud for purposes of cloud-bursting. That’ll be covered in another post.

specifically, when converting ami-1a837773 (Ubuntu-Maverick-32bit)

ec2-run-instances ami-1a837773 -k dak-keypair

When that boots, scp the credentials zip file that you got from the ECC (or your own cloud)  (i.e. scp -i dak-keypair euca2*.zip ubuntu@50.16.60.6:.) (UPDATE: my image didn’t have zip installed, so I repackaged the zip as a tar.gz) Because Ubuntu images don’t allow root login, we can only copy files into the user directory. Ideally, we don’t want credentials on the root filesystem because they’ll end up in the bundle. So, the first thing we’ll need to do after logging into the instance is to move the zip file to /mnt directory (ephemeral store). (There are additional security concerns that may apply. This post at alestic.com covers that well.)

On the instance;

sudo mv euca2*.zip /mnt
cd /mnt
sudo unzip euca2*.zip
source eucarc

To bundle/upload the image, you’ll need the euca2ools. There are some instructions here that help. This Maverick image already has them installed.

If the image has a default kernel specified (as this Maverick one does), that aki id won’t work on eucalyptus. For the ECC, looking at the list of images shows that many of them specify the eki-6CBD12F2 kernel, so I will also use that when overriding the EC2 kernel.  If you run your own Eucalyptus installation, it is easy to get the default kernel id via the management interface on the “Configuration” tab. Take note of the ramdisk id also, since that goes hand-in-hand with the kernel.

In the case of a private Eucalyptus installation, network restrictions probably won't allow the EC2 instance to upload to Eucalyptus directly. One way to do that is downloading a gzipped image to your local machine, run euca-bundle-image prior to upload. That is time consuming and since I'm working with ECC here, all of the operations can be run on the EC2 instance.
sudo -E euca-bundle-vol -p Ubuntu-10.10-Maverick-32bit -s 2048 -d /mnt -r i386 --kernel eki-6CBD12F2 --ramdisk eri-A97113E4</pre>
euca-upload-bundle -b dak-images -m /mnt/Ubuntu-10.10-Maverick-32bit.manifest.xml
euca-register dak-images/Ubuntu-10.10-Maverick-32bit.manifest.xml

At this point, you should be all set to launch the image.

Footnote: I've tested this with a Maverick S3 backed AMI and a Lucid EBS backed AMI.

Connecting to the Eucalyptus Community Cloud with typica

Eucalyptus recently announced a public “cloud” sandbox known as Eucalyptus Community Cloud. It is a place where you can kick the tires to some degree and since they support a subset of the Amazon EC2 API, you can generally point EC2 tools at the ECC. This post will deal with using typica to interact with the ECC from within your Java software.

First thing to do is follow the ECC link above and create an account. If you already have an account to get into the Eucalyptus forums, you can login and apply for an ECC account. Once you get a confirmation e-mail and confirm the account, you’ll be able to login and get your access id and secret key. To do that, visit the ECC, login and select “show keys”, which reveal the QueryID (access id) and Secret Key. While you’re hear, you should also download credentials. This gives you a zip that contains something we’ll need later.


Jec2 ec2 = new Jec2(props.getProperty("aws.accessId"), props.getProperty("aws.secretKey"), true, "ecc.eucalyptus.com", 8773);
ec2.setResourcePrefix("/services/Eucalyptus");

Let me explain this code. The first line creates a new Jec2 object, that is configured to talk to the ECC. The “props” variable came from reading a property file containing the access id and secret key. The next parameter specifies SSL. Then, you pass the hostname for the ECC and the port it uses. After that, it would be business as usual. The EC2 sample code demonstrates some normal operations, and the API docs give a more complete picture.

When running the code, there’s a special option you’ll need as compared to using typica to talk to AWS. Since Eucalyptus clouds are generally installed with self signed SSL certs, you’ll need to specify a file that came with that credentials download in your java options. If you don’t do this, you’ll likely see a “SSLPeerUnverifiedException: peer not authenticated” error.


$ java ... -Djavax.net.ssl.trustStore=<path to files from credentials zip>/jssecacerts ... TestJec2