Deploying the Eucalyptus Management Console on Eucalyptus

The Eucalyptus Management Console can be deployed in a variety of ways, but we’d obviously like it to be scalable, highly available and responsive. Last summer, I wrote up the details of deploying the console with Auto Scaling coupled with Elastic Load Balancing. The Cloud Formations service ties this all together by putting all of the details of how to use these services together in one template. This post will describe an example of how you can do this which works well on Eucalyptus (and AWS) and may guide you with your own application as well.

Let’s tackle a fairly simple deployment for the first round. For now, we’ll setup a LaunchConfig, AS group and ELB. We’ll also set up a security group for the AS group and allow access only to the ELB. Finally, we’ll set up a self signed SSL cert for the console. In another post, we’ll add memcached and and a cloudwatch alarm to automate scaling the console.

Instead of pasting pieces of the template here, why not open the template in another window. Under the “Resources” section, you’ll find the items I listed above. Notice “ConsoleLaunchConfig” pulls some values from the “Parameters” section such as KeyName, ImageId and InstanceType. Also uses is the “CloudIP”, but that gets included in a cloud-init script that is passed to UserData. Also, notice the SecurityGroups section that refers to the “ConsoleSecurityGroup” defined further down.

Right above that is the “ConsoleScalingGroup” which pulls in the launch config we just defined. Next “ConsoleELB” defines an ELB that listens for https traffic on 443 and talks to port 8888 on autoscaled instances. It defines a simple health check to verify the console process is running.

The “ConsoleSecurityGroup” uses attributes of the ELB to allow access only to the ELB security group on port 8888. We also allow for ssh ingress from a provided CIDR via “SSHLocation”.

To automate deploying the console using this Cloud Formations template, I wrote a shell script to pass the required values and create the stack. At the top of the script, there are 3 values you will need to set based on your cloud. CLOUD_IP is the address for your cloud front end. SSH_KEY is the name of the Keypair you’d like to use for ssh access into the instances (if any). IMAGE_ID must be the emi-id of a CentOS 6.6 image on your cloud. There are other values you may wish to change just below that. Those are used to create a self-signed SSL certificate. This cert will be installed in your account and it’s name passed into “euform-create-stack” command along with several other values we’ve already discussed.

If you’ve run this script successfully, you can check the status of the stack by running “euform-describe-stacks console-stack”. Once completed, the output section will show the URL to use to connect to your new ELB front-end.

To adjust the number of instances in the scaling group, you can use euscale-update-auto-scaling-group –desired-capacity=N. Since the template defines max count as 3, you would need to make other adjustments for running a larger deployment.

Check back again to see how to configure a shared memcached instance and auto-scale the console.

Working with Magic Search from Bower

In a previous post, I talked about an AngularJS widget for searching. Now, that has been re-factored a little with an eye towards reuse and is available using Bower. There was a little discussion about re-use in the last post, but now the task is much simpler. If you are already using Bower, you can simply add “angular-magic-search” to your bower.json. The widget can now be integrated into your templates and leverages which ever flavor you use for resource location and parameter passing. For example, using Chameleon templates, we do the following:

<link rel="stylesheet" type-"text/css" href="${request.static_path('eucaconsole:static/js/thirdparty/magic-search/magic_search.css')}"/>
<script src="${request.static_path('eucaconsole:static/js/thirdparty/magic-search/magic_search.js')}"></script>
  strings="{'remove':'${layout.ms_remove}', 'cancel':'${layout.ms_cancel}', 'prompt':'${layout.ms_prompt}'}"
  facets="${search_facets}" filter-keys="${filter_keys}"></magic-search>

The first 2 lines pull in the CSS and JS files. The div tag sets the ng-app and contains the magic-search element. Attributes are used to set that up, including template location, strings (which were run through i18n), as well as facets and filter keys.

The facets value is a JSON structure that looks like this:

 {'name': 'owner_alias',
  'label': 'Images owned by',
    [{'key': '', 'label': 'Anyone'},
     {'key': 'self', 'label': 'Me (or shared with me)'}]
 {'name': 'platform',
  'label': 'Platform',
    [{'key': 'linux', 'label': 'Linux'},
     {'key': 'windows', 'label': 'Windows'}]
 {'name': 'architecture',
  'label': 'Architecture',
    [{'key': 'x86_64', 'label': '64-bit'},
     {'key': 'i386', 'label': '32-bit'}],

It is used to populate the facets and is specific to the data being presented. Note that labels should have been run through your i18n function. The filter-keys value is an array of names. These are passed with the “textSearch” event so that the code you write to perform live text filtering can know which data values to look at.

['architecture', 'description', 'id', 'name', 'owner_alias', 'platform_name', 'root_device_type', 'tagged_name']

The final piece is to listen for events emitted by the search bar.

$scope.$on('searchUpdated', function($event, query) {
$scope.$on('textSearch', function($event, text, filter_keys) {

In the first function, “query” is a query fragment generated by the search facets. One use may be to append that to a URL for an XHR call to retrieve a new data set from the server. The second function gets the “filter_keys” discussed above and “text” which is simply the text the user typed which is not part of a pre-defined facet.

Hopefully, this makes it easier to re-use magic-seach in your application!

(coming next, magic-search and bootstrap)

Magic Search : facets and text in a single widget for efficient search UX

The Eucalyptus console has historically taken different approaches to search. Early versions simply filtered by text on the client. The next version introduced faceted search by using the Visual Search widget. This worked OK since we adopted backbone.js and used a client-side data model. In an effort to free ourselves from so much javascript on the client (and less data on the client), we re-built the console from the ground up in 2014. We went with much more server-side processing and routing along with AngularJS on the client. We also went with a very basic form based filter set which used fetches from the server to apply those filters. We also supported text filtering via a small search-bar.

In all cases, the search was localized to data we were displaying in a table or grid view (user selectable). Search filters were very much context sensitive. For each page, we define a list of columns that text search applies to (and mostly all columns were specified). We also used the query string in the URL to specify filters, so filters became book-markable.

As I write this, we’re finalizing the 4.1 console, which is the second release of the new code-base. In planning for version 4.2, we decided we’d like to revisit the faceted search we tried before. The widget we had used required backbone and had no desire to introduce that into an Angular application. I also surveyed other search widgets and didn’t find anything that matched that level of functionality and UX. The decision was made to develop our own Angular widget. We have a feature branch on github

Screen Shot 2015-01-07 at 9.53.41 AM

How We Built It

Our console uses Chameleon templates for basic DOM structure and SASS for styling on top of Foundation. We started by creating a widget template which defined the magic search layout. It is decorated with a single foundation element to make it full-width as well as many Angular attributes. The Angular controller is what primarily drives the functionality of the search. A foundation dropdown menu is used to display search facets and value lists. Angular’s ng-repeat is used to render selected facets and items on the dropdown. The controller simply maintains lists of these things which are displayed as needed.

How to Use It

To actually use the magic search, you must initialize the controller with a list of facets in JSON and a list of filter keys (which are the columns used for text filtering). The magic search bar emits events when search actions are performed. An event called “searchUpdated” is emitted with a query string when filter facets are changed. An event called “textSearch” is emitted when text search changes. Live text filtering is supported by emitting this event for each character typed in the search input which does not match a facet.

The application can choose what to do with those 2 events. In our case, we use an XHR call to populate our tables. When the “searchUpdated” event is received a new XHR call is made using the query string. This causes a server-side fetch using the new filter values. Our application responds to the “textSearch” event by live filtering the existing list using the filter keys to inspect the objects that make up the list.


We paid a lot of attention to usability in our design. I worked with Jenny Loza to refine all of the user interaction. The user is initially presented with a blank search bar and some placeholder text. Upon clicking anywhere in the search bar, the list of facets is presented and focus is set for text input. The user may select a facet with the mouse or type. If they type and the text matches any text in the facets, the list of facets will be filtered (and matching text bolded). This lets the user use a few mouse clicks to filter items, or continue entirely with the keyboard. If the user chooses the keyboard, typing and tabbing will get them through as many facets as they wish to select (including text search). Alternatively, a user may select facets and values entirely with the mouse (excluding text search). Each facet can be removed by clicking a small X in the facet box. The entire search bar can be cleared by selecting an X to the far right. We chose not to support edit-in-place for existing facets since it is very simple to remove and add facets.

The magic search bar allows multiple facets and those are combined to reduce the set of results (using “and”). If the user selects multiple values for a single facet, those results are combined (using “or”). We find this very intuitive.

Screen Shot 2015-01-07 at 9.51.16 AM

Screen Shot 2015-01-07 at 9.51.53 AM

Screen Shot 2015-01-07 at 9.52.32 AM

Screen Shot 2015-01-07 at 9.53.13 AM

How Your Project Might Use It

I recognize that not everybody will use Foundation, or the same server-side templating (or even SASS). Here are some ways you could approach re-use in some form. In place of Foundation, you could use Bootstrap dropdowns. They use the same DOM structure as Foundation’s and similar activation, so this would be an easy switch. Note that the “hideMenu()” function in the Angular controller uses a Foundation call to close the dropdown, so that would need to be replaced as well.

Our reliance on a server-side template is very minimal. Two places insert hrefs for resources (css and js files). The other 2 template references are to send initialization values to the Angular controller. You could replace those fairly easily in your own application.

The last thing is SASS. We check in the generated css file, so you could simply use that instead of our .scss file. The only external reference our .scss file uses is a dark grey color used more widely in our application. Additionally, our application defines an item-list and item class which are used for facet display. Those can be found as a SASS mix-in here. It’s likely there are other classes in the widget that were re-used either from our application or Foundation’s own styles. I’ll try to document any further exceptions. Please notice we use 2 SVG icons from Foundation, fi-filter and fi-x.

Getting Involved

This feature is still in development, though we think there isn’t much left to change. I do expect to find bugs which we’ll fix in the 4.2 development process. The code can be viewed in the Pull Request where you can also comment. The key files are, magic_search.js and magic_search.scss. There is a ticket which we use to track this feature. It includes a list of test criteria that we’ll use to create functional tests, and it should give you a better idea of the capabilities built into this widget. Feel free to contact me with any comments, suggestions, concerns or otherwise.

Running the Eucalyptus Management Console on Eucalyptus with the triangle services

At Eucalyptus, we’ve leveraged the existing compute infrastructure to deploy some new services. For example, ELB and our imaging service user workers that run as instances. This is useful because the cloud administrator won’t need to configure new machines to handle these tasks. The workers can be dynamically provisioned on top of existing infrastructure and that’s what cloud is all about! The management console can be deployed on top of Eucalyptus as well. In fact, using ELB and Autoscaling, we can provide a single service endpoint for users and runs a scalable back-end. Since Eucalyptus provides RHEL/CentOS packages, I started by installing a CentOS 6 image from This image included cloud-init so I can very easily provision the console on an instance with user data. Here is the cloud-init script you would supply in user data. The one value that needs to be adjusted for your install is the cloud IP address (

# vim: syntax=yaml
# This config installs the eucalyptus and epel repos, then installs and
# configures the eucaconsole package
 - [ yum, -y, install, "" ]
 - [ yum, -y, install, eucaconsole ]
 - [ sed, -i, "s/localhost/", /etc/eucaconsole/console.ini ]
 - [ service, eucaconsole, restart ]

Here are some commands you can run with euca2ools to set things up. First, assume the above script is stored in a file called “console-init”

eulb-create-lb -z PARTI00,PARTI01 -l "lb-port=80, protocol=HTTP, instance-port=8888, instance-protocol=HTTP" console-lb

The cloud I used had 2 clusters shown above. I also set up port 80 on the elb to talk to port 8888 on the instances. We could also set up port 443 and SSL termination instead. Now, run eulb-describe-lbs console-lb –show-long and you’ll notice the owner-alias and group-name values. That’s the internal security group you’ll need to authorize port 8888 ingress for. What that does is indicate these instances only give access to ELB traffic on the port the console runs on. Run the euca-authorize command using the owner-alias and group-name (i.e. euca-authorize -P tcp -p 8888 -o euca-internal-276586128672-console-elb -u 641936683417 console-as-group).

euscale-create-launch-config -i emi-22536a68 -t m1.medium --group console-as-group --key dak-ssh-key --monitoring-enabled -f console-init console-launch-config

The launch config needs the CentOS 6 EMI ID. I also used an m1.medium since it uses more memory, but still a single CPU. You can certainly dedicate more resources to single instances as you see fit. Specifying an ssh key is optional unless things have gone pear-shaped.

euscale-create-auto-scaling-group -l console-launch-config -m 1 --desired-capacity 2 --max-size 4 --grace-period 300 -z PARTI00,PARTI01 --load-balancers console-lb consolegroup

The autoscaling group ties things together. After the last command runs, you should get 2 instances pending. Once those are up, eulb-describe-instance-health console-lb will show you the state of the instances from an end-user perspective. An “InService” instance can handle requests going through the ELB whereas “OutOfService” instances may still be installing/configuring per cloud-init. The grace period determines how long the scaling group waits for those to be ready. There is a lot more we could do with cloud watch data and autoscaling. For now, this setup will let you manually adjust the number of instances you dedicate to the console scaling group. You can point your browser to the ELB DNS name and see the console login screen!

Extra Credit

Let’s setup SSL termination for ELB. You either have your own certs or you could generate your own. Here are the commands to generate self-signed certs:

openssl genrsa 2048 > myssl.pem
openssl req -new -key myssl.pem -out csr.pem
openssl x509 -req -in csr.pem -signkey myssl.pem -days 365 -sha512 -out myssl.crt
chmod 600 myssl.*

Now you have the key and cert you need. The csr.pem file can be discarded. Now, upload the cert

euare-servercertupload -s myssl --certificate-file myssl.crt --private-key-file myssl.pem

To get the ARN for this cert, run “euare-servercertgetattributes -s myssl”

Now, add the listener to the ELB

eulb-create-lb-listeners console-lb --listener "protocol=HTTPS,lb-port=443,instance-port=8888,instance-protocol=HTTP,cert-id=arn:aws:iam::276586128672:server-certificate/myssl"

Now you can use the console with https! To see details of the ELB, run “eulb-describe-lbs console-elb –show-long”. You might want to remove the port 80 listener. To do that, type “eulb-delete-lb-listeners -l 80 console-lb”.


Adventures in memcached integration

As developers, we sometimes run into problems that are somewhat… challenging. That’s part of the fun of writing code though. I like trying to find clever ways to solve a problem. This was the case when trying to integrate memcached into the Eucalyptus Management Console.

Version 4.0 of the console uses Gunicorn which utilizes separate worker processes to handle requests. To implement any kind of effective caching, we’d need a shared cache. Memcached is a pretty obvious choice. Since we were using pyramid, beaker seems like an obvious option. Beaker does have support for memcached, but as the author points out, dogpile.cache is a much better choice as a cache interface library. Dogpile.cache has backends for memcached, redis and others which allow for some more interesting choices architecturally.

Our application uses boto to talk to both Eucalyptus and AWS. To start with, we wanted to cache image lists since they don’t change often and they can be fairly large. Dogpile.cache has regions you configure (generally for different expiration times). We set up short_term, long_term and others for our application. While working on a prototype for this, I ran into 2 main issues which I’ll cover in detail: pickle doesn’t handle all boto object graphs and invalidation of cache data.

Pickled Botos

We have an array of boto.ec2.image.Image objects that need to be cached. The memcached backend for dogpile.cache can use one of a few python interfaces to memcached. I chose to use python-memcached. It pickles the data before sending it to the memcached server. For those who don’t know, pickling is a way to encode python data and can be used to marshall and unmarshall object graphs. Anyway, some boto objects don’t marshall very well. I ran into this about 2 years ago when working on the first version of the console that used the JSONEncoder to send json versions of the boto objects to the browser as AJAX responses. I had to write my own JSONEncoder to handle the objects which didn’t marshall properly. The JSONEncoder supports passing your own implementation which handles object conversion, so that made life a little easier. The Pickler also supports this, but the implementation is buried down in the python-memcached package and there is no way to pass your own pickler down from the dogpile.cache layer. (I feel a pull request coming..) What I chose to do instead was to iterate over the image list and make adjustments to the objects graphs prior to storing in the cache. Certainly, this isn’t ideal, but it works for now. In doing this, I was able to delete some values out of the object graph which I don’t care about which saves time and space in the cache mechanism. I also found that (in this case) the boto.ec2.blockdevicemapping.BlockDeviceType object contained a circular reference which was causing the pickler to barf. I trimmed this out during my iteration and pickling worked fine!

The hard part was figuring out which object was causing the problem. I found a stackoverflow article that helped here. It showed how to extend the Pickler to either log what it was operating on, or catch exceptions (as I added for my purposes). Here’s my code;

class MyPickler (pickle.Pickler):
  def save(self, obj):
    #print 'pickling object', obj, 'of type', type(obj)
    try:, obj)
      print "--------- object dict = "+str(obj.__dict__)

I found it very helpful to see what object was causing the problem and could insert a breakpoint to inspect that object when the problem occurs. In the file of python-memcached, I had to change an import so that cpickler wasn’t used. That’s a native pickler which is much faster, but doesn’t allow me to extend it in this way. This is clearly only a debugging tool and the standard package code should be used in production.

Invalidate == Delete

Each item stored in a cache region has a key generated. When using the @cache_on_arguments decorator, the cache key is created based on the string form of the arguments passed to the cache function. The decorator takes a namespace argument, so I was able to specify an additional key component so that any image values being cached all included “image” in the cache key. By default the key is also run through sha1 to create a digest to get consistent length (and obfuscated) cache keys. This works well and would have been all I had to do except that I couldn’t simply rely on the configured expiration of the cache region. There are cases where we needed to invalidate the set of data in the cache due to changes initiated within the application. In that case, our user would expect to see the new data immediately.

To invalidate, we would need to know the cache key used to refer to the data in the cache and perform a delete on the key. Since the cache key was generated for us, I had no idea what to use for deletion. I could have reverse engineered it, but if something changed in the underlying library, that could be fragile. Fortunately, a cache region can be given a key generator function when it is configured. We could use our own code to generate the cache key and call that again to invalidate the cache. This is the key generator I’m using:

def euca_key_generator(namespace, fn):
  def generate_key(*arg):
    # generate a key:
    # "namespace_arg1_arg2_arg3..."
    key = namespace + "_" + "_".join(str(s) for s in arg[1:])

    # return cache key
    # apply sha1 to obfuscate key contents
    return sha1(key).hexdigest()

  return generate_key

To use this to invalidate a cache (based on args), I wrote another function:

def invalidate_cache(cache, namespace, *arg):
  key = euca_key_generator(namespace, None)(*arg)

The namespace and arg list are passed to the key generator as you can see. This is merely a helper function. To invalidate the image cache, I needed to call the above function with the proper arguments. These are the same arguments passed in to the cache function (which uses the decorator).

The work on shared caching is currently in a branch, but will likely be merged into develop over the next month or so.

Size Problem

After beating my head against a wall for a while, I found there is a size limitation on memcached. It will only take values up to 1MB in size unless you recompile it. Fortunately, there is a handy solution. Since value get pickled, they do really well with compression. The python-memcached library supports compression, but you need to enabled it. By default the min_compress_len is zero, which means it never tries to compress the pickled data. In fact, the code silently returns from the set method having done nothing. This is where the frustration came in. I ended up spending some quality time in pdb to figure out that I could configure a dogpile.cache region with a min_compress_len greater than zero to get the underlying code to compress my data. Bingo! My large data set went from 3MB to 650K. This is how I configured my regions:

     expiration_time = int(settings.get('cache.long_term.expire')),
     arguments = {

I realize that 650K is not that far from 1MB, so perhaps splitting up the data will be needed at some point. The failure mode is simply a performance one, not so fatal.

Memcached Debug Tips

I learned a couple of things about monitoring my memcached server while debugging things. Two tips I found that will help are:

  • run memcached from a shell with -vv option. You’ll get useful output about get, set, send and delete operations.
  • telnet into the server using “telnet localhost 11211”. You can run commands like “stats” and “stats items”

IAM Role support for the Eucalyptus Management Console

If you like using IAM Roles in Eucalyptus, there’s a nice option for you to try out. Since Eucalyptus 4.0 was just released with a brand new management console, I thought it would be a good time to let you know about a feature branch you can try out if you don’t mind installing from source. Please find instructions in the README for installing dependencies and running from source. You can install using “python install”

If you haven’t seen the new console yet, it adds support for IAM Users and Groups. This means you can manage users, groups and policies for the account, all from the console. This branch adds support for IAM Roles. Along side users and groups, you’ll be able to create, view and delete roles. To use roles, you assign them to instances or launch configurations (for AutoScaling). This allows you to assign special privileges to instances. I’ve added the ability to assign a role in both the new instance wizard and launch configuration wizard. The required IAM Instance Profile is handled for you by the console.

Here are some screen shots to show you a few of the changes.





Angular JS inter-controller communication

I’ve been using Angular JS for a few months now and started doing more and more with it. Quickly, I ended up having reusable widgets with their own controllers. When I wanted to use them within another app (page), I included them and ran into another problem. I wanted to be able to expose functions or pass data between them. For example, one controller managed a generic set of data provided by some AJAX request. That table would be embedded within another page.

angular.module('myPage', ['tableWidget']).controller(...)

I had a special case where I wanted to lazy-load some details of the table, but had no way for the main controller to access the scope of the tableWidget controller. I dug around a bit and found I could use $emit and $on to pass events. For example, on the re-usable tableWidget controller, when data had finished loading, I would emit an event.

$scope.$emit('itemsLoaded', $scope.items);

That happens every time the widget loads data, on every page. But, on my page, I wanted to know about that so I set up a listener.

$scope.on('itemsLoaded', function($event, items){
  for (var i=0; i < items.length ; i++) {
    var item = items[i];
    // do something with item

Pretty slick, right? Well, I had another problem. The table widget had a reload button to allow the user to cause another AJAX fetch without reloading the entire page. I needed to trigger that fetch from my main controller. I tried $emit, but no good. I finally found $broadcast.


Within the table widget controller, I listened.

$scope.on('refresh', function($event) {
  // trigger the ajax call

I hope this is helpful! It really allowed me to make the reusable components in my application much more useful!