Installing the Eucalyptus Console from source and packages

I previously posted some information about the new User Console we’ve been working on at Eucalyptus Systems. There has been a lot of activity and we’ve shown it to a lot of users to get feedback. We will be releasing it officially very soon, but till then, you can run it yourself a couple of ways. You can build from source, which is very easy, or install nightly builds which we provide for RHEL 6 and CentOS 5 and 6.

Did You Get the Package?

Packages are available in the nightly directory here:

http://downloads.eucalyptus.com/software/eucalyptus/nightly/3.2/

Configure the repo like this:

rpm -Uvh http://downloads.eucalyptus.com/software/eucalyptus/nightly/3.2/centos/6/x86_64/eucalyptus-release-3.2.0-0.1.el6.noarch.rpm

You’ll also need to have the elrepo configured as well.

rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm

Next, run

yum install -y eucalytpus-console

Next, you’ll need to configure the cloud location and perhaps some other things. The file is located in /etc/eucalyptus-console/console.ini  The config file has a lot of settings, but there are just a few you need to understand to get going quickly.

clchost – this is the IP or dns name for the eucalyptus cloud you’ll connect to

uiport – defaults to 8888, but you can use a different port if you like

sslcert, sslkey – these values are used to configure SSL, which you don’t need to do for development

usemock – this is important if you don’t have a cloud to talk to. Setting this to true instructs the user console to load mock data and many features operate on the mock data (though many also don’t work as well). In this mode, the console can be run standalone for simple demos or to work on the browser side like when you want to changing branding or other look and feel items.

Now, start the service. You’ll see something like this;


# service eucalyptus-console start
Generating self-signed certificate: [ OK ]
Generating cookie secret: [ OK ]
Starting eucalyptus-console: [ OK ]

By default, SSL is enabled. You can connect to the application using https://localhost:8888/ (assuming you’re on the same host, otherwise substitute the right hostname or IP). Skip down to the “Once You’re Running” section below.

Using the Source

A source install is also very easy, but there are some differences you will need to be aware of. First, the code lives on github here: https://github.com/eucalyptus/eucalyptus/tree/maint/3.2/testing

Notice that I’m showing you the maint/3.2/testing branch. That is because we’re putting the very latest fixes there. If you’re a little more risk-averse, you may want to simply run from maint/3.2/master instead. We push changes from testing to master after the code has passed QA to a reasonable degree.

To get started, you’ll need;

  • a git client
  • python 2.6 (or 2.7)
  • boto
  • m2crypto
  • tornado (2.1 or higher)
On CentOS 6,

rpm -Uvh http://elrepo.org/elrepo-release-6-4.el6.elrepo.noarch.rpm

yum install -y git python python-boto m2crypto python-tornado

or, Ubuntu 12.04,

sudo apt-get install git python-boto m2crypto python-tornado

Now, grab the source.

git clone git://github.com/eucalyptus/eucalyptus.git
cd eucalyptus
git checkout maint/3.2/testing
git pull origin maint/3.2/testing

For better or for worse, the user console is in a subdirectory of the main eucalyptus source tree. This means you pull down a whole lot more than you really need if you're only interested in the console. The good news is that in the console directory, you have a completely independent project that doesn't have any build or run-time dependencies on the rest (aside from connecting to the cloud). To get the console running, you'll first need to configure one or two things.  The console/eucaconsole/console.ini file has many settings, which we cover in the package install section above.

Once you've set things up to your liking, simply run

cd console
./launcher.sh

You should see a message like "2012-11-10 23:56:43 INFO Starting Eucalyptus Console". If not, there may be other issues you need to fix first. If you have problems and need help, visit #eucalytpus-ui or one of the other #eucalyptus channels on freenode.net.

If you saw this message, that's great! It's very likely you're ready to connect with the browser. Assuming you are running the console on the same machine as your browser, use the appropriate localhost URL like "http://localhost:8888/" if you kept the default port and ssl config (off).

Once You're Running

You should see a login page. If you're using the mock (usemock: True), you can put anything in the fields, or nothing and simply login. If you're connecting to a Eucalyptus cloud, you'll need to use the regular web login credentials you'd use to login to the Eucalyptus admin console (read on...)

* Warning: Science Content *

The user console uses temporary session credentials from the STS service to make calls, so your actual access key and secret key are never passed to the console app. As a result, accounts that want to use the console must have a) a password set up and b) access credentials assigned. These things can be done by the cloud admin via the admin console or using euca2ools like this;

euare-accountcreate -a testuser1
euare-useraddloginprofile --delegate testuser1 -u admin -p euca123
euare-useraddkey --delegate testuser1 -u admin

Another way to do this would be using the Eucalyptus Admin UI which you can reach in your web browser here: https://<your-cloud-frontend&gt;:8443/ and there are some helpful docs here.

Assuming you were able to login, you should be ready to explore!

VMWare Fusion Error Recovery

I’m running VMWare Fusion on my MacBook Pro. I run an Ubuntu Maverick VM for some development work. One time, I tried restarting it from a suspended state and kept getting an error that “unexpected signal 10 received”. I was told to look for a log file, send it to VMWare support, etc, etc. I was unable to get the VM back up and had no option to boot the VM from scratch.

The answer turns out to be fairly simple. In my ~/Documents/Virtual Machines/ directory, there’s a directory for my VM. If I browse there in the finder, I can right-click and select “show package contents”. In the files contained in the VM directory, there is one ending in vmem and a directory ending in vmem.lck. I moved both of those to the trash and tried launching the VM again. This time, it said there was an error and would I like to preserve the state or not. I chose not to, and then I was able to boot the VM from scratch!

How to build a local NAS backed by Amazon S3

A previous post talked about my need for some local, reliable storage in my home. That project led to investigating some other options. Since I’m a big fan of Amazon S3, it seemed like something I should involve in my storage solution. The Elastician (Mitch Garnaat) and I bought the same hardware and are working through the setup together. Here’s the rundown of the hardware including costs;

Cooler Master Elite 360 m-ATX ATX Mid/Mini Tower Case with 350-Watt Power Supply RC-360-KKR1 $56.97
Gigabyte Core 2 Quad/Intel G41/DDR2/A&V&GbE/MATX/DualBIOS Motherboard GA-G41M-ES2L $56.99
Intel Pentium E5300 2.6GHz 2M L2 Cache 800MHz LGA775 Desktop Processor $66.99
Corsair XMS2 4 GB (2 X 2 GB) PC2-6400 800 MHz 240-PIN DDR2 Dual-Channel Memory Kit – TWIN2X4096-6400C5 $94.99
Western Digital 1 TB Caviar Green SATA Intellipower 64 MB Cache Bulk/OEM Desktop Hard Drive WD10EARS $54.49 * 2
Kingston DataTraveler 112 – 8 GB USB 2.0 Flash Drive DT112K/8GBCL (Black) $13.93 * 2
RadioShack® Molex® to SATA Power Cable $2.99

My previous post discusses the hardware in more detail and some of the choices. Here’s a picture of inside of the case once things were assembled. The observant among you would notice that one of the drives doesn’t have power. That’s because the case power supply didn’t have 2 SATA power connectors and the adapter cable was on order when this picture was taken. I’ll also point out that this case isn’t ideal for mounting several 3.5″ drives. With adapters, I can fit 4 in there, true. However, shopping around for something more to my liking is something I’d do differently next time. Thinking more about the software to run on the NAS has led to several projects including FreeNAS and OpenFiler. We decided to go with something we’re familiar with, Ubuntu. Ubuntu has instructions on their download page for creating a bootable flash drive. I tried the Mac OS-X method and failed, so I resorted the tool from pendrivelinux.com on the family window box. The Universal USB Installer they have works well and created good, bootable flash drives every time.

Creating a Bootable Flash Drive

I tried the Ubuntu Server download, but that seems to be geared towards jumpstarting a server install vs running right off the flash drive. The Ubuntu Desktop was much more to my liking.

To get things going, I needed to connect a mouse/keyboard/monitor. Once I configured the BIOS to boot from the USB HDD, it recognized the bootable flash drive and started bring Ubuntu up. It seems to take “forever” to boot up. I could hit “escape” to watch the console and found that it was timing out on the floppy drive, which I don’t have. I went into the BIOS settings to let it know there wasn’t a floppy drive attached and boot time went WAY down! I let the desktop come up, but since this is an install image, changes made aren’t saved. Having the 2nd flash drive will come in very handy now! Plug it into another USB port before prceeding. Select the “System”->”Administration” menus, then the “Install Ubuntu… ” option. There are steps on the install wizard that require special mention. On step 4, select “erase and use the entire disk”, and select your flash drive (not of the hard drivces!). In step 5, after you’ve entered the required information, select “log in automatically”, which will help when running headless later. Now the most critical part, step 7 has an “advanced” button you need to click. Make sure  you select the proper device, because it defaults to /dev/sda (the first hard drive). You need to select /dev/sdd, which is the last device connected (the target flash drive). Let the install proceed and you’ll have a bootable ubuntu image we can start configuring.

Remote Desktop for Administration

Once it was up, I could use the desktop and configure Remote Desktop. Having played with the default VNC server, it seemed like the wrong option. It didn’t run unless I had a monitor attached, so I did some digging and found that tightVNC is a popular alternative. There are a few steps to getting it installed and running at boot, detailed here.

For another means of access, its a good idea to install ssh (“apt-get install openssh-server”)

Configuring the RAID

The Disk Utility also has a menu option to configure the RAID. It uses mdadm, but I heard some folks talking about using lvm. Linux Mag has an article that talks about both. I decided to go with the built-in option.

Run “apt-get install mdadm” in a termal window. You can then use “Disk Utility” (on the “System”->”Administration” menu). One thing I noticed is that if you play around with RAID config or do your own partitioning of the drives, the RAID wizard isn’t really happy about using those drives. If this is the case, select each drive and then “Format Drive”. Select the “Don’t Partition” option to reset the drive state. You’ll find that you can now select the drives in the RAID setup wizard.

I’ve set the drives up in a RAID 0 config. Prior to doing this, I did a performance test on a single drive and got an average read rate of 84MB/sec. Once the RAID was configured and formatted, I ran the same performance test and got a read rate of 155MB/sec, which is approaching double the speed! Now that’s what I was hoping for!

To get the RAID started at boot time, edit the /etc/mdadm/mdadm.conf file and replace the existing “DEVICE” line with these 2 lines;

DEVICE /dev/sda1 /dev/sdb1
ARRAY /dev/md0 devices=/dev/sda1,/dev/sdb1 auto=yes

Next, run “dpkg-reconfigure mdadm” and accept the defaults. Thanks to goldfisch.at for the help.

Now, to get it mounted, add this to the /etc/fstab

/dev/md0	/mdeia/RAID	ext4	rw,nosuid,nodev,uhelper=udisks	1	2

I might have been able to say “defaults” in that options column, but I took what was there when I mounted the RAID manually using the disk utility.

Sharing the Storage

Initially, I’m setting up Samba to share with my household machines. I found this article at ubuntu.com to help me. I’m concerned with privacy, not because I don’t trust my family, but because I plan on backing up my laptop and I don’t want others messing with my files.

I created a “data” directory on the RAID drive. If you right-click on that folder, select “sharing options”. It brings up a dialog, and if you click “share this folder”, you’ll get prompted to install some packages (do it!). I discovered that I needed to use “smbpasswd” to set the share password. I’ll probably need to do this for each user I create to access the RAID.

The Amazon S3 Backup

For the Amazon S3 backup part, we’ve tossed around a number of different options. S3sync isn’t bad, but doesn’t allow for threaded uploads, and there’s the issue of how often do we kick it off. We asked, “what about running an S3 based filesystem and doing a RAID 1 on top of that and the RAID 0 local drives?”. That might be OK, but how about traffic control? What block size do we use, and what penalty do we pay for a larger block size when storing small files? Where do we store the local cache? Do we even want a local cache since we have a local disk array? Along those lines, we looked at S3Backer and others.

What is the solution when  you don’t really think the available options are great? Right your own! We think that we can write a daemon tied into the file system notification (pynotify) and use boto for the S3 part. Stay tuned… I smell another open source project!

You can now buy “options” for Amazon EC2 instances!

They don’t call it options, They call it “Reserved Instances”. Let’s look at the details first.. then I’ll run some numbers.

Reserved Instances give you the option to make a low, one-time payment for each instance you want to reserve and in turn receive a significant discount on the hourly usage charge for that instance. After the one-time payment for an instance, that instance is reserved for you, and you have no further obligation; you may choose to run that instance for the discounted usage rate for the duration of your term, or when you do not use the instance, you will not pay usage charges on it.

Linux/UNIX One-time Fee  
Standard Reserved Instances 1 yr Term 3 yr Term Usage
Small (Default) $325 $500 $0.03 per hour
Large $1300 $2000 $0.12 per hour
Extra Large $2600 $4000 $0.24 per hour
High CPU Reserved Instances 1 yr Term 3 yr Term Usage
Medium $650 $1000 $0.06 per hour
Extra Large $2600 $4000 $0.24 per hour

Reserved Instances can be purchased for 1 or 3 year terms, and the one-time fee per instance is non-refundable. Usage pricing is per instance-hour consumed. Instance-hours are billed for the time that instances are in a running state; if you do not run the instance in an hour, there is zero usage charge. Partial instance-hours consumed are billed as full hours. 

Reserved Instances are currently available for Linux/UNIX operating systems. We expect reserved Instances to be available for the EU region in the near future. Click here to see FAQs on using Reserved Instances.

As you can see the cents/hour figures have dropped a lot! How does that work if you factor in 1 or 3 years of “reservation” money?

For a small instance, a 1 year term is $325, so that’s $325/365/24 = $0.03710 per hr. Let’s add that up. You can now get a small instance for $0.0671 per hr instead of $0.10. That’s a savings of 0.0329 per hour, or $288.20 per year! If you bought into 3 years, $0.049 per instance per hour and $446.76 saved per year! That’ll pay a pretty nice sized Amazon S3 bill!

As they say, the savings scale with the instance size! To summarize;

Standard Linux/Unix

1 year 3 years
Small – $0.0671 per hr, $288.20 savings per year $0.049 per hour, $446.76 savings per year
Large – $0.2684 per hr, $1152.80 savings per year $0.1961 per hour, $1786.13 savings per year
XLarge – $0.5368 per hr, $2305.60 savings per year $0.3922 per hour, $3572.27 savings per year

High CPU

1 year 3 years
Medium – $0.1342 per hr, $576.40 savings per year $0.09805 per hour, $893.07 savings per year
XLarge – $0.5368 per hr, $2305.60 savings per year $0.3922 per hour, $3572.27 savings per year

 

If you run several instances all the time, this is a huge win! If you buy into 3 years, the savings is about 50%!

Nice work, Amazon!