About

Ian Ozsvald picture

This is Ian Ozsvald's blog (@IanOzsvald), I'm an entrepreneurial geek, a Data Science/ML/NLP/AI consultant, author of O'Reilly's High Performance Python book, co-organiser of PyDataLondon, a Pythonista, co-founder of ShowMeDo and also a Londoner. Here's a little more about me.

High Performance Python book with O'Reilly

View Ian Ozsvald's profile on LinkedIn

ModelInsight Data Science Consultancy London Protecting your bits. Open Rights Group

Archive

11 December 2009 - 16:35Eucalyptus Clustering – follow-up

A month back I tried to build an Ubuntu-based Eucalyptus cloud/cluster environment for a client for a parallel processing research project.  The project was thwarted by an overly aggressive corporate firewall and my lack of understanding of low-level network config-fu.

I’ve revisited the project using the same machines but with an external public internet connection (no firewall – yay!).

Grub2

On the node machine I still needed to dual-boot to Windows.  Unfortunately whilst reboots to Linux are fine, if Windows is booted it ‘does something’ to the MBR and the machine is unbootable.  I delved into the boot-loader and had to learn some Grub2-fu.

Grub2 was introduced in Ubuntu 9.10, it replaces Grub which in turn replaced boot managers like lilo.  The wiki page is pretty good for recovering a boot-loader using an Ubuntu LiveCD but it didn’t work quite to plan.

The step for ‘sudo chroot /mnt’ fails as bash or sh can’t be run from within /mnt (which at this point is looking at the originally installed hd).  There is something odd going on with the LiveCD, much googling didn’t seem to reveal the answer.

To run grub-install on the hd, rather than via the CD (because chroot fails) I used ‘sudo grub-install –root-directory=/mnt /dev/sda’, it reports that ‘(hd0) /dev/sda’ is installed.

Sidenote – on later attempts somehow a reference to (fd0) got involved and this broke the boot process.  I edited /mnt/boot/grub/device.map to remove the fd0 reference, leaving the hd0 reference.  I ran grub-install again and all was fine.  Now the machine can boot again.

Mounting a USB memory stick

Whilst a 8Gb memory stick was recognised, it didn’t get mounted.  I had to edit /etc/fstab and add:

/dev/sdf1 /mnt/stick auto umask=0,user,iocharset=iso8859-1,sync,codepage=850,noauto,exec,users 0 0

After this I used ‘sudo mkdir /mnt/stick’, ‘sudo mount /dev/sdf1’ and it mounted just fine.

Installing Eucalyptus

The install process this time around was much the same as before, except this time without the firewall it all ‘just worked’.  Seeing the fnords part 1 took me through the basic install.

I got the feeling from later steps that the cloud controller needs a static IP so I switched the cluster controller from DHCP to a static IP and rebooted.

The discover nodes process (‘sudo euca_conf –no-rsync –discover-nodes’) for euca_conf also required that I’d setup ssh keys on the Node, step 6 in the NodeInstall doc has the instruction.  Typo note – if you spell ‘eucalyptus’ wrong you’ll go round in circles trying to figure out why the password won’t work!

Sometimes I couldn’t get ‘euca-describe-availability-zones verbose’ to work, it’d just report ‘No route to host’.  It seems that a reboot of the CC and Node are required, plus a minute or so of patience after boot for Apache to sort itself out, before this problems just goes away.

Using the Ubuntu Store

Having installed the CC and registered a Node, next I ran the web interface via ‘https://10.0.0.4:8443’.  Note ‘https’.  If you visit the website too soon after a reboot (i.e. <1 minute) then the webapp won’t respond or maybe it won’t recognise the admin user.  Having logged in, the first login forces a password change.

Next check the ‘Configuration’ tab and verify the IP addresses.  For reasons beyond my understanding our switch rebooted during my first attempt to setup the cluster and it switched from the ‘192.168.x.x’ address range to ’10.x.x.x’ – this royally barfed my configuration.  I chose to re-install the CC from scratch (I was plagued by ‘no route to host’ problems no matter how much tweaking I tried).

Next visit the ‘Store’ tab and download an image, I’m using ‘Ubuntu 9.10 Karmic Koala (i386)’.  Today this works – I’ve spent 2.5 days building and re-building the cluster to get it to this point.  Often the Store would download an image and then report ‘no route to host’.  This process is pretty darned frustrating and seems to lack useful error messages.

But ultimately – no cigar

Rather frustratingly I can’t get my Node to run an image.  I can see that the Node exists though ‘euca-describe-availability-zones verbose’ shows that a Node exists but doesn’t list its IP address which is odd, the online docs say it should be shown.

If I run an image then it enters the ‘pending’ state and then the ‘terminating’ state.  Digging around in Google shows that other people currently have the same problem, it might be related to the lack of Hypervisor instructions on my Node machine (though they’re not supposed to be required…).  Possibly also the current build in unstable, there’s a lot of bug-fixing going on.

Debug notes

Eucalyptus has a trouble-shooting guide, this blog series is very useful.

Conclusion

Eucalyptus should give you an EC2-like cloud that runs on your own machines, using an EC2-compatible API so you could move to the cloud when you want to scale up or are less concerned about the privacy of your data.  Currently I can’t get it to work but others do have it working – it seems to depend upon your hardware.  It also lacks clear error messages so debugging is hard – I resorted to clean installs on three occasions.


Ian applies Data Science as an AI/Data Scientist for companies in ModelInsight, sign-up for Data Science tutorials in London. Historically Ian ran Mor Consulting. He also founded the image and text annotation API Annotate.io, co-authored SocialTies, programs Python, authored The Screencasting Handbook, lives in London and is a consumer of fine coffees.

No Comments | Tags: Life

Comments are closed.