Embedded Pentaho Data Integration - Raspberry Pi

Embedded Pentaho Data Integration - Raspberry Pi

This week I was asked if it was possible to run a Pentaho Data Integration transform in an embedded use case. My initial thoughts were that I didn't know, but I didn't see why not. Best try it out.

In the UK the Raspberry Pi is a very popular, readily available embedded platform that's used in all sorts of fun hacky projects.  Costing about £25 ($40) it's also pretty cheap.  It has a 700 MHz ARM 11 based CPU  and 512MB RAM, so it's no powerhouse, but should be enough.

The board itself is about the size of a credit card and comes with a good selection of ports, HDMI, 4x USB, ethernet and a mini USB for power.  It also has a 40 pin GPIO (General Purpose Inout Output) connector that has a wide range of possibilities.  

The board can be supplied with a 8GB SSD that comes with a collection of operating systems and can then be used for storage and booting.  

To get started I installed Raspbian which is a Debian clone optimized for the PI.  Installing took a couple of minutes and the OS booted, initially I was connected to a monitor and keyboard just to get the setup done.  Once the initial setup was complete and I was on the network, I just had to enable SSH then login over the network.  After this point I dispensed with the keyboard, mouse and monitor.  

I obviously wasn't going to run Spoon, the transformation development environment, but my objective was to see if I could run "a" data transformation on the platform. One way to achieve this was to run a Carte server, this allows you to connect remotely and run transformations.  

The Carte server can be copied over from the data integration design tool folder,  and to my utter amazement with a couple of bugs and errors in the console (some X11 related, could be connected with running headless) the server started up first time.  (I know that's supposed to happen with Java, but still).  

So the next part was to create an ultra simple transformation just to show that things work!  

So this just creates a few rows, waits 10 seconds between each row, gets some system information and writes it to the log, virtually pointless but proves the use case non the less. So the next part is to configure the Carte server, View Tab->Slave Server->New and enter the config.

All configured, now just select run the transform and execute remotely selecting your new slave config, and off it goes.

Just to be sure that it was executing on the Raspberry, here is the console output on the Pi.

So it works, whats next?  That's where the fun can begin, there are a huge range of applications of what this could enable, and additionally lots of options on how to communicate with the remote devices and make them as autonomous as possible.  Hopefully I'll find the time to try some of these out!