This should roughly guide you (albeit on a meandering path), through the process of getting a Raspberry Pi (3) up and running with Docker, and using a simple Docker-contained toolset, create your own data & control hub for the so-called Internet of Things (IoT).
<p>With the complex job or raising a newborn, and the fear put into us about keeping her temperature in some sort of Goldilocks zone, I thought I'd see if I could put an iBeacon <a href="https://twitter.com/de_velopment">a colleague</a> leant me to good use, as it has a temperature sensor (among others).</p>
<p>So, we ordered a Raspberry Pi 3, since it has both Wi-Fi and Bluetooth 4.1 Low Energy built in, and then I began setting it up. Then the first roadblock hit me... getting the latest version of Docker on an ARM-based machine.</p>
<p>While I'm looking at using the Bean, I've also looked at the <a href="http://www.theowl.com/heating/intuition-c/">Owl Intuition-C</a> heating control and energy monitoring device, which I've previously integrated with <a href="https://github.com/dsample/intuition-gem">some Ruby scripts</a>, but I've been meaning to update.</p>
<h2>The aim</h2>
<p><img src="images/post_imgs/rpi_ha_small.png" alt="Diagram showing architecture of Node-RED, MQTT, InfluxDB and Chronograf/Grafana" /></p>
<p>This was the first diagram I drew while thinking about the main building blocks of the architecture was going to be.</p>
<p>What I ended up with (for now) is a bit more simplistic than that, but it's also a bit more coupled together. The reason being that Node-RED was far more capable than I was expecting, and integrated into everything directly.</p>
<h1>Installation</h1>
<h2>Installing (compiling) Docker</h2>
<p>After <a href="http://www.makeuseof.com/tag/setup-wi-fi-bluetooth-raspberry-pi-3/">setting up Wi-Fi</a> the task was to install Docker. I quickly found <a href="http://blog.hypriot.com/">Hypriot</a>, but soon realised that the latest version they had for <a href="http://blog.hypriot.com/downloads/#hypriot-docker-debian-packages-for-raspberry-pi:2a4af035d9e12b64c084b5e7cfb2c420">download</a> was 1.10.3 (the proper latest version at this time is <a href="https://github.com/docker/docker/releases/tag/v1.12.0">1.12.0</a>).</p>
<p>After a day spare-time effort (between changing nappies and keeping mum well hydrated and fed) attempting to convert the <a href="https://github.com/docker/docker/blob/d5cbc57eff0df651bfbfb455608da45747651d0c/Dockerfile.armhf">Dockerfile for Docker itself</a> into a shell script I could run on the Pi, with limited success, Hypriot happened to publish the perfect blog post, <a href="http://blog.hypriot.com/post/building-docker-directly-on-a-raspberry-pi/">Building Docker 1.12 on a Raspberry Pi</a>.</p>
<h3>Basic steps</h3>
<ol>
<li><p>Use Docker, by using the ready-made <a href="hypriot-docker-image-for-raspberry-pi:2a4af035d9e12b64c084b5e7cfb2c420">HypriotOS</a> Raspberry Pi image, to build the Docker package</p>
<ol>
<li><code>sudo apt-get install -y dphys-swapfile</code> (add a swap file so the Raspberry Pi has enough memory to compile Docker)</li>
<li><code>git clone https://github.com/docker/docker.git</code></li>
<li><code>cd docker && git checkout v1.12.0</code></li>
<li><code>git fetch origin pull/25192/head:fix-manpages-on-arm && git cherry-pick fix-manpages-on-arm</code> (apply a necessary fix which hadn't been applied to the master repo)</li>
<li><code>time make deb</code> (compile Docker, and time it)</li>
<li>Play with your newborn baby and change several nappies while waiting </li>
</ol></li>
<li>Transfer the resulting package from the Raspberry Pi, ready for use later (they're in a directory similar to <code>~/docker/bundles/1.12.0/build-deb/*/*.deb</code>)</li>
<li>Reflash with Raspbian (or your preferred flavour) onto an SD card. <em>You can keep using HypriotOS</em>, but I'm more familar with a closer-to-pure Debian distro</li>
<li><code>apt update && apt dist-upgrade</code> (update as normal)</li>
<li><code>apt install libapparmor1</code> (install a dependency of Docker)</li>
<li>Copy the package back to the Pi and run <code>dpkg -i docker-engine_*_armhf.deb</code></li>
<li><code>sudo useradd -G docker pi</code> (add the <code>pi</code> user to the <code>docker</code> group)</li>
<li>Check <code>docker --version</code> and rejoice</li>
<li><code>sudo systemctl enable docker</code> (configure Docker to start on boot as a service)</li>
</ol>
<h3>Useful things to note</h3>
<ul>
<li>Use <code>tmux</code> or <code>screen</code> to keep the terminal session open (if you're doing this over SSH like I did). The compiling step can take a few hours.</li>
<li>You can now just use <code>apt</code> (rather than <code>apt-get</code>) for most Debian package management needs, and you'll get better feedback, progress bars, etc.</li>
<li>Now that you've got Docker installed, don't get complacent. You can't just <code>docker run</code> any image you find on <a href="https://hub.docker.com">Docker Hub</a>. Each one has to be built to run on ARM. Look for images tagged <code>rpi</code> or <code>armhf</code>.</li>
</ul>
<h2>The integration layer (Node-RED)</h2>
<p>If you've ever used an SOA/BPM orchestration tool, you'll be familiar with <a href="http://nodered.org/">Node-RED</a>. The premise is that you use Node-RED as the integration point for your IoT devices and services, and use it to react to events - such as pushing a lightswitch - with some useful action(s) - such as turning on a light, followed by some music.</p>
<h3>Deploying Node-RED</h3>
<p>Node-RED has a handy Docker image, and even a <code>rpi</code> tagged one that we could, in theory, use out-of-the-box. It already has some pre-installed plugins, like GPIO, but it doesn't have Bluetooth, and doesn't have the plugin for devices like the <a href="https://punchthrough.com/bean">PunchThrough Bean</a> (the iBeacon variant I'm using).</p>
<p>The way I improved this was to make my own derivative Dockerfile with Bluetooth configured. I've published my <a href="https://github.com/dsample/node-red-iot">Dockerfile on GitHub</a> and also pushed my built image on <a href="https://hub.docker.com/r/dsample/node-red-iot/">Docker Hub</a>.</p>
<p>The basic <code>docker run -d --name=node-red dsample/node-red-iot:rpi</code> would get Node-RED running, but there would be several issues with this basic setup:</p>
<ul>
<li>The container wouldn't start when the Raspberry Pi is rebooted. Solve this by adding <code>--restart=always</code></li>
<li>The Node-RED flows you create will be lost if you recreate the image. Solve this by adding <code>-v /srv/node-red/data:/data</code> to map the local directory <code>/srv/node-red/data</code> to the container's <code>/data</code> directory</li>
<li>Bluetooth won't work. The network interfaces within the container are virtual, and don't include the <code>hci0</code> device for the Raspberry Pi's bluetooth adapter. We can solve this by adding <code>--net=host</code>, but this will remove the ability to do internal 'links' between this container and other Docker containers. This is the main negative point I've found during this project.</li>
</ul>
<p>With these additions, we now have a command <code>docker run -d --net=host --restart=always --name=node-red -v /srv/node-red:/data dsample/node-red-iot:rpi</code>.</p>
<p>Now open a browser to your Raspberry Pi's IP on port <code>1880</code> and you should see the Node-RED interface.</p>
<h2>The messaging bus (MQTT)</h2>
<p>Thinking that Node-RED would only be the interface towards devices and services for 'IoT' purposes, I figured I'd need a messaging bus to carry the signals to an from other services which would be listening and piping the data into a other systems (eg. a database). I was wrong, Node-RED can integrate into MongoDB, InfluxDB and other backend systems easily, but I thought I'd give myself the option of using MQTT anyway.</p>
<p>I chose <a href="http://mqtt.org/">MQTT</a> rather than <a href="https://www.rabbitmq.com/">RabbitMQ</a> which I'm more familiar with as MQTT is lighter and designed for the purpose of IoT (previously called M2M), so felt like the right tool for the job.</p>
<p>The MQTT Broker I chose is <a href="https://mosquitto.org/">Mosquitto</a>. It doesn't appear to be the most featureful broker, but I didn't need much to begin with.</p>
<p>As with the Node-RED container, I ended up creating my own image. Partially because I wanted to configure slightly different volume paths than the other attempts, but also because I didn't see the need to install either cURL or Wget into the image due to the fact that Docker can download files itself (and save 6MB in the image).</p>
<p>Again, the code is published on <a href="https://github.com/dsample/docker-images-rpi/tree/master/mosquitto">GitHub</a>.</p>
<p>We end up with:</p>
<pre><code>docker run -d \
-p 1883:1883 \
-p 9001:9001 \
-v /srv/mqtt/config:/mqtt/config:ro \
-v /srv/mqtt/log:/mqtt/log \
-v /srv/mqtt/data:/mqtt/data \
--restart=always \
--name mqtt \
dsample/rpi-mosquitto:rpi
</code></pre>
<h2>The metrics (InfluxDB)</h2>
<p>In order to visualise the data we'll need to a database. I'm planning to use MongoDB for long-term storage, but having used MongoDB for the past couple of years to store my heating an electricity readings, I felt it was time to use something else, as the graph generation from MongoDB was getting a bit slow (although that's probably just my poor indexing).</p>
<p>At work we use <a href="https://prometheus.io/">Prometheus</a>, but I'm not sold on the benefits of it over <a href="https://graphiteapp.org/">Graphite</a>, so I thought I'd give another alternative a try, namely <a href="https://influxdata.com/time-series-platform/influxdb/">InfluxDB</a>.</p>
<p>Again, I wrote <a href="">my own Dockerfile</a>. I used the config from the <a href="https://github.com/influxdata/influxdata-docker/blob/master/influxdb/0.13/Dockerfile">official build</a> but they use <code>wget</code> again, and also don't provide an 'rpi' version.</p>
<pre><code>docker run -d \
-p 8083:8083 \
-p 8086:8086 \
-v /srv/influxdb/meta:/influxdb/meta \
-v /srv/influxdb/data:/influxdb/data \
-v /srv/influxdb/wal:/influxdb/wal \
--restart=always \
--name influxdb \
dsample/rpi-influxdb:rpi
</code></pre>
<p>For this one, if you go to port <code>8083</code> on your Pi you should see the InfluxDB interface.</p>
<h2>TODO</h2>
<ul>
<li>I haven't gotten around to installing a dashboard tool yet (<a href="https://influxdata.com/time-series-platform/chronograf/">Chronograf</a> or <a href="http://grafana.org/">Grafana</a>), so no funky energy/temperature graphs yet.</li>
<li>I haven't installed MongoDB yet as I haven't found an image I'm happy with yet (most images are cross-compiled rather than native). The retention policy I've set on InfluxDB is 1 year though, so although I may lose the raw events, I'll be able to visualise the data for now.</li>
</ul>
<h1>Integration</h1>
<p>Now we get onto the more interesting part of making Node-RED do something vaguely useful. For this article, I'll concentrate on just getting data out of the devices. </p>
<h2>PunchThrough Bean</h2>
<p>After spending a long time working out how to get Bluetooth working within the Node-RED container, getting data out of the Bean was pretty straightforward.</p>
<p><img src="images/post_imgs/node-red_bean.png" alt="Screenshot of the Node-RED flow for the Bean" /></p>
<p>What this does is use an <code>inject</code> node to trigger a collection of the Bean temperature every 10 seconds and set a global variable with the result. There is then another flow triggered by an HTTP request which takes the global value and responds with it.</p>
<p>You can import my flow using the JSON below:</p>
<div><script src="https://gist.github.com/dsample/8ab0b036cd4329a3e080730d3b339b07.js?file=node-red_bean.json"></script></div>
<h2>OWL Intuition</h2>
<p>I use an OWL Intuition-C heating control & electicity monitoring system. I bought it as it had an API, and the main API is a UDP Multicast data stream (XML).</p>
<p>I'd previously spent days/weeks coding some Ruby services to collect and transmit and store the data. I managed to do the same thing in Node-RED in around 10 minutes.</p>
<p><img src="images/post_imgs/node-red_owl.png" alt="Screenshot of the Node-RED flow for the OWL Intuition integration" /></p>
<p>This flow is a little more complex. It triggers from a UDP multicast message which contains XML. It deserialises the XML into JSON and maps the result, then it creates an InfluxDB 'measurement' and also publishes it to MQTT. The couple of nodes between just deal with the 'topic' of the message. This topic is used to separate out the heating and electricity reading messages so that we can have smaller functions for each.</p>
<p>You can create another flow that starts with an MQTT subscription, and that was my plan for getting the data into MongoDB (although I wasn't planning to use Node-RED for that part). Node-RED, however, has a simple <code>MongoDB</code> output node which will insert data.</p>
<p>You can import my flow using the JSON below:</p>
<script src="https://gist.github.com/dsample/8ab0b036cd4329a3e080730d3b339b07.js?file=node-red_owl.json"></script>