alvinashcraft
shared this story
from 
Microsoft Press.
This is the 51st in our series of guest posts by Microsoft Most Valued Professionals (MVPs). You can click the “MVPs” tag in the right column of our blog to see all the articles.
Since the early 1990s, Microsoft has recognized technology champions around the world with the MVP Award. MVPs freely share their knowledge, real-world experience, and impartial and objective feedback to help people enhance the way they use technology. Of the millions of individuals who participate in technology communities, around 4,000 are recognized as Microsoft MVPs. You can read more original MVP-authored content on the Microsoft MVP Award Program Blog.
This post is by Visual C++ MVP Alon Fliess. Thanks, Alon!
Introduction to the Internet of Things – From the Device to Microsoft Azure Cloud
Technology advances in “Buzzwords” steps. At the beginning, there is the basic technology. It slowly evolves, and then after a few years, sometimes even many years, everything becomes connected and the world, not the early bird world, but everybody is ready to embrace that technology. This is where the big buzz begins and everybody predicts that in five to ten years the technology will generate incremental revenue exceeding hundreds of billions. With this buzz, all major companies invest in the technology and we begin seeing TV news reports and economy magazine articles about the technology, telling that the everyday life of every human being on the planet is going to change because of that technology!
Of course, I am a bit cynical, but this is exactly what is happening now with regards to the IoT – the Internet of things. The basic technology is already here for almost a decade so far. Amazon Web Services (ASW) started in 2006. Microsoft Azure is 5 years old. Devices such as those based on the Amtel AVR controller are more than 20 years old, and the affordable Arduino family of devices used by IoT hobbyists are 10 years old. It is not (just) the technology that makes IoT what it is, but the concepts, the perception, the commitment and the challenges that the entire industry is dealing with nowadays. IoT is about the machine-to-machine (M2M) communication at scale. Vast numbers of devices using different hardware and software technologies are connected between them and to the cloud. The cloud provides many services, which can handle huge streams of data, analyze and can extract vital information about the current state of the system and can even predict future state.
So what exactly is IoT?
In one simple form, IoT is about a device that can monitor a physical character of the environment and transfer this data over the Internet to a cloud service. In a more complex form, IoT is the combination of many smart devices that can “feel” the environment, read the data, and transfer this information to a collector service in the cloud. This service has to deal with large amounts of devices and huge streams of data. Such services take in information and can extract a vital information from a live stream, or run algorithms such as those based on big-data map-reduce patterns. Services can act on historical and new data, or can provide future insight about the collected data. Services like Azure Machine Learning can use the collected data to predict future behaviors. The cloud can send commands to devices. Take for an example a system that starts the water sprinklers according to an algorithm that reads information from a group of soil moisture level sensors. Based on information that it gets from a forecast service that predicts that no rain is coming, it decides to send a command to an actuator that starts the sprinklers.
Sometimes the end device has the capabilities to communicate directly with the cloud service, and sometimes the device is cheap, weak, and has to conserve power, or has no encryption capabilities. In the latter case, a group of such weak devices is connected to a local gateway – a software that runs on a stronger local device that serves as a mediator between the local device sub-network and the cloud.
To illustrate, let us take a simple personal and fun project – which I have only recently turned into an IoT projectJ). With this project, I can control my home electrical devices such as lights, shutters, hot water boiler, and the air condition. I can use a web browser, Windows Phone or Windows 8.1 Modern application to send a command or read the current state of a device. I have other services that turn on the garden lights on sunset, or raise the shutters automatically every morning except on weekend days. I even have a service that sets the boiler according to the forecast.
My own electrical network is based on a well-known protocol named Instabus. This protocol is based on transmitting messages over a two-wire low-voltage command network. To communicate with the phone, or other devices and applications, I have built my own local gateway service that translates my REST based protocol to Instabus commands and vice versa. Whenever I use my Windows Phone to turn on the light, the phone application calls the REST based local gateway that extracts the command and transfers it to the local Instabus. The bus in turn publishes the command in its own protocol making an actuator that is the end receiver of the command to close the electrical circuit and turn on the light.
An IoT System
Every device that has a way to communicate can take part in an IoT system. A modern IoT system is built from at least one end device and one service that runs over the internet, however most large IoT systems include one or more of the following elements:
1. Many different devices with sensors and actuators
2. Local gateways
3. A collection of cloud services that enables:
a. Registration of devices
b. Management of devices
c. Different communication protocols that provides reliability and security
d. The ability to collect a vast amount of data at a very high input rate
e. The ability to analyze the stream of information in close to real-time manner
f. The ability to analyze the current and historical collected information
g. The ability to show the resulting conclusion and the collected data
In the rest of the article, we will see the two sides of an IoT system. We will start with the device and we will continue with the cloud services focusing on the rich possibilities that Microsoft Azure provides such a system.
The Devices:
Any device that can transmit and receive information is a candidate for an IoT system. Usually these devices are based on a System on a Chip (SoC) – an integrated circuit that has all the components of a computer on a single chip. Many hardware companies build new versions of these chips to be the heart of an IoT device. Some of the devices have an 8 bit based processors, others are based on 32-bit architecture and have the power of a small PC. For example the Intel Galileo 2 that is based on the Quark™ - a Pentium based architecture.
Hobbyists, or professionals that need a prototype device, in most cases, use a device from one of these families:
1. Arduino
2. Netduino
3. Raspberry Pi
4. Intel Galileo & Edison
5. Tessel
6. Spark Photon, Core and Electron
All of these device families have the ability to get information from sensors using digital or analog inputs, have the power to run custom algorithms and they can transmit and receive data using one or more means of communications.
The Arduino family is one of the older existing device families. It serves as a simple and cheap hardware to develop simple electronic devices. The Arduino is an open source hardware and software; anyone can build their own Arduino device. You can find Arduino compatible boards that cost about $1.50. These small devices have no internet capabilities; however, they can communicate using RS232 (Serial COM Port) to a local gateway, or even use a very inexpensive Wi-Fi device such the ESP8266. In case that there is a need for a more powerful and expensive device, which can also serve as a local gateway; the Arduino family has the Arduino Yun board. The Yun is a hybrid of two systems on one board. The first system is the Arduino. The second system is based on OpenWRT – a Linux distribution for embedded devices. The two systems communicate using a bridge library. The Yun has a Wi-Fi and Ethernet capabilities and it can serve as a gateway or as a client and communicate directly with the cloud. Usually, to develop for Arduino, you use the simple dedicated Arduino Editor that lets you write simple C/C++ code, however there is a Visual Studio Add-on that lets you develop and debug Arduino programs from Visual Studio. For the Linux side of Arduino Yun you can use C/C++ with GCC, Python, JavaScript with Node.JS or any other language that has a Linux compiler or interpreter for OpenWRT. The Yun is an IoT ready Arduino device.
Arduino Boards
The Netduino family is another example of open source hardware and software. This family is hardware compatible with the Arduino sensors and shields – Arduino add-on hardware boards. The Netduino software is based on the .NET Micro framework, a tiny .NET version that lets you develop for the device using C# in Visual Studio!
To use the latest versions of Visual Studio and .NET Micro framework, you might need to flash a new firmware. The Netduino plus 2 has an Ethernet on board and it can use a Wi-Fi shield such as the SparkFun WiFly to become an even better IoT device.
The Raspberry Pi is more like a small form factor PC than a hardware device; it can be connected to a monitor or TV using analog output or a HDMI cable. It has USB connectors that can be used to connect mouse, keyboard and even a Wi-Fi dongle. Raspberry Pi runs a special Linux distribution called Raspbian. Microsoft has announced that a special version of Windows 10 for IoT devices will support the Raspberry Pi 2.
Developing for the Pi is quite easy. You can use the built-in Python or Scratch, or you can install any other language compiler including .NET Mono. If you like to use C/C++ and port Arduino code to the Pi, you can use the WiringPi library with GCC.
Like the Arduino and Netduino, the Raspberry Pi can be connected to devices through its GPIO pins. The raspberry uses 3.3 Volt sensors and connecting it poorly or using the wrong voltage may render it inoperable. As opposed to Arduino, the Raspberry Pi has no analog inputs, which in many cases are important for reading a level value from a simple sensor, such as temperature or ambient light level. You can connect an Analog to Digital chip that has a Serial Peripheral Interface Bus (SPI) channel to overcome this problem, or use a combination of inexpensive 3.3V Arduino with the Pi. If you are concerned about your Pi becoming inoperable, you can use boards like the Pibrella that makes it safe to play with it.
Raspberry Pi 2 – Will run Windows 10
There are other Raspberry Pi compatible but more powerful devices such as the Banana-Pi and the Orange-Pi. The Orange Pi has a built-in Gigabit Ethernet and Wi-Fi, which make it a good IoT (gateway) device.
An Orange Pi
Intel supplies both hardware and software for IoT. The Intel Galileo 2 board is a relatively robust machine that runs Linux or the new Windows IoT (still in alpha version). The Galileo is compatible with Arduino shield and software and you can use Visual Studio C++ and sub-set of the Win 32 API. With these C++ and Win 32 capabilities, you can use the C++ REST SDK and Azure C++ Storage API with the Galileo to communicate with Microsoft cloud.
Intel Galileo Gen 2 on Clay’s Breadboard kit – Windows IoT is installed on the micro-sd card.
There is also an Arduino Development Environment for the Galileo. Intel has another smaller IoT device – the Intel Edison. Though I haven’t had the chance to play with it yet, according to specification it appears to be a great IoT capable device with dual-core CPU, integrated Wi-Fi, Bluetooth, 40 GPIO interfaces and libraries to develop using C, C++, Python and JavaScript.
The Tessel is a nice device for beginners and for people that have no electronics knowledge or as the Tessel people describe it – “Hardware development for software developers”. A tiny board that has four connecters on the side for connecting a dedicated set of modules - sensor boards. The Tessel has built-in Wi-Fi and it runs JavaScript. The Tessel is a very easy to use IoT device.
The Tessel
Last but not least in the devices part is a new and very promising device family – the Spark. The family currently has two devices and one in development: the Core – the first Wi-Fi enabled device, the Photon – a relatively inexpensive tiny Wi-Fi capable device and the Electron – A Kickstarter project, an IoT device that connects the Internet via cellular network.
Spark Photon
The common dominator of all above devices is that they have the capability to bring the physical environmental information to the Internet or execute commands. They do that via a set of input and output pins. In many cases, each pin can serve as an input or an output based on its settings. Some of the pins serve as communication protocol connectors such as RS-232, I2C and SPI that were mentioned earlier. Other pins can be used to control motors and servos with Pulse-Width-Modulation (PWM). Sensors are connected to these input/output pins. Sensors provide information such as temperature, moisture, water level, rain, wind speed, weight, motion detection, sonar – range detector, infra-red transmitter and receiver, microphone, camera, Serial over Wi-Fi, Bluetooth, RFID reader/writer and so forth.
In the following example, each time a motion sensor connected to input pin 12 detects a movement, it lights a led:
Smart Refrigerator Scenario
We would like to have a set of devices and a backend system that transforms our refrigerator to an IoT system. The system can tell us that we are running out of milk, it can tell our retail store that they need to deliver a new milk bottle, and with the power of the cloud it can tell the retail chain headquarter the trends of using their goods and predict the near future supply needs. This knowledge provides real business value. There are many different ways to make a smart-refrigerator – for example, each product can come in a smart package that knows the product details such as the expiration date, the price, the nutrition and the current amount that is in the box. For our humble example, we are going to use a device that can weigh the milk bottle; we will convert the result to the current amount that is left in the box. We can set a collection of such devices in the refrigerator for different products.
A prototype setup looks like this:
Once everything works using the breadboard, we can use a prototype shield:
This circuit can operate using a battery or a strong enough power supply. Using the ESP 8266 with the simple Arduino requires going through a local gateway, since most of the cloud services that deal with a stream of events require the use of standard protocols and the use of security layer which only powerful devices such as the Intel Galileo or the Raspberry PI. The gateway runs a simple software that bridges the local device network with the cloud. The gateway can handle many local client connections.
To read the weight, we use the Weight sensor through a HX711 ADC device. The HX711 reads the very small voltage changes that occur when we put a weight on the scale and transfer this information as a stream of bits. A special library lets you read the stream of bits coming from the HX711. The library parses the raw bits data and converts it to a usable floating-point number value. You can download the code for the Arduino from here, or if you like to play with Netduino – I have ported the library to C#.
The Arduino code:
1 #include "HX711.h"
2 #include <SoftwareSerial.h>
3
4 #define SSID "NOKIA Lumia Alon"
5 #define PASS "AF123456"
6 #define DST_IP "192.168.137.36"
7
8 int ledPin = 2; // select the pin for the LED
9 HX711 scale(9, 8); // parameter "gain" is ommited; the default value 128 is used by the library
10
11 SoftwareSerial mySerial(7, 6); // RX, TX
12
13 void blink(int times, bool isGood = true)
14 {
15 int d = isGood ? 100 : 300;
16 for (int i = 0; i < times; ++i)
17 {
18 digitalWrite(ledPin, HIGH);
19 delay(d);
20 digitalWrite(ledPin, LOW);
21 delay(d);
22 }
23 }
24
25 void setup()
26 {
27 // Open serial communications and wait for port to open:
28 pinMode(ledPin, OUTPUT);
29 //scale.set_scale(2280.f); // this value is obtained by calibrating the scale with known weights; see the README for details
30 scale.set_scale(160.f); // this value is obtained by calibrating the scale with known weights; see the README for details
31 scale.tare(); // reset the scale to 0
32
33 mySerial.begin(9600);
34 mySerial.println("Ready");
35 blink(5);
36
37 Serial.begin(115200);
38
39 Serial.setTimeout(5000);
40
41 //test if the module is ready
42 Serial.println("AT+RST");
43 delay(3000);
44
45 //connect to the wifi
46 boolean connected=false;
47 for(int i=0;i<50;i++)
48 {
49 if(connectWiFi())
50 {
51 connected = true;
52 break;
53 }
54 blink(2, false);
55 }
56 if (!connected)
57 {
58 mySerial.println("Not connected");
59 blink(100, false);
60 while(1)
61 ;
62 }
63 delay(5000);
64 blink(5);
65 Serial.println("AT+CIPMUX=0");
66 mySerial.println("Connected");
67 }
68
69 void dumpSerial()
70 {
71 while (Serial.available())
72 {
73 char c = Serial.read();
74 mySerial.write(c);
75 if(c=='\r') mySerial.print('\n');
76 }
77 }
78
79 void loop()
80 {
81 scale.power_up();
82 double sensorValue = scale.get_units(10);
83 scale.power_down(); // put the ADC in sleep mode
84
85 String cmd = "AT+CIPSTART=\"TCP\",\"";
86 cmd += DST_IP;
87 cmd += "\",5001";
88 Serial.println(cmd);
89 //mySerial.println(cmd);
90 delay(250);
91 if(Serial.find("Error"))
92 {
93 blink(4, false);
94 mySerial.println("Error");
95 Serial.println("AT+RST");
96 delay(3000);
97 connectWiFi();
98 delay(3000);
99 return;
100 }
101
102 cmd = String("Weight:") + String(sensorValue) + ",Scale.Arduino\r\n";
103 //mySerial.println("");
104 Serial.print("AT+CIPSEND=");
105 Serial.println(cmd.length());
106 delay(300);
107 Serial.print(cmd);
108 delay(2000);
109 //mySerial.println(cmd);
110 dumpSerial();
111 delay(1000);
112 }
113
114 boolean connectWiFi()
115 {
116 Serial.println("AT+CWMODE=1");
117 String cmd="AT+CWJAP=\"";
118 cmd+=SSID;
119 cmd+="\",\"";
120 cmd+=PASS;
121 cmd+="\"";
122 Serial.println(cmd);
123 delay(3000);
124 if(!Serial.find("OK"))
125 {
126 blink(4);
127 return false;
128 }
129 //else, OK, read assigned IP for debugging
130 Serial.println("AT+CIPSTATUS");
131 delay(250);
132 dumpSerial();
133
134 blink(2, false);
135 delay(250);
136 return true;
137 }
The Netduino code:
1 using System;
2 using System.IO.Ports;
3 using System.Net;
4 using System.Net.Sockets;
5 using System.Threading;
6 using Microsoft.SPOT;
7 using Microsoft.SPOT.Hardware;
8 using SecretLabs.NETMF.Hardware;
9 using SecretLabs.NETMF.Hardware.Netduino;
10
11
12 namespace WiFiScale
13 {
14 public class Program
15 {
16 private const string SSID = "NOKIA Lumia Alon";
17 private const string Password = "AF123456";
18 private const string DestinationIp = "192.168.137.36";
19 private static readonly OutputPort LedPin = new OutputPort(Pins.GPIO_PIN_D2, false);
20 private static readonly SerialPort DebugSerial = new SerialPort(SerialPorts.COM4, 9600, Parity.None, 8,
21 StopBits.One);
22 private static readonly HX711 Scale = new HX711(Pins.GPIO_PIN_D9, Pins.GPIO_PIN_D8);
23 private static readonly SerialPort WiFiSerial = new SerialPort(SerialPorts.COM1, 115200, Parity.None, 8, StopBits.One);
24
25 public static void Main()
26 {
27 Scale.Scale = 160;
28 WiFiSerial.Open();
29 DebugSerial.Open();
30
31
32 // this value is obtained by calibrating the scale with known weights; see the README for details
33 Scale.Tare(); // reset the scale to 0
34 DebugSerial.Write("Ready");
35
36 Blink(5);
37 // write your code here
38
39 WiFiSerial.WriteLine("AT+RST");
40 Thread.Sleep(3000);
41
42 //connect to the WiFi
43 bool connected = false;
44
45 for (int i = 0; i < 50; i++)
46 {
47 if (ConnectWiFi())
48 {
49 connected = true;
50 break;
51 }
52 Blink(2, false);
53 }
54 if (!connected)
55 {
56 DebugSerial.WriteLine("Not connected");
57 while (true)
58 Blink(100, false);
59 }
60
61 Thread.Sleep(3000);
62 Blink(5);
63 WiFiSerial.WriteLine("AT+CIPMUX=0");
64 DebugSerial.WriteLine("Connected");
65 Thread.Sleep(2000);
66 while (true)
67 {
68 Loop();
69 }
70 }
71
72 private static void Blink(int times, bool isGood = true)
73 {
74 int d = isGood ? 100 : 300;
75 for (int i = 0; i < times; ++i)
76 {
77 LedPin.Write(true);
78 Thread.Sleep(d);
79 LedPin.Write(false);
80 Thread.Sleep(d);
81 }
82 }
83
84 private static void Loop()
85 {
86 Scale.PowerUp();
87 double sensorValue = Scale.GetUnits(10);
88 Scale.PowerDown(); // put the ADC in sleep mode
89
90 String cmd = "AT+CIPSTART=\"TCP\",\"";
91 cmd += DestinationIp;
92 cmd += "\",5001";
93 WiFiSerial.WriteLine(cmd);
94
95 Thread.Sleep(250);
96 if (WiFiSerial.Find("Error"))
97 {
98 Blink(4, false);
99 DebugSerial.WriteLine("Error");
100 WiFiSerial.WriteLine("AT+RST");
101 Thread.Sleep(3000);
102 ConnectWiFi();
103 Thread.Sleep(3000);
104 return;
105 }
106
107 cmd = "Weight:" + sensorValue + ",Scale.Net\r\n";
108
109 WiFiSerial.Write("AT+CIPSEND=");
110 WiFiSerial.WriteLine(cmd.Length);
111 Thread.Sleep(300);
112 WiFiSerial.Write(cmd);
113 Thread.Sleep(2000);
114 //DebugSerial.WriteLine(cmd);
115 DumpSerial();
116 Thread.Sleep(1000);
117 }
118
119 static void DumpSerial()
120 {
121 while (WiFiSerial.BytesToRead > 0)
122 {
123 var line = WiFiSerial.ReadLine();
124 DebugSerial.WriteLine(line);
125 }
126 }
127
128 private static bool ConnectWiFi()
129 {
130 WiFiSerial.WriteLine("AT+CWMODE=1");
131 String cmd = "AT+CWJAP=\"";
132 cmd += SSID;
133 cmd += "\",\"";
134 cmd += Password;
135 cmd += "\"";
136 WiFiSerial.WriteLine(cmd);
137 Thread.Sleep(3000);
138 if (!WiFiSerial.Find("OK"))
139 {
140 Blink(4);
141 return false;
142 }
143 //else, OK, read assigned IP for debugging
144 WiFiSerial.WriteLine("AT+CIPSTATUS");
145 Thread.Sleep(250);
146 DumpSerial();
147
148 Blink(2, false);
149 Thread.Sleep(250);
150 return true;
151 }
152 }
153 }
154
In this demo, the Server IP address and the Access Point SSID and password are all hardcoded. In a real world scenario, you can have a setting switch on the board that can command the device to move to setup mode. In setup mode, the ESP 8266 is set to an Access Point mode where a simple client running on a laptop, tablet or even Raspberry Pi can connect to it and set these parameters. You will also need a SD card as a drive and a card to store the information – The Netduino Plus 2 has a built-in one, and it is very easy to add such a device to the Arduino. Whenever the controller is turned on, it tests the setting-switch and decides whether to go to setup mode or read the IP address and other parameters from the SD card and set the ESP 8266 as a client. This is similar to the way you initialize an Arduino Yun board for the first time.
The Gateway and the Cloud:
Up until now, we have seen all about the device, but IoT is mainly about connecting devices to the cloud and handling the stream of data.
Microsoft Azure provides the required framework to build a large-scale IoT backend:
Devices and applications collect the data and create a stream of events. These devices may directly call the Event Hub or use a local gateway to do that. Azure Stream Analytics is a Complex Event Processing (CEP) service that can handle millions of events per second. It can transform the data to other representations, can filter, aggregate or search for a specific information. Using Azure Stream Analytics, we can extract vital information from the current data. We can also store the data and use Azure big data - HDInsight and Azure Machine Learning - to get a deeper analysis and even get a projection and decisions that can be used to send a command back to the device.
In a case that we need a system that can answer queries from the old and the new data, we can implement the Lambda Architecture (LA) – an architecture that aims to provide scalable and fault-tolerant data processing system for low latency queries. The basic idea behind this architecture is to use scalable services such as Azure Event Hubs or Apache Kafka, for handling the huge data stream. The data event stream is dispatched into two layers – a batch layer – for old data and a speed layer for recent, short, near real time data. In Azure, the batch layer is based on storing the information in one of the Azure storage options (Blob, Table, DocumentDB, Azure SQL DB) and the speed layer can be built with Stream Analytics or with Azure HDInsight Apache Storm. To implement the Lambda Architecture, you need to create an Azure service (Web-Site REST service for example) that for any incoming query, it distributes the query to both layers and then merges the query result.
I would like to show you a simple IoT system, from the device through a gateway to the cloud. Of course, I will not touch many IoT aspects such as device registration and management, but I will handle secure communication and build a scalable solution.
I already shared the code that runs on the Arduino and Netduino, that sends the weight information to the gateway. For the example, I choose to use the Raspberry PI as a gateway. I use Raspberry PI Model 2 with a Wi-Fi dongle, for which I developed the gateway code with Python. I chose Python since I found it very easy to use and talk with Azure Event Hub Service. There is a great blog post of Kloud that shows how to use Azure event Hub directly from the Arduino Yun – a device that can run C++ and Python. There are also many other options such as using Java Script with Node.JS or Using C++ with Apache QPid that communicate with the Azure Event Hub using the standard AMQP protocol.
The Python code is a multi-threaded TCP server that accepts connections from the local network. Once it has a new connection it spawns a thread that reads the stream of events coming from a simple device and sends the raw information to Azure Event Hub.
This is the Gateway code:
1 #!/user/bin/python
2 import sys
3 import azure
4 import socket
5 import SocketServer
6 import threading
7
8 from azure.servicebus import (
9 _service_bus_error_handler
10 )
11
12 from azure.servicebus.servicebusservice import (
13 ServiceBusService,
14 ServiceBusSASAuthentication
15 )
16
17 from azure.http import (
18 HTTPRequest,
19 HTTPError
20 )
21
22 from azure.http.httpclient import _HTTPClient
23
24 import time
25
26 class EventHubClient(object):
27
28 def sendMessage(self,body,partition):
29 eventHubHost = "aloniot.servicebus.windows.net"
30
31 httpclient = _HTTPClient(service_instance=self)
32
33 sasKeyName = "iotpolicy"
34 sasKeyValue = "l6TGnew90yy0Pw1u1VWTt0awHOezwEB315+BCeNfH8c="
35
36 authentication = ServiceBusSASAuthentication(sasKeyName,sasKeyValue)
37
38 request = HTTPRequest()
39 request.method = "POST"
40 request.host = eventHubHost
41 request.protocol_override = "https"
42 request.path = "/iot/publishers/" + partition + "/messages"
43 request.body = body
44
45 print(body)
46
47 request.headers.append(('Content-Type', 'application/atom+xml;type=entry;charset=utf-8'))
48
49 authentication.sign_request(request, httpclient)
50
51 request.headers.append(('Content-Length', str(len(request.body))))
52
53 status = 0
54
55 try:
56 resp = httpclient.perform_request(request)
57 status = resp.status
58 except HTTPError as ex:
59 status = ex.status
60
61 return status
62
63 class EventDataParser(object):
64
65 def getMessage(self,payload,sensorId):
66 host = socket.gethostname()
67 body = "{ \"Time\" : \"" + time.asctime(time.localtime(time.time())) + "\", \"GatewayId\" : \"" + sys.argv[1] + "\", \"DeviceId\" : \"" + host + "\","
68
69 sensorType = payload.split(":")[0]
70 sensorValue = payload.split(":")[1]
71 body += "\"SensorId\" : \"" + sensorId + "\", \"SensorType\" : \"" + sensorType + "\", \"SensorValue\" : \"" + sensorValue + "\"}"
72 return body
73
74
75 class IoTAzureGatewayTCPHandler(SocketServer.BaseRequestHandler):
76 """
77 The RequestHandler class for our server.
78
79 It is instantiated once per connection to the server, and must
80 override the handle() method to implement communication to the
81 client.
82 """
83
84 def handle(self):
85 hubClient = EventHubClient()
86 parser = EventDataParser()
87 hostname = socket.gethostname()
88
89 while True:
90 # self.request is the TCP socket connected to the client
91 self.data = self.request.recv(1024).strip()
92 if not self.data:
93 print("Nothing to read. Closing conncetion")
94 break
95 print("{} wrote:".format(self.client_address[0]))
96 print(self.data)
97
98 splitData = self.data.split(",")
99 sensor = splitData[1]
100 body = parser.getMessage(splitData[0],sensor)
101 hubStatus = hubClient.sendMessage(body,hostname)
102 # return the HTTP status to the caller
103 print(hubStatus)
104 # just send back
105 self.request.sendall(str(hubStatus))
106
107
108 class IoTAzureGatewayTCPServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
109 pass
110
111 if __name__ == "__main__":
112 HOST, PORT = "0.0.0.0", 5001
113
114 server = IoTAzureGatewayTCPServer((HOST, PORT), IoTAzureGatewayTCPHandler)
115 ip, port = server.server_address
116
117 # Start a thread with the server -- that thread will then start one
118 # more thread for each request
119 server_thread = threading.Thread(target=server.serve_forever)
120 # Exit the server thread when the main thread terminates
121 server_thread.daemon = True
122 server_thread.start()
123 print("Server loop running in thread:", server_thread.name)
124
125 # this will keep running until you
126 # interrupt the program with Ctrl-C
127 while True:
128 pass
129 server.shutdown()
130
The result:
This sums the device part of the story; we are sending telemetry information to Azure Event Hub. Each gateway can connect many local devices and we can deploy many gateway devices, for example, one gateway per refrigerator and one end device per product box or a place on the refrigerator shelf.
Azure Event Hub
Azure Event Hub is a highly scalable service that enables ingesting and processing millions of events per second. Queues and topics are great for building reliable and scalable enterprise solutions but event hubs are particularly suitable to the IoT pattern of scale. In any scenario that involves scalability – you must tackle the potential bottlenecks of the system. Queues, topics and event hubs handle message throughput differently. Queues synchronize producers and consumers – for example, queues provide a mechanism that hides a message once a consumer starts to handle it, a mechanism to delete it when the consumer finishes handling it, or the consumer can return the message to the queue if it could not process it to completion. Topics are similar, with the exception that multiple consumers can process a message. To enable improved scalability, low latency and performance, event hubs have a different structure and approach. Event hubs have many data structures similar to queues called partitions. When you create an event hub, you provide the number of partitions – currently up to 32, but you can request additional partitions if this is insufficient for your solution. Each message sent to the event hub can include a target partition but you can also use a key such as a device id and allow the event hub infrastructure to co-located messages from the same device id to the same destination partition. In case of the latter – the device id goes through a hash function that chooses the partition for its stream of data. This enables scalability and message order from a device source. To further increase the performance, event hubs do not delete events immediately on
