In a race to optimize everything, developers often go to extremes to build software that performs routine tasks. MissionControl is a system that allows users to program a control center that stores interfaces with attached hardware sensors, allowing the users to control any other devices that can be activated via the underlying protocol. For demo purposes, the MissionControl build at this point is compatible with the Phidgets IR hybrid sensor.
The system has two core components:
A server application, which is a Win32 console application that handles incoming queries and returns data to the connected clients. This application runs on the desktop machine with the connected sensor.
The Windows Phone application that sends requests to the target server and can trigger a variety of pre-programmed commands.
The Basics
Hardware and Communication Infrastructure
One of the most important parts of the project is the signal capture and replication hardware. For the purposes of this project, I decided to use a dual-mode Phidgets IR sensor. It supports both IR code capture and subsequent replication. From a user’s perspective, this device also eliminates a substantial code-learning overhead as well as the potential error rate. Instead of searching for a device-specific hexadecimal sequence that later has to be transformed in a working IR code, the user simply has to point his remote control at the sensor and press the button that he wants accessible from a mobile device. Given that the capturing software is running on the target machine, once the sensor detects that a code can be repeated within an acceptable precision range, it will be automatically captured and stored, with all required transformations worked out in the backend using the free Phidgets SDK.
Even though I can, I don’t have to handle the binary code content received through the sensor—the Phidgets .NET libraries carry built-in types that contain all the processed metadata that I will discuss later in this article.
This sensor is connected through a USB port to a machine that acts as a communication gateway. This server should have port 6169 open for inbound connections.
NOTE: The port number can be changed, but you have to keep it consistent between your server and client applications.
The communication between the phone and the computer running the client is performed via a TCP channel—sockets are used to perform the initial connections and serialized data transfer. You can see the generalized data flow between the devices that are involved in the procedure in the graphic below:
The server (desktop client) handles the local storage and release of all incoming IR codes. The mobile client has to know the location of the server—once specified and confirmed, it can send one of the pre-defined commands to it and either query the server for existing command groups (sets) or invoke one of the stored IR codes. When I pass data between devices, I use JSON for the serializable components. The data is also processed before being sent in order to speed-up the process—for example, on the server side the sets are serialized together with the associated codes. Like this:
The inherent problem with the JSON data above is the fact that the phone client does not need the information related to the code binary sequence and all the metadata that goes with it. So it is effectively stripped down and reduced to the names of the sets (when a list of sets is requested) and commands (when a list of commands is requested).
The Data Model
As you saw from the description above, the server organizes individual infrared codes in sets. A single set is a bundle of codes that may or may not be related to each other—ultimately, this is the user’s decision. A good example of using sets is organizing IR commands by rooms, devices or code types. Each set has a unique name on the server, therefore eliminating the possibility of a request conflict.
Each set stores individual commands built around the Command model:
Despite the obvious Name property, you can see that I am using a SerializableIRCode instance that is specific to each model. Before going any further, I need to mention that the Phidgets SDK offers the IRLearnedCode model to store code contents. I could have used it instead, but there is an issue that prevents me from doing that—there is no public constructor defined for IRLearnedCode, therefore there is no way to serialize it, either with the built-in .NET serialization capabilities or JSON.NET, which I am using in the context of the project.
Instead, I have this:
It is an almost identical 1:1 copy of the original class, storing both the layout of the IR code and additional information related to its replication mechanism. You can learn more about each property listed in the model above by reading the official document on the topic.
ToggleMask, the identity bit carrier that helps marking the code as repeated or not, is also implemented through a built-in Phidgets SDK model, and it has the same problem as IRLearnedCode. I implemented this model to replace it in the serializable code:
I also needed an easy way to store all sets at once and carry all associated codes in a single instance retrieved from the storage. Here is the Set class:
Notice that there is an IsList flag that allows me to specify how to display this specific list on the connecting device. This adds some level of flexibility for situations where the user wants to build a virtual remote for closely-related keys, such as digits. With that in mind, displaying those as a list might be inconvenient, wasting visual space on the client. But if the flag is set to false, the list can be displayed as a pad.
Also, when the server performs the data exchange, it provides a single “envelope” that allows the connecting device to easily understand what the server is trying to do:
The Identifier property carries the server IP address. That way, when a device receives a response, it is able to either accept it, because it knows that a response is requested from a target location, or discard it because the user is no longer using the specific server.
Marker carries the command type of the sent command, therefore giving the Windows Phone application a hint as to what to do with the data. The server can send the following commands:
SET_LIST – returns the list of sets that are currently available on the server.
SET_COMMANDS:SET_NAME:IS_LIST – returns the list of commands that are associated with a given set that is currently stored on the server.
NOTIFICATION – send a simple notification to the client; no further action is required.
Last but not least, Content is used to push the necessary data that is associated with the given Marker. It can be either a JSON-based string that lists the sets or commands, or a plain-text message that is used as an alert for the end-user.
Server Architecture
The server is the only component of this entire system that does all the heavy lifting. It learns commands, stores them and then generates new IR signal requests, as controlled from any of the connected clients. Let’s take a closer look at what happens behind the scenes—to start, I am going to document the network infrastructure.
The Network Layer
In order to be a reliable system, the server needs to be always ready to accept an incoming connection. For that purpose, it is possible to use the TcpListener class—an “always on” receiver that can handle incoming TCP connections. I integrated it in my CoreStarter class that is used to start the listener when the application is launched:
When LaunchSocket is called, the listener is activated on the current machine. As I mentioned above, the port number can be arbitrarily assigned, but has to be consistent between connecting apps in order for the TCP links to be established. Because I expect that more than one device will be connecting to the service at a time, the listener is set as active across a constant number of threads.
NOTE: By default, a there is a maximum limit of 5 simultaneous clients. Although this number can be adjusted, be aware of the requirements of each environment in which a limited number of potential devices can connect. Even though the performance footprint of each thread is minimal, it can have a negative effect if used in unnecessarily large instances.
ListenForData is used to read the incoming stream. When an inbound connection is accepted, the data is read with the help of a fixed content buffer. Then a read timeout is specified to prevent situations where the stream was completely read but the application still waits to pull non-existent data. Once the timeout milestone is hit, an exception is thrown, which marks the end of the stream—at this point, the plain text data that was received (remember that both the server and client exchange text data only) is passed to the command interpreter—CommandHelper, with a reference to the source of the command.
The commands from the device are passed as serialized key-value pairs (KeyValuePair
), the key being the command with any possible suffixes, and the value being the contents of the command itself that helps the server identify the specific item in the local storage.
InterpretCommand,in this case, does three things sequentially:
Deserialize the incoming string and create a KeyValuePair
instance.
Process the command and check whether it is recognizable.
Send a response to the client, if deemed necessary by the command type.
The serialization and deserialization is done via JSON.NET. You can install this package in your console managed Win32 project and the Windows Phone application project via NuGet:
The deserialization step is as simple as one line of C# code:
The string is sanitized to ensure that only JSON content is being passed to the serializer.
Because of a relatively limited command set, I can put together the entire interpretation stack like this:
All commands are constants, declared in the local helper class:
Notice that these are not the commands that the server sends back, but rather the commands it receives from connecting Windows Phone devices.
Let’s now take a look at the breakdown for each command.
SendSets:
When this command is received, the server does not have to do much processing. It is only invoked when the client establishes the initiating link and needs to know what possible sets it can get from the target machine. The request is logged in the console and a server response is prepared that contains a serialized list of set names, which is later serialized as well and sent back to the source machine location.
StorageHelper and NetworkHelper will be documented later in this article.
CreateSet:
When a mobile device attempts to create a new set on the server, it sends a command in the following format:
CREATE_SET:list/pad, SET_NAME
CreateSet will get the type of the set that was created, will check whether a set with the same name already exists and will either create it or ignore the command altogether. No notification is sent to the connecting device, but either the failure or the success of the command is registered in the local console.
SendCommands:
Commands are sent in the same manner as sets—once the set is recognized, the names of the associated commands are retrieved and serialized inside a ServerResponse instance and then pushed back to the requesting device.
LearnCommand:
Once a request was received that the server needs to learn a new command, an initial verification is done to make sure that the requested command name and set are not already taken. If neither the command nor the set exist, both will be created.
After the basic setup is complete, the IR sensor is activated and will be waiting for the command to be learned. The way it works is quite simple – the sensor will remain in learning mode until the point where it recognizes a command without error, being 100% sure that it can be reproduced internally. You will need to point your remote towards the sensor and hold the button you want captured for one or two seconds in order for the command to be learned.
NOTE: To ensure that a proper transmission is done, I manually set the minimal repeat value to 5. This is the number of times the sensor will fire the same code towards the target. That is the optimal value for a target device to receive the code if the remote is pointed directly at it without necessarily triggering the same command twice or more.
After the command is learned, the code is processed and transformed into a serializable instance. The connecting client is then notified about whether the command was learned.
ExecuteCommand:
Command execution relies on the hardware sensor. The phone sends a command execution request in the following format:
EXECUTE:SET_NAME, COMMAND_NAME
Once the command is parsed out and found in the local storage, the IR code is transformed back to a model that is recognizable by the Phidgets SDK and transmitted towards the location where the sensor is pointed at the time of the execution.
DeleteSet:
When deleting a set, only the name of the set should be specified. The user will get a warning on the client side that requires a confirmation of the deletion. The server will blindly execute the command.
DeleteCommand:
Not only can the user remove entire sets, but he can also target specific commands from a given set. Once a DELETE_COMMAND directive is recognized, the set name is parsed out from the original string, that follows the DELETE_COMMAND:SET_NAME, COMMAND_NAME format, and a simple LINQ query extracts the command instance, removes it and stores the set content on the local hard drive.
Notice that for some commands, particularly for set creation, deletion and command deletion, the server will return a list of the remaining items. The contents will be automatically updated on the devices, which will be waiting for that response. This measure was deliberately introduced to minimize the chances of a user triggering a command that was already deleted or trying to query a previously removed set.
Transforming Codes
You might have noticed that I am using IRCodeWorker.GetSerializableCodeType to transform a Phidgets SDK native IR code model into a serializable one. This is a helper function that performs a field copy of the existing object. Because of the differences in the model structure, it has to be done manually:
The reverse process is easier because I can pass each of the existing properties to the IRCodeInfo constructor. The only difference is the fact that I need to use Reflection to create an instance of IRLearnedCode because there is no public constructor defined and a dynamic object has to be created:
Command and Set Management
Looking back at the code that I put together for the command interpreter, there is one class that does all local content manipulation—StorageHelper. This is a simple class that performs LINQ queries on set as well as command collections, and makes sure that all the changes are preserved in the sets.xml file in the application folder that is used as the only storage place for all the content that is being manipulated by the server.
Sending Data Back to the Client
SendData in the NetworkHelper class handles all outbound connections. Here is its structure:
A new stream socket is created in order to connect to the target machine over the TCP pipe. If IP sanitization is enabled, the port is stripped from the address in order to pass a valid IP. A Socket instance cannot directly handle IPs of the format:
255.255.255.0:PORT_NUMBER
Later, in a synchronous manner, a connection is established and the data is sent.
At this point, you can see that the barebones service offers a flexible way to manage content. It can be accessed by any application type as long as the server can be accessed and the application can send commands in the pre-defined format and the content requested is actually located on the target server. This allows for high levels of extensibility and interoperability, as the server usage is not limited to a single platform. If I decide to create a Windows Store application that would allow me to control my TV, I simply need to add socket connection layer that will send plain strings to the machine where the IR sensor is connected.
Similarly, if some functionality needs to be added, it is possible to do so without ever touching the client applications. A modification in the endpoint will be reflected with no direct effect on all connection applications as long as all handled returned and requested values are preserved. The only additional requirement is that if the client applications want to take advantage of newly introduced capabilities, they need to have an updated command transmission layer for the new command types.
In Program.cs, I simply need to start the server through the CoreStarter class:
Mobile client overview
The mobile client does not have the capability to send commands directly to the IR sensor. Instead, it connects to a remote machine that has the IR sensor plugged in and attempts to invoke a command from the list returned by the service. A single mobile client can support control over multiple servers.
NOTE: Make sure that at the time of working with the Windows Phone client, the server is actually running on your local machine. To make it easier to test, also open port 6169 for incoming connections in Windows Firewall.
When building a Windows Phone application, make sure you have the proper version of the SDK installed, as well as a SLAT-compatible machine if you plan on testing the application in the emulator.
Networking Infrastructure
The Windows Phone application also relies on a network infrastructure somewhat similar to that of the server. There is a TCP listener that is created when the application is started:
Here, listener is an instance of TcpSocketListener—a custom class designed to handle incoming network connections:
A StreamSocketListener is used for the connection core. When a connection is received, a continuous loop reads the entire contents of the incoming stream. OnConnectionCompleted is declared in the base class—SocketConnectorBase.
ConnectionEventArgs here is used to identify the content that is passed to the client. DeviceID gives access to the source IP, IsSuccessful tells the developer whether the established connection is active and the Token carries the raw string if any was received.
Sending data is simplified to the maximum with the help of the SocketClient class, which relies on a StreamSocket instance that handles outbound connections and writing to the output stream:
As with the listener class, SocketClient supports OnConnectionCompleted to notify the application that the connection attempt completed.
Back in App.xaml.cs, the data from the incoming connection captured by the TcpSocketListener instance is passed to the ResponseHelper class:
This class reads the possible three commands sent by the server and interprets them, creating internal collections from the raw data if the current server IP matches the one obtained in the ServerResponse (the same model in the desktop application):
If the response comes from a server that is different than the one that is currently active, the data is discarded as the user no longer needs it. Also, for specific commands, the mobile application will be on standby, waiting for a response (unless the user decides to cancel the request) – the IsWaiting flag is an application-wide indicator that a pending server action is in the queue.
Same as with the server, the commands in the Windows Phone application are represented through pre-defined constants:
Let’s now take a closer look at how it is handled internally to build the visual layer.
Handling the Data
The first thing users will see when the application is launched is the list of registered servers:
This is ServiceListPage.xaml. The list of servers that were added is retrieved from the isolated storage on application startup, with the help of the standard serialization routine implemented in the Coding4Fun Toolkit—specifically, its storage subset (you can get it via NuGet):
The one-liner that initializes the internal server collection is as follows:
Here, the SERVERS_FILE constant is eq