Publish using MQTT

Overview

Publish/Subscribe model

MQTT is a lightweight publish/subscribe messaging protocol designed for machine to machine telemetry in low bandwidth environments. It is useful for use with low power sensors, but is applicable to many scenarios. The MQTT protocol is based on the principle of publishing messages and subscribing to topics. Multiple clients connect to a server often termed as a "broker". The broker is a central communication point that transmits the messages between senders and the clients who subscribed to receive the messages on a certain topic. Clients include the topic in the message, when publishing to the broker. The topic is the routing information for the broker. The broker delivers all the messages on a specific topic to the client, if the client has subscribed to it. The broker and MQTT act as a simple, common interface for everything to connect to. The data producers and the data consumers are independant of each other.

Figure 1

Subscription to MQTT topics

Messages in MQTT are published on topics. There is no need to configure a topic, publishing on it is enough. Topics are simple strings treated as a hierarchy, using a slash (/) as a separator.

Clients can receive messages by subscribing to a topic.

In figure 1, we see Client 1 (the Publisher, a temperature sensor) sending temperature to the broker. Client 2 and Client 3 subscribed to the topic and will receive all the messages on that topic. In this case, a sample topic for sending temperature of room S25 in Duboue building of UPPA could be UPPA/Duboue/S25/temp.

Figure 2

A subscription can be to an exact topic, in which case the client will receive all the messages on that exact topic, or it may include wildcards. Two wildcards are available, + or #. The + sign can be used as a wildcard for a single level of hierarchy. The subscription to UPPA/Duboue/+/temp would result in all the messages sent to UPPA/Duboue/S25/temp as well as any topic with an arbitrary value in the place of S25, for example, UPPA/Duboue/S24/temp. The # sign would then replace several level of hierarchy and is very useful when the detailed hierarchy is not known. For instance UPPA/# would get all data, and topics, from UPPA university. A detailed example is shown in figure 2.

Multiple applications have been developed on MQTT which include Amazon Web Services, EVERYTHING IoT platform, Facebook Messenger and many others which are available on PlayStore for Android and AppStore for iOS. An example of Google PlayStore is shown in figure 3 below.

Figure 3

Further documentation on MQTT is available on their official website mqtt.org.

Brief description of the code

We will briefly explain the steps used in the example below to use the MQTT protocol.

Include the publish/subscribe library

 
#include <PubSubClient.h>

Test parameters for MQTT: topic, temperature and MQTT server address

 
char *topicin = "UPPA/test"; 
char *topicout = "UPPA/Duboue/S25/temp"; 
char *msgTemp  = "22.5"; 
char* mqtt_server = "test.mosquitto.org";

Define the publish/subscribe client

WiFiClient espClient; 
PubSubClient client(espClient);

Define a function callback() to process incoming message from a given subscribed topic. This is not needed if you only publish, which is more typical of an end-device.

 
void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }

Define a function reconnect() to keep trying to connect to the MQTT broket until it is successful. Here the client id can be random because the MQTT broker we use does not require authentication.

 
void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");

    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
    // Wait 5 seconds before retrying  
      delay(5000);
    }
  }
}

In function setup(), setup the MQTT server

 
client.setServer(mqtt_server, 1883);

In function loop(), check if client is connected, if not then reconnect

 
void loop() {

  if (!client.connected()) {
    reconnect();
  }

At the end, publish the message on specified topic

   
  int e=client.publish(topicout, msgTemp);

Complete working example

      
// if you have an ESP8266 based board
#define ESP8266

#if defined ESP8266 || defined ARDUINO_ESP8266_ESP01
include <ESP8266WiFi.h>
#else
#include <WiFi.h>
#endif

#include <PubSubClient.h>

// Update these with values suitable for your network.

char* ssid = "iPhoneD";
char* password = "345hello";

char *topicin = "UPPA/test"; 
char *topicout = "UPPA/Duboue/S25/temp"; 
char *msgTemp  = "22.5"; 
char* mqtt_server = "test.mosquitto.org";

WiFiClient espClient;
PubSubClient client(espClient);

int WiFi_status = WL_IDLE_STATUS;

void setup_wifi() {

  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  randomSeed(micros());

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Create a random client ID
    String clientId = "ESP8266Client-";
    clientId += String(random(0xffff), HEX);
    // Attempt to connect
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");

    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
    // Wait 5 seconds before retrying  
      delay(5000);
    }
  }
}

void setup() {
  delay(3000);
  Serial.begin(38400);

// Print a start message  
  Serial.println(F("Simple MQTT demo")); 
  setup_wifi();
  client.setServer(mqtt_server, 1883);
  //client.setCallback(callback);
}

void loop() {

  if (!client.connected()) {
    reconnect();
  }
  
  //client.loop();  
  WiFi_status = WiFi.status();
  
  if ( WiFi_status != WL_CONNECTED) {
  while ( WiFi_status != WL_CONNECTED) {
      Serial.print("Attempting to connect to WPA SSID: ");
      Serial.println(ssid);
      // Connect to WPA/WPA2 network
      WiFi_status = WiFiWiFi.begin(ssid, password);
      delay(500);
    }
    Serial.println("Connected to AP");   
  } 
  Serial.print("Publishing: status=");

  int e=client.publish(topicout, msgTemp);
  Serial.println(e); 
  delay(7500);   
 }                 

The raw source of the sketch example is visible here.

Testing and receiving the published MQTT message

To test the MQTT publishing from the IoT device, we will use a computer with an MQTT client to listen for a specific topic.

IMPORTANT. In all the following examples and assignments, if you do not have access to a computer with a terminal window to use mosquitto_sub and mosquitto_pub command line, you can use the HiveMQ MQTT web client to subscribe and publish in place of mosquitto_sub and mosquitto_pub. In this case, also replace the MQTT broker test.mosquitto.org by broker.hivemq.com in the Arduino example.

 
char* mqtt_server = "broker.hivemq.com";

From now on, we will describe examples and assignments using mosquitto_sub and mosquitto_pub command line and assume you will adapt according to your local settings. With Eclipse Mosquitto in command line mode, we will use the mosquitto_sub client. If it is not installed, you need to install it (or see above to use HiveMQ MQTT web client instead).

mosquitto_sub takes at least 2 parameters: -h and -t to indicate respectively the MQTT broker and the topic to subscribe to. In the Arduino example, the MQTT broker used was test.mosquitto.org and the published topic was "UPPA/Duboue/S25/temp". The -v option will display information in verbose mode to get the complete topic in presence of a wildcard.

The command would then look like: mosquitto_sub -v -h test.mosquitto.org -t UPPA/#, which means subscribe to all topics under UPPA/.

Each time that the IoT device is publishing, you should see on your computer terminal an output similar to:

 
> mosquitto_sub -v -h test.mosquitto.org -t UPPA/#
UPPA/Duboue/S25/temp 22.5
UPPA/Duboue/S25/temp 22.5
UPPA/Duboue/S25/temp 22.5
...

Going a step further: use Node-RED to create complex data workflows

Node-RED

Node-RED "is a programming tool for wiring together hardware devices, APIs and online services in new and interesting ways. It provides a browser-based editor that makes it easy to wire together flows using the wide range of nodes in the palette that can be deployed to its runtime in a single-click. Node-RED is a flow-based programming tool, originally developed by IBM’s Emerging Technology Services team and now a part of the JS Foundation".

It is built on Node.js and is available on a large variety of platforms. You can use Node-RED on a Raspberry PI, a desktop computer or your laptop. If it is not already installed on your computer, you can follows instructions from the Node-RED web site. See this video illustrating Node-RED's main features.

You need first to start Node-RED locally with:

> node-red
or
> node-red-start

Then, you can use a web browser to connect to the local Node-RED web interface on http://127.0.0.1:1880.

Assignment 1: write a simple Node-RED flow with MQTT nodes

Create a Node-RED flow with:

  • an MQTT input node listening for data on "UPPA/Duboue/S25/temp" topic from test.mosquitto.org MQTT broker
  • a Function node that would decrease the received temperature by 1.8 degree Celcius
  • an MQTT output node that will publish the new temperature on "UPPA/Duboue/S25/realtemp" topic on test.mosquitto.org MQTT broker

In the Function node, you can add simple Javascript code to process msg.payload which will normally contain the received temperature (string format) on "UPPA/Duboue/S25/temp". You can also add a debug node after the MQTT input node to verify that you can correctly receive on "UPPA/Duboue/S25/temp". You can then test your Node-RED flow by deploying it (Deploy button) and use mosquitto_pub and mosquitto_sub commands to respectively publish on "UPPA/Duboue/S25/temp" and receive on "UPPA/Duboue/S25/#".

The commands would then look like:

  • mosquitto_pub -h test.mosquitto.org -t UPPA/Duboue/S25/temp -m "21.6", to publish a temperature of 21.6 degree Celsius
  • mosquitto_sub -v -h test.mosquitto.org -t UPPA/Duboue/S25/#, to subscribe to all topics under UPPA/Duboue/S25

Use a terminal to subscribe. Then use another terminal to publish. Normally, each time you publish "21.6" on "UPPA/Duboue/S25/temp", you should receive a modified temperature on "UPPA/Duboue/S25/realtemp", if your Node-RED flow is correctly developed.

 
> mosquitto_sub -v -h test.mosquitto.org -t UPPA/Duboue/S25/#
UPPA/Duboue/S25/realtemp 19.8
UPPA/Duboue/S25/realtemp 19.8
UPPA/Duboue/S25/realtemp 19.8
...

Assignment 2: add a ThingSpeak node to upload on ThingSpeak cloud

In a second step, you will add a ThingSpeak node to publish the modified temperature on a ThingSpeak channel. To do so, you may need to install the ThingSpeak node with:

> cd
> cd .node-red/node_modules
> npm install node-red-contrib-thingspeak42

You can then use our ThingSpeak LoRa test channel (https://thingspeak.com/channels/66794, the write key is "SGSH52UGPVAUYG3S") and assign the modified temperature to the field of your choice (field 1 to field 8). The Node-RED ThingSpeak node works as follows: for each field, you need to indicate a matching topic that will trigger the upload on that field. So, somewhere in the Function node, you need to assign msg.topic to the topic you chose for that field. In this example, you can just take msg.topic= UPPA/Duboue/S25/realtemp. If your Node-RED flow is correct, you should also see the temperature value you published on "UPPA/Duboue/S25/realtemp" uploaded on the ThingSpeak channel as well. Check on the channel page for the data you are uploading.

Assignment 3: emulating a real-world IoT deployment scenario

Here, we are going to emulate a more realistic scenario where an IoT gateway will run the Node-RED flow and will receive data from sensor nodes through an adhoc wireless communication using long-range radios such as LoRa for instance. In that case, the Node-RED flow will not listen on an MQTT topic but rather will receive and process data from sensor nodes.

To add an additional data sub-flow in the current flow, simply add new processing nodes to the current Node-RED flow and chain them accordingly. All Node-RED data sub-flows are executed in parallel.

Here, you will use an Inject node to inject the string "Sensor6/temp/21.6" to the Node-RED flow. That would emulate reception of data from a physical sensor node. From a Node-RED perspective, msg.payload will be set to "Sensor6/temp/21.6". This string will then be injected into a Change node. At this point, to better understand how Node-RED messages work and their format you may have a look at Steves Node-Red Guide on "Understanding and Using The Node-Red Message Object" and Node-RED documentation on "Working with messages"

Configure the Change node to translate "Sensor6" into "UPPA/Duboue/S25" and "temp" into "realtemp". Then add a new Function node to process "UPPA/Duboue/S25/realtemp/21.6" in order to have msg.topic set to "UPPA/Duboue/S25/realtemp" and msg.payload changed to "21.6". You can use Javascript msg.payload.lastIndexOf(), msg.payload.substring() and msg.payload.length to write your processing code. Remember that you can add debug nodes to verify at each step of your Node-RED flow that it performs as expected.

The message that will come out from the new Function node will then be injected to a copy of your previous Function node that would decrease the received temperature by 1.8 degree Celcius. The final MQTT output node that publishes on "UPPA/Duboue/S25/realtemp" will now simply need to use msg.topic as the MQTT topic and msg.payload as the new temperature value to publish. Therefore leave the MQTT Topic field empty in the MQTT node configuration tab.

As in Assignment 1, use a terminal to subscribe to "UPPA/Duboue/S25/#". Normally, each time you inject the string "Sensor6/temp/21.6" in your Node-RED flow, you should receive a modified temperature on "UPPA/Duboue/S25/realtemp", if your Node-RED flow is correctly developed.

 
> mosquitto_sub -v -h test.mosquitto.org -t UPPA/Duboue/S25/#
UPPA/Duboue/S25/realtemp 19.8
UPPA/Duboue/S25/realtemp 19.8
UPPA/Duboue/S25/realtemp 19.8
...

Going even further: add additional Node-RED processing nodes

Play with Node-RED, search the web and add additional processing nodes such as email or social media nodes. For email, install the Node-RED node as follows:

> cd
> cd .node-red/node_modules
> npm install node-red-node-email

Unleash your imagination and be creative!

Enjoy!