[90] MQTT Support

Use this forum to discuss possible implementation of a new feature before opening a ticket.
A developer shall edit the topic title with "[xxx]" where xxx is the id of the accompanying tracker id.
Duplicate posts about the same id. +1 posts are not allowed.
Post Reply
User avatar
gizmocuz
Posts: 8491
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: MQTT Support

Post by gizmocuz » Sunday 03 November 2013 16:08

14 June 2015:

This forum topic has been cleanup.

MQTT is implemented, and additional info can be found on the wiki:
http://www.domoticz.com/wiki/MQTT

Please note that the next SDCard image of the stable release will include native NodeJS and Node Red.
Quality outlives Quantity!

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Monday 04 November 2013 22:02

I can see (imagine) some of the power that MQTT could bring us, but how many devices already "talk" MQTT that Domoticz would benefit from?

kylegordon
Posts: 96
Joined: Friday 01 November 2013 0:58
Target OS: Linux
Domoticz version: Trunk
Contact:

Re: MQTT Support

Post by kylegordon » Friday 08 November 2013 16:05

maxtrash wrote:I can see (imagine) some of the power that MQTT could bring us, but how many devices already "talk" MQTT that Domoticz would benefit from?
Finding MQTT enabled hardware can be tricky as it's often rebranded. A badly formatted list follows... :-)

Firstly, there's a list at https://xively.com/dev/hardware/
Android apps to monitor the environment, location, hardware stats, etc.
Anything that connects to an Arduino, Netduino or Launchpad microprocessor can also be MQTT enabled (so, in my house, that's PIRs, LED strips, temperature sensors, magnetic switches, Lux sensors, Bluetooth proximity).
Ford has about 80 cars in their range that can be sensor enabled (even when 'parked') using the OpenXC platform (think Geofencing, windshield rain sensors, engine air temperature and pressure sensors, humidity, etc - your car is a mobile weather station)
Sprint are using it for their in-vehicle systems

It's not so much about the 'what hardware uses it right now?', but more 'what hardware will it enable?' - it's the same argument that stunted IPv6 deployment for over a decade.

A useful article about home automation with MQTT is by the inventor of it here http://electronicdesign.com/embedded/mq ... net-things
1 x HP DL380 & KVM
1 x RPI
1 x RFXtrx433 V78
11 x LWRF Switches
1 x LIFX
2 x Echo Dots
6 x Byron PIRs
2 x Nexa PIRs
2 x Kodi
2 x ESP8266 MQTT
1 x GPIO/MQTT PIR
1 x GPIO(PWM)/MQTT LEDs
1 x GPIO(SPI)/MQTT LPD6803 LEDs
Lots of Node-Red glue

kylegordon
Posts: 96
Joined: Friday 01 November 2013 0:58
Target OS: Linux
Domoticz version: Trunk
Contact:

Re: MQTT Support

Post by kylegordon » Monday 09 December 2013 14:49

If anyone else is using MQTT at the moment, there's a great plugin for XBMC that will display MQTT notifications on your big screen. Great for watching out for those updates :-)
1 x HP DL380 & KVM
1 x RPI
1 x RFXtrx433 V78
11 x LWRF Switches
1 x LIFX
2 x Echo Dots
6 x Byron PIRs
2 x Nexa PIRs
2 x Kodi
2 x ESP8266 MQTT
1 x GPIO/MQTT PIR
1 x GPIO(PWM)/MQTT LEDs
1 x GPIO(SPI)/MQTT LPD6803 LEDs
Lots of Node-Red glue

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Saturday 28 December 2013 18:35

I'm experimenting a little, and trying to learn a bit about javascript, node.js, sockets, mqtt etc. in the process. I managed to install node.js on my raspberry and got basic MQTT functionality working. The nodejs script server.js will subscribe to a particular topic and show the temperature payload that is published to it.
I will continue my experimenting, this is just basic prototyping I guess.

Code: Select all

var h2d = function(h) {
	return parseInt(h,16);
};

var	HID = '3'; 											//Hardware ID of dummy in Domoticz
	IP = '127.0.0.1'; 									//IP address of Domoticz (127.0.0.1 for same machine)
	Port = '8080';										//Port of Domoticz

// for types please refer to http://sourceforge.net/p/domoticz/code/HEAD/tree/domoticz/main/RFXtrx.h
// for Maxbuddy API please refer to http://bugs.maxbuddy.de/projects/maxbuddy/wiki/Buddy_API
var	WMT_dtype1 = '80', WMT_subtype1 = '9'; 				//WallMountedThermostat Domoticz types for current temperature (temperature , RUBiCSON)
var	HT_dtype1 = h2d('0x50'), HT_subtype1 = '9'; 		//HeatingThermostat Domoticz types for current temperature (temperature , RUBiCSON)
var	HT_dtype2 = h2d('0x52'), HT_subtype2 = h2d('0xA'); 	//HeatingThermostat Domoticz types for current temperature (temperature , RUBiCSON)
var	SC_dtype1 = h2d('0x20'), SC_subtype1 = '01';		//ShutterContact 

var sys = require('sys');
var net = require('net');
var mqtt = require('./MQTTClient.js');
var request = require('request');

var io  = require('socket.io').listen(5000);
var client = new mqtt.MQTTClient(1883, '127.0.0.1', 'pusher');

var toString = function(text) {
	return text + '';
};

 
io.sockets.on('connection', function (socket) {
  socket.on('subscribe', function (data) {
    console.log('Subscribing to '+data.topic);
    client.subscribe(data.topic);
  });
});
 
client.addListener('mqttData', function(topic, payload){
  var DeriveUrl;
  
  sys.puts(topic+'='+payload);
  console.log('listening');
  io.sockets.emit('mqtt',{'topic':String(topic),
    'payload':String(payload)});
  
  DeriveUrl = 	'http://'+IP+':'+Port+'/json.htm?type=command&param=udevice&hid='+HID+'&did='+topic+'&dunit='+'1'+'&dsubtype='+HT_subtype1+
				'&dtype='+HT_dtype1+'&nvalue='+'0'+'&svalue='+payload;
  console.log( DeriveUrl);	
  request( DeriveUrl, function (error, response, body) {
  if (!error && response.statusCode == 200) {
    console.log(body) // Print the web page.
  }
  })
});

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Monday 30 December 2013 13:44

not sure if anyone is interested but let's just document my progress here.
the previous example was about domoticz "subscribing" to mqtt and showing a value in it's GUI.
The following LUA-script will publish every event in Domoticz to MQTT. The accompanying node.js script is part of the comments.

Code: Select all

--[[
MQTT demo device script
 
devicechanges are pushed to localhost port 5001 if there's any value to publish
a node.js script like this is waiting and publishes using mqtt

var url = require("url");
var http = require('http');
var pubclient = new mqtt.MQTTClient(1883, '127.0.0.1', 'domoticz');

http.createServer(function (req, res) {
	res.writeHead(200, {'Content-Type': 'text/plain'});
	res.end('Response from Node.js \n');
	pubclient.publish('/events/domoticz'+url.parse(req.url).pathname, url.parse(req.url).query);	
}).listen(5001, '127.0.0.1');

NB. Domoticz will add _ and property to the name of certain sensors (e.g. _temperature, _humidity). This is passed as lowest level of message in mqtt
--]]

commandArray = {}

logging = true
d = otherdevices
device = ""
for i, v in pairs(devicechanged) do
  if (#device == 0 or #i < #device) then device = i end
  text = v..''
  if #text > 0 then
	text = string.gsub(i, "_", '/')
	text = '127.0.0.1:5001/'..text..'?'..v
	text = string.gsub(text, "%s+", '%%20')
	commandArray['OpenURL']=text
	if (logging) then print(text) end
  end
end

if (logging) then print('--- LUA Device script MQTT --- Triggered by ' .. device .. ' now ' .. d[device]) end

return commandArray

alfred_j_kwak
Posts: 253
Joined: Friday 20 September 2013 18:49
Target OS: Raspberry Pi
Domoticz version: 2.3530
Location: Finland
Contact:

Re: MQTT Support

Post by alfred_j_kwak » Monday 30 December 2013 13:55

I'm interested, but I haven't find any MQTT for dummies documentation and starting feels quite difficult. So it's nice to read your progress especially with Domoticz.

-Jussi-

User avatar
gizmocuz
Posts: 8491
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: MQTT Support

Post by gizmocuz » Wednesday 01 January 2014 5:33

Yep, thats my thoughts too ;)

What is the difference compared to NodeJS and would the later not be better to get implemented?
Quality outlives Quantity!

sebas
Posts: 171
Joined: Tuesday 17 September 2013 7:46
Target OS: Linux
Domoticz version:
Location: Deventer, The Netherlands
Contact:

Re: MQTT Support

Post by sebas » Wednesday 01 January 2014 14:09

This looks as a great solution to connect an Arduino to Domoticz. I've got an Arduino setup that measures temperature and humidity and controls lighting that I'd like to connect to Domoticz. I've found a library for Arduino that can push and receive data to/from a MQTT broker.

@Maxtrash: your script, if I understand it correctly, subscribes to a broker and that pushes the data to virtual devices in Domoticz via the JSON API. Do you also run a MQTT broker (server) on your Pi or do you use an externally hosted one? I've found m2m.io as an externally hosted broker.

I'll try to get the combo with an Arduino up and running the coming days.
Last edited by sebas on Wednesday 01 January 2014 15:08, edited 1 time in total.

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Wednesday 01 January 2014 14:26

I did struggle a little, but thats was mainly because I was clueless about it all. Node.js is a runtime environment which allows running javascripts on the server. I think I more or less followed this guide to get it running.
http://blog.rueedlinger.ch/2013/03/rasp ... sic-setup/

with the help of nodejs it's relatively easy to create some sort of "star topology". E.g. I could have a file server.js running within nodejs looking like this:

Code: Select all

// https://github.com/maartendamen/DomoMQTT/wiki
// actions/device_id/property
// events/device_type/device_id/property
// config/id
// logging/id

var domoticz = require('./domoticz.js');
var pusher = require('./pusher.js');
var elvmax = require('./elvmax.js');
var x10 = require('./x10.js');
var plugwise = require('./plugwise.js')
var eventghost = require('./eventghost.js');
var usbuirt = require('./usbuirt.js');
This way domoticz only needs to have one interface. As long as all drivers use the same protocol (structures)

For all of this we don't need MQTT, node.js alone would be enough. However instead of just sending messages point to point (e.g. x10 to domoticz), it would make sense to publish them to a bus. That way anyone/anything interested in the messages coming from the x10-driver could subscribe to it.

@gizmocuz: you don't need to implement node.js because you've already done it :-) using the http json-api node.js can send information to domoticz and with lua/openurl domoticz can send info back to nodejs. Implementing a native mqtt interface could be nice in the future, but frankly there's not that much practical use for MQTT at the moment. But who knows, for instance this is also interesting stuff: https://play.google.com/store/apps/deta ... .mqttitude

on thing though: it would be nice to have more information available in the lua-scripts. Now I can publish messages using the name of devices and it's state, but it would also be very practical to have the devicetype and room available in LUA
Last edited by maxtrash on Wednesday 01 January 2014 22:44, edited 1 time in total.

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Wednesday 01 January 2014 14:27

sebas wrote:This looks as a great solution to connect an Arduino to Domoticz. I've got an Arduino setup that measures temperature and humidity and controls lighting that I'd like to connect to Domoticz. I've found a library for Arduino that can push and receive data to/from a MQTT broker.

@Maxtrash: your script, if I understand it correctly, subscribes to a broker and that pushes the data to virtual devices in Domoticz via the JSON API. Do you also run a MQTT broker (server) on your Pi or do you use an externallay hosted one? I've found m2m.io as an externally hosted broker.

I'll try to get the combo with an Arduino up and running the coming days.
correct and I'm using the broker Mosquitto on the PI.

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Wednesday 01 January 2014 14:59

Maybe to avoid confusion: the script subscribes to a broker. Then I can publish messages to the topic and the nodejs script then pushes them to Domoticz

sebas
Posts: 171
Joined: Tuesday 17 September 2013 7:46
Target OS: Linux
Domoticz version:
Location: Deventer, The Netherlands
Contact:

Re: MQTT Support

Post by sebas » Wednesday 01 January 2014 15:11

maxtrash wrote:Maybe to avoid confusion: the script subscribes to a broker. Then I can publish messages to the topic and the nodejs script then pushes them to Domoticz
Thanks for the info and the great work in the script. I've got node.js and mosquitto up and running and I'm able to send messages from my Android phone to my Macbook via Mosquitto and back.

I'm now figuring out how I can get all the values for the device id's etc. to push things to Domoticz. I've created a virtual device and added a Temp + Humidity sensor to that.

sebas
Posts: 171
Joined: Tuesday 17 September 2013 7:46
Target OS: Linux
Domoticz version:
Location: Deventer, The Netherlands
Contact:

Re: MQTT Support

Post by sebas » Wednesday 01 January 2014 18:20

I've managed to update Domoticz using the post of maxtrash as a guide. I did change the server.js file a bit. Wanted to make it more "general" without the need for configuration per sensor upfront. It expects a json string with all the data as an aray in it.

It's still a work in progress.

Code: Select all

/*Notes
Removed Socket.io as it is not needed for subscribing to MQTT and pushing to Domoticz
Installed MQTT via NPM as node.js module. npm install mqtt
Installed Request via NPM to do HTTP requests to Domoticz. npm install request

Input is JSON
Example data:
{
   "data":[{
      "idx":247,
      "svalue":"21.1;70%"
   }]
}

*/

var IP = '127.0.0.1';                            //IP address of Domoticz (127.0.0.1 for same machine)
var Port = '8080';                              //Port of Domoticz

var sys = require('sys');
var net = require('net');
var request = require('request');
var mqtt = require('mqtt')

client = mqtt.createClient(1883, 'localhost');

var toString = function(text) {
   return text + '';
};

client.subscribe('test');
client.publish('test', 'This is the Arduino');

client.on('message', function (topic, message) {
  console.log(message);
  
  try {
    var payload = JSON.parse(message);  
    console.log(payload);
    var url = 'http://'+IP+':'+Port;
    var svalue = 0;
    var idx = 0;
    var nvalue = 0;
    console.log(payload.data);

    for (var i = payload.data.length - 1; i >= 0; i--) {
      var data = payload.data[i];
      if(data.idx) {
        idx = data.idx;
      }
      if(data.nvalue) {
        nvalue = data.nvalue;
      }
      if(data.svalue) {
        svalue = data.svalue;
      }
      url = url + "/json.htm?type=command&param=udevice&idx="+idx+"&nvalue="+nvalue+"&svalue="+svalue;
      console.log(url);
      request(url, function(error, response, body){
        console.log("Sending requesd");
        console.log(url);
        console.log(error);
        console.log(response);
        console.log(body);
      });
    };
  }
  catch(e) {
    console.log("Could not parse JSON.");
    console.log(e);
  }

});

andriej
Posts: 302
Joined: Tuesday 10 December 2013 23:27
Target OS: Linux
Domoticz version: beta
Contact:

Re: MQTT Support

Post by andriej » Wednesday 01 January 2014 22:02

I guess it looks just too complicated for me ATM to report everything from my mega with this protocol.
Any sketches out there to see example of Arduino-Raspberry-Domoticz cooperation?
Everything sounds great and I'd really like to transfer data via RF24 module from nodes (nanos) to MEGA and then via LAN to Domoticz.

--edit:
I've done some research and MQTT sounds perfect for using around Domoticz and arduino.
Can't wait to see progress in this.
Last edited by andriej on Thursday 02 January 2014 1:51, edited 1 time in total.
Orange Pi (@ Debian) / MySensors Serial / GPIO / Custom Serial 433 MHz Transmitter (for plug switches and livolo wall switches) / JSON&Bash API scripts

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Thursday 02 January 2014 1:49

nice work Sebas. You seem to be more experienced in java script?
I like to use the standard NPM MQTT library as well, thanks for pointing me to it. I'm not too sure about using json for payloads, it might complicate things a bit but that might change once I get the hang of it.
Personally in the json to domoticz I like to use hid/did/dunit/dsubtype instead of idx because that way you can push new devices to domoticz that would automaticly show up in the unused devices tab. But it should be possible to support both ways. I'll come up with some code suggestions later on!

sebas
Posts: 171
Joined: Tuesday 17 September 2013 7:46
Target OS: Linux
Domoticz version:
Location: Deventer, The Netherlands
Contact:

Re: MQTT Support

Post by sebas » Thursday 02 January 2014 9:22

andriej wrote:I guess it looks just too complicated for me ATM to report everything from my mega with this protocol.
Any sketches out there to see example of Arduino-Raspberry-Domoticz cooperation?
Everything sounds great and I'd really like to transfer data via RF24 module from nodes (nanos) to MEGA and then via LAN to Domoticz.

--edit:
I've done some research and MQTT sounds perfect for using around Domoticz and arduino.
Can't wait to see progress in this.
The example sketches I've found for the Arduino seemed easy to use. There's a library that takes care of all the communication over MQTT. You'll only need to create the payload message for the data you want to send. It can also receive messages so it would be possible to control an Arduino using Domoticz.

Once I've got the server setup and figured out how to update sensors I start on the Arduino side of things. I'll publish the sketch code when I get it to work.
maxtrash wrote:nice work Sebas. You seem to be more experienced in java script?
I like to use the standard NPM MQTT library as well, thanks for pointing me to it. I'm not too sure about using json for payloads, it might complicate things a bit but that might change once I get the hang of it.
Personally in the json to domoticz I like to use hid/did/dunit/dsubtype instead of idx because that way you can push new devices to domoticz that would automaticly show up in the unused devices tab. But it should be possible to support both ways. I'll come up with some code suggestions later on!
I've done some web development that involved javascript, that helps :-) I'm new to Node.js so it's partly copy-paste from tutorials I've found.

JSON is nice to use as it is easy to create. Allmost all languages have support for it and in the end it's just a string of text that's easy to construct on an Arduino. When it's parsed in JS it's an object that you can loop through. That way it's possible to work with an unlimted number of sensors without the need to do any coding in the server.js file.

Is there any documentation around the hid/did/dunit/dsubtype? The only thing I've found is a small description about idx, nvalue and svalue in the Wiki and that also leaves me with more questions. I can update the values in Domoticz but somehow my temp + humidity sensor expects 3 values where I would expect to have 2 and also it's storing the temp and showing it in the graph it's not showing it on the temperature screen in the web front-end. Code examples will help.

maxtrash
Posts: 165
Joined: Tuesday 06 August 2013 1:31
Target OS: -
Domoticz version:
Contact:

Re: MQTT Support

Post by maxtrash » Thursday 02 January 2014 9:51

In my max buddy script I'm using temp + humidity sensors. I'll have a look tonight. Back to work again...
Do you happen to know an easy way to parse a topic string in parts (eg. Split a/b/c in a, b and c?)

sebas
Posts: 171
Joined: Tuesday 17 September 2013 7:46
Target OS: Linux
Domoticz version:
Location: Deventer, The Netherlands
Contact:

Re: MQTT Support

Post by sebas » Thursday 02 January 2014 11:08

maxtrash wrote:In my max buddy script I'm using temp + humidity sensors. I'll have a look tonight. Back to work again...
Do you happen to know an easy way to parse a topic string in parts (eg. Split a/b/c in a, b and c?)
var parts = topic.split("/");
Where topic is the string of topics. The variable parts will be an array of the topic parts. parts[0] will return a, parts[1] returns b etc.

andriej
Posts: 302
Joined: Tuesday 10 December 2013 23:27
Target OS: Linux
Domoticz version: beta
Contact:

Re: MQTT Support

Post by andriej » Thursday 02 January 2014 11:56

The best way I see the cooperation between Domoticz and multiple Arduino sensors is MQTT on i.e. Arduino MEGA and nodes (i.e. Nanos).
That's the setup I'm currently using, so Arduino Mega is connected to LAN via Ethernet and it uses RF24Network library (2.4GHz rx/tx) to get values from sensors (up to >2000 possible nodes).

The only problem I see is that RF24Network can send/receive only 24bytes of data in one packet, so I guess it's not enough for longer commands/data for MQTT?
I can't wait to see your examples/sketches. I'll try to help in this thread, as it's the only way I can get my remotes, sensors and weather station connected to Domoticz (RFXCom currently doesn't support it, so I'm left with unusable device...)

Regards

--edit:
Did some googe'ing for RF24 and MQTT to implement.
Found this library (for Raspberry): https://github.com/mannkind/Ripcord
May be a hint.
But still - better use would come from "RF24Network" library on MEGA with proper rx/tx of MQTT.
Orange Pi (@ Debian) / MySensors Serial / GPIO / Custom Serial 433 MHz Transmitter (for plug switches and livolo wall switches) / JSON&Bash API scripts

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests