[90] MQTT Support

Use this forum to discuss possible implementation of a new feature before opening a ticket.
A developer shall edit the topic title with "[xxx]" where xxx is the id of the accompanying tracker id.
Duplicate posts about the same id. +1 posts are not allowed.
wblock
Posts: 1
Joined: Saturday 12 December 2015 5:46
Target OS: -
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by wblock » Saturday 12 December 2015 6:54

I have built a bunch of sensors based on ESP8266/NodeMCU hardware. They report to a Mosquitto MQTT server with topics like office/temperature and office/humidity, or shed/temperature and shed/batteryvoltage. This all works, and I can see the published data with mosquitto_sub. (I looked for existing protocols for sensors, and MQTT seemed to be the thing to use.) For example:

Code: Select all

office/temperature 65
office/humidity 39
shed/temperature 33
shed/humidity 64
shed/batteryvoltage 12.75
The next step is to provide web graphs and, hopefully, alarms. Domoticz looked good because it does not rely on PHP or Java, but the MQTT graphic in the wiki (https://www.domoticz.com/wiki/MQTT) has me wondering if it is the right tool for the job.

I can add the broker (running on the same machine as Domoticz), but then don't see a way to add sensors attached to that broker. It appears that setting this up might involve Lua, Node.js, JSON, and reprogramming all the sensors. I was hoping to just add the topics in a GUI, though.

Maybe there is a plugin or add-on that simplifies this?

Thanks!

wildfire
Posts: 29
Joined: Friday 18 September 2015 11:04
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by wildfire » Saturday 12 December 2015 11:07

wblock,
at this point you domoticz only listens on domoticz/in and you have to use Json.
also you have to add the idx of the device and you have to create it first, at least thats how i managed to get this working.
so you make a dummy hardware then a virtual device, then check the idx and then send to that idx.
example for temperature
{
"idx" : 8,
"nvalue" : 0,
"svalue" : "26.2"
}
example for humi
{
"idx" : 14,
"nvalue" : 50,
"svalue" : ""
}

i added a node-red flow to change this.
also i noticed it was buggy when you post some sensors a the same time, i changed to the json post method for now.

shkaff
Posts: 24
Joined: Friday 11 December 2015 11:29
Target OS: Raspberry Pi
Domoticz version: Stable
Contact:

Re: [90] MQTT Support

Post by shkaff » Monday 14 December 2015 16:30

Dear community, let me please ask you to confirm my understanding that domoticz isn't actually an MQTT client and has no settings to specify to which MQTT topic to subscribe.
The platform Domoticz only allows external applications like Node-RED script to manipulate with MQTT messages and send / receive them to / from Domoticz, right?

I looked thru settings of Domoticz and found no way to specify an address of Mosquito or any other MQTT broker to subscribe

Kouseri
Posts: 76
Joined: Sunday 04 January 2015 22:24
Target OS: Raspberry Pi
Domoticz version:
Location: Finland
Contact:

Re: [90] MQTT Support

Post by Kouseri » Monday 14 December 2015 21:37

shkaff wrote:Dear community, let me please ask you to confirm my understanding that domoticz isn't actually an MQTT client and has no settings to specify to which MQTT topic to subscribe.
The platform Domoticz only allows external applications like Node-RED script to manipulate with MQTT messages and send / receive them to / from Domoticz, right?

I looked thru settings of Domoticz and found no way to specify an address of Mosquito or any other MQTT broker to subscribe
Have you created a MQTT Client Gateway with LAN Interface device in Setup/Hardware?

If you haven't, please take a look available settings of the MQTT Client Gateway with LAN Interface device. According to my current understanding it's not possible
to subscribe to any other topic than "domoticz/in" but there are some options to modify publish topics though...

Nautilus
Posts: 639
Joined: Friday 02 October 2015 12:12
Target OS: Raspberry Pi
Domoticz version: beta
Location: Finland
Contact:

Re: [90] MQTT Support

Post by Nautilus » Tuesday 15 December 2015 0:03

Hi,

In my Raspberry Pi setup there has been some inconsistency with the MQTT hardware in Domoticz (not receiving anything although everything should be in order with both Domo & Mosquitto / Node-red). Therefore I set the "Data Timeout" to 30 minutes in hopes of the connection would reset itself if no data received through this hardware for 30 minutes. The result, however, is that it resets the connection every 30 minutes like there's no data received ever (it is receiving messages every 5 minutes in domoticz/in and publishing as well to domoticz/out). Is this because I don't have any devices created where MQTT would be set as the hardware? I just have devices that are updated through messages coming through this channel...

Also, the inconsistency with this channel continues, e.g. this morning there was in the log:

Code: Select all

2015-12-14 06:42:31.028  Error: Restarting: MQTT
and after this one update and then two hours of "down-time" (until restart of Domoticz). So it did not even try to restart it after that anymore even though now clearly there was a timeout in receiving data through this channel for more than 30 minutes. Have others had problems with this and any ideas what I could do to monitor that MQTT channel is working?

shkaff
Posts: 24
Joined: Friday 11 December 2015 11:29
Target OS: Raspberry Pi
Domoticz version: Stable
Contact:

Re: [90] MQTT Support

Post by shkaff » Tuesday 15 December 2015 0:26

Kouseri wrote:...MQTT Client Gateway with LAN Interface device...
Aha! That's the way to go! Thanks, Kouseri!
I would agree with the topic starter of How to use "MQTT Client Gateway with LAN interface" that the description of MQTT at WIKI isn't clear.
Will be digging further.

Nautilus
Posts: 639
Joined: Friday 02 October 2015 12:12
Target OS: Raspberry Pi
Domoticz version: beta
Location: Finland
Contact:

Re: [90] MQTT Support

Post by Nautilus » Wednesday 16 December 2015 10:09

Nautilus wrote:

Code: Select all

2015-12-14 06:42:31.028  Error: Restarting: MQTT
and after this one update and then two hours of "down-time" (until restart of Domoticz).
Same thing as above happening again, except start was at around 3:30AM this time and restart happened at 9:30. I noticed now that after this also Wunderground and Weather IO stop updating until the next restart (other services still update). Also, the restart seems to happen at the time of logging in to Domoticz (which I believe is because the service freezes - well at least the page keeps loading and loading - and then Monit kicks in).

So maybe it is MQTT, maybe Weather IO or Wunderground or some combination of these that is causing the issue. I've now disabled the weather hardware, let's see if this improves the situation :)

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by Toni » Friday 18 December 2015 9:15

My MQTT-based sensors have also randomly stopped working in the latest stable version (I think it was V2.3530). Before this I have been running on MQTT for months without problems.

Yesterday, in hope of new fixes, I switched to the beta channel and updated to V2.3852, but the same problem continues. For example, I restarted Domoticz yesterday, and this morning all sensors had stopped updating at around 7AM (EET, it's now 9AM). The 'domoticz/in' topic does get messages, I verified this with 'mosquitto_sub -t domoticz/in', and the sensors will again start working when I restarted Domoticz.

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by Toni » Friday 18 December 2015 13:25

I made a Lua script to watch when my outdoor sensor has been last updated, and keep track of the number of Lua script executions with a user variable. If this is not updated frequently enough (should be updated every minute), the script will restart Domoticz. It also increments another user variable, so I have an idea on how frequently this happens.

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by Toni » Saturday 19 December 2015 16:46

11 restarts due to MQTT stopped responding, since yesterday. It's definately broken... Any ideas on where to start looking for the root cause?

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by Toni » Monday 18 January 2016 8:57

Anybody? I have updated to the latest beta a number of times, and I still face this problem daily. Looking at 'mosquitto_sub' output, I can see that MQTT messages are coming as they should, and there doesn't seem to be any issues with anything like corrupted data etc. Still Domoticz stops responding to the MQTT messages, and only a restart of Domoticz helps.

wildfire
Posts: 29
Joined: Friday 18 September 2015 11:04
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by wildfire » Monday 18 January 2016 12:50

Tony,
i also had problems with input from mqtt and moved away from updating sensors using mqtt, it keeps crashing domoticz.
i use node-red to get the mqtt input and then post it using HTTP requests.
I have 10 sensors up and running, posting every 10 seconds, i do notice domoticz slows down of all the http posts, i realy wish mqtt would get fixed.
if you need help using node-red, ill help if i can.

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by Toni » Monday 18 January 2016 13:24

I can actually switch into the HTTP JSON api fairly easily, I'd just like to use MQTT interface instead :( Bummer...

There's absolutely nothing in the Domoticz log, so I don't even know where to start looking... Perhaps I should start by writing some small breakpoint script with gdb. First I would just need to learn gdb. It almost sounds like the MQTT handling thread would exit or something. Mosquitto and lsof both show that Domoticz is still connected to the tcp socket, it just doesn't seem to process anything.

wildfire
Posts: 29
Joined: Friday 18 September 2015 11:04
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by wildfire » Monday 18 January 2016 13:43

yup, thats what i get to, and there are no logs or noting to find what happens

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [90] MQTT Support

Post by Toni » Monday 18 January 2016 13:52

I would be happy to produce logs etc. if somebody could help me in getting started. This happens several times a day on my Raspberry. Currently I'm just logging the domoticz/in topic into a file (including timestamps), as well as the domoticz.log. I think a gdb breakpoint script would be useful, I'd just need to find time (or preferably get help from somebody) to get started with in. I.e. what function calls to break, and what to run in the breakpoint script etc.

kylegordon
Posts: 96
Joined: Friday 01 November 2013 0:58
Target OS: Linux
Domoticz version: Trunk
Contact:

Re: [90] MQTT Support

Post by kylegordon » Tuesday 19 January 2016 17:54

Ok so this is the same behaviour that I've been living with for months. I thought it was just my configuration, but apparently not. I'll have another search around to see what's going on. I'm tempted to try starting domoticz under strace, and logging everything...

Sent from my A0001 using Tapatalk
1 x HP DL380 & KVM
1 x RPI
1 x RFXtrx433 V78
11 x LWRF Switches
1 x LIFX
2 x Echo Dots
6 x Byron PIRs
2 x Nexa PIRs
2 x Kodi
2 x ESP8266 MQTT
1 x GPIO/MQTT PIR
1 x GPIO(PWM)/MQTT LEDs
1 x GPIO(SPI)/MQTT LPD6803 LEDs
Lots of Node-Red glue

kylegordon
Posts: 96
Joined: Friday 01 November 2013 0:58
Target OS: Linux
Domoticz version: Trunk
Contact:

Re: [90] MQTT Support

Post by kylegordon » Tuesday 19 January 2016 20:54

Came home to a dark house with the heating on. This morning I woke up to cold dark house... The past few iterations of the master branch have been getting worse and worse, but I put it down to something in my configuration. I've been kill -9'ing it it at least twice a day this year.

Anyway, it's now stracing to a log file. It's not quite gdb, but it might show something.
1 x HP DL380 & KVM
1 x RPI
1 x RFXtrx433 V78
11 x LWRF Switches
1 x LIFX
2 x Echo Dots
6 x Byron PIRs
2 x Nexa PIRs
2 x Kodi
2 x ESP8266 MQTT
1 x GPIO/MQTT PIR
1 x GPIO(PWM)/MQTT LEDs
1 x GPIO(SPI)/MQTT LPD6803 LEDs
Lots of Node-Red glue

User avatar
nayr
Posts: 430
Joined: Tuesday 11 November 2014 19:42
Target OS: Linux
Domoticz version: github
Location: Denver, CO - USA
Contact:

Re: [90] MQTT Support

Post by nayr » Tuesday 19 January 2016 22:43

Might go create an issue on github so the devs notice this issue, with all these reports of crashes its unlikely to be caused by user error.

There is enough people here indicating a severe bug in the code, so I think creating an issue report at this point is justified and unlikely to annoy any of the devs.

Sounds like its deadlocking the MQTT interface, any traces you can get would be a big help.. might force a core when its not responding and run a gdb backtrace on it and report the output.

Add:

Code: Select all

ulimit -c unlimited
to your init script so Domoticz dont have any limit on core sizes, then kill -3 to force a core on the proccess after MQTT is hung, finally run a backtrace

Code: Select all

gdb ~/domoticz/domoticz corefile
thread apply all bt full
It'll give alot of output, but with any luck we'll see what function the MQTT thread is hung on.
Debian Jessie: CuBox-i4 (Primary) w/Static Routed IP and x509 / BeagleBone with OpenSprinkler / BeagleBone Planted Aquarium / 3x Raspbery Pi2b GPIO Slaves
Elemental Theme - node-domoticz-mqtt - Home Theatre Controller - AndroidTV Simple OSD Remote - x509 TLS Auth

kylegordon
Posts: 96
Joined: Friday 01 November 2013 0:58
Target OS: Linux
Domoticz version: Trunk
Contact:

Re: [90] MQTT Support

Post by kylegordon » Tuesday 19 January 2016 23:20

Cool, I'll do that too. I'll look for, or open, an issue when I have some more output from strace. Will also dump it using gdb.

At this rate... 12 hours and I should have something :-)
1 x HP DL380 & KVM
1 x RPI
1 x RFXtrx433 V78
11 x LWRF Switches
1 x LIFX
2 x Echo Dots
6 x Byron PIRs
2 x Nexa PIRs
2 x Kodi
2 x ESP8266 MQTT
1 x GPIO/MQTT PIR
1 x GPIO(PWM)/MQTT LEDs
1 x GPIO(SPI)/MQTT LPD6803 LEDs
Lots of Node-Red glue

User avatar
nayr
Posts: 430
Joined: Tuesday 11 November 2014 19:42
Target OS: Linux
Domoticz version: github
Location: Denver, CO - USA
Contact:

Re: [90] MQTT Support

Post by nayr » Saturday 23 January 2016 8:51

added a small example script to the Wiki to help people get started: https://www.domoticz.com/wiki/MQTT#Publish
It has functions for all basic publishing needs, nothing yet for parsing subscriptions.

I am thinking of writing a node-domoticz-mqtt module, that exports functions for watching incoming json for specific devices and firing off calls, and other functions for easy device control.

It shouldn't be all that difficult, put it on NPM and I think MQTT will become alot more friendly of an interface.. I just havent written anything for combing through /domoticz/out yet, but I need to for projects I am working on, anyone have anything already?
Last edited by nayr on Sunday 24 January 2016 5:41, edited 1 time in total.
Debian Jessie: CuBox-i4 (Primary) w/Static Routed IP and x509 / BeagleBone with OpenSprinkler / BeagleBone Planted Aquarium / 3x Raspbery Pi2b GPIO Slaves
Elemental Theme - node-domoticz-mqtt - Home Theatre Controller - AndroidTV Simple OSD Remote - x509 TLS Auth

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests