Python Plugin - SupressTriggers = True doesn't update Database !

Python and python framework
pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Python Plugin - SupressTriggers = True doesn't update Database !

Post by pipiche » Friday 28 September 2018 15:15

The Options fields seems to not be updated in the DB.

I'm using Options to store a number of plugin related information. At creation there is not issue. But when I want to update it, there is no update of the Options field !

Here is what options looks like

Code: Select all

Options = {'ClusterType': 'Plug', 'Zigate': {'Ep': {'01': {'0000': {}, '0006': '00', '000c': {}, 'Type': 'Plug'}, '02': {'000c': '0.0', 'Type': 'Power/Meter'}}, 'Status': 'inDB', 'Heartbeat': '16', 'RIA': '7', 'RSSI': 51, 'Battery': {}, 'Model': 'lumi.plug', 'MacCapa': '8e', 'IEEE': '00158d00024d873d', 'Type': 'Plug/PowerMeter', 'ProfileID': {}, 'ZDeviceID': {}, 'DomoID': '1cb6', 'SQN': {}}}
And here after is how the device is created and how it is updated

Code: Select all

Domoticz.Device(DeviceID=str(DeviceID),Name=str(t) + "-" + str(DeviceID) + "-" + str(Ep), Unit=FreeUnit(self, Devices), TypeName="kWh", Options={"Zigate":str(self.ListOfDevices[DeviceID]), "ClusterType":t}).Create()

Code: Select all

Devices[Unit].Update(nValue=int(nValue), sValue=str(sValue), Options=str(Options), SignalLevel=int(rssi), BatteryLevel=int(BatteryLvl) , Color = Color_)
Last edited by pipiche on Saturday 27 October 2018 15:55, edited 1 time in total.

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Friday 28 September 2018 15:16

I'm really stuck. I tried to add some debug in hardware/plugin/PythonObjects.cpp but without success

I just been able to identify the test : if (pOptionsDict && PyDict_Check(pOptionsDict)) in line 947 of hardware/plugin/PythonObjects.cpp is false. So either my object is not a Python Dict, which I would be surprise as you can see here after!

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Friday 28 September 2018 17:48

I have been able to put SetDeviceOptions in debug and here are the log that I'm getting:

Code: Select all

Sep 28 17:34:32 pi3 domoticz[3028]: 2018-09-28 17:34:32.654 (Zigate) UpdateDevice_v2 Option before Update = {'ClusterType': 'LvlControl', 'Zigate': {'ZDeviceID': '0200', 'Heartbeat': 0, 'Type': {}, 'IEEE': '20918a0000038f65', 'Ep': {'01': {'0015': {}, '0008': {}, '0100': {}, '0004': {}, '0005': {}, '0006': {}, 'fc20': {}, '0000': {}, '000a': {}, 'fc21': {}, '0003': {}}}, 'Model': 'shutter.Profalux', 'Battery': {}, 'MacCapa': '8e', 'Status': 'inDB', 'RIA': '2', 'ProfileID': '0104', 'DomoID': 'e86c', 'RSSI': 66, 'SQN': {}}}
Here is Options before call Update

Code: Select all

Sep 28 17:43:44 pi3 domoticz[3028]: 2018-09-28 17:43:44.390 (Zigate) UpdateDevice_v2 Option before Update = {'ClusterType': 'LvlControl', 'Zigate': {'ZDeviceID': '0200', 'Heartbeat': 0, 'Type': {}, 'IEEE': '20918a0000038f65', 'Ep': {'01': {'0015': {}, '0008': {}, '0100': {}, '0004': {}, '0005': {}, '0006': {}, 'fc20': {}, '0000': {}, '000a': {}, 'fc21': {}, '0003': {}}}, 'Model': 'shutter.Profalux', 'Battery': {}, 'MacCapa': '8e', 'Status': 'inDB', 'RIA': '0', 'ProfileID': '0104', 'DomoID': 'e86c', 'RSSI': 57}}

Code: Select all

Devices[Unit].Update(nValue=int(nValue), sValue=str(sValue), Options=Options)
Here after we see reaching SetDeviceOptions in SQLHelper

Code: Select all

Sep 28 17:43:44 pi3 domoticz[3028]: 2018-09-28 17:43:44.416  Status: DEBUG : reaching SetDeviceOptions
Sep 28 17:43:44 pi3 domoticz[3028]: 2018-09-28 17:43:44.416  Status: DEBUG : setting options 'ClusterType:THZsQ29udHJvbA==;Zigate:eydaRGV2aWNlSUQnOiAnMDIwMCcsICdIZWFydGJlYXQnOiAnMScsICdUeXBlJzoge30sICdJRUVFJzogJzIwOTE4YTAwMDAwMzhmNjUnLCAnRXAnOiB7JzAxJzogeycwMDE1Jzoge30sICcwMDA4Jzoge30sICcwMTAwJzoge30sICcwMDA0Jzoge30sICcwMDA1Jzoge30sICcwMDA2Jzoge30sICdmYzIwJzoge30sICcwMDAwJzoge30sICcwMDBhJzoge30sICdmYzIxJzoge30sICcwMDAzJzoge319fSwgJ01vZGVsJzogJ3NodXR0ZXIuUHJvZmFsdXgnLCAnQmF0dGVyeSc6IHt9LCAnTWFjQ2FwYSc6ICc4ZScsICdTdGF0dXMnOiAnaW5EQicsICdSSUEnOiAnMCcsICdQcm9maWxlSUQnOiAnMDEwNCd9' on device 13
Here after is Option after the Update. It is not matching what was in Options before

Code: Select all

Sep 28 17:43:44 pi3 domoticz[3028]: 2018-09-28 17:43:44.418 (Zigate) UpdateDevice_v2 Option after Update = {'ClusterType': 'LvlControl', 'Zigate': "{'ZDeviceID': '0200', 'Heartbeat': '1', 'Type': {}, 'IEEE': '20918a0000038f65', 'Ep': {'01': {'0015': {}, '0008': {}, '0100': {}, '0004': {}, '0005': {}, '0006': {}, 'fc20': {}, '0000': {}, '000a': {}, 'fc21': {}, '0003': {}}}, 'Model': 'shutter.Profalux', 'Battery': {}, 'MacCapa': '8e', 'Status': 'inDB', 'RIA': '0', 'ProfileID': '0104'}"}

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Friday 28 September 2018 22:28

Here attached is a plugin to reproduce the probleme !!!!

Here is a short plugin.py to reproduce the problem.

A device is created in the onStart (if not yet created).

Then in onHearbeat, we display the
(1) Display Devices[1].Options Look at Heartbeat -- It is 0
(2) Display Options after increment of hearbeat

In the next cyce, you'll see that Hearbeat is not incremented !!!!!!!!!!!!
plugin.py.zip
(1.59 KiB) Downloaded 9 times

User avatar
Dnpwwo
Posts: 767
Joined: Sunday 23 March 2014 10:00
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Python Plugin - Options not updated in DB

Post by Dnpwwo » Saturday 29 September 2018 15:04

@pipiche,

Your test plugin crashed Domoticz on Windows for me rather than failing to update, I will have a look into it.

I suspect your problem is because during the create you correctly pass in a dictionary with string leys and string values but in the update you pass in a nested dictionary rather than a string.

It seemed to work if I changed line 78 to be:

Code: Select all

Options['Zigate']=str({'Ep': {'01': {'0000': {}, '0003': {}, '0004': {}, '0005': {}, '0006': {}, '0008': {}, '0300': {}, '0b05': {}, '1000': {}}}, 'Status': 'inDB', 'Heartbeat': str(self.hb), 'RIA': '0', 'RSSI': {}, 'Battery': {}, 'Model': 'Ampoule.LED1624G9.Tradfri', 'MacCapa': '8e', 'IEEE': '90fd9ffffee77f7e', 'Type': {}, 'ProfileID': 'c05e', 'ZDeviceID': '0200', 'DomoID': '3145', 'SQN': {}})
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Saturday 29 September 2018 15:21

@Dnpwwo, indeed it works. So I need to jump in the real case to understand the issue.

Might be great that the Domoticz code would complain instead of doing nothing, no ?

Thanks

User avatar
Dnpwwo
Posts: 767
Joined: Sunday 23 March 2014 10:00
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Python Plugin - Options not updated in DB

Post by Dnpwwo » Sunday 30 September 2018 9:11

@pipiche,

I pushed a fix today that will effectively force a 'str()' convertion inside the Device.Update so your old code will now work.

By the way, the SuppressTriggers parameter takes a Boolean value (True/False) not a string. Any string other than an empty one will always evaluate to True (see https://documentation.help/Python-3.4/s ... html#truth)
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Sunday 30 September 2018 10:00

@Dnpwwo, BTW as I think you master this part, is there any specific reason why DeviceID cannot be changed ?

I tried to update the code to accommodate it, but I'm getting an error message "CDevice_update' failed 'SystemError':'More keyword list entries (17) than format specifiers (16)'" that I cannot identify where it is coming from.

Basically trying to implement in CDevice_update() the same way as for Name update.


PS/ this is in the context of a plugin where we have to upgrade all of the DeviceID,

[EDIT]
I found it ;-)

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Tuesday 09 October 2018 11:45

@Dnpwwo , just to confirm your recent fix. It works very well.

Unfortunately, you might have seen my recent post viewtopic.php?f=65&t=25187 ; where I understood from Gizmocuz that Options is a column for the only purpose of Domoticz and must not be used by any plugin.

User avatar
Dnpwwo
Posts: 767
Joined: Sunday 23 March 2014 10:00
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Python Plugin - Options not updated in DB

Post by Dnpwwo » Tuesday 09 October 2018 14:20

@pipiche,

The Options column tends to be used in certain ways by native Domoticz code, such as selector switches. The behanviour you have described where additional Options get lost would apply to Selctor switches created by any hardware and is not a limitation of the plugin framework.

The question is: What are you trying to achieve by attempting to store additional data in the Options Column for a device?

If you want to store additional information you can write it to a file local to the plugin using native Python file handling.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Tuesday 09 October 2018 14:43

@Dnpwwo
If you want to store additional information you can write it to a file local to the plugin using native Python file handling
.

You right, this is indeed what we are going to do. We were - and wrongly - under the impression to be able to store extra information in Options to accommodate the Plugin needs to store informations.
The question is: What are you trying to achieve by attempting to store additional data in the Options Column for a device?
I do consider , that distributing information across several locations increase the risk of inconsistency and troubles. And we saw suhc benefits to use this Options field with a centrally managed set of information

More wider and outside of my current problem, I feel that it would be good to have the Plugin able to access Domoticz database as scuh as Creating 1 extra table and corresponding address, with the possibility to Create, Update and Delete rows in it.
( this is more-less what you can do with a Native plugin), and this becomes a limitation from the Python framework.

Taking in consideration, that I see Domoticz becoming more and more complex due to the large number of Native Plugin, I see a lot of benefits of the Python Framework as it allows a full plugin development in a relative light coupling with Domoticz.

This I would say a very nice feature ... Don't you agree ?

StasDemydiuk
Posts: 44
Joined: Saturday 14 January 2017 22:37
Target OS: Linux
Domoticz version: Beta
Location: Ukraine
Contact:

Re: Python Plugin - Options not updated in DB

Post by StasDemydiuk » Wednesday 10 October 2018 9:08

I also thought that we can use Options field to store additional information for device from plugin.
Currently it doesn't look like an option because Domoticz itself use this field and overwrites plugin custom keys.

@Dnpwwo Because of that maybe we could think about adding such ability to python plugins framework to store additional information inside the device somehow, as it could be extremely useful for some plugins (mine as example viewtopic.php?f=65&t=24975 :roll: ) instead of reinventing the wheel in each plugin individually.

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Wednesday 10 October 2018 10:46

@StasDemydiuk, looks like we are working on similar topic but from different angle .

As stated earlier, I really like what the Python Framework is bringing , enhancing it would make it even more valuable and would help us to reach more complex plugin and I thing what @StasDemydiuk and I are working fall into this.

User avatar
Dnpwwo
Posts: 767
Joined: Sunday 23 March 2014 10:00
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Python Plugin - Options not updated in DB

Post by Dnpwwo » Monday 15 October 2018 1:32

@pipiche and @StasDemydiuk,

I'm not against adding extra functionality to the framework, I'm just trying to better understand what you are trying to achieve to inform the design.

One of the goals of the framework was to make plugins work 100% the same was as native hardware implementations. Currently there is a clear separation within Domoticz between the code that interfaces to hardware and the code that handles devices which is why the code supporting Selector Switches believes it has the right to update the DeviceStatus.Options field. If we are considering adding a per device storage capability it should not be 'plugin only' but rather it should be consumable by the broader application.

I'm currently looking at a Z-Wave plugin which may have a similar requirement because Z-Wave devices declare their capabilities at run time so remembering them from previous runs would make start up much faster.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Monday 15 October 2018 8:16

Dnpwwo wrote:
Monday 15 October 2018 1:32
@pipiche and @StasDemydiuk,

I'm not against adding extra functionality to the framework, I'm just trying to better understand what you are trying to achieve to inform the design.

One of the goals of the framework was to make plugins work 100% the same was as native hardware implementations. Currently there is a clear separation within Domoticz between the code that interfaces to hardware and the code that handles devices which is why the code supporting Selector Switches believes it has the right to update the DeviceStatus.Options field. If we are considering adding a per device storage capability it should not be 'plugin only' but rather it should be consumable by the broader application.

I'm currently looking at a Z-Wave plugin which may have a similar requirement because Z-Wave devices declare their capabilities at run time so remembering them from previous runs would make start up much faster.
I'm fully with you that it was a mistake willing to use Device.Options, so no issue on that.

As you stated
One of the goals of the framework was to make plugins work 100% the same was as native hardware implementations
, and The Framework should allow the creation of one Table specific to the plugin itself. This is what a number of Native plugin are doing. EnoceanSensors, ZWaveNodes for examples.

In such you will give all freedom in the Plugin to create its own storage capability.

This for me would be the right enhacement

- Creation of 1 Plugin Table
- Insrert/Update/Delete/Query rows in that table
- Remove of the Plugin Table
- Update of Plugin Table

Logread
Posts: 209
Joined: Sunday 28 August 2016 7:48
Target OS: Raspberry Pi
Domoticz version:
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by Logread » Monday 15 October 2018 19:41

pipiche wrote:
Monday 15 October 2018 8:16
The Framework should allow the creation of one Table specific to the plugin itself. This is what a number of Native plugin are doing. EnoceanSensors, ZWaveNodes for examples.

In such you will give all freedom in the Plugin to create its own storage capability.

This for me would be the right enhacement

- Creation of 1 Plugin Table
- Insrert/Update/Delete/Query rows in that table
- Remove of the Plugin Table
- Update of Plugin Table
I like this idea very much... for now I have had to use a hack for plugins I wrote that need their own persistent data storage capability, like the SmartVirtualThermostat... I store/read plugin data to/from the uservariables table via the json interface... not very clean but at least the data is backed up together with the entire database (which is not the case if you use a local file as storage). Having a dedicated table for plugins data is a great idea.

User avatar
Dnpwwo
Posts: 767
Joined: Sunday 23 March 2014 10:00
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Python Plugin - Options not updated in DB

Post by Dnpwwo » Thursday 18 October 2018 13:40

I always assumed that if people really wanted to get to get to the database or wanted to create their own tables and the like they would do so using:

Code: Select all

import sqlite3

conn = sqlite3.connect('..\..\Domoticz.db')

or something similar as documented on the Python website https://docs.python.org/3.7/library/sqlite3.html. The library ships with all Python distributions apparently. YouTube tutorials available as well (such as https://www.youtube.com/watch?v=SQj17D1Q_6s)

Adding general database functionality to the framework seems like a lot of effort and there does not seem to be a real consensus as to a specific solution.

To be clear though: Apart from testing the 'import' statement worked I've never looked at or tested this so use on a development database until you are certain it works for you.
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Thursday 25 October 2018 20:54

Dnpwwo wrote:
Sunday 30 September 2018 9:11
@pipiche,

I pushed a fix today that will effectively force a 'str()' convertion inside the Device.Update so your old code will now work.

By the way, the SuppressTriggers parameter takes a Boolean value (True/False) not a string. Any string other than an empty one will always evaluate to True (see https://documentation.help/Python-3.4/s ... html#truth)
@Dnpwwo, could you kindly clarify the expected behaviour of SuppressTriggers when True ? As this is not matching my expectations.

For instance when SuppressTriggers is set to True, indeed there is no notification done , but there is NO UPDATE at all done to the Database !

User avatar
Dnpwwo
Posts: 767
Joined: Sunday 23 March 2014 10:00
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Melbourne, Australia
Contact:

Re: Python Plugin - Options not updated in DB

Post by Dnpwwo » Saturday 27 October 2018 2:07

The SuppressTriggers was added by someone else and I’ve not used it myself.

I looked at the code the other day when I commented on the parameter type and it didn’t look right but I didn’t change anything because nobody on the forum had a problem with it as it is.

If it really isn’t working it would be an easy fix
The reasonable man adapts himself to the world; the unreasonable one persists to adapt the world to himself. Therefore all progress depends on the unreasonable man. George Bernard Shaw

pipiche
Posts: 156
Joined: Monday 02 April 2018 20:33
Target OS: Raspberry Pi
Domoticz version: beta
Location: France
Contact:

Re: Python Plugin - Options not updated in DB

Post by pipiche » Saturday 27 October 2018 9:54

Dnpwwo wrote:
Saturday 27 October 2018 2:07
The SuppressTriggers was added by someone else and I’ve not used it myself.

I looked at the code the other day when I commented on the parameter type and it didn’t look right but I didn’t change anything because nobody on the forum had a problem with it as it is.

If it really isn’t working it would be an easy fix
I'm using for instance when there is not update for sValue/nValue but we want to update SignalQuality, so I do confirm that it doesn't work as expected as if we put SuppressTriggers to True then there is no write to the Database and this is how it looks like in the code.

However I'm not good enought to fix it in the code, so your help would be fantastic

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests