Plugin system gets stuck on a specific HTTP reply

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post

Please mark your topic as Solved when the problem is solved.
Post Reply
galinette
Posts: 19
Joined: Monday 11 December 2017 23:57
Target OS: Raspberry Pi
Domoticz version:
Contact:

Plugin system gets stuck on a specific HTTP reply

Post by galinette » Saturday 16 December 2017 15:07

I am trying to implement an electrical counter device with a HTTP interface. The counter has an ethernet connection and a webserver which can provide:
A nice webpage on current and past consumptions on the base URL, or JSON data on a specific URL.

Currently I'm just trying to get the JSON reply. I have a stub plugin based on the google example from the wiki, which works getting a reply from any website or webserver. I properly get the "onMessage" event and the reply data.

But, if I just change the host and URL to my electrical counter device, it seems the plugin system crashes or gets stuck (all my devices disappear and I need to restart Domoticz to get a proper behavior)

The HTTP request is properly sent to the webserver and the server replies, but it seems the plugin system gets in trouble when trying to process that reply, before onMessage is called.

SweetPants
Posts: 2255
Joined: Friday 12 July 2013 21:24
Target OS: Linux
Domoticz version: V4.10007
Location: The Netherlands
Contact:

Re: Plugin system gets stuck on a specific HTTP reply

Post by SweetPants » Saturday 16 December 2017 15:10

Can you add some more information? (like what is in the red forum rules) and maybe the script also. It's a wild guess now.
Ubuntu 16.04.3 LTS, Intel NUC, MySensors 2.3.0-alpha, Pro Mini 3.3/8M RFM69(HW), ESP8266 (SDK2.2.0), Sonoff, RFLink, RFXcom, MQTT, NodeRed, Domoticz Beta (from source)

galinette
Posts: 19
Joined: Monday 11 December 2017 23:57
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Plugin system gets stuck on a specific HTTP reply

Post by galinette » Saturday 16 December 2017 16:24

Sorry, here is it:

Version: 3.8790
Platform: Windows (also happening on RasPi)
Plugin/Hardware: Legrand Eco Compteur 412000
Description: The HTTP request API of the plugin system seems to crash when receiving the HTTP response from this specific device. If the host/url is set on anything else (website, other device...) it works



Log from Domotics:

Code: Select all

 2017-12-16 15:16:32.410 (Legrand) Initialized version 1.1.0, author 'Galinette'
2017-12-16 15:16:32.410 (Legrand) Entering work loop.
2017-12-16 15:16:32.566 PluginSystem: Restarting I/O service thread.
2017-12-16 15:16:32.566 Error: PluginSystem: Exception processing message.
2017-12-16 15:16:32.566 (Legrand) onDisconnect called for connection to: 192.168.0.10:80
2017-12-16 15:17:34.671 PluginSystem: Restarting I/O service thread.
2017-12-16 15:17:34.671 Error: PluginSystem: Exception processing message.
2017-12-16 15:17:34.671 Error: PluginSystem: Exception processing message.
2017-12-16 15:17:34.671 (Legrand) onDisconnect called for connection to: 192.168.0.10:80
2017-12-16 15:18:37.368 PluginSystem: Restarting I/O service thread.
2017-12-16 15:18:37.368 Error: PluginSystem: Exception processing message.
2017-12-16 15:18:37.368 Error: PluginSystem: Exception processing message.
2017-12-16 15:18:37.368 (Legrand) onDisconnect called for connection to: 192.168.0.10:80 



Code: Select all

"""
<plugin key="Legrand_412000" name="Legrand Eco-Compteur" author="Galinette" version="1.1.0" externallink="https://www.legrand.fr/sites/default/files/le_manuel_utilisateur_ecocompteur.pdf">
    <params>
        <param field="Address" label="IP Address" width="200px" required="true" default="192.168.0.10"/>
        <param field="Port" label="Port" width="30px" required="true" default="80"/>
        <param field="Mode6" label="Debug" width="75px">
            <options>
                <option label="True" value="Debug"/>
                <option label="False" value="Normal"  default="true" />
                <option label="Logging" value="File"/>
            </options>
        </param>
    </params>
</plugin>
"""
import Domoticz
import json

class BasePlugin:
    httpConn = None
    runAgain = 6
   
    def __init__(self):
        return

    def onStart(self):
        if Parameters["Mode6"] == "Debug":
            Domoticz.Debugging(1)
        DumpConfigToLog()
        self.httpConn = Domoticz.Connection(Name="HTTP Test", Transport="TCP/IP", Protocol="HTTP", Address=Parameters["Address"], Port=Parameters["Port"])
        self.httpConn.Connect()

    def onStop(self):
        Domoticz.Log("onStop - Plugin is stopping.")

    def onConnect(self, Connection, Status, Description):
        if (Status == 0):
            Domoticz.Debug("Connected successfully.")
            headers = { 'Content-Type': 'text/xml; charset=utf-8', \
                                       'Connection': 'keep-alive', \
                                       'Accept': 'Content-Type: text/html; charset=UTF-8', \
                                       'Host': Parameters["Address"]+":"+Parameters["Port"], \
                                       'User-Agent':'Domoticz/1.0' }
            url = "/inst.json"
            Connection.Send({"Verb":"GET", "URL":url, "Headers": headers})
        else:
            Domoticz.Log("Failed to connect ("+str(Status)+") to: "+Parameters["Address"]+":"+Parameters["Port"]+" with error: "+Description)

    def onMessage(self, Connection, Data):
        Domoticz.Debug("onMessage called")
        
        # Status = int(Data["Status"])
        # if (Status == 200):
            # Domoticz.Log("Good Response")
            # compteurs = json.loads(Data["Data"].decode("utf-8", "ignore"))
            # c1 = float(compteurs["Data1"])
            # c2 = float(compteurs["Data2"])
            # c3 = float(compteurs["Data3"])
            # c4 = float(compteurs["Data4"])
            # c5 = float(compteurs["Data5"])
            # total = c1+c2+c3+c4+c5
            # Domoticz.Log(str(total))
            # self.httpConn.Disconnect()
        # elif (Status == 302):
            # Domoticz.Log("Server returned a Page Moved Error.")
        # elif (Status == 400):
            # Domoticz.Error("Server returned a Bad Request Error.")
        # elif (Status == 500):
            # Domoticz.Error("Server returned a Server Error.")
        # else:
            # Domoticz.Error("Server returned a status: "+str(Status))

    def onCommand(self, Unit, Command, Level, Hue):
        Domoticz.Debug("onCommand called for Unit " + str(Unit) + ": Parameter '" + str(Command) + "', Level: " + str(Level))

    def onDisconnect(self, Connection):
        Domoticz.Log("onDisconnect called for connection to: "+Connection.Address+":"+Connection.Port)

    def onHeartbeat(self):
        Domoticz.Debug("onHeartbeat called")
        if (self.httpConn.Connecting() or self.httpConn.Connected()):
            Domoticz.Debug("onHeartbeat called, Connection is alive.")
        else:
            self.runAgain = self.runAgain - 1
            if self.runAgain <= 0:
                self.httpConn.Connect()
                self.runAgain = 6
            else:
                Domoticz.Debug("onHeartbeat called, run again in "+str(self.runAgain)+" heartbeats.")

global _plugin
_plugin = BasePlugin()

def onStart():
    global _plugin
    _plugin.onStart()

def onStop():
    global _plugin
    _plugin.onStop()

def onConnect(Connection, Status, Description):
    global _plugin
    _plugin.onConnect(Connection, Status, Description)

def onMessage(Connection, Data):
    global _plugin
    _plugin.onMessage(Connection, Data)

def onCommand(Unit, Command, Level, Hue):
    global _plugin
    _plugin.onCommand(Unit, Command, Level, Hue)

def onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile):
    global _plugin
    _plugin.onNotification(Name, Subject, Text, Status, Priority, Sound, ImageFile)

def onDisconnect(Connection):
    global _plugin
    _plugin.onDisconnect(Connection)

def onHeartbeat():
    global _plugin
    _plugin.onHeartbeat()

# Generic helper functions
def LogMessage(Message):
    if Parameters["Mode6"] == "File":
        f = open(Parameters["HomeFolder"]+"http.html","w")
        f.write(Message)
        f.close()
        Domoticz.Log("File written")

def DumpConfigToLog():
    for x in Parameters:
        if Parameters[x] != "":
            Domoticz.Debug( "'" + x + "':'" + str(Parameters[x]) + "'")
    Domoticz.Debug("Device count: " + str(len(Devices)))
    for x in Devices:
        Domoticz.Debug("Device:           " + str(x) + " - " + str(Devices[x]))
        Domoticz.Debug("Device ID:       '" + str(Devices[x].ID) + "'")
        Domoticz.Debug("Device Name:     '" + Devices[x].Name + "'")
        Domoticz.Debug("Device nValue:    " + str(Devices[x].nValue))
        Domoticz.Debug("Device sValue:   '" + Devices[x].sValue + "'")
        Domoticz.Debug("Device LastLevel: " + str(Devices[x].LastLevel))
    return

def DumpHTTPResponseToLog(httpDict):
    if isinstance(httpDict, dict):
        Domoticz.Log("HTTP Details ("+str(len(httpDict))+"):")
        for x in httpDict:
            if isinstance(httpDict[x], dict):
                Domoticz.Log("--->'"+x+" ("+str(len(httpDict[x]))+"):")
                for y in httpDict[x]:
                    Domoticz.Log("------->'" + y + "':'" + str(httpDict[x][y]) + "'")
            else:
                Domoticz.Log("--->'" + x + "':'" + str(httpDict[x]) + "'")

SweetPants
Posts: 2255
Joined: Friday 12 July 2013 21:24
Target OS: Linux
Domoticz version: V4.10007
Location: The Netherlands
Contact:

Re: Plugin system gets stuck on a specific HTTP reply

Post by SweetPants » Saturday 16 December 2017 17:00

Do you know what is returned from the device? I do not have one so cannot test it, but as you said other sites/url/host works, so it seems something in the response triggering this. When looking into the code it fails (exception) on message processing
try
{
Message->Process();
}
catch(...)
{
_log.Log(LOG_ERROR, "PluginSystem: Exception processing message.");
}
Ubuntu 16.04.3 LTS, Intel NUC, MySensors 2.3.0-alpha, Pro Mini 3.3/8M RFM69(HW), ESP8266 (SDK2.2.0), Sonoff, RFLink, RFXcom, MQTT, NodeRed, Domoticz Beta (from source)

galinette
Posts: 19
Joined: Monday 11 December 2017 23:57
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Plugin system gets stuck on a specific HTTP reply

Post by galinette » Saturday 16 December 2017 22:53

The server replies a short JSON data. I do not know the exact full HTTP header, just the content.

I figured out by parsing the JSON data manually with python json module, that it throws an exception because the JSON is slightly uncompliant (there are non-zero integers with leading zeros).

So maybe the plugin system is seeing that the MIME type is JSON in HTTP header, and tries to parse it without catching the exception properly. If this is the case, it should catch a json parser error and fall back to raw data. But why would the plugin system try to parse "preventively" some JSON?

febalci
Posts: 328
Joined: Monday 03 July 2017 19:58
Target OS: NAS (Synology & others)
Domoticz version:
Contact:

Re: Plugin system gets stuck on a specific HTTP reply

Post by febalci » Sunday 17 December 2017 1:25

UPDATE:
Looking at your plugin file, i realized the header for Accept is only html, you should put 'application/json' instead to properly get json instead of html json: 'Accept': 'Content-Type: text/html; charset=UTF-8'

UPDATE 2:
Also you didn't used DumpHTTPResponseToLog(Data) in your onmessage. So the first reference Data["Status"] will most probably give error.

---------------------
Have you tried using JSON or None Protocol instead of HTTP?

You can construct the HTTP Get header on plugin side if you use 'None' as protocol:

Code: Select all

    requestHeader = "GET /index.html HTTP/1.1\r\n" \
                "Host: www.something.com\r\n" \
                "User-Agent: Domoticz/1.0\r\n" \
                "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n" \
                "Accept-Language: en-US,en;q=0.5\r\n" \
                "Accept-Encoding: gzip, deflate\r\n" \
                "Origin: http://www.something.com\r\n" \
                "DNT: 1\r\n" \
                "Connection: keep-alive\r\n" \
                "Pragma: no-cache\r\n" \
                "Cache-Control: no-cache\r\n" \
and

Code: Select all

self.wsConn = Domoticz.Connection(Name="TestConn", Transport="TCP/IP", Protocol="None", Address="www.something.com", Port="80")
self.wsConn.Connect()


Connection.Send(self.requestHeader)
Because if you use None protocol, the plugin doesn't care if the response is a formatted JSON or whatever. Then you can get the response and do whatever requred before processing it with json.dump

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests