How to use Data from a Dust Sensor

Others (MiLight, Hue, Toon etc...)
Post Reply
matzeb74
Posts: 49
Joined: Sunday 07 January 2018 20:23
Target OS: Raspberry Pi
Domoticz version: newest
Location: Stuttgart
Contact:

How to use Data from a Dust Sensor

Post by matzeb74 » Tuesday 13 February 2018 18:28

Hello,
I build my own Dustsensor (https://luftdaten.info). I now would like to use the Data in Domoticz. There are only Scripts for Fhem and HAss.
The Sensor also showa data in an Json File like this:

Code: Select all

{"software_version": "NRZ-2017-099", "age":"139", "sensordatavalues":[{"value_type":"SDS_P1","value":"7.63"},{"value_type":"SDS_P2","value":"4.23"},{"value_type":"temperature","value":"8.30"},{"value_type":"humidity","value":"27.30"},{"value_type":"samples","value":"617412"},{"value_type":"min_micro","value":"229"},{"value_type":"max_micro","value":"25617"},{"value_type":"signal","value":"-67"}]}
Also its possible to use an own api. Now my question is how to get the Data into domoticz?

Sensors used with an Nodemcu are DHT22 and SDS011.
Attachments
Luftdaten.JPG
Luftdaten.JPG (92.72 KiB) Viewed 1418 times

matzeb74
Posts: 49
Joined: Sunday 07 January 2018 20:23
Target OS: Raspberry Pi
Domoticz version: newest
Location: Stuttgart
Contact:

Re: How to use Data from a Dust Sensor

Post by matzeb74 » Wednesday 21 February 2018 22:14

Is here no one who Knows how to parse the Data to Domoticz?

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: How to use Data from a Dust Sensor

Post by Toulon7559 » Thursday 22 February 2018 16:11

@matzeb74

Struggling with the same question.

Perhaps not a direct answer to your question, but in this thread a solution was found to 'dissect' an XML-file. Conversion from json to xml can be rather easily be made by a python-script.
For the xml-files in that thread the values are conveniently enclosed between unique 'header'-markers and 'tail'-markers like <gauge power> and </gauge power>, and then you can perform a string-search to determine within the string the unique start- and stop-positions for the values, enabling to pinpoint the extraction of the values, even if variable length.

But such 'marker-grip' is missing in the json-file coming from the Luftdaten-firmware:
that json-file does not have unique markers, but just has plenty of couples consisting of "value_type" and "value".
The derived XML-file is not much better, with only couples consisting of <value_type>+</value_type> and <value>+</value>
Simply counting characters in the string is also not a solution, because the values can have variable number of characters.

To demonstrate what I mean, below you find a json-file from Luftdaten and the equivalent XML-file.

Code: Select all

{"age": "94", "sensordatavalues": [{"value_type": "SDS_P1", "value": "5.50"}, {"value_type": "SDS_P2", "value": "4.03"}, {"value_type": "temperature", "value": "19.80"}, {"value_type": "humidity", "value": "44.70"}, {"value_type": "samples", "value": "607696"}, {"value_type": "min_micro", "value": "234"}, {"value_type": "max_micro", "value": "966087"}, {"value_type": "signal", "value": "-66"}], "software_version": "NRZ-2017-099"

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?><root><age type="str">94</age><sensordatavalues type="list"><item type="dict"><value_type type="str">SDS_P1</value_type><value type="str">5.50</value></item><item type="dict"><value_type type="str">SDS_P2</value_type><value type="str">4.03</value></item><item type="dict"><value_type type="str">temperature</value_type><value type="str">19.80</value></item><item type="dict"><value_type type="str">humidity</value_type><value type="str">44.70</value></item><item type="dict"><value_type type="str">samples</value_type><value type="str">607696</value></item><item type="dict"><value_type type="str">min_micro</value_type><value type="str">234</value></item><item type="dict"><value_type type="str">max_micro</value_type><value type="str">966087</value></item><item type="dict"><value_type type="str">signal</value_type><value type="str">-66</value></item></sensordatavalues><software_version type="str">NRZ-2017-099</software_version></root>
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: How to use Data from a Dust Sensor

Post by Toulon7559 » Monday 26 February 2018 9:26

With hints from friends at Stackoverflow achieved the conversion from the basic file-layout from Luftdaten.info into a format with unique fieldnames.
Also the related script running, which then extracts the information and transfers towards Domoticz.

2-Step approach:

Step1 = Python-script running under cronjob, which
- calls the local measuring setup,
- changes the format of the json-file,
- makes a 2nd set of json-file + xml-file for upload, and
- uploads those 2 files to a server
[Reason for upload of json+xml: in that way the data is accessible for multiple applications also by other users]

Step2 = lua-script which
- reads the 2nd xml-file with the name given in step1 [therefore do no change filenames in Step1 unless you know what you do!],
- dissects that file and extracts temperature, humidity, PM2.5 and PM10, and
- uploads the extracted information to Domoticz
[upload of temperature+humidity is towards a virtual sensor which mimicks a DHT22, and towards 2 Custum sensors for PM2.5 respectively PM10]

Python-script for Step1

Code: Select all

#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2017 script compiled by Toulon7559 from various material from forums, version 00
# (c)2018 extended by extra step to extract info from json-file from Luftdaten.info
# --------------------------------------------------
# Line006 = PREPARATION & SETTING
# --------------------------------------------------
# Imports for script-operation
import json
import urllib
import dicttoxml
# SDS011 is dust-sensor at IP=192.168.0.141
# IP-address obviously to be tuned to YOUR configuration
# --------------------------------------------------
# Line015 Read data and compile json-file + xml-file
# --------------------------------------------------
page = urllib.urlopen('http://192.168.0.141/data.json')
content_test = page.read()
obj_test = json.loads(content_test)
print(obj_test)
with open('json_sds011_upload1.json', 'w') as outfile:
    json.dump(obj_test, outfile)
xml_test = dicttoxml.dicttoxml(obj_test)
print(xml_test)
xml_output = open("xml_sds011_upload1.xml",'w')
xml_output.write(xml_test)
xml_output.close()

# --------------------------------------------------
# Line030 = Extract Field-names and values from json
# --------------------------------------------------
import json
import pprint

with open('json_sds011_upload1.json') as j:
    data = json.load(j)
for sdv in data.pop('sensordatavalues'):
    data[sdv['value_type']] = sdv['value']
print(data)
pprint.pprint(data)

# --------------------------------------------------
# Line043 Compile 2nd set of json-/xml-file
# --------------------------------------------------
with open('json_sds011_upload2.json', 'w') as outfile:
    json.dump(data, outfile)
xml_test2 = dicttoxml.dicttoxml(data)
print(xml_test2)
xml_output = open("xml_sds011_upload2.xml",'w')
xml_output.write(xml_test2)
xml_output.close()

# --------------------------------------------------
# Line054 = FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
import ftplib
import os
# Definition of Upload_function
def upload(ftp, file):
    ext = os.path.splitext(file)[1]
    if ext in (".txt", ".htm", ".html"):
        ftp.storlines("STOR " + file, open(file))
    else:
        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Line068 = Actual FTP-Login & -Upload
# --------------------------------------------------
ftp = ftplib.FTP("<ftp-server>")
ftp.login("<ftp-username>", "<ftp-password>")

upload(ftp, "json_sds011_upload2.json")
upload(ftp, "xml_sds011_upload2.xml")
Lua-script for Step2

Code: Select all

------------------------------------------------------------------------------------------
-- Version 00 20180226
--
-- Domoticz lua script to convert XML output for SDS011-sensor plus related DHT22-sensor
-- Reads the status based on the unique web-address and passes values
-- to virtual sensor(s) in Domoticz
-- 
--------------------------------------------------------------------------------------------------
-- Original script by RATA1 at Domoticz-forum, adapted by Toulon7559 (c)2016-2018
--------------------------------------------------------------------------------------------------
-- Line 11: Start of script
commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file, set to false is not.
debug = true

-- Line 29: Define the idx of YOUR virtual sensor(s) in Domoticz
-- => what you have called your sensor(s) and the related IDX number(s)
PT_SDS011M = 1582 -- Virtual device for DHT22, temp + baro
PT_SDS011A = 1583 -- Virtual device for SDS PM2.5 [= SDS_P2 in file], custom sensor  
PT_SDS011B = 1584 -- Virtual device for SDS PM10  [= SDS_P1 in file], custom sensor

-- Line 35: Define your device IP-address: the xml-filename was set in Step1!
-- Lines 37 ~ 45 define where the XML-file can be read
Device_IP = "xxxx.nl" -- if you read from a remote website xxxx.nl
-- Device_IP = "192.168.1.x/home/pi" --if you read from your local Domoticz@Raspberry 
if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/xml_sds011_upload2.xml'")
end

-- Line 43: Read the XML data from your device and from your XML-string

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/xml_sds011_upload2.xml'",1)

valid = string.find(XML_string, "<root>")    -- check that we are looking in the right place
if debug == true then
    print("SDS011-string = "..XML_string)
end
   
if valid == nil then
    print ("Bad XML status read - info NOT updated")
else
 
-- Line 57: Find in the XML_string the info-fields based on their labels
-- Line 58: read position of SDS_strings
    s1 = string.find(XML_string,"<SDS_P1")  -- start of header SDS_P1
    print(s1)
    t1 = string.find(XML_string,"</SDS_P1")  -- start of tail SDS_P1
    print(t1)
    s2 = string.find(XML_string,"<SDS_P2") -- start of header SDS_P2
    print(s2)
    t2 = string.find(XML_string,"</SDS_P2") -- start of tail SDS_P2
    print(t2)
-- Line 67: read position of temperature & humidity strings
    s3 = string.find(XML_string,"<temperature")   -- start of header temperature
    print(s3)
    t3 = string.find(XML_string,"</temperature")   -- start of tail temperature
    print(t3)
    s4 = string.find(XML_string,"<humidity")      -- start of header humidity
    print(s4)
    t4 = string.find(XML_string,"</humidity")      -- start of tail humidity
    print(t4)
-- Line 76: Manually determine/set start/end-positions of the values within the substrings
    p1 = s1+19 -- number = characters of header-string SDS_P1
    print(p1)
    p2 = s2+19 -- number = characters of header-string SDS_P2
    print(p2)
    p3 = s3+24 -- number = characters of header-string temperature
    print(p3)
    p4 = s4+21 -- number = characters of header-string humidity
    print(p4)
-- Line 85: Extract the values and process for upload to Domoticz
-- Line 86: read SDS_P1 string
    SDSP1 = string.sub(XML_string,p1,t1-1) 
    SDS_P1 = tonumber(SDSP1)
    print(SDS_P1)
-- Line 90: read SDS_P2 string
    SDSP2 = string.sub(XML_string,p2,t2-1) 
    SDS_P2 = tonumber(SDSP2)
    print(SDS_P2)
-- Line 94: read temperature string
    tmp1 = string.sub(XML_string,p3,t3-1) 
    DHT_T = tonumber(tmp1)
    print(DHT_T)
-- Line 98: read humidity string
    hm1 = string.sub(XML_string,p4,t4-1) 
    DHT_H = tonumber(hm1)
    print(DHT_H)

-- Line 103: Upload to Domoticz
     
    commandArray[1] = {['UpdateDevice'] = PT_SDS011M.."|0|"..DHT_T..";"..DHT_H..";1"}  -- send updated values to Domoticz
--  with the virtual device set in line 31 the above { layout } as result mimicks a DHT22, except for the 3rd (static) value
    commandArray[2] = {['UpdateDevice'] = PT_SDS011A.."|0|"..SDS_P2}  -- send updated value for PM2.5 to Domoticz
    commandArray[3] = {['UpdateDevice'] = PT_SDS011B.."|0|"..SDS_P1}  -- send updated value for PM10 to Domoticz
       
        if debug == true then
            print("Temp/Hum = ".."'"..DHT_T.."/"..DHT_H.."'")
            print("PM2.5 = ".."'"..SDS_P2.."'")
            print("PM10  = ".."'"..SDS_P1.."'")
        end   

end 
 
return commandArray
For testing & checking the scripts have plenty of print-lines, which obviously can be removed to get leaner scripts (with simultaneous adaptation or removal of the line-numbers)
Last edited by Toulon7559 on Sunday 11 March 2018 9:24, edited 1 time in total.
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

matzeb74
Posts: 49
Joined: Sunday 07 January 2018 20:23
Target OS: Raspberry Pi
Domoticz version: newest
Location: Stuttgart
Contact:

Re: How to use Data from a Dust Sensor

Post by matzeb74 » Monday 26 February 2018 15:54

Many thanks @Toulon7559.

I´ll Try it this evening. For me its good with the printlines, helps me to understand and learn.

Hello @Toulon7559.

I just tried it, and after first error message trying the first script, i figured out that i had to add: "import urllib.request".

Then i got another error, and now i´m stuck.

Code: Select all

 Traceback (most recent call last):
  File "feinstaubstep1.py", line 20, in <module>
    obj_test = json.loads(content_test)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'
i´m using python 3.5.3

I found this https://github.com/llSourcell/antivirus_demo/issues/3, and exactly this helped me.

Now my Script looks like:

Code: Select all

#!/usr/bin/python
# -*- coding = utf-8 to enable reading by simple editors -*-
# (c)2017 script compiled by Toulon7559 from various material from forums, version 00
# (c)2018 extended by extra step to extract info from json-file from Luftdaten.info
# --------------------------------------------------
# Line006 = PREPARATION & SETTING
# --------------------------------------------------
# Imports for script-operation
import json
import urllib
import dicttoxml
import urllib.request
# SDS011 is dust-sensor at IP=192.168.0.141
# IP-address obviously to be tuned to YOUR configuration
# --------------------------------------------------
# Line015 Read data and compile json-file + xml-file
# --------------------------------------------------
page = urllib.request.urlopen('http://192.168.178.71/data.json')
content_test = page.read().decode('utf-8')
obj_test = json.loads(content_test)
print(obj_test)
with open('json_sds011_upload1.json', 'w') as outfile:
    json.dump(obj_test, outfile)
xml_test = dicttoxml.dicttoxml(obj_test)
print(xml_test)
xml_output = open("xml_sds011_upload1.xml",'wb')
xml_output.write(xml_test)
xml_output.close()

# --------------------------------------------------
# Line030 = Extract Field-names and values from json
# --------------------------------------------------
import json
import pprint
with open('json_sds011_upload1.json') as j:
    data = json.load(j)
for sdv in data.pop('sensordatavalues'):
    data[sdv['value_type']] = sdv['value']
print(data)
pprint.pprint(data)

# --------------------------------------------------
# Line043 Compile 2nd set of json-/xml-file
# --------------------------------------------------
with open('json_sds011_upload2.json', 'w') as outfile:
    json.dump(data, outfile)
xml_test2 = dicttoxml.dicttoxml(data)
print(xml_test2)
xml_output = open("xml_sds011_upload2.xml",'wb')
xml_output.write(xml_test2)
xml_output.close()

# --------------------------------------------------
# Line054 = FTP_UPLOAD to Server
# --------------------------------------------------
# Imports for script-operation
#import ftplib
#import os
# Definition of Upload_function
#def upload(ftp, file):
#    ext = os.path.splitext(file)[1]
#    if ext in (".txt", ".htm", ".html"):
#        ftp.storlines("STOR " + file, open(file))
#    else:
#        ftp.storbinary("STOR " + file, open(file, "rb"), 1024)

# --------------------------------------------------
# Line068 = Actual FTP-Login & -Upload
# --------------------------------------------------
#ftp = ftplib.FTP("<adress>")
#ftp.login("<login>", "<pass>")

#upload(ftp, "json_sds011_upload2.json")
#upload(ftp, "xml_sds011_upload2.xml")
I Commented the ftp part out, cause the Script runs on same RPI like my domoticz. It writes me 4 files
json_sds011_upload1.json
json_sds011_upload2.json
xml_sds011_upload1.xml
xml_sds011_upload2.xml

Now trying to get the second script working. I just put the Script as lua event into domotictz and get an error in the log:

Code: Select all

2018-02-26 22:35:00.436 LUA: Reading values from: 'http://192.168.178.74/home/domoticz/scripts/python/feinstaub/xml_sds011_upload2.xml'
2018-02-26 22:35:00.475 LUA: SDS011-string =
2018-02-26 22:35:00.475 LUA: Bad XML status read - info NOT updated
My Script:

Code: Select all

------------------------------------------------------------------------------------------
-- Version 00 20180226
--
-- Domoticz lua script to convert XML output for SDS011-sensor plus related DHT22-sensor
-- Reads the status based on the unique web-address and passes values
-- to virtual sensor(s) in Domoticz
-- 
--------------------------------------------------------------------------------------------------
-- Original script by RATA1 at Domoticz-forum, adapted by Toulon7559 (c)2016-2018
--------------------------------------------------------------------------------------------------
-- Line 11: Start of script
commandArray = {}

function XML_Capture(cmd,flatten)
   local f = assert(io.popen(cmd, 'r'))
   local s = assert(f:read('*a'))
   f:close()
   if flatten  then
      s = string.gsub(s, '^%s+', '')
      s = string.gsub(s, '%s+$', '')
      s = string.gsub(s, '[\n\r]+', ' ')
   end
   return s
end

-- Line 26: Set debug to true to print to log file, set to false is not.
debug = true

-- Line 29: Define the idx of YOUR virtual sensor(s) in Domoticz
-- => what you have called your sensor(s) and the related IDX number(s)
Aussensensor = 89 -- Virtual device for DHT22, temp + baro
Feinstaub_PM2 = 88 -- Virtual device for SDS PM2.5 [= SDS_P2 in file], custom sensor  
Feinstaub_PM10 = 87 -- Virtual device for SDS PM10  [= SDS_P1 in file], custom sensor

-- Line 35: Define your device IP-address: the xml-filename was set in Step1!
-- Lines 37 ~ 45 define where the XML-file can be read
--Device_IP = "xxxx.nl" -- if you read from a remote website xxxx.nl
Device_IP = "192.168.178.74/home/domoticz/scripts/python/feinstaub" --if you read from your local Domoticz@Raspberry 
if debug == true then
        print("Reading values from: 'http://"..Device_IP.."/xml_sds011_upload2.xml'")
end

-- Line 43: Read the XML data from your device and from your XML-string

XML_string=XML_Capture("curl -s 'http://"..Device_IP.."/xml_sds011_upload2.xml'",1)

valid = string.find(XML_string, "<root>")    -- check that we are looking in the right place
if debug == true then
    print("SDS011-string = "..XML_string)
end
   
if valid == nil then
    print ("Bad XML status read - info NOT updated")
else
 
-- Line 57: Find in the XML_string the info-fields based on their labels
-- Line 58: read position of SDS_strings
    s1 = string.find(XML_string,"<SDS_P1")  -- start of header SDS_P1
    print(s1)
    t1 = string.find(XML_string,"</SDS_P1")  -- start of tail SDS_P1
    print(t1)
    s2 = string.find(XML_string,"<SDS_P2") -- start of header SDS_P2
    print(s2)
    t2 = string.find(XML_string,"</SDS_P2") -- start of tail SDS_P2
    print(t2)
-- Line 67: read position of temperature & humidity strings
    s3 = string.find(XML_string,"<temperature")   -- start of header temperature
    print(s3)
    t3 = string.find(XML_string,"</temperature")   -- start of tail temperature
    print(t3)
    s4 = string.find(XML_string,"<humidity")      -- start of header humidity
    print(s4)
    t4 = string.find(XML_string,"</humidity")      -- start of tail humidity
    print(t4)
-- Line 76: Manually determine/set start/end-positions of the values within the substrings
    p1 = s1+19 -- number = characters of header-string SDS_P1
    print(p1)
    p2 = s2+19 -- number = characters of header-string SDS_P2
    print(p2)
    p3 = s3+24 -- number = characters of header-string temperature
    print(p3)
    p4 = s4+21 -- number = characters of header-string humidity
    print(p4)
-- Line 85: Extract the values and process for upload to Domoticz
-- Line 86: read SDS_P1 string
    SDSP1 = string.sub(XML_string,p1,t1-1) 
    SDS_P1 = tonumber(SDSP1)
    print(SDS_P1)
-- Line 90: read SDS_P2 string
    SDSP2 = string.sub(XML_string,p2,t2-1) 
    SDS_P2 = tonumber(SDSP2)
    print(SDS_P2)
-- Line 94: read temperature string
    tmp1 = string.sub(XML_string,p3,t3-1) 
    DHT_T = tonumber(tmp1)
    print(DHT_T)
-- Line 98: read humidity string
    hm1 = string.sub(XML_string,p4,t4-1) 
    DHT_H = tonumber(hm1)
    print(DHT_H)

-- Line 103: Upload to Domoticz
     
    commandArray[1] = {['UpdateDevice'] = Aussensensor.."|0|"..DHT_T..";"..DHT_H..";1"}  -- send updated values to Domoticz
--  with the virtual device set in line 31 the above { layout } as result mimicks a DHT22, except for the 3rd (static) value
    commandArray[2] = {['UpdateDevice'] = Feinstaub_PM2.."|0|"..SDS_P2}  -- send updated value for PM2.5 to Domoticz
    commandArray[3] = {['UpdateDevice'] = Feinstaub_PM10.."|0|"..SDS_P1}  -- send updated value for PM10 to Domoticz
       
        if debug == true then
            print("Temp/Hum = ".."'"..DHT_T.."/"..DHT_H.."'")
            print("PM2.5 = ".."'"..SDS_P2.."'")
            print("PM10  = ".."'"..SDS_P1.."'")
        end   

end 
 
return commandArray
What happened?

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: How to use Data from a Dust Sensor

Post by Toulon7559 » Monday 26 February 2018 23:38

:-( Different versions of Python sometimes cause trouble ....

Looking at the error report IMHO you should look at the Device_IP:
is in line 39 the path towards the XML-file correct?
The scriptline looking for the XML-String obviously looks for the occurrence of <root> in the XML-file, and apparently not finding.
Double-check also the contents of the XML-file.
It should look like

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?><root><max_micro type="str">26645</max_micro><temperature type="str">19.20</temperature><software_version type="str">NRZ-2017-099</software_version><age type="str">36</age><SDS_P1 type="str">6.35</SDS_P1><humidity type="str">41.50</humidity><min_micro type="str">243</min_micro><samples type="str">587059</samples><SDS_P2 type="str">5.20</SDS_P2><signal type="str">-73</signal></root>
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

matzeb74
Posts: 49
Joined: Sunday 07 January 2018 20:23
Target OS: Raspberry Pi
Domoticz version: newest
Location: Stuttgart
Contact:

Re: How to use Data from a Dust Sensor

Post by matzeb74 » Tuesday 27 February 2018 0:26

The path to my xml is right. my xml is in /home/domoticz/scripts/python/feinstaub and the ip is the ip from my rpi

My xml file looks like:

Code: Select all

<?xml version="1.0" encoding="UTF-8" ?><root><min_micro type="str">230</min_micro><age type="str">10</age><SDS_P2 type="str">3.83</SDS_P2><max_micro type="str">25769</max_micro><humidity type="str">34.00</humidity><software_version type="str">NRZ-2017-099</software_version><samples type="str">615520</samples><signal type="str">-70</signal><temperature type="str">1.40</temperature><SDS_P1 type="str">4.27</SDS_P1></root>

matzeb74
Posts: 49
Joined: Sunday 07 January 2018 20:23
Target OS: Raspberry Pi
Domoticz version: newest
Location: Stuttgart
Contact:

Re: How to use Data from a Dust Sensor

Post by matzeb74 » Tuesday 27 February 2018 1:30

I got the Error!!?!

I tried to upload the xml files to my external root server and point the script to there, and it works. So calling the file from the same RPI must be different i think.

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: How to use Data from a Dust Sensor

Post by Toulon7559 » Tuesday 27 February 2018 9:38

;-) ;-) Always those puzzles ........
The construction you used to read the file within your RPI looked OK, but apparently still something amiss.

:-( Have now same kind of problem on next part of the trajectory:
- both virtual sensors in Domoticz for dust are identical technical layout [singlefield, custom sensor]
- display etc. in Domoticz in widgets and in svalues is OK
- reading by script of value for PM2.5 is OK
- reading by same script using comparable line for PM10 results in report that value is nil [?????], although certainly the value is not nil

Headscratching to find the differences .....
[;-) Probably when solution has been found, it is an aspect of the kind 'Of course .....']

Addition 27th of Feb:
pragmatic, quick & dirty, temporary solution is that the lua-script from this message is extended to do the desired job with the available data from SDS011 and DHT22 .....
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

matzeb74
Posts: 49
Joined: Sunday 07 January 2018 20:23
Target OS: Raspberry Pi
Domoticz version: newest
Location: Stuttgart
Contact:

Re: How to use Data from a Dust Sensor

Post by matzeb74 » Tuesday 27 February 2018 23:41

Mine runs with xml Upload to my root-server and reload via the Lua script into domoticz perfectly. Fetching the xml file local will not work for me. If i try this, i get no data.

Now i tried something:

I just put the URL from Remote Site into Local.

Code: Select all

2018-02-27 22:44:34.811 LUA: Reading values from: 'http://beliarsblog.de/xml_sds011_upload2.xml'
2018-02-27 22:44:34.926 LUA: SDS011-string =
2018-02-27 22:44:34.926 LUA: Bad XML status read - info NOT updated
If i use the same URL for external request:

Code: Select all

2018-02-27 22:45:00.490 LUA: Reading values from: 'http://beliarsblog.de/xml_sds011_upload2.xml'
2018-02-27 22:45:00.645 LUA: SDS011-string = 13.9061680522.231232301.20-6825723NRZ-2017-09928.70
2018-02-27 22:45:00.645 LUA: 115
2018-02-27 22:45:00.645 LUA: 139
2018-02-27 22:45:00.645 LUA: 46
2018-02-27 22:45:00.645 LUA: 70
2018-02-27 22:45:00.645 LUA: 210
2018-02-27 22:45:00.645 LUA: 238
2018-02-27 22:45:00.645 LUA: 382
2018-02-27 22:45:00.645 LUA: 408
2018-02-27 22:45:00.645 LUA: 134
2018-02-27 22:45:00.645 LUA: 65
2018-02-27 22:45:00.645 LUA: 234
2018-02-27 22:45:00.645 LUA: 403
2018-02-27 22:45:00.645 LUA: 22.23
2018-02-27 22:45:00.645 LUA: 13.9
2018-02-27 22:45:00.645 LUA: 1.2
2018-02-27 22:45:00.645 LUA: 28.7
2018-02-27 22:45:00.645 LUA: Temp/Hum = '1.2/28.7'
2018-02-27 22:45:00.645 LUA: PM2.5 = '13.9'
2018-02-27 22:45:00.645 LUA: PM10 = '22.23'
2018-02-27 22:45:00.647 EventSystem: Script event triggered: Feinstaub

PieterS
Posts: 36
Joined: Wednesday 31 May 2017 16:06
Target OS: NAS (Synology & others)
Domoticz version: V3.9483
Location: Netherlands
Contact:

Re: How to use Data from a Dust Sensor

Post by PieterS » Saturday 03 March 2018 16:42

Strugling with same issue. How to upload data from my Luftdatensensor to Domoticz.

Did anyone have success and can you write an HowTo? This thread is not clear to me.

Succeeded to upload straight into my own InfluxDB. Or is there a way to import from InfluxDB into the database of Domoticz? :?

Thanks in advance! :D

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: How to use Data from a Dust Sensor

Post by Toulon7559 » Wednesday 21 March 2018 21:53

@Matzeb74

Looking at your result for the External Request you now get all info desired!
Same 'practical' method I presently use:
- first, make XML-files and upload XML-files to (remote) server
- second, read back and dissect for application of values

Locally reading the xml-file caused me (too many) headaches as well:
line 39 of the second script should do that job, but (during my experiments) occasionally also refuses for no clear reason.
;-) Then you accept 'less logical' solutions like uploading more remote and reading back.

Wondering what happens if you locally upload to a different directory at the Raspberry, and then read back.
The problem seems to be something very subtle in the handling of files .........

The present 'solution' it is a very big detour to get from the dustsensor-values into Domoticz, but who knows a practical shortcut?
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: How to use Data from a Dust Sensor

Post by Toulon7559 » Wednesday 21 March 2018 22:20

Application of the InFluxDB for Domoticz is described in https://www.domoticz.com/wiki/HttpLink
At first sight does not look 'simple & short' ........
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

User avatar
Involver
Posts: 3
Joined: Thursday 18 August 2016 22:27
Target OS: Raspberry Pi
Domoticz version: 3.4834
Location: Amsterdam
Contact:

Re: How to use Data from a Dust Sensor

Post by Involver » Saturday 21 April 2018 14:51

Hi All,

I have converted a script I found elsewhere (I'm by no means a Python expert!) and that works for me. I have 1 sensor for now and in Domoticz I have added two custom sensors (one for 2,5um and one for 10um readings). In the script you need to change the variables to match your setup, see the comments in the script.

First get the required components:

Code: Select all

sudo apt install git-core python-serial python-enum
Then you need to add this script to /home/pi/domoticz/scripts/pyhton:

Code: Select all

#!/usr/bin/python
# coding=utf-8
# "DATASHEET": http://cl.ly/ekot
# https://gist.github.com/kadamski/92653913a53baf9dd1a8
from __future__ import print_function
import serial, struct, sys, time, json, urllib

DEBUG = 0
CMD_MODE = 2
CMD_QUERY_DATA = 4
CMD_DEVICE_ID = 5
CMD_SLEEP = 6
CMD_FIRMWARE = 7
CMD_WORKING_PERIOD = 8
MODE_ACTIVE = 0
MODE_QUERY = 1

# domoticz values, pls fill in your device ID for 2,5 and 10 um and domoticz IP and port
IDX25 = 197
IDX10 = 198
DOMOTICZ = 'http://10.11.12.23:8080'

# get your tty device ID using dmesg command first
ser = serial.Serial()
ser.port = "/dev/ttyUSB1"
ser.baudrate = 9600

ser.open()
ser.flushInput()

byte, data = 0, ""

def dump(d, prefix=''):
    print(prefix + ' '.join(x.encode('hex') for x in d))

def construct_command(cmd, data=[]):
    assert len(data) <= 12
    data += [0,]*(12-len(data))
    checksum = (sum(data)+cmd-2)%256
    ret = "\xaa\xb4" + chr(cmd)
    ret += ''.join(chr(x) for x in data)
    ret += "\xff\xff" + chr(checksum) + "\xab"

    if DEBUG:
        dump(ret, '> ')
    return ret

def process_data(d):
    r = struct.unpack('<HHxxBB', d[2:])
    pm25 = r[0]/10.0
    pm10 = r[1]/10.0
    checksum = sum(ord(v) for v in d[2:8])%256
    return [pm25, pm10]
    #print("PM 2.5: {} μg/m^3  PM 10: {} μg/m^3 CRC={}".format(pm25, pm10, "OK" if (checksum==r[2] and r[3]==0xab) else "NOK"))

def process_version(d):
    r = struct.unpack('<BBBHBB', d[3:])
    checksum = sum(ord(v) for v in d[2:8])%256
    print("Y: {}, M: {}, D: {}, ID: {}, CRC={}".format(r[0], r[1], r[2], hex(r[3]), "OK" if (checksum==r[4] and r[5]==0xab) else "NOK"))

def read_response():
    byte = 0
    while byte != "\xaa":
        byte = ser.read(size=1)

    d = ser.read(size=9)

    if DEBUG:
        dump(d, '< ')
    return byte + d

def cmd_set_mode(mode=MODE_QUERY):
    ser.write(construct_command(CMD_MODE, [0x1, mode]))
    read_response()

def cmd_query_data():
    ser.write(construct_command(CMD_QUERY_DATA))
    d = read_response()
    values = []
    if d[1] == "\xc0":
        values = process_data(d)
    return values

def cmd_set_sleep(sleep=1):
    mode = 0 if sleep else 1
    ser.write(construct_command(CMD_SLEEP, [0x1, mode]))
    read_response()

def cmd_set_working_period(period):
    ser.write(construct_command(CMD_WORKING_PERIOD, [0x1, period]))
    read_response()

def cmd_firmware_ver():
    ser.write(construct_command(CMD_FIRMWARE))
    d = read_response()
    process_version(d)

def cmd_set_id(id):
    id_h = (id>>8) % 256
    id_l = id % 256
    ser.write(construct_command(CMD_DEVICE_ID, [0]*10+[id_l, id_h]))
    read_response()

if __name__ == "__main__":
    while True:
        cmd_set_sleep(0)
        cmd_set_mode(10);
        for t in range(4):
            values = cmd_query_data();
            if values is not None:
                print("PM2.5: ", values[0], ", PM10: ", values[1])
                time.sleep(2)

# Disabled the ## lines since no need for output to file as in original script

        # open stored data
##        with open('/home/pi/domoticz/scripts/python/data/aqi.json') as json_data:
##            data = json.load(json_data)

        # check if length is more than 100 and delete first element
##        if len(data) > 100:
##            data.pop(0)

        # append new values
##        data.append({'pm25': values[0], 'pm10': values[1], 'time': time.strftime("%d.%m.%Y %H:%M:%S")})

	# post it to domoticz device
	httpresponse = urllib.urlopen(str(DOMOTICZ)+"/json.htm?type=command&param=udevice&idx="+str(int(IDX25))+"&nvalue=0&svalue=" + str(values[0]))
	httpresponse = urllib.urlopen(str(DOMOTICZ)+"/json.htm?type=command&param=udevice&idx="+str(int(IDX10))+"&nvalue=0&svalue=" + str(values[1]))

        # save it
##       with open('/home/pi/domoticz/scripts/python/data/aqi.json', 'w') as outfile:
##            json.dump(data, outfile)

        print("Going to sleep for 5min...")
        cmd_set_mode(0);
        cmd_set_sleep()
        time.sleep(300)
This script needs to be changed to executable:

Code: Select all

 chmod +x aqi.py
You can manually execute:

Code: Select all

$ ./aqi.py
Which will produce a few lines with measurements (10 seconds to make sure the air is flowing sufficiently), write the data to domoticz and than halts for 5 minutes (to save the sensor).

When done testing (everything works for you) you need to add the script to the crontab:

Code: Select all

$ crontab -e
... and add the following to the end:

Code: Select all

@reboot cd /home/pi/domoticz/scripts/python/ && ./aqi.py
That should do it...
Domoticz - RPi - RFXCom - KaKu - Synology - HikVision - Oregon Scientific - YouLess - Anna

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests