Fetch data from external servers without locking the event system!

Easy to use, 100% Lua-based event scripting framework.
Post Reply
BakSeeDaa
Posts: 551
Joined: Thursday 17 September 2015 10:13
Target OS: Raspberry Pi
Domoticz version:

Fetch data from external servers without locking the event system!

Post by BakSeeDaa » Friday 08 September 2017 17:46

I have seen so many scripts here lately that fetches data from external servers on the Internet.

Most if not all of those scripts use code similar to this:

Code: Select all

local file=assert(io.popen('curl http://someserver.json'))   
local raw = file:read('*all')
file:close()   
(Unfortunately even the Wiki is suggesting this method) It's a very bad practice to do like this. The Domoticz event system will wait for the response. That is, with a little bit of bad luck nothing will happen in your Domoticz system for up to 10 seconds.

So, all of a sudden when you least expect it your Domoticz system freezes and you have no clue why. The whole Domoticz system seems unstable and unpredictable. So my advise is to get rid of all those scripts immediately. We all know that there are delays on the Internet ...You can use such scripts in your LAN if you are 100% sure that your LAN servers are up and running 24/7 and always replies very fast.

What can you do about it then?

For a linux system it's not difficult at all to write a script that first calls the url without waiting for the server to respond and then returns one minute later to check the result. Below is a fully working dzVents skeleton script that does exactly that. It calls an URL every 5 minutes. The server is expected to respond with a json and the result is redirected to a temporary file. Then the next minute it checks for the result file and encodes it. Please feel free to use my script.

Code: Select all

-- This example dzVents Lua script shows how to retrieve data from the Internet
-- without risking to lock the Domoticz event system.

local jsonResultFile = '/tmp/tmpJson.json' -- Temporary file to store the servers response
local fetchIntervalMins = 5 -- (Integer) (Minutes, Range 5-60) How often the data shall be retrieved 
local scriptVersion = '1.0.2'

return {
	active = true,
	logging = {
		level = domoticz.LOG_INFO, -- Uncomment to override the dzVents global logging setting
		marker = 'callNoWait '..scriptVersion
	},
	on = {
		timer = {'every minute'} -- Don't change! Verander niet! Ne changez pas!
	},
	execute = function(domoticz, device)
		-- Every 5 minutes, call the URL and exit.
		-- The following minute, return to read the result file 
		local callUrl = false
		if (os.date('*t').min % fetchIntervalMins) == 0 then
			callUrl = true
		elseif ((os.date('*t').min -1) % fetchIntervalMins) ~= 0 then
			return
		end

		if callUrl then
			local url = 'https://jsonplaceholder.typicode.com/posts/1'
			domoticz.log('Requesting data from the jsonplaceholder server...', domoticz.LOG_INFO)
			domoticz.log('URL used: '..url, domoticz.LOG_DEBUG)
			os.execute('curl -s "'..url..'" > '..jsonResultFile..'&')
			return -- Nothing more to do for now, we'll be back in a minute to read the data!
		end

		---loads a json file with the provided name and returns it as a table (if it exists)
		local function readLuaFromJsonFile(fileName)
			local file = io.open(fileName, 'r')
			if file then
				package.path = './scripts/lua/?.lua;'..package.path
				local jsonParser = require('JSON')
				local _json = file:read('*a')
				local json = jsonParser:decode(_json)
				io.close(file)
				return json
			end
			return nil
		end

		local demoData = readLuaFromJsonFile(jsonResultFile)
		if not demoData then
			domoticz.log('Could not read demoData from file: '.. jsonResultFile, domoticz.LOG_ERROR)
			return
		end
		domoticz.log('json data file has been read. "title" = "'..demoData.title..'"', domoticz.LOG_INFO)
	end
}
For the future I'm looking for a way to make the script call a bash script that makes a "call-back" when the external server has responded so that we wouldn't have to wait that extra minute for the result. @dannybloe might have some ideas to to re-trigger the script using a callback, we'll see about that later maybe. But for now I really recommend you to rewrite all your scripts that's using assert(io.popen('curl

Cheers!
Last edited by BakSeeDaa on Sunday 10 September 2017 10:42, edited 2 times in total.

DutchHans
Posts: 340
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Fetch data from external servers without locking the event system!

Post by DutchHans » Friday 08 September 2017 17:58

Perfect... But how about a lua version? There are many lua-users around here.

Regards, Hans

BakSeeDaa
Posts: 551
Joined: Thursday 17 September 2015 10:13
Target OS: Raspberry Pi
Domoticz version:

Re: Fetch data from external servers without locking the event system!

Post by BakSeeDaa » Friday 08 September 2017 18:09

DutchHans wrote:
Friday 08 September 2017 17:58
Perfect... But how about a lua version? There are many lua-users around here.

Regards, Hans
It's not difficult at all to do exactly the same thing in "vanilla" Lua Hans and I'm sure someone likes to translate it.

I used to do all coding in vanilla Lua before but I have realized how much better it gets when using a good framework like dzVents. Actually dzVents makes a lot of work for you on a higher level.

If you are new to dzVents you can just try the script out by copying it and paste it into your internal Domoticz code editor (SETUP-->EVENTS) select dzVents in the drop down, name the script "callNoWait", check "Event active" and save. Now you have your first dzVents script up and running! :lol:

BakSeeDaa
Posts: 551
Joined: Thursday 17 September 2015 10:13
Target OS: Raspberry Pi
Domoticz version:

Re: Fetch data from external servers without locking the event system!

Post by BakSeeDaa » Friday 08 September 2017 18:54

DutchHans wrote:
Friday 08 September 2017 17:58
Perfect... But how about a lua version? There are many lua-users around here.

Regards, Hans
OK, I might be banned for this (posting in wrong forum) but... here you go:

Code: Select all

commandArray = {}

local jsonResultFile = '/tmp/tmpJson.json' -- Temporary file to store the servers response
local fetchIntervalMins = 5 -- (Integer) (Minutes, Range 5-60) How often the data shall be retrieved 
local scriptVersion = '1.0.2'

-- Every 5 minutes, call the URL and exit.
-- The following minute, return to read the result file 
local callUrl = false
if (os.date('*t').min % fetchIntervalMins) == 0 then
	callUrl = true
elseif ((os.date('*t').min -1) % fetchIntervalMins) ~= 0 then
	return
end

if callUrl then
	local url = 'https://jsonplaceholder.typicode.com/posts/1'
	print('Requesting data from the jsonplaceholder server...')
	print('URL used: '..url)
	os.execute('curl -s "'..url..'" > '..jsonResultFile..'&')
	return -- Nothing more to do for now, we'll be back in a minute to read the data!
end

---loads a json file with the provided name and returns it as a table (if it exists)
local function readLuaFromJsonFile(fileName)
	local file = io.open(fileName, 'r')
	if file then
		package.path = './scripts/lua/?.lua;'..package.path
		local jsonParser = require('JSON')
		local _json = file:read('*a')
		local json = jsonParser:decode(_json)
		io.close(file)
		return json
	end
	return nil
end

local demoData = readLuaFromJsonFile(jsonResultFile)
if not demoData then
	print('Could not read demoData from file: '.. jsonResultFile)
	return
end
print('json data file has been read. "title" = "'..demoData.title..'"')

return commandArray
Last edited by BakSeeDaa on Sunday 10 September 2017 10:42, edited 3 times in total.

elmortero
Posts: 237
Joined: Sunday 29 November 2015 21:46
Target OS: Raspberry Pi
Domoticz version: 3.9639
Location: Spain
Contact:

Re: Fetch data from external servers without locking the event system!

Post by elmortero » Saturday 09 September 2017 12:41

Great work bakseedaa, Will try that today

DutchHans
Posts: 340
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Fetch data from external servers without locking the event system!

Post by DutchHans » Saturday 09 September 2017 13:38

@ Bakseedaa

Thank you... Give that man a cigar....
Cheers, Hans

BakSeeDaa
Posts: 551
Joined: Thursday 17 September 2015 10:13
Target OS: Raspberry Pi
Domoticz version:

Re: Fetch data from external servers without locking the event system!

Post by BakSeeDaa » Saturday 09 September 2017 19:32

Thanks for your kind words.

I just updated the script to version 1.0.1

The readLuaFromJsonFile function is now using require for loading the JSON library. The variable holding the path to the JSON library has been removed. It should be more efficient and hopefully also platform independent.

BakSeeDaa
Posts: 551
Joined: Thursday 17 September 2015 10:13
Target OS: Raspberry Pi
Domoticz version:

Re: Fetch data from external servers without locking the event system!

Post by BakSeeDaa » Sunday 10 September 2017 10:46

I just updated the script to version 1.0.2

It's a minor change.

Changed the line

Code: Select all

package.path = package.path ..";./scripts/lua/?.lua;"
To become

Code: Select all

package.path = './scripts/lua/?.lua;'..package.path

rrozema
Posts: 124
Joined: Thursday 26 October 2017 13:37
Target OS: Raspberry Pi
Domoticz version: beta
Location: Delft
Contact:

Re: Fetch data from external servers without locking the event system!

Post by rrozema » Thursday 14 December 2017 15:48

I haven't the foggiest if this is actually possible, but can you make the output of the curl command go into a text control instead of into a temp file? If you can do that somehow, you can have the 2nd part of the script, i.e. reading the output, execute when domoticz signals the text control as changed.

dannybloe
Posts: 1481
Joined: Friday 29 August 2014 11:26
Target OS: Raspberry Pi
Domoticz version:
Location: Ermelo
Contact:

Re: Fetch data from external servers without locking the event system!

Post by dannybloe » Thursday 14 December 2017 17:02

Just wait for dzVents 2.4 :)
Creator dzVents - RPi3, loads of zwave devices, esp8266, evohome.

User avatar
EddyG
Posts: 320
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9999
Location: Rhenen, Netherlands
Contact:

Re: Fetch data from external servers without locking the event system!

Post by EddyG » Saturday 16 December 2017 16:52

Nice script template, I use it several times.
It would be nice to have the same kind of script for outputting to PVoutput without locking the event system.
Is there a common way to do that?
Regards,
Eddy

Post Reply

Who is online

Users browsing this forum: No registered users and 0 guests