Page 1 of 1

Fetch data from external servers without locking the event system!

Posted: Friday 08 September 2017 17:46
by BakSeeDaa
I have seen so many scripts here lately that fetches data from external servers on the Internet.

Most if not all of those scripts use code similar to this:

Code: Select all

local file=assert(io.popen('curl http://someserver.json'))   
local raw = file:read('*all')
file:close()   
(Unfortunately even the Wiki is suggesting this method) It's a very bad practice to do like this. The Domoticz event system will wait for the response. That is, with a little bit of bad luck nothing will happen in your Domoticz system for up to 10 seconds.

So, all of a sudden when you least expect it your Domoticz system freezes and you have no clue why. The whole Domoticz system seems unstable and unpredictable. So my advise is to get rid of all those scripts immediately. We all know that there are delays on the Internet ...You can use such scripts in your LAN if you are 100% sure that your LAN servers are up and running 24/7 and always replies very fast.

What can you do about it then?

For a linux system it's not difficult at all to write a script that first calls the url without waiting for the server to respond and then returns one minute later to check the result. Below is a fully working dzVents skeleton script that does exactly that. It calls an URL every 5 minutes. The server is expected to respond with a json and the result is redirected to a temporary file. Then the next minute it checks for the result file and encodes it. Please feel free to use my script.

Code: Select all

-- This example dzVents Lua script shows how to retrieve data from the Internet
-- without risking to lock the Domoticz event system.

local jsonResultFile = '/tmp/tmpJson.json' -- Temporary file to store the servers response
local fetchIntervalMins = 5 -- (Integer) (Minutes, Range 5-60) How often the data shall be retrieved 
local scriptVersion = '1.0.2'

return {
	active = true,
	logging = {
		level = domoticz.LOG_INFO, -- Uncomment to override the dzVents global logging setting
		marker = 'callNoWait '..scriptVersion
	},
	on = {
		timer = {'every minute'} -- Don't change! Verander niet! Ne changez pas!
	},
	execute = function(domoticz, device)
		-- Every 5 minutes, call the URL and exit.
		-- The following minute, return to read the result file 
		local callUrl = false
		if (os.date('*t').min % fetchIntervalMins) == 0 then
			callUrl = true
		elseif ((os.date('*t').min -1) % fetchIntervalMins) ~= 0 then
			return
		end

		if callUrl then
			local url = 'https://jsonplaceholder.typicode.com/posts/1'
			domoticz.log('Requesting data from the jsonplaceholder server...', domoticz.LOG_INFO)
			domoticz.log('URL used: '..url, domoticz.LOG_DEBUG)
			os.execute('curl -s "'..url..'" > '..jsonResultFile..'&')
			return -- Nothing more to do for now, we'll be back in a minute to read the data!
		end

		---loads a json file with the provided name and returns it as a table (if it exists)
		local function readLuaFromJsonFile(fileName)
			local file = io.open(fileName, 'r')
			if file then
				package.path = './scripts/lua/?.lua;'..package.path
				local jsonParser = require('JSON')
				local _json = file:read('*a')
				local json = jsonParser:decode(_json)
				io.close(file)
				return json
			end
			return nil
		end

		local demoData = readLuaFromJsonFile(jsonResultFile)
		if not demoData then
			domoticz.log('Could not read demoData from file: '.. jsonResultFile, domoticz.LOG_ERROR)
			return
		end
		domoticz.log('json data file has been read. "title" = "'..demoData.title..'"', domoticz.LOG_INFO)
	end
}
For the future I'm looking for a way to make the script call a bash script that makes a "call-back" when the external server has responded so that we wouldn't have to wait that extra minute for the result. @dannybloe might have some ideas to to re-trigger the script using a callback, we'll see about that later maybe. But for now I really recommend you to rewrite all your scripts that's using assert(io.popen('curl

Cheers!

Re: Fetch data from external servers without locking the event system!

Posted: Friday 08 September 2017 17:58
by DutchHans
Perfect... But how about a lua version? There are many lua-users around here.

Regards, Hans

Re: Fetch data from external servers without locking the event system!

Posted: Friday 08 September 2017 18:09
by BakSeeDaa
DutchHans wrote:
Friday 08 September 2017 17:58
Perfect... But how about a lua version? There are many lua-users around here.

Regards, Hans
It's not difficult at all to do exactly the same thing in "vanilla" Lua Hans and I'm sure someone likes to translate it.

I used to do all coding in vanilla Lua before but I have realized how much better it gets when using a good framework like dzVents. Actually dzVents makes a lot of work for you on a higher level.

If you are new to dzVents you can just try the script out by copying it and paste it into your internal Domoticz code editor (SETUP-->EVENTS) select dzVents in the drop down, name the script "callNoWait", check "Event active" and save. Now you have your first dzVents script up and running! :lol:

Re: Fetch data from external servers without locking the event system!

Posted: Friday 08 September 2017 18:54
by BakSeeDaa
DutchHans wrote:
Friday 08 September 2017 17:58
Perfect... But how about a lua version? There are many lua-users around here.

Regards, Hans
OK, I might be banned for this (posting in wrong forum) but... here you go:

Code: Select all

commandArray = {}

local jsonResultFile = '/tmp/tmpJson.json' -- Temporary file to store the servers response
local fetchIntervalMins = 5 -- (Integer) (Minutes, Range 5-60) How often the data shall be retrieved 
local scriptVersion = '1.0.2'

-- Every 5 minutes, call the URL and exit.
-- The following minute, return to read the result file 
local callUrl = false
if (os.date('*t').min % fetchIntervalMins) == 0 then
	callUrl = true
elseif ((os.date('*t').min -1) % fetchIntervalMins) ~= 0 then
	return
end

if callUrl then
	local url = 'https://jsonplaceholder.typicode.com/posts/1'
	print('Requesting data from the jsonplaceholder server...')
	print('URL used: '..url)
	os.execute('curl -s "'..url..'" > '..jsonResultFile..'&')
	return -- Nothing more to do for now, we'll be back in a minute to read the data!
end

---loads a json file with the provided name and returns it as a table (if it exists)
local function readLuaFromJsonFile(fileName)
	local file = io.open(fileName, 'r')
	if file then
		package.path = './scripts/lua/?.lua;'..package.path
		local jsonParser = require('JSON')
		local _json = file:read('*a')
		local json = jsonParser:decode(_json)
		io.close(file)
		return json
	end
	return nil
end

local demoData = readLuaFromJsonFile(jsonResultFile)
if not demoData then
	print('Could not read demoData from file: '.. jsonResultFile)
	return
end
print('json data file has been read. "title" = "'..demoData.title..'"')

return commandArray

Re: Fetch data from external servers without locking the event system!

Posted: Saturday 09 September 2017 12:41
by elmortero
Great work bakseedaa, Will try that today

Re: Fetch data from external servers without locking the event system!

Posted: Saturday 09 September 2017 13:38
by DutchHans
@ Bakseedaa

Thank you... Give that man a cigar....
Cheers, Hans

Re: Fetch data from external servers without locking the event system!

Posted: Saturday 09 September 2017 19:32
by BakSeeDaa
Thanks for your kind words.

I just updated the script to version 1.0.1

The readLuaFromJsonFile function is now using require for loading the JSON library. The variable holding the path to the JSON library has been removed. It should be more efficient and hopefully also platform independent.

Re: Fetch data from external servers without locking the event system!

Posted: Sunday 10 September 2017 10:46
by BakSeeDaa
I just updated the script to version 1.0.2

It's a minor change.

Changed the line

Code: Select all

package.path = package.path ..";./scripts/lua/?.lua;"
To become

Code: Select all

package.path = './scripts/lua/?.lua;'..package.path

Re: Fetch data from external servers without locking the event system!

Posted: Thursday 14 December 2017 15:48
by rrozema
I haven't the foggiest if this is actually possible, but can you make the output of the curl command go into a text control instead of into a temp file? If you can do that somehow, you can have the 2nd part of the script, i.e. reading the output, execute when domoticz signals the text control as changed.

Re: Fetch data from external servers without locking the event system!

Posted: Thursday 14 December 2017 17:02
by dannybloe
Just wait for dzVents 2.4 :)

Re: Fetch data from external servers without locking the event system!

Posted: Saturday 16 December 2017 16:52
by EddyG
Nice script template, I use it several times.
It would be nice to have the same kind of script for outputting to PVoutput without locking the event system.
Is there a common way to do that?