Energy consumption incorrectly computed by Domoticz?

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
Post Reply
kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Energy consumption incorrectly computed by Domoticz?

Post by kimot » Saturday 09 December 2017 13:59

Version: 3.8153
Platform: RaspberyPi
Plugin/Hardware: Type: general, kWh
Description:
I am sending constant power consumption of 660W from my ESPeasy heating regulator in interval cca 6 hours.

From ESPeasy every minute with command:
SendToHTTP 192.168.1.253,8080,/json.htm?type=command&param=udevice&idx=28&nvalue=0&svalue=660;0

Domoticz is set to compute energy itself. ( Energy read - computed )

In graph, power is displayed correctly ( 660w ), but energy consumption is computed between 602 to 606 Wh in this hours.
I think it must be 660wh in this interval.

User avatar
jvdz
Posts: 1191
Joined: Tuesday 30 December 2014 20:25
Target OS: Raspberry Pi
Domoticz version: Stable
Location: Westland/Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by jvdz » Saturday 09 December 2017 16:01

Are the updates send exactly after 60 seconds or could it be that it slips a couple of seconds with each update?
I can imagine this causes this issue.

Jos

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Saturday 09 December 2017 16:19

I study log and times are:

2017-12-09 15:04:11.478 (ESP Easy) General/kWh (Kacka_spotreba)
2017-12-09 15:05:11.879 (ESP Easy) General/kWh (Kacka_spotreba)
2017-12-09 15:06:11.870 (ESP Easy) General/kWh (Kacka_spotreba)
2017-12-09 15:07:09.891 (ESP Easy) General/kWh (Kacka_spotreba)
2017-12-09 15:08:12.830 (ESP Easy) General/kWh (Kacka_spotreba)
2017-12-09 15:09:12.720 (ESP Easy) General/kWh (Kacka_spotreba)
2017-12-09 15:10:11.083 (ESP Easy) General/kWh (Kacka_spotreba)

But I think, that Domoticz uses last value until it is changed.

When I am heating, I send every minute:
SendToHTTP 192.168.1.253,8080,/json.htm?type=command&param=udevice&idx=28&nvalue=0&svalue=660;0

When not, then every minute:
SendToHTTP 192.168.1.253,8080,/json.htm?type=command&param=udevice&idx=28&nvalue=0&svalue=0;0

I learned how to attach images, so I am sending my graphs now
Energy_1.png
Energy_1.png (315.75 KiB) Viewed 826 times
Energy_2.png
Energy_2.png (315.45 KiB) Viewed 826 times
Energy_3.png
Energy_3.png (307.35 KiB) Viewed 826 times

gordonb3
Posts: 607
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gordonb3 » Sunday 10 December 2017 15:27

It's quite simple. Both hourly and daily usage are computed by taking the difference of the last value of the given time frame and the first value. As a result the actual time frame used is not a full hour or day and thus the calculated value is always smaller than the real hourly (or daily) usage.

The straight forward solution id of course to take the difference between the last value of an hour and the last value of the hour before. However if such a value does not exist this will result in a much larger error and even if it does this will in just about every situation still not lead to an exact hour.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Sunday 10 December 2017 22:21

Sorry, but I do not understand.
What difference, last value, previous value etc.

Maybe you interchange therms sending power and energy usage.

But I am sending power - so energy consumption is integration by time, it is suma, not difference.
When node sends power value, then is clear wich consumption is in every second, minute, until node sends different value.

When node send energy consumption ( like pulse meter counters ), it is another case and you are right.
But this is not my case.

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Sunday 10 December 2017 23:26

In SQLHelper.cpp i found:

Code: Select all

time_t now = time(0);
		struct tm ltime;
		localtime_r(&now,&ltime);
		//Commit: If Option 1: energy is computed as usage*time
		//Default is option 0, read from device
		if (sOption == "1" && devType == pTypeGeneral && subType == sTypeKwh)
		{
			std::vector<std::string> parts;
			struct tm ntime;
			double interval;
			float nEnergy;
			char sCompValue[100];
			std::string sLastUpdate = result[0][6];
			time_t lutime;
			ParseSQLdatetime(lutime, ntime, sLastUpdate, ltime.tm_isdst);

			interval = difftime(now,lutime);
			StringSplit(result[0][5].c_str(), ";", parts);
			nEnergy = static_cast<float>(strtof(parts[0].c_str(), NULL)*interval / 3600 + strtof(parts[1].c_str(), NULL)); //Rob: whats happening here... strtof ?
			StringSplit(sValue, ";", parts);
			sprintf(sCompValue, "%s;%.1f", parts[0].c_str(), nEnergy);
			sValue = sCompValue;
Interesting line:

Code: Select all

nEnergy = static_cast<float>(strtof(parts[0].c_str(), NULL)*interval / 3600 + strtof(parts[1].c_str(), NULL)); //Rob: whats happening here... strtof ?
Maybe " interval = difftime(now,lutime); " return not correct time inerval

But I do not know what is time(0), ltime, etc.
It seems that suma of "interval" in one hour interval is not 3600, but less.

I think correct method will be something like this.

measure_time = now
interval = difftime(measure_time,last_measure_time);
last_measure_time = measure_time

Maybe the same is done with:

ParseSQLdatetime(lutime, ntime, sLastUpdate, ltime.tm_isdst);

I am not a programmer.

gordonb3
Posts: 607
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gordonb3 » Sunday 10 December 2017 23:58

Same difference... ;)

When asking for the usage during a specific hour you are asking for a sum that is relative to a non zero starting point. So taking the usage between 3PM and 4PM your starting point will be something like 15:00:12.720 and your end time will be something like 15:59:11.083. Meaning that your `hour` will be 3,527.28 seconds long, resulting in a calculated usage of 648.7Wh with your constant feed of 660W. That is ignoring any misses and periodic consolidating of the shortlog to five minute periods inside Domoticz.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Monday 11 December 2017 21:28

I understand and thank you for your explanation.

But still I mean, it is a bug.
We have got right data, and because wrong computation we obtain cca -9% error

But I look to domoticz.db and I found it.
Pic_1.png
Pic_1.png (217.36 KiB) Viewed 716 times
I think, we lose last 5 minutes in every hour.
It corresponded my wrong graph values.

660*(3600-300)/3600= 605

what is right what I see in my graph. ( not cca 648 like you wrote )

At day consumption is something wrong too:
pic_2.png
pic_2.png (171.82 KiB) Viewed 716 times
128796 - 116743 = 12 053 NOT 12 008

Solution ?
Exist - Linear interpolation of values to correct time interval
wiki

But my programmers skills are not enough to contribute this to source code of Domoticz,
so I must write some LUA script to compute correct hourly and daily energy consumption from power usage.

Thank you for your time.

gerardwr
Posts: 73
Joined: Saturday 28 January 2017 12:30
Target OS: -
Domoticz version:
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gerardwr » Monday 11 December 2017 23:22

Several others have come to the same conclusion, there's a bug in Domoticz here.

So far the developers gave no hight priority to correcting this bug.

Would be great if you support my suggestion here to get this bug on the bug-correction list:
http://www.domoticz.com/forum/viewtopic.php?f=6&t=20712

gordonb3
Posts: 607
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gordonb3 » Tuesday 12 December 2017 17:14

kimot wrote:
Monday 11 December 2017 21:28
what is right what I see in my graph. ( not cca 648 like you wrote )
The calculation I showed was of course just an example. But yes, not every value received is stored in the database for historic reference. Probably the initial thought of the short log was just to be able to draw usage graphs. They may have cared less about the values being completely accurate than they did about showing how usage changes over time. The current method may also be the result of changes made over time to solve crashes or huge spikes due to missing entries required by other methods.

In any case linear interpolation will not be the answer as the two points we require are both outside the (currently) requested range. Which of course brings us to the next issue that if we do happen to be able to show two consecutive periods of time it will be very unlikely that the calculations we did on these two ranges resulted in the exact same (extrapolated) usage value for the full hour that separates them.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Wednesday 13 December 2017 23:49

as the two points we require are both outside the (currently) requested range.
08:55:02 660W
09:00:04 720W

aaa new hour?
So compute power ( watts ) from this two points exactly at 09:00:00 ( linear interpolation ), then compute energy (Ws) from 08:55:02 to 09:00:00 and add it to the energy counter for one hour ( 8:00 - 9:00 )
Maybe assuming power at 9:00:00 is the same like at 9:00:04 will be enough.
Then compute first interval for next hour from 09:00 to 09:04 and next intervals to 09:55:xx like usually.

By the way, it seems, that this a few second shift is done by domoticz itself due delay with computing all sensors.
So we can rounded to 5 minutes and result will be better then now.

And interesting things happened, when I change interval of sending data from sensor.

At first one time per 2 hours 5 min.
Pic_3.png
Pic_3.png (298.41 KiB) Viewed 632 times
And from 15:25 every 30 minutes - and voala, it works now.
Pic_4.png
Pic_4.png (298.26 KiB) Viewed 632 times
Remember, time in domoticz and my sensor is not synchronized. I have got no RTC in my sensor.

We can see, Domoticz calculates values for next intervals ( max one hour ahead ) itself,
and ads some seconds to minutes, of course :o)
Pic_5.png
Pic_5.png (201.7 KiB) Viewed 632 times
And it seems, that energy calculation is done only when new data from sensor arrived, not when new value of power is stored to database.
( which is every 5 minutes )

gordonb3
Posts: 607
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gordonb3 » Thursday 14 December 2017 1:43

You're not reading my argument correctly. If even at all. You are assuming that there is a hh:55:ss entry in the database. What if there is not? What if there is no entry for the hh-1 hour at all? Or the hh+1 hour? The latter of course being a certainty when you are requesting the current hour or day. It's not about the math - the math is easy. It's about what to do if the value you need to work with is either unexpected or missing. Which for you is a blink of the eye, but the trick is to translate that into program lines so that it won't crash the application.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Friday 15 December 2017 0:44

And why hh:55:ss entry is not in the database.
As you can see, although no data are send from sensor, Domoticz assumes last value and stored it into database for next hour in 5 min intervals.
But somebody decided to stop this after one hour after last data arrived. So we can continue write this value to database ahead.
You can say, that it is not valid data then.
Ok, but when new data arrive ( after 2 hours for example ) , domoticz now normally uses this previous "not valid" data and calculate energy for the whole previous interval.

And why use database for this calculation?
Pic_6.png
Pic_6.png (174.48 KiB) Viewed 610 times
Simply take this value when changed or every 5 minutes ( when writing to database routine ) and calculate energy for previous interval and add to one_hour _energy_counter.
When start next hour, store Energy for previous hour and calculate again the same way for this next hour.
I do not see any risk.

Or, when we must use database for this calculation, simply store power every 5 min. ,cut seconds from time and this way correctly add last 5min interval for each hour into one hour energy counter.

gordonb3
Posts: 607
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gordonb3 » Friday 15 December 2017 10:26

So you mean to tell that you system is always up. You never do software updates and you never experience any (power) failures? I guess you must be special.

No. Domoticz does not assume anything. It simply copies the data from what is then known as the current state to the shortlogs at five minute intervals. There is no prediction whatsoever going on there.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

kimot
Posts: 30
Joined: Saturday 25 November 2017 18:18
Target OS: Raspberry Pi
Domoticz version: v3.8153
Location: Czech Rep.
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by kimot » Sunday 17 December 2017 0:41

No, I am not special.
I am only looking for solution to correctly compute energy consumption from correct data.

No crashes, no restarts, even constant data ( no changes ).
I do not imagine better conditions.
And result?
Pic_7.png
Pic_7.png (300.84 KiB) Viewed 544 times
Bad some per hour data, bad per day data...

If it is good for you, OK.
For me not.
Will be good for users to know, that selecting choice, when domoticz compute energy consumption from power, gives only approximately results.
So they must compute it itself or send energy directly from sensors.

blauwebuis
Posts: 258
Joined: Wednesday 21 December 2016 10:11
Target OS: Raspberry Pi
Domoticz version: current
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by blauwebuis » Sunday 21 January 2018 19:23

How could this be improved?

gordonb3
Posts: 607
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by gordonb3 » Sunday 21 January 2018 20:38

Simple: you need to extend the database lookup to include one measurement before and one measurement after what is currently requested to calculate a specific period's usage. Using those values you can make an estimate of the missing values by assuming a constant usage between the two points surrounding it.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

lennart49e
Posts: 9
Joined: Thursday 01 December 2016 9:59
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Energy consumption incorrectly computed by Domoticz?

Post by lennart49e » Sunday 21 January 2018 21:43

You can also use the existing data in Domoticz database to calculate the hourly and daily consumption.
See example: viewtopic.php?f=38&t=21504&sid=4c1970fc ... a014b13d17

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests