Weird values because of daylight saving time (summer/winter)

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Tuesday 01 November 2016 18:35

@gordonb3: Hopefully we can get some point we both agree on the concatenate ;-)
@gizmocus: Hope to understand correctly: Logging data in db is now time_t and not already in UTC?

Would love to try to build it. Can't be hard ;-) However source is in C# or C++? No experience... Pascal/Perl/Python is more to my liking.

Anyway...
Can we agree, as mentioned by others, to record withh a UTC timestamp.
And solve the display errors make a few extra settings in the configuration:
- Option to use systems TimeZone Yes/No
- Display Time on Sensors and/or Graphs based on UTC or Local TimeZone

There was also a reply above displaying and mrtg-like graph with an extra hour... Looks odd, however still correct...


Sent from my K00C using Tapatalk

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 21:34

I would suggest having a look at the code (c++) to see how the datetime is handled... part of it is automated by triggers
Quality outlives Quantity!

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Tuesday 01 November 2016 21:37

stlaha2007 wrote:@gordonb3: Hopefully we can get some point we both agree on the concatenate ;-)
Having looked at the code I can inform you that it is in fact far from straight forward to enable concatenating the values, especially when using UTC timestamps. But as said: if I used 2700 Watts at 2:05 CETD and 3000 Watts at 2:05 CET I would not want the graph to show that I used 5700 Watts at 2:05, because that is simply not true, although granted that would be way better than the 600kW it displayed last Sunday. The thing here is: if we like to show both values, do we know what happens if we feed the graph generator X-axis values that jump back from 2:55 to 2:00 (assuming CET)? Does that draw a straight line so we get three lines in total between 2:00 and 3:00? If so: will that make the graph better to read than the sawtooth one might expect from alternating between the two ranges (as is done now, but then with the numbers fixed)? Moving on to March next year: what should the graph display between 2:00 and 3:00? A straight line between the values from 1:55 and 3:00 (the latter which in the present code will be 8% of the value that it should show), or a baseline at zero? Again assuming it will not be possible to have the graph generator display a gap.

Point being that there is no single answer on how to handle this. Everyone will want something else, but then the question becomes if you can actually read every individual value from the graph and also be able to determine whether that particular value belongs to CET or CETD. And more important: would you? If the value does not seem out of the ordinary that is.
stlaha2007 wrote:@gizmocus: Hope to understand correctly: Logging data in db is now time_t and not already in UTC?
It is in fact SQL datetime, meaning that you get a string representation of the value when retrieving it (and need a string representation to load it - or NULL to have to have the SQL engine automatically fill it with the configured value now()) The application interprets this string to come to a C++ timestamp value and since time out is supposed to be uniform, but localized, it simply assumes a constant setting for DST (non zero in this case).
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 21:40

Again... if we log in UTC you won't have two values (on the same time)
Quality outlives Quantity!

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Tuesday 01 November 2016 22:31

gizmocuz wrote:Again... if we log in UTC you won't have two values (on the same time)
That would be true if we display in UTC as well. The thing is: we don't.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
robpow
Posts: 82
Joined: Friday 19 July 2013 19:28
Target OS: Raspberry Pi
Domoticz version: Beta :)
Location: .se
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by robpow » Tuesday 01 November 2016 23:25

The day of switching over in October would have 25 hours and in march 23 hours as that's essentially how a human experiences those two days. The graphs would have one more, and one less, hour interval, respectively.

http://ux.stackexchange.com/questions/4 ... ogged-data

Matt

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Wednesday 02 November 2016 0:40

@gordonb3:
If we gonna log time in UTC (so rewriting SQL query to insert the correct UTC timestamp)
And we display the graph with TZ info added eg. 02.00 CETD | 02.30 CETD | 03.00 CETD | 02.00 CET | 02.30 CET | 03.00 CET | 03.30 CET
This way we can read and display all logged time-data values and graph them... creating a 25hour graph on the Summer-winter change and a 23 hour graph when we switch from winter to summer.

Personally it will give me visually the impression it has recorded the data on the right time. I think this will only need some code to feed the graph that way and should be easier to implement.


Sent from my K00C using Tapatalk

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Wednesday 02 November 2016 9:14

This DST change was not the big problem... the one in march is...
There we also get another daily log entry.
Made a fix for that:

CSQLHelper::FixDaylightSaving
Quality outlives Quantity!

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Wednesday 02 November 2016 12:25

stlaha2007 wrote:@gordonb3:
If we gonna log time in UTC (so rewriting SQL query to insert the correct UTC timestamp)
And we display the graph with TZ info added eg. 02.00 CETD | 02.30 CETD | 03.00 CETD | 02.00 CET | 02.30 CET | 03.00 CET | 03.30 CET
This way we can read and display all logged time-data values and graph them... creating a 25hour graph on the Summer-winter change and a 23 hour graph when we switch from winter to summer.

Personally it will give me visually the impression it has recorded the data on the right time. I think this will only need some code to feed the graph that way and should be easier to implement.
Doesn't work that way. I ran a little test with fake data in the order you propose and this is what came out:
Spoiler: show
{
"d" : "2016-11-02 02:55:00 +0200",
"eu" : "712",
"r1" : "0",
"r2" : "0",
"v" : "253",
"v2" : "0"
},
{
"d" : "2016-11-02 02:00:00 +0100",
"eu" : "483",
"r1" : "0",
"r2" : "0",
"v" : "1168",
"v2" : "0"
},
Image
Which answers my previous question whether it would draw a straight line back to 2:00 on the X-axis. Yes it does. The reason that the values for the non-DST run from 2:00 to 3:00 are up high is because I added an extra 1,000 Watts to these values. It is however clear that this will become very messy indeed if you'd put sane values in there. Additionally I received a "Highcharts Error #15" in my web console, which states that data is supposed to be pre-sorted.

As you can see TZ info is effectively ignored, which I found to be a time parser issue. After adjusting the code to implement the time zone I got the result which I should have expected beforehand: the graph shifts to display the values at UTC.
Image
Seems like a bad idea to me.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Wednesday 02 November 2016 13:48

gizmocuz wrote:This DST change was not the big problem... the one in march is...
There we also get another daily log entry.
Made a fix for that:

CSQLHelper::FixDaylightSaving
Could that be related to this issue?
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: RE: Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Wednesday 02 November 2016 14:08

gordonb3 wrote:
stlaha2007 wrote:@gordonb3:
If we gonna log time in UTC (so rewriting SQL query to insert the correct UTC timestamp)
And we display the graph with TZ info added eg. 02.00 CETD | 02.30 CETD | 03.00 CETD | 02.00 CET | 02.30 CET | 03.00 CET | 03.30 CET
This way we can read and display all logged time-data values and graph them... creating a 25hour graph on the Summer-winter change and a 23 hour graph when we switch from winter to summer.

Personally it will give me visually the impression it has recorded the data on the right time. I think this will only need some code to feed the graph that way and should be easier to implement.
Doesn't work that way. I ran a little test with fake data in the order you propose and this is what came out:
Spoiler: show
{
"d" : "2016-11-02 02:55:00 +0200",
"eu" : "712",
"r1" : "0",
"r2" : "0",
"v" : "253",
"v2" : "0"
},
{
"d" : "2016-11-02 02:00:00 +0100",
"eu" : "483",
"r1" : "0",
"r2" : "0",
"v" : "1168",
"v2" : "0"
},
Image
Which answers my previous question whether it would draw a straight line back to 2:00 on the X-axis. Yes it does. The reason that the values for the non-DST run from 2:00 to 3:00 are up high is because I added an extra 1,000 Watts to these values. It is however clear that this will become very messy indeed if you'd put sane values in there. Additionally I received a "Highcharts Error #15" in my web console, which states that data is supposed to be pre-sorted.

As you can see TZ info is effectively ignored, which I found to be a time parser issue. After adjusting the code to implement the time zone I got the result which I should have expected beforehand: the graph shifts to display the values at UTC.
Image
Seems like a bad idea to me.
Ok... Thats what is to be expected correctly.
Graphing with UTC Timestamp is fine with me.
How to solve the rewriting backover is what needs to be inspected again...

As UTC is able to do consecutive numbering, GMTD+1 to GMT+1 isn't. Would there be a way to create this line with consecutive numbers representing the correct date/time when the graph becomes displayed.

I haven't looked into the code to build the graphs, so i can be wrong in this matter. (Busy rebuilding the frontpage from Gerard..3)

Sent from my D6603 using Tapatalk

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by Toni » Wednesday 02 November 2016 14:13

See this post: viewtopic.php?f=6&t=8583#p59803

RRD does it right.

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Wednesday 02 November 2016 16:01

Right...

As it happens there is an option in the graph drawing library to fix the display issue:
Image
There's a catch to this though. As this is javascript, the DST switch is based on the client's locale instead of the server. And in fact the server time is also not what we want, because we want the time from the meter. So while this looks like a solution, it is actually unsafe practice.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Thursday 03 November 2016 9:39

gordonb3 wrote:
gizmocuz wrote:This DST change was not the big problem... the one in march is...
There we also get another daily log entry.
Made a fix for that:

CSQLHelper::FixDaylightSaving
Could that be related to this issue?
Very interesting ! I think this is the reason why the timers are an hour off 2 times a year.
The reason i use localtime_r and localtime_s is because they are thread-safe.

seems we need to call tzset() each minute then ?
Quality outlives Quantity!

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Thursday 03 November 2016 10:06

Assuming people could in theory change the time zone on their server at any given moment that might be appropriate. As a more practical approach I'd say we could restrict ourselves to doing a tzset() at :01 and :31 which I'm certain will catch every DST switch in any country. You are thinking about a schedule, right?
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Thursday 03 November 2016 11:27

That will be another if/then , we already have a minute function in the mainworker. I assume this will not take cpu time as it is normally called in localtime ?
Quality outlives Quantity!

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Thursday 03 November 2016 12:17

Ah! In that case: by all means.

Don't know how localtime() does it. It would stand to reason that it implements tzset() but it could use some other isolated method as well. In any case one would expect localtime_r() to be more processor friendly and I think we can spare the additional CPU cycles for running tzset() once every minute to allow the tens (hundreds?) of localtime_r() instances being triggered in between to return the correct time.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Thursday 03 November 2016 12:29

Quality outlives Quantity!

gordonb3
Posts: 477
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Friday 04 November 2016 0:10

Excellent. Any way to test this or do we wait for next March to verify? :mrgreen:

With respect to the day graphs, could you share your thoughts on how to proceed?
  • Bottom line: the only way to have the graph making library consider a DST shift is to rely on the client's locale. While in practice that will likely work for the vast majority, including an estimated 99%+ of my own usage, I'd still say that's a no-no.
  • With the server specifying the time values, the only way to display a straight consecutive line therefore becomes showing UTC values. This does not seem very user friendly to me, but it could be made an option on the settings page and possibly allow setting a fixed offset as well.
  • Every other option will mean that there will be two values for a one hour period when moving away from DST:
    • We can simply fix the values and leave the rest as is
    • We can fix the order of the records so there will be a loop and a javascript warning message
    • We can discard the second occurrence of a time value
Most of these options will require redefining the table properties and/or altering the SQL statements that access it to have it store UTC values for the correct sort order. That doesn't appear to be a very big deal as we can easily convert those values to localtime. We can in fact make SQLight do this for us. We can even make SQLight return the datetime value as INT-1970 so we wouldn't have to additionally parse the result as a string to convert it back to a time value.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
gizmocuz
Posts: 8380
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 04 November 2016 8:53

We have an automated build system, you should already be able to test the tzset patch

The major problem is the short-log
We need to store these in UTC

the daily log we (of course) store in localtime, according the users timezone

When we request the short-log data, we (of course) convert them to localtime

There will be no graph issues

UTC does not change with a change of seasons, but local time or civil time may change if a time zone jurisdiction observes daylight saving time or summer time.

We do not need to change any datatables, only the datetime stamps of the short logs, and of course all queries, but thats not difficult
Quality outlives Quantity!

Post Reply

Who is online

Users browsing this forum: darlomrh and 3 guests