Weird values because of daylight saving time (summer/winter)

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 21:40

Again... if we log in UTC you won't have two values (on the same time)
Quality outlives Quantity!

User avatar
robpow
Posts: 84
Joined: Friday 19 July 2013 19:28
Target OS: Raspberry Pi
Domoticz version: Beta :)
Location: .se
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by robpow » Tuesday 01 November 2016 23:25

The day of switching over in October would have 25 hours and in march 23 hours as that's essentially how a human experiences those two days. The graphs would have one more, and one less, hour interval, respectively.

http://ux.stackexchange.com/questions/4 ... ogged-data

Matt

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Wednesday 02 November 2016 0:40

@gordonb3:
If we gonna log time in UTC (so rewriting SQL query to insert the correct UTC timestamp)
And we display the graph with TZ info added eg. 02.00 CETD | 02.30 CETD | 03.00 CETD | 02.00 CET | 02.30 CET | 03.00 CET | 03.30 CET
This way we can read and display all logged time-data values and graph them... creating a 25hour graph on the Summer-winter change and a 23 hour graph when we switch from winter to summer.

Personally it will give me visually the impression it has recorded the data on the right time. I think this will only need some code to feed the graph that way and should be easier to implement.


Sent from my K00C using Tapatalk

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Wednesday 02 November 2016 9:14

This DST change was not the big problem... the one in march is...
There we also get another daily log entry.
Made a fix for that:

CSQLHelper::FixDaylightSaving
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: RE: Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Wednesday 02 November 2016 14:08

gordonb3 wrote:
stlaha2007 wrote:@gordonb3:
If we gonna log time in UTC (so rewriting SQL query to insert the correct UTC timestamp)
And we display the graph with TZ info added eg. 02.00 CETD | 02.30 CETD | 03.00 CETD | 02.00 CET | 02.30 CET | 03.00 CET | 03.30 CET
This way we can read and display all logged time-data values and graph them... creating a 25hour graph on the Summer-winter change and a 23 hour graph when we switch from winter to summer.

Personally it will give me visually the impression it has recorded the data on the right time. I think this will only need some code to feed the graph that way and should be easier to implement.
Doesn't work that way. I ran a little test with fake data in the order you propose and this is what came out:
Spoiler: show
{
"d" : "2016-11-02 02:55:00 +0200",
"eu" : "712",
"r1" : "0",
"r2" : "0",
"v" : "253",
"v2" : "0"
},
{
"d" : "2016-11-02 02:00:00 +0100",
"eu" : "483",
"r1" : "0",
"r2" : "0",
"v" : "1168",
"v2" : "0"
},
Image
Which answers my previous question whether it would draw a straight line back to 2:00 on the X-axis. Yes it does. The reason that the values for the non-DST run from 2:00 to 3:00 are up high is because I added an extra 1,000 Watts to these values. It is however clear that this will become very messy indeed if you'd put sane values in there. Additionally I received a "Highcharts Error #15" in my web console, which states that data is supposed to be pre-sorted.

As you can see TZ info is effectively ignored, which I found to be a time parser issue. After adjusting the code to implement the time zone I got the result which I should have expected beforehand: the graph shifts to display the values at UTC.
Image
Seems like a bad idea to me.
Ok... Thats what is to be expected correctly.
Graphing with UTC Timestamp is fine with me.
How to solve the rewriting backover is what needs to be inspected again...

As UTC is able to do consecutive numbering, GMTD+1 to GMT+1 isn't. Would there be a way to create this line with consecutive numbers representing the correct date/time when the graph becomes displayed.

I haven't looked into the code to build the graphs, so i can be wrong in this matter. (Busy rebuilding the frontpage from Gerard..3)

Sent from my D6603 using Tapatalk

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by Toni » Wednesday 02 November 2016 14:13

See this post: viewtopic.php?f=6&t=8583#p59803

RRD does it right.

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Thursday 03 November 2016 9:39

gordonb3 wrote:
gizmocuz wrote:This DST change was not the big problem... the one in march is...
There we also get another daily log entry.
Made a fix for that:

CSQLHelper::FixDaylightSaving
Could that be related to this issue?
Very interesting ! I think this is the reason why the timers are an hour off 2 times a year.
The reason i use localtime_r and localtime_s is because they are thread-safe.

seems we need to call tzset() each minute then ?
Quality outlives Quantity!

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Thursday 03 November 2016 11:27

That will be another if/then , we already have a minute function in the mainworker. I assume this will not take cpu time as it is normally called in localtime ?
Quality outlives Quantity!

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Thursday 03 November 2016 12:29

Quality outlives Quantity!

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 04 November 2016 8:53

We have an automated build system, you should already be able to test the tzset patch

The major problem is the short-log
We need to store these in UTC

the daily log we (of course) store in localtime, according the users timezone

When we request the short-log data, we (of course) convert them to localtime

There will be no graph issues

UTC does not change with a change of seasons, but local time or civil time may change if a time zone jurisdiction observes daylight saving time or summer time.

We do not need to change any datatables, only the datetime stamps of the short logs, and of course all queries, but thats not difficult
Quality outlives Quantity!

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 04 November 2016 11:44

We are talking about the same... DATETIME DEFAULT (datetime('now','localtime')));
A simple patch to DATETIME DEFAULT (datetime('now')));
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Friday 04 November 2016 11:53

@gordonb3 This way i agree, but also introduces a major problem...

How can we make the server (Domoticz daemon) be sure its running with UTC...
This all depends on users intervention when they install there system (RPi or Syno-NAS) with Timezone And Local Date / Time. Even on these systems with integrated timeserver-daemons they can be setup incorrectly.

@Gizmocus and gordonb3:
As long as Domoticz IS running on UTC then the problem from shortlog could be solved with logging data into a table in memory, as the sensors like temp only sent their data through air or otherwise ''directly'' {ignoring method/intermediate device for now} to Domoticz-daemon.



Sent from my D6603 using Tapatalk

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: RE: Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Friday 04 November 2016 13:09

gordonb3 wrote:
stlaha2007 wrote:@gordonb3 This way i agree, but also introduces a major problem...

How can we make the server (Domoticz daemon) be sure its running with UTC...
We can't. But it really doesn't matter. The other option would be hardware clock and that is still linear time. This is even true if the user would then set a timezone with DST, as this would then reference the hardware clock as if it were UTC. So yes this will mean some random offset towards the real UTC, but that will be up to the user to fix that himself.
I think we can. Why is time so important for directory structures like eDir/Active Directory/LDAP or any other structure? They DO record events with UTC stamp and DO depend on NTP(-like)-servers and Synchronization {yes i'm a SysAdmin from before Novell Directory Services and ActiveDirectory ;-) }. Caveat has always been Time Issues to drive other minds crazy because of misbehaviour because of faulty configured Time and Zone.

Therefor synching with atomclock (nowdays NTP-servers closeby) and be sure to set time based on UTC (hardware clock also!) and after that correctly implement your Timezone (User Point-of-View).

RPi's also do sync there time [os raspbian based] through NTP.

Almost any OS these days have nice cli/gui and auto-setup which are pretty precise, and you have to know what your doing to deribably input/reconfigure wrong values. Sure i made some mistakes sometime ago.., but Next/Next/Finish is near close to perfection.


Sent from my D6603 using Tapatalk

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Friday 04 November 2016 15:05

ok, ok.

I think we are not looking from above (helicopter-view). I'm sorry if i misunderstand.

I would suggested to try the following approach:

For me i see to two devices;
1) domoticz as a server with a viewpoint
2) user with a viewpoint

Those can be the same, and i do agree with you about the user or NAS which don't care where time is coming from, or different.

I'm curtain that if server point-of-view is Time X independend from timezone and DST and is just ticking consequtive [so to UTC], theres no problem.

I'm also certain that from users point-of-view time is viewed as like their clock is presenting them. As long as those viewpoint are equal theres no problem.

The problem arises when the viewpoint from server and user are different.


Sent from my D6603 using Tapatalk

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 04 November 2016 15:42

Pfff it's so easy actually, and i already mentioned this a long time ago here in this thread

- domoticz (the server) will log in UTC
- domoticz (the server) also know the timezone of the user , as the user set the correct timezone on the system where domoticz is installed on

It does not matter if the user is in his timezone, or even if the user is +10 hours away from his timezone and looking at the web page
The time that should be displayed in the web page is always the timezone of the domoticz machine

For example we have a graph displaying energy usage from 08:00am till 11:00am

When the user is at +12, he still wants to see 08:00am as it happens at the users home, where the domoticz server is running

Recap:
- We should log in UTC (see first page of this post)
- Yes, there needs to be done some work, which is not much actually
- Yes, time is always the problem (specially after a new stable release)
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: RE: Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Friday 04 November 2016 15:52

gizmocuz wrote:Pfff it's so easy actually, and i already mentioned this a long time ago here in this thread

- domoticz (the server) will log in UTC
- domoticz (the server) also know the timezone of the user , as the user set the correct timezone on the system where domoticz is installed on

It does not matter if the user is in his timezone, or even if the user is +10 hours away from his timezone and looking at the web page
I agree with this, domoticz needs to talk time internally. Was this perhaps initially a designflaw?
gizmocuz wrote: The time that should be displayed in the web page is always the timezone of the domoticz machine
I Don't agree: I think it needs to be user's/client's point-of-view (as is browser or tablet whatever)
gizmocuz wrote: Recap:
- We should log in UTC (see first page of this post)
- Yes, there needs to be done some work, which is not much actually
- Yes, time is always the problem (specially after a new stable release)
Don't worry, Time Will Come ;-)

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 04 November 2016 16:03

Design flaw... yes, many many moons ago...

No, we should ALWAYS see it from the local time of the domoticz server ALWAYS
That would be really funky otherwise

A switch at 09:00am in the log

User is in Chinastan, +xx, he looks at the switch time, it is 19:30pm

Nope
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Friday 04 November 2016 16:22

Ouch, you're right... Did my best to explain the users point-of-view. However: Server Will always be right displaying HIS time.

Sent from my K00C using Tapatalk

User avatar
gizmocuz
Posts: 8572
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 04 November 2016 16:24

Any fixed offset towards UTC would not pose a problem as well. Really the only issue here is DST, which is essentially a timezone change, which I think is mostly solved already by tzset() being executed every minute now (in the beta). The only thing that remains is how to fix the graph so that it becomes acceptable to the human eye.
This will be solved if we log in UTC

Anyway, let's try to get it to log on UTC first (with the other needed changes) then have a look at 'a' graph
Quality outlives Quantity!

Toulon7559
Posts: 441
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by Toulon7559 » Friday 04 November 2016 17:07

;-) Don't make the problem bigger than necessary.
Registration is a matter to be solved by the Domoticz-people (whatever) and in the display accept the fact of 'double' time at autumn-timeshift, or a 'hole' for spring-timeshift. The below examples may help.

Other (meteo)registration software solves this 'problem' by always storing and always displaying the data relative to actual local time allowing 'double' entries in the database and in the graph: in the attached graph it produces the 'zigzag' between 02:00 and 03:00 hours, due to hopping back to 02:00 o'clock during the generation of the graph.
20161030.gif
autumntimeshift20161030_WsWingraph
20161030.gif (38.53 KiB) Viewed 1184 times
The WUnderground solution is slightly different.
Also by recording and showing from the database the values relative to actual local time: see the table below.
However, in their graphic presentation (with continuous time-scale) they only take the latest values from the table => no 'zig-zag'.
wunderground_timeshift161030.png
autumntimeshift20161030_WU_table
wunderground_timeshift161030.png (75.26 KiB) Viewed 1181 times
Last edited by Toulon7559 on Friday 04 November 2016 20:48, edited 7 times in total.
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests