Weird values because of daylight saving time (summer/winter)

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Friday 04 November 2016 11:30

gizmocuz wrote:The major problem is the short-log
We need to store these in UTC

the daily log we (of course) store in localtime, according the users timezone

When we request the short-log data, we (of course) convert them to localtime

There will be no graph issues

UTC does not change with a change of seasons, but local time or civil time may change if a time zone jurisdiction observes daylight saving time or summer time.

We do not need to change any datatables, only the datetime stamps of the short logs, and of course all queries, but thats not difficult


Okay, I'll try to be more specific.

The thing is: I looked at the code. What you call short-log I presume to be table [MultiMeter] for the P1 meter and this table is maintained by a timed event that takes data from the current device status. At present that function does not load any timestamp associated with that data into the table, but leaves it to SQLight to generate a timestamp value according to a rule that is part of the table properties.

Of course, the application can still provide any other valid datetime string to overrule the automatic assigned value, but at present the table properties state to enter localtime if the client omits that value.

Which by itself is not a problem. The values are still good. The trouble arises with how these values are processed. Since we have double time values when moving away from DST we get a sort issue. With the times currently stored as localtime this results in a list of values like this:

Code: Select all

... 01:50  01:55  02:00  02:00  02:05  02:05  02:10  02:10  02:15  02:15  02:20  ...

This is in fact how the graph drawing library likes to receive the X-axis values. So that's good. We just don't handle this situation very well and end up with crazy numbers. This is not very hard to fix, except for the 02:00 value that follows 02:55 which we cannot reference at that point.

If we store times as UTC we do not get the sort issue, but if we convert to localtime that will still give us the double time values but in a different order:

Code: Select all

... 02:45  02:50  02:55  02:00  02:05  02:10  02:15  ...

This is NOT how the graph drawing library likes to receive the X-axis values and it will generate a warning (see post @11/02/2016 11:25) , but it will fix our crazy numbers without further code change.

Either way there are graph issues, unless we export UTC values (with or without a fixed offset) to the client. When using localtime our options are: mixing, superimposing, or skipping/hiding.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

User avatar
gizmocuz
Posts: 8045
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gizmocuz » Friday 04 November 2016 11:44

We are talking about the same... DATETIME DEFAULT (datetime('now','localtime')));
A simple patch to DATETIME DEFAULT (datetime('now')));
Quality outlives Quantity!

stlaha2007
Posts: 414
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby stlaha2007 » Friday 04 November 2016 11:53

@gordonb3 This way i agree, but also introduces a major problem...

How can we make the server (Domoticz daemon) be sure its running with UTC...
This all depends on users intervention when they install there system (RPi or Syno-NAS) with Timezone And Local Date / Time. Even on these systems with integrated timeserver-daemons they can be setup incorrectly.

@Gizmocus and gordonb3:
As long as Domoticz IS running on UTC then the problem from shortlog could be solved with logging data into a table in memory, as the sensors like temp only sent their data through air or otherwise ''directly'' {ignoring method/intermediate device for now} to Domoticz-daemon.



Sent from my D6603 using Tapatalk

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Friday 04 November 2016 12:02

I don't think so. That is a table property, so that is effectively changing a table.

Which does not answer the question if this can in fact be considered a solution as it is also the source of a new problem that needs to be addressed.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Friday 04 November 2016 12:14

stlaha2007 wrote:@gordonb3 This way i agree, but also introduces a major problem...

How can we make the server (Domoticz daemon) be sure its running with UTC...

We can't. But it really doesn't matter. The other option would be hardware clock and that is still linear time. This is even true if the user would then set a timezone with DST, as this would then reference the hardware clock as if it were UTC. So yes this will mean some random offset towards the real UTC, but that will be up to the user to fix that himself.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

stlaha2007
Posts: 414
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: RE: Re: Weird values because of daylight saving time (summer/winter)

Postby stlaha2007 » Friday 04 November 2016 13:09

gordonb3 wrote:
stlaha2007 wrote:@gordonb3 This way i agree, but also introduces a major problem...

How can we make the server (Domoticz daemon) be sure its running with UTC...

We can't. But it really doesn't matter. The other option would be hardware clock and that is still linear time. This is even true if the user would then set a timezone with DST, as this would then reference the hardware clock as if it were UTC. So yes this will mean some random offset towards the real UTC, but that will be up to the user to fix that himself.

I think we can. Why is time so important for directory structures like eDir/Active Directory/LDAP or any other structure? They DO record events with UTC stamp and DO depend on NTP(-like)-servers and Synchronization {yes i'm a SysAdmin from before Novell Directory Services and ActiveDirectory ;-) }. Caveat has always been Time Issues to drive other minds crazy because of misbehaviour because of faulty configured Time and Zone.

Therefor synching with atomclock (nowdays NTP-servers closeby) and be sure to set time based on UTC (hardware clock also!) and after that correctly implement your Timezone (User Point-of-View).

RPi's also do sync there time [os raspbian based] through NTP.

Almost any OS these days have nice cli/gui and auto-setup which are pretty precise, and you have to know what your doing to deribably input/reconfigure wrong values. Sure i made some mistakes sometime ago.., but Next/Next/Finish is near close to perfection.


Sent from my D6603 using Tapatalk

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Friday 04 November 2016 14:10

I understand where you come from, but really: we can't. We're dealing home users mostly here and many may not care beyond time being roughly correct. They may be paranoid and not have their machine attached to the internet. I see people run Domoticz on Synology - does a NAS even care having its time run in sync with NASA or whatever? How do people set time on their machine if it's two hours behind on local time? Do they change the time zone setting (if they know it's there) or simply adjust the clock?

This is really the user's own responsibility.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

stlaha2007
Posts: 414
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby stlaha2007 » Friday 04 November 2016 15:05

ok, ok.

I think we are not looking from above (helicopter-view). I'm sorry if i misunderstand.

I would suggested to try the following approach:

For me i see to two devices;
1) domoticz as a server with a viewpoint
2) user with a viewpoint

Those can be the same, and i do agree with you about the user or NAS which don't care where time is coming from, or different.

I'm curtain that if server point-of-view is Time X independend from timezone and DST and is just ticking consequtive [so to UTC], theres no problem.

I'm also certain that from users point-of-view time is viewed as like their clock is presenting them. As long as those viewpoint are equal theres no problem.

The problem arises when the viewpoint from server and user are different.


Sent from my D6603 using Tapatalk

User avatar
gizmocuz
Posts: 8045
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gizmocuz » Friday 04 November 2016 15:42

Pfff it's so easy actually, and i already mentioned this a long time ago here in this thread

- domoticz (the server) will log in UTC
- domoticz (the server) also know the timezone of the user , as the user set the correct timezone on the system where domoticz is installed on

It does not matter if the user is in his timezone, or even if the user is +10 hours away from his timezone and looking at the web page
The time that should be displayed in the web page is always the timezone of the domoticz machine

For example we have a graph displaying energy usage from 08:00am till 11:00am

When the user is at +12, he still wants to see 08:00am as it happens at the users home, where the domoticz server is running

Recap:
- We should log in UTC (see first page of this post)
- Yes, there needs to be done some work, which is not much actually
- Yes, time is always the problem (specially after a new stable release)
Quality outlives Quantity!

stlaha2007
Posts: 414
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: RE: Re: Weird values because of daylight saving time (summer/winter)

Postby stlaha2007 » Friday 04 November 2016 15:52

gizmocuz wrote:Pfff it's so easy actually, and i already mentioned this a long time ago here in this thread

- domoticz (the server) will log in UTC
- domoticz (the server) also know the timezone of the user , as the user set the correct timezone on the system where domoticz is installed on

It does not matter if the user is in his timezone, or even if the user is +10 hours away from his timezone and looking at the web page


I agree with this, domoticz needs to talk time internally. Was this perhaps initially a designflaw?

gizmocuz wrote:The time that should be displayed in the web page is always the timezone of the domoticz machine

I Don't agree: I think it needs to be user's/client's point-of-view (as is browser or tablet whatever)

gizmocuz wrote: Recap:
- We should log in UTC (see first page of this post)
- Yes, there needs to be done some work, which is not much actually
- Yes, time is always the problem (specially after a new stable release)


Don't worry, Time Will Come ;-)

User avatar
gizmocuz
Posts: 8045
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gizmocuz » Friday 04 November 2016 16:03

Design flaw... yes, many many moons ago...

No, we should ALWAYS see it from the local time of the domoticz server ALWAYS
That would be really funky otherwise

A switch at 09:00am in the log

User is in Chinastan, +xx, he looks at the switch time, it is 19:30pm

Nope
Quality outlives Quantity!

stlaha2007
Posts: 414
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby stlaha2007 » Friday 04 November 2016 16:22

Ouch, you're right... Did my best to explain the users point-of-view. However: Server Will always be right displaying HIS time.

Sent from my K00C using Tapatalk

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Friday 04 November 2016 16:22

stlaha2007 wrote:The problem arises when the viewpoint from server and user are different.

That is not the current problem, but it will be the problem if we like to be able to have the DST shift represented in the X-axis of the graph. Because the only way to do that with this (external) library is to use the client's locale, which we can't trust.

stlaha2007 wrote:I'm also certain that from users point-of-view time is viewed as like their clock is presenting them. As long as those viewpoint are equal theres no problem.

Not exactly. As it is the client interface now tells the graph library to assume time values are UTC and that it should not convert them. Of course the values are not UTC, but represent the server's locale. So you may be -0500 away from home, e.g. New York, and then if you check home at 10:00 AM you'll see the graph extend to 03:00 PM as the server registered it.

stlaha2007 wrote:I'm curtain that if server point-of-view is Time X independend from timezone and DST and is just ticking consequtive [so to UTC], theres no problem.

Any fixed offset towards UTC would not pose a problem as well. Really the only issue here is DST, which is essentially a timezone change, which I think is mostly solved already by tzset() being executed every minute now (in the beta). The only thing that remains is how to fix the graph so that it becomes acceptable to the human eye.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

User avatar
gizmocuz
Posts: 8045
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gizmocuz » Friday 04 November 2016 16:24

Any fixed offset towards UTC would not pose a problem as well. Really the only issue here is DST, which is essentially a timezone change, which I think is mostly solved already by tzset() being executed every minute now (in the beta). The only thing that remains is how to fix the graph so that it becomes acceptable to the human eye.


This will be solved if we log in UTC

Anyway, let's try to get it to log on UTC first (with the other needed changes) then have a look at 'a' graph
Quality outlives Quantity!

Toulon7559
Posts: 334
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby Toulon7559 » Friday 04 November 2016 17:07

;-) Don't make the problem bigger than necessary.
Registration is a matter to be solved by the Domoticz-people (whatever) and in the display accept the fact of 'double' time at autumn-timeshift, or a 'hole' for spring-timeshift. The below examples may help.

Other (meteo)registration software solves this 'problem' by always storing and always displaying the data relative to actual local time allowing 'double' entries in the database and in the graph: in the attached graph it produces the 'zigzag' between 02:00 and 03:00 hours, due to hopping back to 02:00 o'clock during the generation of the graph.

20161030.gif
autumntimeshift20161030_WsWingraph
20161030.gif (38.53 KiB) Viewed 323 times

The WUnderground solution is slightly different.
Also by recording and showing from the database the values relative to actual local time: see the table below.
However, in their graphic presentation (with continuous time-scale) they only take the latest values from the table => no 'zig-zag'.

wunderground_timeshift161030.png
autumntimeshift20161030_WU_table
wunderground_timeshift161030.png (75.26 KiB) Viewed 320 times
Last edited by Toulon7559 on Friday 04 November 2016 20:48, edited 7 times in total.
Set1 = RPI-B+RFXCom433 + S0PCM + Linksprite-shield for BMP180/DS18B20/RS485
Set2 = RPI-3+RFLinkGTW + ESP8266s + WS7000
Common = 2*PVLogger + TFA_Nexus + KAKUs
=> Energy & Data Management based on Time and on measured PV&Consumption&Meteo

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Friday 04 November 2016 17:18

That is essentially what happens if we convert the dates in the database to UTC and back to local again for displaying. I have/had an image of this in one of yesterday's posts, but it appears tinypic is currently down.

Edit: ah. here. It's back
Image

BUT
Although the javascript library that is used to draw the graphs does handle this, it also reports it as an error and I don't know how each and every browser may respond to that. And that is an issue.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Monday 07 November 2016 18:19

Oops!

Converting to UTC actually turns out to have some pitfalls. And I found a whole bunch of other DST related problems that made my head spin. Having tzset() now in reference to the usage of the keyword 'now' is most likely a very good idea, but I'm convinced now that the "March DST switch"-issue will not be solved by it.This is most likely caused by date calculations that return incorrect results when a DST switch occurs.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

User avatar
gizmocuz
Posts: 8045
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gizmocuz » Tuesday 08 November 2016 8:58

I know, thats why UTC would solve the DST issue
Quality outlives Quantity!

gordonb3
Posts: 360
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gordonb3 » Tuesday 08 November 2016 11:10

gizmocuz wrote:I know, thats why UTC would solve the DST issue

No it doesn't. Really. Not.

You cannot detach from localtime. You'll always need to be able to reference midnight relative to local time, which will give you either 01:00AM or 23:00PM the previous day if a DST switch occurred. Also, there is no portable code in C++ that allows handling datetime strings as UTC and since SQLight also relies on localtime_r for translating datetime values there's no real sense in using any other intermediate format to defeat that limitation.

Effectively the only thing that storing dates as UTC will do is fix the sort order in the shortlogs if these contain a transition from Summer time to Winter time, at the cost of a lot of added complexity to translate between localtime and UTC many many times. It's complex, it's ugly and I think we can achieve the same much easier by simply adding a one byte field to hold the DST state.

No worries though. I know how to fix the time calculations. There appears to be a lot of copy paste repeated code that needs to be changed though. I'm thinking of cleaning that up while addressing the issue.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.6584

User avatar
gizmocuz
Posts: 8045
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Postby gizmocuz » Tuesday 08 November 2016 12:28

Okey ;) i'm not sure if we need a DST field.... but i will wait your test results
Quality outlives Quantity!


Return to “Bugs and Problems”

Who is online

Users browsing this forum: No registered users and 9 guests