Weird values because of daylight saving time (summer/winter)

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
OLLIFDG
Posts: 6
Joined: Saturday 20 February 2016 14:48
Target OS: Raspberry Pi
Domoticz version: V2.3530
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by OLLIFDG » Saturday 20 February 2016 16:45

I'm having a time issue. Perhaps it's easy to solve but until now I did not find a solution.
There is a time difference from exactly 1 hours in the Netherlands. Domoticz system is saying 14:44 and the time is one hours later.
How can I fix this? :oops:

Thanks

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Saturday 20 February 2016 17:18

did you run

sudo raspi-config

and set the correct time zone ?

Did you install the ntp package and synced the time ?
Quality outlives Quantity!

OLLIFDG
Posts: 6
Joined: Saturday 20 February 2016 14:48
Target OS: Raspberry Pi
Domoticz version: V2.3530
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by OLLIFDG » Saturday 20 February 2016 18:15

gizmocuz wrote:did you run

sudo raspi-config

and set the correct time zone ?

Did you install the ntp package and synced the time ?

I only set the location degrees.
I'm going to see how to do the sudo, ntp and synching time.
I'm new. Do you now a link where I can find how to do this?

Thanks

I managed to change in config via putty.

Thanks again

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Monday 28 March 2016 10:48

A new beta #5019 is released to solve the summer time issue (2 values on one day) till next year
Quality outlives Quantity!

RidingTheFlow
Posts: 78
Joined: Friday 11 March 2016 19:23
Target OS: Raspberry Pi
Domoticz version:
Location: Essex, UK
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by RidingTheFlow » Tuesday 29 March 2016 0:20

Toni wrote:The general rule of thumb for anything related to timestamps is:
  • ALWAYS internally store all timestamps in the UTC format, or for example in Unix time
  • ONLY convert the timestamps to local time when showing the timestamp to the user
This way it will always work correctly. If it's done in any other way, it will always be wrong in some cases.
.
I second this. I did a lot of programming with timestamps and dealt with a lot of daylight related bugs.

But its easy to solve really (well, its easy recipe, but still lot of work) - you just need process all data internally as UTC. E.g. if you want to calculate meter update, store it in DB, pass data around via JSON, etc - you always use UTC.

You only convert to local when you about to print value in UI.
And you only convert from local directly when getting value from input box.

It all UTC otherwise, and UTC is continuous time same in all regions of the world and no DST periods - so its always consistent and no gaps in duration calculations and such.

Though from my experience it quite a tremendous work to convert project (like Domoticz) which was all using local time to all using UTC :(

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 29 March 2016 9:38

Time is the problem.....

And i do not think it is a lot of work... how many tables we have which uses date/time ? 10 ? 15? Not much

Most work will be converting back/forward user interface stuff like timers, blockly stuff

But all needs to be tested very good, and also we should convert the current date/times to UTC in the database
Quality outlives Quantity!

RidingTheFlow
Posts: 78
Joined: Friday 11 March 2016 19:23
Target OS: Raspberry Pi
Domoticz version:
Location: Essex, UK
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by RidingTheFlow » Tuesday 29 March 2016 10:41

Databases are not hard, but I would be more concerned about all time-using code spread around in various places both on C++ and JS side. Because all that needs to be changed for a proper fix - mostly places where "current time" is obtained, but some times when interfacing with some other library/API/hardware - and sometimes this will have option to use UTC and should do that, but some times will have to convert to/from local time.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 29 March 2016 10:56

from and to the web gui should be converted, that is all, but the most work
Quality outlives Quantity!

rafal98
Posts: 3
Joined: Tuesday 03 November 2015 10:42
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by rafal98 » Tuesday 29 March 2016 12:09

Hi, I also have an issue following the daylight savint time where both days are stacked. I'm not sure it's a new problem. Here a screenshot

Regards
Screenshot domoticz.png
Screenshot domoticz.png (231.99 KiB) Viewed 1250 times

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 29 March 2016 12:24

gizmocuz wrote:A new beta #5019 is released to solve the summer time issue (2 values on one day) till next year
Quality outlives Quantity!

louisL
Posts: 4
Joined: Thursday 11 August 2016 23:14
Target OS: Linux
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by louisL » Friday 12 August 2016 0:15

The usual way for using times is often to store times as the number of seconds since the epoch (00:00 hours, Jan 1, 1970 UTC) (time_t) or as a struct tm containing a broken down time and date. See http://www.cplusplus.com/reference/ctime/ for more information. The best solution often is to use time_t (the second counter) in the database and use gmtime or localtime for presentation. The conversion to localtime will use the system's timezone setting to convert the time_t data to the local time. So if you are in another timezone the display will still use whatever timezone setting is used on the server. During daylight saving switches you will still have days of 23 or 25 hours though. Leap seconds can be taken care of in these interfaces as well they may be included on the system's timezone definitions.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 12 August 2016 8:53

Its stored as time_t in the database
But we need to store all in UTC, and convert to localtime in presentation
Time... (changing the code) is sparse ;)
Quality outlives Quantity!

User avatar
BartdaMan
Posts: 22
Joined: Sunday 22 May 2016 22:58
Target OS: Raspberry Pi
Domoticz version:
Location: Amersfoort
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by BartdaMan » Sunday 30 October 2016 13:39

Well, although I am using V3.5834 I think the clock change from 03 to 02 hours last night caused some weird readings:
2016-10-30_123836.png
2016-10-30_123836.png (98.39 KiB) Viewed 898 times
Any clue how this happened?
And how to fix it?

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Sunday 30 October 2016 15:37

"how to fix this"

There have to be made some code changes for this to get solved
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Monday 31 October 2016 23:35

Seen this, and others, and probably can't be fixed.

For example most logfiles [eg. /var/log/messages] also have duplicate logging.

Better stop the winter-summer-wintertime changes :-)

It looks a bit weird for sure, however its only cosmeticly wrong.

As mentioned earlier, when its recorded correctly, and daily totals (haven't checked but was ok i believe) are correct. Don't spend time trying to solve a cosmetic misbehaviour.

Grtz Stephan

Sent from my D6603 using Tapatalk

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 8:56

Sure it can be fixed if we use UTC for logging... have to watch out for the daily log
Quality outlives Quantity!

gordonb3
Posts: 566
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Tuesday 01 November 2016 11:08

Should probably note that even with UTC the graph will still look weird because you will still get the double entries between two and three AM after applying the locale for display.

The trouble here is that time is used as an index in a function that produces derived numbers, i.e. numbers that are not actually in the database itself, while assuming that index contains unique values which of course it doesn't because of the time leap. Switching to UTC would fix that but it also introduces many new problems, such as what time is midnight? Does my locale day start somewhere yesterday or end somewhere tomorrow? And yes I realize that I suggested switching to UTC myself, but I don't think that is a really good idea any more.

As @stlaha2007 stated, this is merely a display issue, but it would still be nice to be able to see it generate "normal" output. The json output hints that we are dealing with a query in the form SELECT * FROM [table] WHERE [timestamp] BETWEEN [time] AND [time2] ORDER BY [timestamp]. The function that spits out the delta values obviously does this by comparing each line in the resultset with its predecessor, which is wrong in this particular case. Notably domoticz already detects this, which is clear from the fact that it refuses to divide by zero (same time) and also resets negative numbers to zero. So essentially all that needs to be done is change the code that is currently set to using a constant if these conditions apply to referencing one line further back.

Does that make sense?
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Tuesday 01 November 2016 13:08

Like stated before: switching to UTC should be the correct option. (Thought it was, however missed that part probably)

Any server installed by me will be setup in UTC. And depending on the locale (Holland W-CET = GMT +1 or Great Brittain GMT + 0 = UTC) displaying the locale date/time correctly. And depending on Daylight Savings On it will be WCET = GMT + 1 ({Daylight On} +1).
So anyway.... When recording with the UTC timestamp it will record data correctly with time and value(s) from the sensors. And they do NOT overlap eachother when going from daylight off to on or daylight on to off. Thats the database-part.

The real problem is displaying: And to read the records from the database, apply the local timezone and after thats done, you need to concatenate the values when talking about the smartmeters and rfxmeters. However eg. themperature-sensors.... these values need to be accumulate and divided by 2 to get a more realistic value.

Perhaps this will solve the (Cosmetic)display-issue for anybody not familiar with timezones and the underlying datastructure.

Sent from my K00C using Tapatalk

gordonb3
Posts: 566
Joined: Friday 22 January 2016 11:15
Target OS: Linux
Domoticz version: custom
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gordonb3 » Tuesday 01 November 2016 13:57

Disagree on the concatenate. This would give the impression that usage is higher than normal between those times, which is (probably) not the case. I'd rather see a double line, but I doubt the graph function (which is jQuery I presume) will support that.

In any case I had some spare time to look at the code and found the expected code without too much trouble. There is in fact no time filter in the database query, so I suspect that graph drawing will gradually become slower as history grows. That's one point that could use improving. As it turns out the code does not only check for negative values, but also for overly high values (>100,000). Shamefully the latter occurs before the values are multiplied by 3600, so while the intention is there it does not prevent the extreme usage spikes people are seeing.

Edit: strike that improvement remark. As it turns out the table referenced is apparently cleaned as it is written and thus only contains the last 24 hour of data.
Excito B3 running Gentoo Linux, P1, Rfxtrx433 to read and control TFA, KaKu, EvoHome RFG100
Custom patched Domoticz v3.8000

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 16:05

Logging in UTC is not the problem for the shortlog, we just use UTC timestamps instead of local, and where fine.
Converting UTC to local is 'just' a sqlite query thing, no problem at all... and we know the local of the system... easy !

But... here comes the tricky part, at midnight, we might want to calculate the total amount of energy/gas/water used, or log the day value for a temperature sensor
This has to be done at the users local time (for example GMT+1)

Maybe also this is not the big problem....

And we need to convert the current shortlog values (one time) at startup to UTC... maybe thats also a simple SQLite query

Maybe it is not difficult at all ;)... only time is

If someone want to try it, please go ahead
Quality outlives Quantity!

Post Reply

Who is online

Users browsing this forum: galinette and 6 guests