Weird values because of daylight saving time (summer/winter)

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
OLLIFDG
Posts: 6
Joined: Saturday 20 February 2016 14:48
Target OS: Raspberry Pi
Domoticz version: V2.3530
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by OLLIFDG » Saturday 20 February 2016 16:45

I'm having a time issue. Perhaps it's easy to solve but until now I did not find a solution.
There is a time difference from exactly 1 hours in the Netherlands. Domoticz system is saying 14:44 and the time is one hours later.
How can I fix this? :oops:

Thanks

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Saturday 20 February 2016 17:18

did you run

sudo raspi-config

and set the correct time zone ?

Did you install the ntp package and synced the time ?
Quality outlives Quantity!

OLLIFDG
Posts: 6
Joined: Saturday 20 February 2016 14:48
Target OS: Raspberry Pi
Domoticz version: V2.3530
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by OLLIFDG » Saturday 20 February 2016 18:15

gizmocuz wrote:did you run

sudo raspi-config

and set the correct time zone ?

Did you install the ntp package and synced the time ?

I only set the location degrees.
I'm going to see how to do the sudo, ntp and synching time.
I'm new. Do you now a link where I can find how to do this?

Thanks

I managed to change in config via putty.

Thanks again

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Monday 28 March 2016 10:48

A new beta #5019 is released to solve the summer time issue (2 values on one day) till next year
Quality outlives Quantity!

RidingTheFlow
Posts: 78
Joined: Friday 11 March 2016 19:23
Target OS: Raspberry Pi
Domoticz version:
Location: Essex, UK
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by RidingTheFlow » Tuesday 29 March 2016 0:20

Toni wrote:The general rule of thumb for anything related to timestamps is:
  • ALWAYS internally store all timestamps in the UTC format, or for example in Unix time
  • ONLY convert the timestamps to local time when showing the timestamp to the user
This way it will always work correctly. If it's done in any other way, it will always be wrong in some cases.
.
I second this. I did a lot of programming with timestamps and dealt with a lot of daylight related bugs.

But its easy to solve really (well, its easy recipe, but still lot of work) - you just need process all data internally as UTC. E.g. if you want to calculate meter update, store it in DB, pass data around via JSON, etc - you always use UTC.

You only convert to local when you about to print value in UI.
And you only convert from local directly when getting value from input box.

It all UTC otherwise, and UTC is continuous time same in all regions of the world and no DST periods - so its always consistent and no gaps in duration calculations and such.

Though from my experience it quite a tremendous work to convert project (like Domoticz) which was all using local time to all using UTC :(

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 29 March 2016 9:38

Time is the problem.....

And i do not think it is a lot of work... how many tables we have which uses date/time ? 10 ? 15? Not much

Most work will be converting back/forward user interface stuff like timers, blockly stuff

But all needs to be tested very good, and also we should convert the current date/times to UTC in the database
Quality outlives Quantity!

RidingTheFlow
Posts: 78
Joined: Friday 11 March 2016 19:23
Target OS: Raspberry Pi
Domoticz version:
Location: Essex, UK
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by RidingTheFlow » Tuesday 29 March 2016 10:41

Databases are not hard, but I would be more concerned about all time-using code spread around in various places both on C++ and JS side. Because all that needs to be changed for a proper fix - mostly places where "current time" is obtained, but some times when interfacing with some other library/API/hardware - and sometimes this will have option to use UTC and should do that, but some times will have to convert to/from local time.

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 29 March 2016 10:56

from and to the web gui should be converted, that is all, but the most work
Quality outlives Quantity!

rafal98
Posts: 4
Joined: Tuesday 03 November 2015 10:42
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by rafal98 » Tuesday 29 March 2016 12:09

Hi, I also have an issue following the daylight savint time where both days are stacked. I'm not sure it's a new problem. Here a screenshot

Regards
Screenshot domoticz.png
Screenshot domoticz.png (231.99 KiB) Viewed 1463 times

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 29 March 2016 12:24

gizmocuz wrote:A new beta #5019 is released to solve the summer time issue (2 values on one day) till next year
Quality outlives Quantity!

louisL
Posts: 4
Joined: Thursday 11 August 2016 23:14
Target OS: Linux
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by louisL » Friday 12 August 2016 0:15

The usual way for using times is often to store times as the number of seconds since the epoch (00:00 hours, Jan 1, 1970 UTC) (time_t) or as a struct tm containing a broken down time and date. See http://www.cplusplus.com/reference/ctime/ for more information. The best solution often is to use time_t (the second counter) in the database and use gmtime or localtime for presentation. The conversion to localtime will use the system's timezone setting to convert the time_t data to the local time. So if you are in another timezone the display will still use whatever timezone setting is used on the server. During daylight saving switches you will still have days of 23 or 25 hours though. Leap seconds can be taken care of in these interfaces as well they may be included on the system's timezone definitions.

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Friday 12 August 2016 8:53

Its stored as time_t in the database
But we need to store all in UTC, and convert to localtime in presentation
Time... (changing the code) is sparse ;)
Quality outlives Quantity!

User avatar
BartdaMan
Posts: 22
Joined: Sunday 22 May 2016 22:58
Target OS: Raspberry Pi
Domoticz version:
Location: Amersfoort
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by BartdaMan » Sunday 30 October 2016 13:39

Well, although I am using V3.5834 I think the clock change from 03 to 02 hours last night caused some weird readings:
2016-10-30_123836.png
2016-10-30_123836.png (98.39 KiB) Viewed 1111 times
Any clue how this happened?
And how to fix it?

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Sunday 30 October 2016 15:37

"how to fix this"

There have to be made some code changes for this to get solved
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Monday 31 October 2016 23:35

Seen this, and others, and probably can't be fixed.

For example most logfiles [eg. /var/log/messages] also have duplicate logging.

Better stop the winter-summer-wintertime changes :-)

It looks a bit weird for sure, however its only cosmeticly wrong.

As mentioned earlier, when its recorded correctly, and daily totals (haven't checked but was ok i believe) are correct. Don't spend time trying to solve a cosmetic misbehaviour.

Grtz Stephan

Sent from my D6603 using Tapatalk

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 8:56

Sure it can be fixed if we use UTC for logging... have to watch out for the daily log
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Tuesday 01 November 2016 13:08

Like stated before: switching to UTC should be the correct option. (Thought it was, however missed that part probably)

Any server installed by me will be setup in UTC. And depending on the locale (Holland W-CET = GMT +1 or Great Brittain GMT + 0 = UTC) displaying the locale date/time correctly. And depending on Daylight Savings On it will be WCET = GMT + 1 ({Daylight On} +1).
So anyway.... When recording with the UTC timestamp it will record data correctly with time and value(s) from the sensors. And they do NOT overlap eachother when going from daylight off to on or daylight on to off. Thats the database-part.

The real problem is displaying: And to read the records from the database, apply the local timezone and after thats done, you need to concatenate the values when talking about the smartmeters and rfxmeters. However eg. themperature-sensors.... these values need to be accumulate and divided by 2 to get a more realistic value.

Perhaps this will solve the (Cosmetic)display-issue for anybody not familiar with timezones and the underlying datastructure.

Sent from my K00C using Tapatalk

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 16:05

Logging in UTC is not the problem for the shortlog, we just use UTC timestamps instead of local, and where fine.
Converting UTC to local is 'just' a sqlite query thing, no problem at all... and we know the local of the system... easy !

But... here comes the tricky part, at midnight, we might want to calculate the total amount of energy/gas/water used, or log the day value for a temperature sensor
This has to be done at the users local time (for example GMT+1)

Maybe also this is not the big problem....

And we need to convert the current shortlog values (one time) at startup to UTC... maybe thats also a simple SQLite query

Maybe it is not difficult at all ;)... only time is

If someone want to try it, please go ahead
Quality outlives Quantity!

stlaha2007
Posts: 437
Joined: Monday 05 October 2015 10:16
Target OS: -
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by stlaha2007 » Tuesday 01 November 2016 18:35

@gordonb3: Hopefully we can get some point we both agree on the concatenate ;-)
@gizmocus: Hope to understand correctly: Logging data in db is now time_t and not already in UTC?

Would love to try to build it. Can't be hard ;-) However source is in C# or C++? No experience... Pascal/Perl/Python is more to my liking.

Anyway...
Can we agree, as mentioned by others, to record withh a UTC timestamp.
And solve the display errors make a few extra settings in the configuration:
- Option to use systems TimeZone Yes/No
- Display Time on Sensors and/or Graphs based on UTC or Local TimeZone

There was also a reply above displaying and mrtg-like graph with an extra hour... Looks odd, however still correct...


Sent from my K00C using Tapatalk

User avatar
gizmocuz
Posts: 8560
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/winter)

Post by gizmocuz » Tuesday 01 November 2016 21:34

I would suggest having a look at the code (c++) to see how the datetime is handled... part of it is automated by triggers
Quality outlives Quantity!

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests