Weird values because of daylight saving time (summer/winter)

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
Plantje
Posts: 128
Joined: Friday 16 October 2015 7:58
Target OS: Windows
Domoticz version:
Contact:

Weird values because of daylight saving time (summer/winter)

Post by Plantje » Sunday 25 October 2015 23:18

This must have been asked a dozen of times before, but I wasn't able to find it.

Last night in the Netherlands we went from summer to winter time. Nothing too fancy. I have my Domoticz connected to my Eneco Toon and in the graph I saw really weird readings. 10800 kW or something like that...

In the database I didn't see those strange readings neither in the backups. Now I read that using shift I can remove values. So I did...and now all values for 2.00 - 3.00 are gone from the database as well. What is happening here? How can that be fixed?
Last edited by ThinkPad on Monday 26 October 2015 11:35, edited 2 times in total.
Reason: Made topictitle a bit more clear, moved to bugs subforum, made sticky

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values for summer --> winter time

Post by gizmocuz » Monday 26 October 2015 10:07

Its a problem we are facing for a long time now, and hope someone could fix this....
its because the data gets logged twice between 02:00 and 03:00
Quality outlives Quantity!

Plantje
Posts: 128
Joined: Friday 16 October 2015 7:58
Target OS: Windows
Domoticz version:
Contact:

Re: Weird values for summer --> winter time

Post by Plantje » Monday 26 October 2015 10:09

Any clue to why this is only visible in the graph? All values in the database seem normal. And now the values in the database are gone, because I deleted them from the graph :)

I'd like to put them back using SQLite, but I think I'll have to stop the Domoticz service/program on my server first.
Edit: just noticed I cannot put them back anymore, because the data of Sunday morning has been purged.
Last edited by Plantje on Monday 26 October 2015 10:16, edited 1 time in total.

SweetPants
Posts: 1607
Joined: Friday 12 July 2013 21:24
Target OS: Linux
Domoticz version: V3.8789
Location: The Netherlands
Contact:

Re: Weird values for summer --> winter time

Post by SweetPants » Monday 26 October 2015 10:16

gizmocuz wrote:Its a problem we are facing for a long time now, and hope someone could fix this....
Our governement can if they stop using summer/wintertime. In a mostly 24x7 economy I don't see the use of it anyway.

Plantje
Posts: 128
Joined: Friday 16 October 2015 7:58
Target OS: Windows
Domoticz version:
Contact:

Re: Weird values for summer --> winter time

Post by Plantje » Monday 26 October 2015 10:18

Agree :)

User avatar
ThinkPad
Posts: 1754
Joined: Tuesday 30 September 2014 8:49
Target OS: Linux
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Weird values for summer --> winter time

Post by ThinkPad » Monday 26 October 2015 11:29

Maybe this has something useful: http://stackoverflow.com/a/2532962
and https://msdn.microsoft.com/en-us/librar ... ime_topic6

https://en.wikipedia.org/wiki/Daylight_ ... Complexity
A common strategy to resolve these problems in computer systems is to express time using the Coordinated Universal Time (UTC) rather than the local time zone. For example, Unix-based computer systems use the UTC-based Unix time internally.
https://en.wikipedia.org/wiki/Daylight_ ... #Computing
ThinkTheme - theme for Domoticz
My (Dutch) blog: http://thinkpad.tweakblogs.net - My Domoticz scripts: Bitbucket
I'm not (very) active anymore on this forum as i don't use Domoticz anymore.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by gizmocuz » Thursday 29 October 2015 20:31

i dont think it will be a difference if using utc...

because if the clock goes back one hour... you still get duplicate entries

the clock just goes back one hour.... so we have one more hour to log ....
Quality outlives Quantity!

User avatar
cyberclwn
Posts: 105
Joined: Thursday 20 August 2015 22:53
Target OS: Raspberry Pi
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by cyberclwn » Friday 30 October 2015 0:18

Hi,

For UTC versus GMT see http://www.timeanddate.com/time/gmt-utc-time.html

Spoiler: GMT is a timezone, UTC is a standard. So UTC doesn't go back 1 hour. The timestamp in the DB will only occur once.
3xPi 2B (Domoticz "live", Domoticz "sandbox", PhotoFrame)
RFXCom433(E), KaKu, Oregon Scientific, Keyes 2-relay, Logitech Media Server, MiLight, Smartwares heating controller(2x), IR Send/Receive, Keyes PIR, XH-M131 DuskSensor, DHT22/11

dannybloe
Posts: 1086
Joined: Friday 29 August 2014 11:26
Target OS: Raspberry Pi
Domoticz version:
Location: Ermelo
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by dannybloe » Friday 30 October 2015 9:52

Interesting... didn't know this UTC thingy. But that would certainly solve it. You always display the charts in the perspective of the current time-zone/daylight-savings setting so you will never have this problem. The only thing is that wherever you display something that is time related, you have to convert it to the locale settings of the user.
Creator dzVents - RPi3, loads of zwave devices, esp8266, evohome.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by gizmocuz » Friday 30 October 2015 9:54

dannybloe wrote:Interesting... didn't know this UTC thingy. But that would certainly solve it. You always display the charts in the perspective of the current time-zone/daylight-savings setting so you will never have this problem. The only thing is that wherever you display something that is time related, you have to convert it to the locale settings of the user.
i dont think you have read my comment above, or did i miss something there?
Quality outlives Quantity!

dannybloe
Posts: 1086
Joined: Friday 29 August 2014 11:26
Target OS: Raspberry Pi
Domoticz version:
Location: Ermelo
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by dannybloe » Friday 30 October 2015 10:17

:-)

I think you did. At first I was with you on this but when I thought about it a bit longer I wasn't anymore:

The thing is that with UTC is there is no daylight saving. Every sample that you store in the database is UTC based. There will be no sample with that same UTC time-stamp (per sensor of course). So let's say we have sensor that is sampled once every hour. If it is 2:59 you take a sample and it will get a UTC time stamp of let's say UTCx (don't want to look it up right now). Then, at 3:00 the clock is set back to 2:00 and after 59 minutes you take another sample. Our clocks will show 2:59 again but UTC time will be UTCx+60 minutes. A similar thing happens in spring when you move the clock forward. GMT is just a representation. Given a time-zone and a UTC you can convert it to the current 'display' time of the viewer (user) and only then you apply DST settings.

Code: Select all

Database:
T (GMT)        T (UTC)       Value
-----------------------------------
<when time is moved set back in autumn>
01:59          UTCx-60       24C
02:59          UTCx          25C
02:59          UTCx+60       26C
03:59          UTCx+120      27C

... 
<when time is moved forward in spring>
01:59          UTCy          23C
03:59          UTCy+60       25C
04:59          UTCy+120      25C


The only challenge you may have is with charts. So in the example above you have two readings UTCx and UTCx+59 (the first taken at 02:59 and the second again at 02:59(wintertime)). How do you display this in the charts? But that is more a UI problem than a storage problem.
Last edited by dannybloe on Friday 30 October 2015 11:02, edited 2 times in total.
Creator dzVents - RPi3, loads of zwave devices, esp8266, evohome.

User avatar
ThinkPad
Posts: 1754
Joined: Tuesday 30 September 2014 8:49
Target OS: Linux
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by ThinkPad » Friday 30 October 2015 10:58

Good explanation. I think UTC is the best option for this. I think that professional systems also work this way.
ThinkTheme - theme for Domoticz
My (Dutch) blog: http://thinkpad.tweakblogs.net - My Domoticz scripts: Bitbucket
I'm not (very) active anymore on this forum as i don't use Domoticz anymore.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by gizmocuz » Friday 30 October 2015 11:31

Stupid me.... indeed your right !

Got some rebuilding todo (graphs/logs/timers etc...)

regarding the graphs (and timers/and others) i think we just convert it in the get/set functions in domoticz
Quality outlives Quantity!

Timple
Posts: 27
Joined: Tuesday 20 January 2015 9:05
Target OS: Linux
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by Timple » Friday 30 October 2015 11:36

Agreed, UTC is more useful for programs.

Everything in the database can be UTC and only at places where the user has to look at the times (graphs, clock, log-display) a conversion is made to the timezone of the user.

So a user that does not look in the database directly (most endusers) will never know ;)

Bonus: if you have a domoticz system in your caravan and drive over a timezone border, everything will still work :mrgreen:

dannybloe
Posts: 1086
Joined: Friday 29 August 2014 11:26
Target OS: Raspberry Pi
Domoticz version:
Location: Ermelo
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by dannybloe » Friday 30 October 2015 11:57

Timple wrote:Agreed, UTC is more useful for programs.
Bonus: if you have a domoticz system in your caravan and drive over a timezone border, everything will still work :mrgreen:
Yeah, thought about that but is that what you expect? After all, the sensors are still in the same timezone. Wouldn't you want to know what time it was for the sensor when it took the reading? If I am in Indonesia and look at my sensors at home (The Neths) and I see that a sensor took a reading of 16C at 02:00. I think I still want it to show 02:00 instead of 11:00 (Indonesian time). After all, I want to know that it was night-time at the location of the sensor. Perhaps I want a switch somewhere on the page to show the log/charts/times/whatever in my local time zone, as an option.
Creator dzVents - RPi3, loads of zwave devices, esp8266, evohome.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by gizmocuz » Friday 30 October 2015 19:43

Okey, but.... we need to check for a day change with localtime (and add the daily logs), because otherwise for example the smartmeter results are (when you are utc+1) from 23:00 - 23:00 local time, and you dont want this

But all very much doable...
Quality outlives Quantity!

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by Toni » Friday 30 October 2015 20:45

The general rule of thumb for anything related to timestamps is:
  • ALWAYS internally store all timestamps in the UTC format, or for example in Unix time
  • ONLY convert the timestamps to local time when showing the timestamp to the user
This way it will always work correctly. If it's done in any other way, it will always be wrong in some cases.

For example, when it comes to the time when the clocks are changed, the internal time just rolls on. It's only the representation of time in the local format which changes:
  • 1445734799 = Last second of the summer time this year on Eastern European time = 3.59.59 GMT+3:00 DST
  • 1445734800 = First second of the winter time this year on Eastern European time = 3.00.00 GMT+2:00
There are other 'weird things', like 26th of October 2015 is 25 hours long in local time. And in the springtime, the corresponding day is only 23 hours long in local time. But the UTC time just moves on, second by second. Never jumps anywhere, except when extra leap seconds are added.

Week numbers are another strange thing, when you want to express the week and the year of a certain date. For example, the week & year of 1st Jan 2010 is 'week 53 of 2009'. Or 29st Dec 2014 is 'week 1 of 2015'. This is by the standard ISO-8601.

User avatar
gizmocuz
Posts: 8526
Joined: Thursday 11 July 2013 18:59
Target OS: Raspberry Pi
Domoticz version: beta
Location: Top of the world
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by gizmocuz » Friday 30 October 2015 20:51

True, but do you think the above post makes sense ?

so when the day changes 'locally' we add the logs for the day logs, but using the sql query to convert the date stamps in local ?

select datetime(timestamp, 'localtime')

... or something, have to google first how to retrieve a database field in localtime
Quality outlives Quantity!

Toni
Posts: 86
Joined: Monday 20 July 2015 14:12
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by Toni » Saturday 31 October 2015 11:13

It is a bit difficult to get your head around the way the time works. But in the case of Domoticz, I think it's enough if it follows the timezone of the server, instead of the web browser (assume that you use Domoticz from the same place where you installed it).

I added some pictures of a system which does it right, i.e. the Round Robin Database. The oct25 picture is from 00:00 to 05:00, and you still see SIX hours in the graph (which is correct btw). The mar29 picture is from 00:00 to 07:00, and again you see SIX hours in the graph, again correctly.
Attachments
thgr228ntemp_mar29.png
thgr228ntemp_mar29.png (16.22 KiB) Viewed 4094 times
thgr228ntemp_oct25.png
thgr228ntemp_oct25.png (16.67 KiB) Viewed 4094 times

ubfssF
Posts: 43
Joined: Monday 02 November 2015 15:12
Target OS: Linux
Domoticz version: 2.2364
Location: Netherlands
Contact:

Re: Weird values because of daylight saving time (summer/win

Post by ubfssF » Monday 16 November 2015 10:29

At https://en.wikipedia.org/wiki/List_of_t ... time_zones one sees that a timezonedatabase is available. The timezone database itself can be retrieved at: ftp://ftp.iana.org/tz/releases/tzdata2015g.tar.gz
These files, orginized per continent, contain current and earlier timezone arrangements for a certain country (or parts of a country, when it has more than one timezone).

Storing all data with an UTC-timestamp&location and presenting it is therefore is possible, when one would be able to parse the current timezonedata for each country or part of a country.
Only, the files are a drag to plough through.......

What should be feasible, is to take the list on https://en.wikipedia.org/wiki/List_of_t ... time_zones and to use the columns TZ* and UTC DST offset. If one would add a picklist in settings with the timezones, DMZ would be able to present the data in the local timezone.

Still, at the time daylight saving time starts or stops, DMZ wouldn't be able to present the data fully. However, it would give correct data when one would ask the data of the full day or week, when one would keep the non-DST times for those.

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests