Memory Leakage?

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
User avatar
EddyG
Posts: 145
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8789
Location: Netherlands
Contact:

Memory Leakage?

Post by EddyG » Thursday 31 August 2017 12:10

Version: 3.8153
Platform: Linux domoticz 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux update to the lastest Debian version
Plugin/Hardware: Raspberry PI 3 with RFXCOM, RazBerry, etc....
Description:
It looks like Domoticz has some memory leakage. I restart Domoticz every morning at approx. 05:30 as is visible.
Memory consumption grow during the day.
chart.png
chart.png (70.58 KiB) Viewed 1768 times
B.t.w. I made this image from the context menu. It would be helpfull to have the name of the device added to the picture.
Like on Domoticz it self.
Regards,
Eddy

SweetPants
Posts: 1624
Joined: Friday 12 July 2013 21:24
Target OS: Linux
Domoticz version: V3.8793
Location: The Netherlands
Contact:

Re: Memory Leakage?

Post by SweetPants » Thursday 31 August 2017 14:07

Are you sure this is a memory leakage? 4 to 8% memory usage is low.
Linux systems tent to build a cache in memory, hence the increase after a reboot.
Try to wait a little longer and see if it reaches 100%, then you might have a memory leak.
Ubuntu 16.04.3 LTS on Intel NUC, MySensors, Pro Mini 3.3 with RFM69, ESP8266 (SDK2.1.0), Sonoff, RFLink, RFXcom.

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Thursday 31 August 2017 17:38

If you want to investigate, put an entry in crontab to list meminfo every hour or so and dump that in a file (or better, create a domoticz device). simplest to issue:
egrep 'MemAvailable' /proc/meminfo | awk '{ print (1-($2 / 947732)) * 100}'
Do the same for the domoticz process:
ps aux|grep "domoticz/domoticz"|grep -v grep| awk '{print $4}'
And see what happens.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
EddyG
Posts: 145
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8789
Location: Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Friday 01 September 2017 12:58

The above graph is from the output of:

Code: Select all

MEMUSG=`top -b -n1 | grep domoticz | awk '{print $10}'`
and that is close to the output of the suggested ps command.
My normal memory usage is stable, the only one growing is the memory use of Domoticz.
Domoticz is restarted every morning by logrotate.
The Domoticz log is going to a tmpfs and is growing fast, so I have to do that every day.
Is it possible to logrotate without a restart of Domoticz?
Below the picture from the last year.
As you can see I updated Domoticz to the latest stable in the beginning of August.
chart (1).png
chart (1).png (102.46 KiB) Viewed 1684 times
Regards,
Eddy

jvandenbroek
Posts: 134
Joined: Wednesday 12 March 2014 17:37
Target OS: Raspberry Pi
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Memory Leakage?

Post by jvandenbroek » Tuesday 05 September 2017 16:06

EddyG wrote:
Friday 01 September 2017 12:58
The above graph is from the output of:

Code: Select all

MEMUSG=`top -b -n1 | grep domoticz | awk '{print $10}'`
and that is close to the output of the suggested ps command.
My normal memory usage is stable, the only one growing is the memory use of Domoticz.
Domoticz is restarted every morning by logrotate.
The Domoticz log is going to a tmpfs and is growing fast, so I have to do that every day.
Is it possible to logrotate without a restart of Domoticz?
Below the picture from the last year.
As you can see I updated Domoticz to the latest stable in the beginning of August.

chart (1).png
You can send kill -HUP signal to only reload the logging system, with the supplied init file you can simply call domoticz.sh reload.
What / how many hardware are you using? It could help to disable them one at a time to see if the usage will no longer increase and report back here, so we can take a look at the specific hardware's code. RFXCom and Razberry are unlikely to be the cause, I also use those, so these can be skipped (or you should try disabling them at last).

User avatar
EddyG
Posts: 145
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8789
Location: Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Tuesday 05 September 2017 16:35

The system is a production system, so I cannot, for longer time, disable hardware.
The hardware I am using is RFXCOM, RaZberry, Weather, Motherboard, SolarEdge, Milight LAN, System Alive Checker, WOL, and a lot of dummies for all kind of sensors, like this one.
I will try the -HUP signal

I run a almost empty TEST system with only Motherboard and some dummy sensors.
That system does not consume extra memory.
I investigate further to find the "bad apple"

I have a slave system with other hardware. I will implement the same Domoticz Memory usage dummy sensor and see if I can narrow it down.
Regards,
Eddy

User avatar
EddyG
Posts: 145
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8789
Location: Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Wednesday 06 September 2017 18:41

I use -HUP now with logrotate.
The memory consumption is steady growing.
I use the following to find the part that is growing:

Code: Select all

sudo memstat -p 17065 > memstat.txt
after some time:
sudo memstat -p 17065 > memstat1.txt
and find the differance with:
diff memstat.txt memstat1.txt
The only part that is growing is the domoticz process.
Currently it is more than ever before.
I keep this setting going for some days to see how much memory will be consumed.
Regards,
Eddy

User avatar
EddyG
Posts: 145
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8789
Location: Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Thursday 07 September 2017 13:05

Currently Domoticz memory usage has grown to more then 11 %.
I also have a production slave system with less hardware.
The slave is also growing in Domoticz memory usage but at regular intervals like 4-5 hours with a step.
Unfortunately it is not a fixed interval.
I also have a TEST system with only the motherboard and 2 dummy sensors.
The TEST system is NOT growing in memory consumption.

So to me it looks like a real problem. The only challenge is how to discover what is causing this behaviour.

I keep it running for a few more days. After that I revert back to stopping and starting Domoticz every day.
Regards,
Eddy

manjh
Posts: 252
Joined: Saturday 27 February 2016 13:49
Target OS: Raspberry Pi
Domoticz version: 3.8153
Location: NL
Contact:

Re: Memory Leakage?

Post by manjh » Saturday 09 September 2017 21:34

I agree, there is a slight memory leak somewhere. I run Domoticz on a Pi-3, and the RAM usage graph shows a gradual but consistent increase.
The graph shows a sharp dip, this is where I rebooted the Pi (for other reasons).
As you can see the memory usage is gradually increasing.
I will keep an eye on it.

Image
Attachments
Schermafdruk 2017-09-09 21.27.27.png
Schermafdruk 2017-09-09 21.27.27.png (276.22 KiB) Viewed 1501 times
Hans

pj-r
Posts: 245
Joined: Wednesday 17 December 2014 18:30
Target OS: Linux
Domoticz version: V3.8650
Location: Jyväskylä, Finland
Contact:

Re: Memory Leakage?

Post by pj-r » Tuesday 12 September 2017 7:37

@manjh: What HW you have running? Scripts, blocky etc? This info could be helpful for developers.
LXC(x64 Ubuntu Xenial), RFXtrx433E, MySensors

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Wednesday 13 September 2017 12:12

Unfortunately for you (fortunately for me) I cannot confirm. With the previous beta, I did see a gradual increase during the day for the entire but the usage would drop after midnight to the base (not related to logrotate). The domoticz process usage has been very stable for me around 4.5%.

http://imgur.com/a/ubqrR
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

manjh
Posts: 252
Joined: Saturday 27 February 2016 13:49
Target OS: Raspberry Pi
Domoticz version: 3.8153
Location: NL
Contact:

Re: Memory Leakage?

Post by manjh » Monday 18 September 2017 21:19

pj-r wrote:
Tuesday 12 September 2017 7:37
@manjh: What HW you have running? Scripts, blocky etc? This info could be helpful for developers.
Memory usage increases from 8% to 10% over the past four days. I understand that 10% is nothing to worry about, but the slope of the graph is: is goes up consistently.
I use the Pi for Domoticz only, run lua scripts (not Blockly).
Hans

pj-r
Posts: 245
Joined: Wednesday 17 December 2014 18:30
Target OS: Linux
Domoticz version: V3.8650
Location: Jyväskylä, Finland
Contact:

Re: Memory Leakage?

Post by pj-r » Wednesday 20 September 2017 13:06

What HW do you use? RFXTrx, z-wave...?
LXC(x64 Ubuntu Xenial), RFXtrx433E, MySensors

jvandenbroek
Posts: 134
Joined: Wednesday 12 March 2014 17:37
Target OS: Raspberry Pi
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Memory Leakage?

Post by jvandenbroek » Wednesday 20 September 2017 15:10

manjh wrote:
Monday 18 September 2017 21:19
Memory usage increases from 8% to 10% over the past four days. I understand that 10% is nothing to worry about, but the slope of the graph is: is goes up consistently.
I use the Pi for Domoticz only, run lua scripts (not Blockly).
Are you sure there are no errors in de log which might indicate that Lua scripts never finishes? Something like "warning script took more than 10 seconds to execute", it's not guarantied that they will ever finish (thus holding the allocated memory). I suppose you could test it easily by temporary disabling all Lua scripts and see what happens in a few hours. 8 to 10% is way too much, my experience is that it will almost stay the same after a few min after starting up.

@manjh I now see that you're not the TS. You mentioned rebooting the Pi, but then we don't know if it's Domoticz causing the climb. Does restarting just Domoticz result in the same dip? In that case, try what I said above (disabling Lua).

DutchHans
Posts: 280
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Memory Leakage?

Post by DutchHans » Thursday 21 September 2017 21:08

Same here since i reverted to the latest stable.
It wasn't there in the beta versions

Grtz,Hans

RFMuser
Posts: 19
Joined: Wednesday 13 April 2016 13:03
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Memory Leakage?

Post by RFMuser » Friday 29 September 2017 20:54

Hi all,

I confirm the same issue here. See graph below. I noticed behavior first around a year ago when I was running beta's because the system stopped working and discovered the high memory consumption up to almost 100%. Then stepped over to the latest stable version back then. Still the memory was increasing.
I found that the problem was coming from the "blocky" scripting. I migrated the few scripts I had, to "LUA" and the problem was solved for month's. Suddenly, after the latest stable release(without changing anything) the memory increase started again. As you can see below. You can also see the point where I stop the event system and the mem increase stops immediately. Unfortunately the log doesn't reveal anything. I also switched on "debug" level logging but no errors are posted.
To be clear all this memory is used by the Domoticz process.

Running on RPI3
Version: 3.8153
Build Hash: 494fff7
Compile Date: 2017-07-30 12:19:41

I I'm a little suspicious on the JSON virtual device updates, but not sure why this would be linked to the event system, although some events use this virtual sensors data. I use a lot of these virtual sensors which get updated by wired Arduino gateways using the JSON urls.

Buts since it seems other people experience the same issue this might not be the problem, unless they use the same setup. Anyway I will try to switch on the event system and disable the "Arduino gateway" devices to see if the mem leak stops.

Hope we can find the culprit as it looks like a long term problem.

Small update: I also switched off all the scripts, but left the event system active and still the memory increases...
Attachments
mem_leak.PNG
mem_leak.PNG (228.32 KiB) Viewed 1089 times

RFMuser
Posts: 19
Joined: Wednesday 13 April 2016 13:03
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Memory Leakage?

Post by RFMuser » Friday 06 October 2017 19:33

I've tested some things but the only way to stop the memory increase is to disable the event system from the settings. Of course this is not a solution.
So for now I'm stuck. I narrowed it down to the event system but further analysis probably requires source code knowledge. Even disabling the scripts individually doesn't help. I'm wondering who to contact about the issue.
At the moment working on a Domoticz implementation in my new build project. Altough Domoticz is a very stable and flexible system I'm a little concerned about this persistent issue.

RFMuser
Posts: 19
Joined: Wednesday 13 April 2016 13:03
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Memory Leakage?

Post by RFMuser » Saturday 07 October 2017 22:50

To be sure I deleted all scripts from the event system. Also removed all timers and notifications just to be sure. So the event system is running idle. As is visible in the attachment the memory increase continues unchanged. This time a screenshot from the Domoticz process only.
Attachments
Screenshot_20171007-200011-03.jpeg
Screenshot_20171007-200011-03.jpeg (42.37 KiB) Viewed 928 times

jvandenbroek
Posts: 134
Joined: Wednesday 12 March 2014 17:37
Target OS: Raspberry Pi
Domoticz version: beta
Location: The Netherlands
Contact:

Re: Memory Leakage?

Post by jvandenbroek » Tuesday 10 October 2017 17:38

DutchHans wrote:
Thursday 21 September 2017 21:08
Same here since i reverted to the latest stable.
It wasn't there in the beta versions

Grtz,Hans
Thanks for this info, so I guess for now just use the beta version to fix this issue, or patiently wait for a new stable to be released ;)

HenFre
Posts: 18
Joined: Thursday 12 May 2016 15:48
Target OS: Raspberry Pi
Domoticz version: 3.8153
Location: Sweden
Contact:

Re: Memory Leakage?

Post by HenFre » Wednesday 11 October 2017 10:58

Running on RPI 3
Version: 3.8153
Build Hash: 494fff7
Compile Date: 2017-07-30 12:19:41

I have got the same problem since I installed the stabile version in sept.
Domoticz runs for about 5 days and then the systems hangs and I have to restart it!

I have used dzVents from the first edition and still doe's.

Added comment:
Today I inactivated script running for 8 h but the memory uses still increases in the same speed.
.
2017-10-11 Minnesanvändning.JPG
2017-10-11 Minnesanvändning.JPG (243.65 KiB) Viewed 861 times
//Henry
Domoticz on Raspberry PI 3 since 2016.

Post Reply

Who is online

Users browsing this forum: Quax1507 and 4 guests