Memory Leakage?

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post
User avatar
EddyG
Posts: 145
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8789
Location: Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Sunday 12 November 2017 17:03

What's your script to get the Domoticz memory usage?
Regards,
Eddy

DutchHans
Posts: 280
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Memory Leakage?

Post by DutchHans » Sunday 12 November 2017 17:58

There were more than 700 updates in the beta channel after 8073.

Activate hardware monitor - motherboard..than you will have several hardware sensors activated.

Cheers ...

DutchHans
Posts: 280
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Memory Leakage?

Post by DutchHans » Wednesday 15 November 2017 22:30

IMG_20171115_212705.png
IMG_20171115_212705.png (125.94 KiB) Viewed 720 times
It drops when Domoticz is restarted (after an update)..but than rising again...
Any news?..or anything i can do to help?
Cheers,Hans

Pjedr
Posts: 104
Joined: Friday 27 December 2013 4:13
Target OS: Linux
Domoticz version: Beta
Location: Friesland
Contact:

Re: Memory Leakage?

Post by Pjedr » Friday 17 November 2017 19:50

Same with me.
Now i am up to latest beta again.
Turned events off for 1 hour.
Now turned it on.
I am afrain something in lua is eating memory, cpu and time.
Need to check later.

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Monday 20 November 2017 21:40

domo_mem.jpg
domo_mem.jpg (51.87 KiB) Viewed 633 times
Slowly happening to me as well. As of Nov 9th, memory increases by 1.7% per day. Not massive but, before that, it was completely stable, no increase at all from oct 20th to Nov 9th. Weird.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

RFMuser
Posts: 19
Joined: Wednesday 13 April 2016 13:03
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Memory Leakage?

Post by RFMuser » Monday 20 November 2017 22:58

Hi all, a few weeks ago I crashed the RPI3 while updating, trying to fight mem problem. So I ended up imaging the SD card again with Jessie light.

Linux raspberrypi 4.9.41-v7+ #1023 SMP Tue Aug 8 16:00:15 BST 2017 armv7l GNU/Linux

After a new Domoticz install
Version: 3.8564
Build Hash: 09e677d
Compile Date: 2017-10-10 00:48:08
dzVents Version: 2.3.0

I recovered the database so most of the settings were the same. I stopped using LUA and setup some dzVents scripts.

I keep an sharp eye on the mem but as you can see a flat line all the time...
chart.png
chart.png (66.75 KiB) Viewed 628 times

User avatar
Siewert308SW
Posts: 499
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 21 November 2017 18:48

Quick question...
Those who suffer from increasing memory.
Are you using (enabled) http aswell as https port when you setup Domoticz?
Setup:
1x RPi3 - PIco hv3.0A Plus / Domoticz
1x RPi3 - PIco hv3.0A Plus / PiHole
1x RFXtrx433E / Aeotec ZW090
3x FI9803P Cam
2x Youless Elec/Gas
4x KD101 detectors
a lot of KaKu/z-wave stuff

Domoticz scripts: @ GitHub
pico_installer: @ GitHub

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Tuesday 21 November 2017 18:50

I am.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
Siewert308SW
Posts: 499
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 21 November 2017 19:31

heggink wrote:
Tuesday 21 November 2017 18:50
I am.
Thx but i'm also curious id there are users who don't use https.
I asked this as for 2 years ago there was also a leak which had a connection to https.
I disabled the port a few hours ago, it is to early to say but maybe this issues has been popped up again.
For now the memory looks stable but will report tomorrow about my findings.
Setup:
1x RPi3 - PIco hv3.0A Plus / Domoticz
1x RPi3 - PIco hv3.0A Plus / PiHole
1x RFXtrx433E / Aeotec ZW090
3x FI9803P Cam
2x Youless Elec/Gas
4x KD101 detectors
a lot of KaKu/z-wave stuff

Domoticz scripts: @ GitHub
pico_installer: @ GitHub

DutchHans
Posts: 280
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Memory Leakage?

Post by DutchHans » Tuesday 21 November 2017 20:36

I am not sure if I installed https ..how can I check (rpi3)

Regards, Hans

User avatar
Siewert308SW
Posts: 499
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Thursday 23 November 2017 19:44

DutchHans wrote:
Tuesday 21 November 2017 20:36
I am not sure if I installed https ..how can I check (rpi3)

Regards, Hans
I bit late ;-)
But what does your log say?

Code: Select all

WebServer(HTTP) started on address: :: with port 8080
WebServer(HTTP) started on address: :: with port 433
After a few days of testing i don't suspect https as the main culprit.
Yesterday i took a second spare Pi3 with a clean Rasbian Jessie and Domoticz beta installation.
The memory usage was stable after 20hr.
A few hours ago i installed libpython3. which is needed to be able to fire up the python plugin system.
Since then the memory is rising and current mem usage after 2hr is already higher then i saw after 20hr idle

edit:
Already asked is someone on github with more experiance could take a look.
I do suspect libpython3. which is used for enabling python plugin system could be the culprit.
After a running my 2 Rpi3 system idle overnight the one with libpython3. without script has a memory increase.
https://github.com/domoticz/domoticz/issues/1955
Setup:
1x RPi3 - PIco hv3.0A Plus / Domoticz
1x RPi3 - PIco hv3.0A Plus / PiHole
1x RFXtrx433E / Aeotec ZW090
3x FI9803P Cam
2x Youless Elec/Gas
4x KD101 detectors
a lot of KaKu/z-wave stuff

Domoticz scripts: @ GitHub
pico_installer: @ GitHub

User avatar
Siewert308SW
Posts: 499
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Saturday 25 November 2017 0:24

After a closer look on both my system it looks like the culprit isn't the libpython3. install
Both system do have a increase in Memory usage.
But the test Pi3 didn't increase that fast as it isn't running any scripts.
For now i disabled the event system on my test Pi3 to see what happens in the upcoming hours.

10min after i disabled the event system i already see a drop in memory usage.
It dropped from 5.89 to 5.6% and still decreasing.
Take in mind no scripts running.
After a few hours without the event system enabled the memory dropped even more till 4.9% and is now running stable.
Will let it run overnight and see what happens.

But someone with more experience do have to look where the culprit is.
As on even a bare system without scripts the memory increases and decreases then run stable if the eventsystem is disabled.
Setup:
1x RPi3 - PIco hv3.0A Plus / Domoticz
1x RPi3 - PIco hv3.0A Plus / PiHole
1x RFXtrx433E / Aeotec ZW090
3x FI9803P Cam
2x Youless Elec/Gas
4x KD101 detectors
a lot of KaKu/z-wave stuff

Domoticz scripts: @ GitHub
pico_installer: @ GitHub

RFMuser
Posts: 19
Joined: Wednesday 13 April 2016 13:03
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Memory Leakage?

Post by RFMuser » Saturday 25 November 2017 21:16

Hi,

like in my previous post another screenshot.
Knipsel.PNG
Knipsel.PNG (65.77 KiB) Viewed 449 times
Pretty flat line. I experienced the mem problem already for a long time. The only thing I changed, after the fresh install detailed above with the same database was, I use dzVentz with the event system active. Now 3 scripts are running.

So you might try a fresh install, and your actual database without LUA. Than run a while and see if it stays flat. If it does, try some dzVentz test scripts. If it remains stable run some LUA scripts and check again. I'm wondering what happens...

Pchatill
Posts: 51
Joined: Wednesday 16 September 2015 5:52
Target OS: Raspberry Pi
Domoticz version: V2.3774
Contact:

Re: Memory Leakage?

Post by Pchatill » Monday 04 December 2017 13:31

Hi.

For information and if that can help:
I have this probleme since beginning of november when I updated Raspbian from Wheezy to Jessie,
and in the same time updated domoticz from last stable to V3.8702.
No others changes in domoticz config.

DutchHans
Posts: 280
Joined: Friday 03 April 2015 20:44
Target OS: Raspberry Pi
Domoticz version:
Location: Germany (near dutch border)
Contact:

Re: Memory Leakage?

Post by DutchHans » Thursday 07 December 2017 16:08

A little update for the developers:

I have Domoticz 3.8771 on a RPI3 (Jessie) with about 200 devices and eventsystem running (No DZvents) and memory is increasing.
I have (for test) another RPI3 (Stretch) with no devices and no eventsystem running and memory is also increasing.

Looks like its not the eventsystem...

Regards, Hans

User avatar
jannl
Posts: 905
Joined: Thursday 02 October 2014 6:36
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Geleen
Contact:

Re: Memory Leakage?

Post by jannl » Thursday 07 December 2017 16:25

Wondering....what happens when the memory is "full", does the (or some) garbage collector clean it up? Or does the system actually crash?

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Thursday 07 December 2017 16:26

System stops. If a process allocates mem, no garbage collector can free anything other than cached.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Thursday 07 December 2017 16:35

Btw, you can use monit to monitor process and system and restart processed when they exceed a threshold. Prevents locks.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
jannl
Posts: 905
Joined: Thursday 02 October 2014 6:36
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Geleen
Contact:

Re: Memory Leakage?

Post by jannl » Thursday 07 December 2017 16:42

And you actually tested that?

I mean, I know that if a process allocates memory and does not free it, the system will stop.

I am just wondering why that does not happen at my system. I just have a crontab that frees some unused memory regularly and the system runs as long a I let it.

edit: actually "echo 3 > /proc/sys/vm/drop_caches" is what I do

User avatar
heggink
Posts: 337
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.7067
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Thursday 07 December 2017 17:55

Yes, have it up and running. Driver was the blockly mem bug. Caused my system to lock up.
In terms of clearing cache, on linux there is no need and it's bad practice (some people use the drop_caches) since it interferes with linux own capabilities that do precisely what it needs to.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

Post Reply

Who is online

Users browsing this forum: Quax1507 and 6 guests