Memory Leakage?

Please use template to report bugs and problems. Post here your questions when not sure where else to post
Only for bugs in the Domoticz application! other problems go in different subforums!
Forum rules
Before posting here, make sure you are on the latest Beta or Stable version.
If you have problems related to the web gui, clear your browser cache + appcache first.

Use the following template when posting here:

Version: xxxx
Platform: xxxx
Plugin/Hardware: xxxx
Description:
.....

If you are having problems with scripts/blockly, always post the script (in a spoiler or code tag) or screenshots of your blockly

If you are replying, please do not quote images/code from the first post

Please mark your topic as Solved when the problem is solved.
User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Sunday 31 December 2017 12:32

EddyG wrote:
Sunday 31 December 2017 12:29
I took in my chart the difference with the start of domoticz, so the graph tells you how much it increases from the initial Kb's
I just had to reboot my Raspberry Pi, so I start again and let it running for some days now.
What do you mean, if you mean the entire lenght from start till end then this chart does show from start to end.
Or how do you setup those charts?
See attachment for my log file.
Attachments
Memory_Leakage.zip
(1.41 KiB) Downloaded 11 times
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
EddyG
Posts: 279
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9788
Location: Rhenen, Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Sunday 31 December 2017 13:40

Your charts looks like below. It's simple, use formulas. Or substract in every relevant column the first data, and that make a graph from the new set.
Your VSZ does not grow, and your RSS grow's in the beginning, but seems to settle later on.
Attachments
2017-12-31_12-36-01.png
2017-12-31_12-36-01.png (16.44 KiB) Viewed 852 times
Regards,
Eddy

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Sunday 31 December 2017 15:54

Thx @EddyG

But still weird why my system settles it's vmem.
It's normal that the RSS is increasing as the log gets filled with lines and some other stuff.
This will be released overtime but the main cause is the Vmem which is stable now on my system.
Have both systems RPi3 production and test system running the script.
Will let it run and see what the outcome is next year or so ;-)
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
EddyG
Posts: 279
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9788
Location: Rhenen, Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Monday 01 January 2018 11:50

EddyG wrote:
Sunday 31 December 2017 13:40
Your VSZ does not grow, and your RSS grow's in the beginning, but seems to settle later on.
You read it wrong.
Regards,
Eddy

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 02 January 2018 1:59

EddyG wrote:
Monday 01 January 2018 11:50
EddyG wrote:
Sunday 31 December 2017 13:40
Your VSZ does not grow, and your RSS grow's in the beginning, but seems to settle later on.
You read it wrong.
Maybe...
Will show my new log tomorrow.
Seems stable to me.

btw:
Are you running ChopperRob his python device online script?
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
EddyG
Posts: 279
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9788
Location: Rhenen, Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Tuesday 02 January 2018 9:21

No I do not have the ChopperRob script running.
It's an external script, how should that increase the memory of Domoticz.
The only way I could thing of that the API call is doing something weird internally in Domoticz.
I do a lot of 'external' API calls from different scripts to Domoticz.
Is it worth investigating?
Regards,
Eddy

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 02 January 2018 10:11

It is worth it, thats why i'm asking.
I had a few external scripts which used APi calls and stopped using them a few days ago.
Since then the leakage stabilized.
At this point i don't know if it's related or a lucky shot.
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
heggink
Posts: 483
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.9530
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Tuesday 02 January 2018 11:23

Did any of you run valgrind? It's pretty simple and quickly shows where the leak is.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 02 January 2018 11:29

heggink wrote:
Tuesday 02 January 2018 11:23
Did any of you run valgrind? It's pretty simple and quickly shows where the leak is.
Yep, but seems i can't get it running as it spits out a lot of errors while google doesn't provide me the answers i need.
And as asked several times over the last few weeks in github to those who have nice valgrind reports haven't/ can't / will answer.
There for it's seeking for a needle in a haw stack for me...
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
heggink
Posts: 483
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.9530
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Tuesday 02 January 2018 11:31

I can probably recompile my Ubuntu test system with the latest sw and run valgrind for you. Just give me a test case. Ever since the latest fix, my memory is completely stable after 2 weeks of running.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 02 January 2018 11:43

heggink wrote:
Tuesday 02 January 2018 11:31
I can probably recompile my Ubuntu test system with the latest sw and run valgrind for you. Just give me a test case. Ever since the latest fix, my memory is completely stable after 2 weeks of running.
Are you running Ubuntu on a RPi3, If so which version?
I run Rasbian Jessie and tested Stretch and on both i can't seem to get valgrind working.
Appreciate the offer but would be nice if i can get it running to debug myself.

What do you wanna know as testcase?

*thinking
Could it be Rasbian has a issue causing the leak while other OSs don't.
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
heggink
Posts: 483
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.9530
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Tuesday 02 January 2018 11:46

Ah, ok. I have Ubuntu on an Intel box where I can test. I have no Pi to test against so if the problem is raspberry specific then no joy :-(. If generic then I can test.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 02 January 2018 11:51

heggink wrote:
Tuesday 02 January 2018 11:46
Ah, ok. I have Ubuntu on an Intel box where I can test. I have no Pi to test against so if the problem is raspberry specific then no joy :-(. If generic then I can test.
I see, thought so due to you Signature.
Then its no joy indeed...
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
heggink
Posts: 483
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.9530
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Tuesday 02 January 2018 11:53

My 2 prod systems are RPi's. As indicated, no issues for me atm.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
Siewert308SW
Posts: 607
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

Re: Memory Leakage?

Post by Siewert308SW » Tuesday 02 January 2018 12:27

heggink wrote:
Tuesday 02 January 2018 11:53
My 2 prod systems are RPi's. As indicated, no issues for me atm.
Which OS do you run on your prod Rpi?
Does Valgrind work for you on your RPi?

At the moment reviewing both my RPI3 system and rebuilding step by step from scratch.
Also reviewing my scripts to avoid any curl in bash scripts and simplify my Lua scripts.
Took the dust of my old rpi2 and moved Pihole and my auto weather tweet script to it to avoid any disturbance.
Both RPi3 and RPi2 running Rasbian Lite Jessie see screenshot below.
Both RPi3 have a PIco UPS HV3.0a stacked, RPi2 doesn't.
Rpi3's contain a from source build Domoticz up to date till this morning.
And running rootfs from a external HDD.

I have setup both system as following:

1. Installed Jessie Lite
2. sudo raspi-config and setup timezone and password
3. sudo raspberry update && sudo raspberry upgrade
4. Installed the following packages:

Code: Select all

libudev-dev
libpython3.
cmake
make
gcc
g++
libssl-dev
git
curl
libcurl4-openssl-dev
libusb-dev
wiringpi
libsqlite3-dev
subversion
zlib1g-dev
libssl-dev
git-core
build-essential
libblkid-dev
e2fslibs-dev
libboost-all-dev 
libaudit-dev

python-rpi.gpio
python3-dev 
python-serial
python-smbus
python-jinja2
python-xmltodict
python-psutil
python-pip
i2c-tools
5. sudo apt-get install rpi-update && sudo rpi-update
6. sudo nano /etc/dphys-swapfile
CONF_SWAPFILE=1024
7. sudo reboot

8. Ran my own script to install and set PIco UPS
picotoolkit.zip
(8.35 KiB) Downloaded 13 times
9.
cd ~

Code: Select all

git clone https://github.com/OpenZWave/open-zwave open-zwave-read-only
cd open-zwave-read-only
git pull
make -j 3
cd ..
10.

Code: Select all

git clone https://github.com/domoticz/domoticz.git domoticz
cd domoticz
git pull
cmake -DCMAKE_BUILD_TYPE=Debug . CMakeLists.txt
make -j 2
cd ..

(buildtime 35min without errors)
11.

Code: Select all

cd domoticz
sudo cp domoticz.sh /etc/init.d
sudo chmod +x /etc/init.d/domoticz.sh
sudo update-rc.d domoticz.sh defaults
12.

Code: Select all

sudo ./domoticz
sidenote:
As for the build from source wiki.
I can't use the latest boost library as mentioned there due the raspbian repo isn't up to date and giving issue.
Schermafdruk 2018-01-02 11.25.08.png
Schermafdruk 2018-01-02 11.25.08.png (64.22 KiB) Viewed 751 times
Setup:
- RPi3 - PIco HV3.0A / Domo Beta 3.9XXX / RFXtrx433E / Aeotec Gen5
- RPi3 - PIco HV3.0A / PiHole / PiVPN / NAS / Print Server
- Youless Elec&Gas
- FI9803P Cams
- KD101 detectors
- Zwave & KaKu

MyDomoticz scripts: @GitHub
PIco UPS Toolkit: @GitHub

User avatar
heggink
Posts: 483
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.9530
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Tuesday 02 January 2018 13:00

I'll have to take a look when I get home but I compiled domoticz on my pi3 before with no issues (not with debug enabled). I run the latest raspbian version.
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
EddyG
Posts: 279
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9788
Location: Rhenen, Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Saturday 06 January 2018 13:03

Version: V3.8153 (stable)
Platform: Linux domoticzslave 4.9.72-v7+ #1069 SMP Fri Dec 29 17:21:52 GMT 2017 armv7l GNU/Linux
Hardware: Raspberry Pi 3 with P1 USB and RFLink V48

I left the system running with switching between Eventsystem and dzVents enabled/disabled, see picture.
The graph represents the memory difference between current and first memory data.
From that I conclude that the Eventsystem is causing the Memory Leakage.
I have no idea what caused the system crash in the middle of the night. Normally this does not happen.
Attachments
2018-01-06_11-48-12.png
2018-01-06_11-48-12.png (64.93 KiB) Viewed 682 times
Regards,
Eddy

User avatar
EddyG
Posts: 279
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9788
Location: Rhenen, Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Saturday 06 January 2018 19:21

I used valgrind on my production system with Domoticz version 3.8798
At the end of the valgrind log I got 2 errors from libarmmem.so.
This is the info: -rw-r--r-- 1 root root 18920 Jul 8 2015 /usr/lib/arm-linux-gnueabihf/libarmmem.so

Code: Select all

disInstr(arm): unhandled instruction: 0xF1010200
                 cond=15(0xF) 27:20=16(0x10) 4:4=0 3:0=0(0x0)
==29383== valgrind: Unrecognised instruction at address 0x48426f4.
==29383==    at 0x48426F4: ??? (in /usr/lib/arm-linux-gnueabihf/libarmmem.so)
==29383== Your program just tried to execute an instruction that Valgrind
==29383== did not recognise.  There are two possible reasons for this.
==29383== 1. Your program has a bug and erroneously jumped to a non-code
==29383==    location.  If you are running Memcheck and you just saw a
==29383==    warning about a bad jump, it's probably your program's fault.
==29383== 2. The instruction is legitimate but Valgrind doesn't handle it,
==29383==    i.e. it's Valgrind's fault.  If you think this is the case or
==29383==    you are not sure, please let us know and we'll try to fix it.
==29383== Either way, Valgrind will now raise a SIGILL signal which will
==29383== probably kill your program.
==29383==
==29383== Process terminating with default action of signal 4 (SIGILL)
==29383==  Illegal opcode at address 0x48426F4
==29383==    at 0x48426F4: ??? (in /usr/lib/arm-linux-gnueabihf/libarmmem.so)
==29383== Use of uninitialised value of size 4
==29383==    at 0x4B2C934: free_mem (in /lib/arm-linux-gnueabihf/libc-2.19.so)
==29383==    by 0x4B2C3C3: __libc_freeres (in /lib/arm-linux-gnueabihf/libc-2.19.so)
==29383==    by 0x4023633: _vgnU_freeres (vg_preloaded.c:61)
==29383==    by 0x52281B: CNotificationHelper::AddNotifier(CNotificationBase*) (in /home/pi/domoticz/domoticz)
==29383==  Uninitialised value was created by a stack allocation
==29383==    at 0x4001BE4: handle_ld_preload (rtld.c:983)
==29383==
==29383==
==29383== HEAP SUMMARY:
==29383==     in use at exit: 129,072 bytes in 3,887 blocks
==29383==   total heap usage: 4,562 allocs, 675 frees, 149,731 bytes allocated
==29383==
==29383== 516 bytes in 23 blocks are possibly lost in loss record 3 of 9
==29383==    at 0x4833970: malloc (vg_replace_malloc.c:263)
==29383==    by 0x7D8E07: operator new(unsigned int) (in /home/pi/domoticz/domoticz)
==29383==
==29383== LEAK SUMMARY:
==29383==    definitely lost: 0 bytes in 0 blocks
==29383==    indirectly lost: 0 bytes in 0 blocks
==29383==      possibly lost: 516 bytes in 23 blocks
==29383==    still reachable: 128,556 bytes in 3,864 blocks
==29383==         suppressed: 0 bytes in 0 blocks
==29383== Reachable blocks (those to which a pointer was found) are not shown.
==29383== To see them, rerun with: --leak-check=full --show-reachable=yes

Anybody any idea???
Regards,
Eddy

User avatar
heggink
Posts: 483
Joined: Tuesday 08 September 2015 21:44
Target OS: Raspberry Pi
Domoticz version: V3.9530
Location: NL
Contact:

Re: Memory Leakage?

Post by heggink » Saturday 06 January 2018 19:25

Not really other than that it seems to be a domoticz crash. For how long did it run? No memory was lost so far...
Pi3, latest beta
RFXCOM, z-wave (AEOTEC, switches, temhum, pir, contacts),
Plugwise2py, P1 'smart'meter & solar panel
Alexa, Wifi Cams motion detection
ESP: relays, PIR & Temp/TempHum
Geofence iCloud, Bluetooth & Wifi ping
Harmony hub, Nest

User avatar
EddyG
Posts: 279
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 4.9788
Location: Rhenen, Netherlands
Contact:

Re: Memory Leakage?

Post by EddyG » Saturday 06 January 2018 20:02

It did not run, it crashed.
I found a solution by disabling the preload of libarmmem.so
Just an other question: how should I stop valgrind with domoticz running.
Only a sudo kill -9 seems to work.
Regards,
Eddy

Post Reply

Who is online

Users browsing this forum: JimmyH1969 and 10 guests