Script to restart Domoticz if it crashes

All kinds of 'OS' scripts
User avatar
Derik
Posts: 2100
Joined: Friday 18 October 2013 23:33
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Arnhem/Nijmegen Nederland
Contact:

Re: Script to restart Domoticz if it crashes

Post by Derik » Tuesday 03 October 2017 19:41

yep i use only a cable and no wifi..
Are tere the solutions to?
Xu4: Beta Extreme antenna RFXcomE,WU Fi Ping ip P1 Gen5 PVOutput Harmony HUE SolarmanPv OTG Winddelen Alive ESP Buienradar MySensors WOL Winddelen counting RPi: Beta SMAspot RFlinkTest Domoticz ...Different backups

User avatar
Egregius
Posts: 2667
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: Script to restart Domoticz if it crashes

Post by Egregius » Tuesday 03 October 2017 20:37

I mean the name of the network interface. You can find it with ifconfig. Useally that's eth0 or wlan0

User avatar
jannl
Posts: 955
Joined: Thursday 02 October 2014 6:36
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Geleen
Contact:

Re: Script to restart Domoticz if it crashes

Post by jannl » Tuesday 03 October 2017 21:27

Why stop the network interface? I use tons of unix/linux systems, including some pi's, I never stop/start a network interface unless I am working on it.

Verstuurd vanaf mijn SM-G930F met Tapatalk


User avatar
Egregius
Posts: 2667
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: Script to restart Domoticz if it crashes

Post by Egregius » Tuesday 03 October 2017 22:07

Because his device looses network connection. It's better to first try if a restart of network avoids the problem than rebooting.
http://www.domoticz.com/forum/viewtopic ... 20#p152494

User avatar
jannl
Posts: 955
Joined: Thursday 02 October 2014 6:36
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Geleen
Contact:

Re: Script to restart Domoticz if it crashes

Post by jannl » Tuesday 03 October 2017 22:37

But what is the cause of the network loss?

Verstuurd vanaf mijn SM-G930F met Tapatalk


Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: Script to restart Domoticz if it crashes

Post by Toulon7559 » Saturday 25 November 2017 18:51

Really a pain-in-the-ass if a 'counter-measure' is not reliable.
Besides a 'user' cron with more lines, in the 'sudo' cron I put a simple, single line for periodic reboot, but apparently even that does not work.
Even worse, sometimes I see that after the ordered reboot the applications/scripts do not start ( ;-) 'obviously' that occurs most of the times at the 'night-reboot' of 04:11).

Code: Select all

11 4/12/20 * * * sudo reboot
Presently don't see another practical solution than a 'habit' to remotely/manually reboot when opening respectively closing my PC-sessions:
stupid/rude, but at least a frequent kick to the Raspberries for clean start.

For other part not really a crash of Domoticz, but more a matter of one or more lua-script(s) which stop(s) running.
Stop occurring semi-randomly, sometimes at 00:00 and sometimes elsewhere.
Nuisance if you want continuity e.g. for power-meaurement or meteo-dataregistration (as described before).
Can usually be solved by the manual reboot, but nevertheless gaps in registration, because you are never in time to detect&close the gap.
Looked at Power Supply as possible source for corruption, but that seems OK.

Looking for other solutions:
1) Manual reboot always works through PuttySSH by manual CLI-command sudo reboot.
Is there a way to 'automate' such function as remote input from another computer?
2) Not use cron, but use a separate, dedicated script to order reboot (hoping :) that that 'reboot'-script stays running).
That 'reboot'-script acting on a flag from the suspect script which indicates heartbeat-present. Time-out on heartbeat = action = reboot-command
Any idea how to fit a 'flag' in the suspect script, which can be read as heartbeat-signal by the 'reboot'-script?
Simple&Rude Setup1 seems to be
a) the suspect script transmits it's time to a Dummy,
b) if the script stops, no further transmission and the time in the Dummy freezes
c) 'reboot'-script reads the Dummy and compares against current systemtime,
d) Defined difference = Time-out detection = order reboot.

Myself have been looking/experimenting on mechanisms directly getting&checking validity-time of data through JSON-calls (which Setup2 is essentally the same as Setup1, but without the Dummy as intermediary), but not yet found/made a reliable script: for status see here.
Setup1 and Setup2 could both be run locally at the Raspberry (for 'close-loop' reboot) or can be remotely run on another computer (for 'remote' reboot).
Last edited by Toulon7559 on Saturday 16 December 2017 12:24, edited 9 times in total.
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: Script to restart Domoticz if it crashes

Post by Toulon7559 » Monday 27 November 2017 12:18

Reading back info from a website as an indicator whether "the link & upload & the uploading computer is operational" has it's limitations, because sometimes the website or the receiving application extrapolates data, and then in the read-out you get info including validity time, which apparently is valid, although the related upload has stopped.

Perhaps another approach ...

My setup periodically uploads info to several websites.
- WUnderground and AWEKAS (for meteo-data) report after een time-out by mailnotification that upload is missing:
- WUnderground has a fixed time-out of 2 hours
- AWEKAS is even better with a time-out adjustable by the user.
- At PVOutput.org I found under the title Register Notification Service something which could be applicable, because you can make a http-call to see whether/which alert is applicable, but that function misses an alert for time-out.

Is there a way to link such incoming notification to one of the crash-scripts in this thread (or to another script that can remotely trigger a reboot)?
Obviously such mechanism only operates if the processor & Domoticz is still running, but only the subject uploading script has stopped.
If the processor really crashed, nothing helps but brute, manual forced reboot etc.
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

NetEye
Posts: 31
Joined: Friday 17 June 2016 7:57
Target OS: Raspberry Pi
Domoticz version: Beta
Location: Spijkenisse
Contact:

Re: Script to restart Domoticz if it crashes

Post by NetEye » Tuesday 06 February 2018 16:48

Hello,
When I have use one of the on my RPi 3, how do I check if it works?
Installation of it went well !
Test it by a command or something?
regards
Jan

User avatar
Derik
Posts: 2100
Joined: Friday 18 October 2013 23:33
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Arnhem/Nijmegen Nederland
Contact:

Re: Script to restart Domoticz if it crashes

Post by Derik » Tuesday 06 February 2018 16:54

Sudo service domoticz.sh stop

Status
Start
Restart
Xu4: Beta Extreme antenna RFXcomE,WU Fi Ping ip P1 Gen5 PVOutput Harmony HUE SolarmanPv OTG Winddelen Alive ESP Buienradar MySensors WOL Winddelen counting RPi: Beta SMAspot RFlinkTest Domoticz ...Different backups

ben53252642
Posts: 381
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: Script to restart Domoticz if it crashes

Post by ben53252642 » Wednesday 07 February 2018 2:24

I wrote an alternate version:

Code: Select all

#!/bin/bash
while true; do

domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:PASSWORD@192.168.0.5/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
if [[ $domoticzcheck != "OK" ]]; then
        echo "Domoticz not running, checking again in 30 seconds."
        sleep 30
        domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:PASSWORD@192.168.0.5/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
        if [[ $domoticzcheck != "OK" ]]; then
                echo "Watchguard is restarting Domoticz"
                service domoticz stop
                service domoticz start
        fi
else
echo "Domoticz is running..."
fi

sleep 10
done
Notes:
1) The curl command is set to allow unsigned SSL certificates.
2) You need change the service start and stop section of the script to the right name of your Domoticz service.
Eg if you type: "service domoticz stop" and that works on your system then great you got the right name. Some systems may have custom names that people manually specified or could be: "service domoticz.sh stop" as an example.
Last edited by ben53252642 on Wednesday 07 February 2018 14:54, edited 1 time in total.

User avatar
Derik
Posts: 2100
Joined: Friday 18 October 2013 23:33
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Arnhem/Nijmegen Nederland
Contact:

Re: Script to restart Domoticz if it crashes

Post by Derik » Wednesday 07 February 2018 8:16

ben53252642 wrote:
Wednesday 07 February 2018 2:24
I wrote an alternate version:

Code: Select all

#!/bin/bash
while true; do

domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:PASSWORD@192.168.0.5/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
if [[ $domoticzcheck != "OK" ]]; then
        echo "Domoticz not running, checking again in 30 seconds."
        sleep 30
        domoticzcheck=$(curl --max-time 60 -k -s "https://USERNAME:PASSWORD@192.168.0.5/json.htm?type=command&param=getversion" | grep '"status"' | awk '{ print $3 }' | sed 's/[\,"]//g')
        if [[ $domoticzcheck != "OK" ]]; then
                echo "Watchguard is restarting Domoticz"
                service domoticz stop
                service domoticz start
        fi
else
echo "Domoticz is running..."
fi

sleep 10
done
Note:
The curl command is set to allow unsigned SSL certificates.
mmm please can you tell me the different s between the other?
Where can i place this?
Need a crontab?
Xu4: Beta Extreme antenna RFXcomE,WU Fi Ping ip P1 Gen5 PVOutput Harmony HUE SolarmanPv OTG Winddelen Alive ESP Buienradar MySensors WOL Winddelen counting RPi: Beta SMAspot RFlinkTest Domoticz ...Different backups

ben53252642
Posts: 381
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: Script to restart Domoticz if it crashes

Post by ben53252642 » Wednesday 07 February 2018 14:40

The original version may fail to detect if Domoticz is really running in some circumstances due to is simply doing a port check.

The version in my above post makes a request to the Domoticz JSON API and only accepts Domoticz as running if it receives a response which confirms that it "is" actually running AND responding.

If it detects that Domoticz is not running, it waits 30 seconds, checks a second time and if the second attempt fails it then restarts Domoticz.

To use the script:

1) sudo apt-get install screen

2) nano /etc/init.d/watchguard

3) paste the contents below, editing the start section as needed to point to where you saved the watchguard.sh file

Code: Select all

#!/bin/sh
### BEGIN INIT INFO
# Provides: watchguard
# Required-Start: $network
# Required-Stop: $network
# Default-Start: 2 3 5
# Default-Stop:
# Description: Starts Watchguard
### END INIT INFO

case "$1" in
'start')
        sudo /usr/bin/screen -S watchguard -d -m sudo /root/scripts/watchguard/watchguard.sh
        ;;
'stop')
        sudo pkill -f "watchguard"
        ;;
*)
        echo "Usage: $0 { start | stop }"
        ;;
esac
exit 0
4) chmod 755 /etc/init.d/watchguard

5) update-rc.d watchguard defaults

6) reboot the system, then type: screen -x

If you see the watchguard running then you have successfully installed it.

If you need to stop the watchguard for example while updating Domoticz then use:

service watchguard stop

Toulon7559
Posts: 442
Joined: Sunday 23 February 2014 18:56
Target OS: Raspberry Pi
Domoticz version: latest
Location: Hengelo(Ov)/NL
Contact:

Re: Script to restart Domoticz if it crashes

Post by Toulon7559 » Sunday 18 February 2018 12:12

Earlier I reported on some 'difficulties' with the crash-recovery-script.

Sometimes a planned reboot completely fails, and that is easy to see, because Domoticz stops, and the Raspberry needs manual help for restart.
But sometimes the reboot-failure is only 'half' as demonstrated in this graph.

pitemp2_180218.png
graph_with_gap
pitemp2_180218.png (41.37 KiB) Viewed 1402 times
What puzzles, is that the effect is not consistent:
- 3 similar graphs with this kind of layout are generated at one and same Raspberry
- one graph usually happily continues after the planned periodic reboot
- the other 2 graphs (of which this is one) stop at a reboot and revive at the next reboot
The gap is not predictable and randomly happens.
- the only functional difference is that the 'continuing' graph is coming from 1 rrd-database-file, and that the other 2 graphs come from another rrd-database-file. Those 2 rrd-database-files are maintained by 2 different, but very comparable lua-scripts.

Seems a 'soft, unpredictable' problem, but those are the most difficult to solve .........
Set1 = RPI-B+RFXCom433+S0PCM+Linksprite-shield for BMP180/DS18B20/RS485+DDS238ZN1
Set2 = RPI-3+RFLinkGTW+ESP8266s+PWS_WS7000
Common = 2*PVLogger+PWS_TFA_Nexus+KAKUs
=> Energy & Data Management based on Time and on PV&Consumption&Meteo

pratty
Posts: 2
Joined: Saturday 28 April 2018 15:08
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: Script to restart Domoticz if it crashes

Post by pratty » Saturday 28 April 2018 16:44

Hi all, I am very new here but have been playing with domoticz for a number of months now.

I noticed a lot of people are stuck on where and how to implement a script to check domoticz.

I have written a little guide on how I have set it up.

Thanks to the handy work of so many people on this forum, it makes life very easy when issues are encountered with domoticz and the domoticz community is around to help.

The script i use is also very simple. I modified the earlier scripts that checks the domoticz state, but I only check if it is NOT equal to active running, therefore, any other state other than "active running" domoticz is restarted. I also get it to restart the homebridge service, as i have found that sometimes it stops responding after domoticz crashes. it obviously isn't a total solution but has been working well for me so far.

if you want to set it up this is how i have implemented it on my pi 3.

The script should work with a standard raspberry pi installation using raspbian.

go into domoticz scripts folder

Code: Select all

cd domoticz/scripts
then make a file called restart_domoticz.sh

Code: Select all

nano restart_domoticz.sh
copy and paste the script below

Code: Select all


#!/bin/bash

IsDomoticzRunning=`sudo systemctl status domoticz.service`

if [[ $IsDomoticzRunning != *"active (running)"* ]]
        then
                echo 'Domoticz is not running.Restarting Domoticz.'
                sudo systemctl stop domoticz.service
                sudo systemctl restart homebridge.service
                echo "Domoticz & Homebridge restarted."
                
	else
                echo "Domoticz is running. Nothing to do."
 fi
 
save the script by pressing CTRL x and then y and enter

then add execute permission to the script

Code: Select all

chmod +x restart_domoticz.sh
if all went well you should now be able to test your script by typing

Code: Select all

./restart_domoticz.sh
then all that is left to do is add to a cronjob to run every minute.

Code: Select all

crontab -e
then add the cronjob at the bottom of the file

Code: Select all

*/1 * * * * ./home/pi/domoticz/scripts/restart_domoticz.sh
save the file by pressing CTRL x and then y and enter.

A big thankyou to everyone who contributed to this idea to keeps domoticz running if or when it fails.

Hopefully, this can help everyone who is having trouble implementing these amazing ideas from the community.

User avatar
Egregius
Posts: 2667
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: Script to restart Domoticz if it crashes

Post by Egregius » Saturday 28 April 2018 20:53

A running service doesn’t mean it’s running ;)

ben53252642
Posts: 381
Joined: Saturday 02 July 2016 5:17
Target OS: Linux
Domoticz version: Beta
Contact:

Re: Script to restart Domoticz if it crashes

Post by ben53252642 » Saturday 28 April 2018 21:59

Egregius wrote:
Saturday 28 April 2018 20:53
A running service doesn’t mean it’s running ;)
Absolutely correct! You must use an alternate method that tests if the service is actually running.

I wrote a simple bash script here:
viewtopic.php?f=63&t=3016&sid=a4e9c3646 ... 40#p170196

Alternatively use the Monit setup:
https://www.domoticz.com/wiki/Monitorin ... gure_Monit

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest