backup every 12 hours on qnap ftp

On various Hardware and OS systems: pi / windows / routers / nas, etc
Post Reply
guantolento
Posts: 172
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

backup every 12 hours on qnap ftp

Post by guantolento » Wednesday 21 December 2016 1:45

good evenig. actually i use the domoticz on my rpi2, i have some script LUA made with the built in domoticz and some script in BASH in another folder.
i want to create a script to save on an external ftp a full backup. in the wiki i can see the procedure to create the script, but i don't understand what folder i need to backup. What do you recommend ? a full SD backup or some single folder ? thanks a lot.
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

User avatar
Egregius
Posts: 2446
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 2:29

I only backup the complete domoticz folder to a versioned system with rsync.

guantolento
Posts: 172
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

Re: backup every 12 hours on qnap ftp

Post by guantolento » Wednesday 21 December 2016 16:04

How you can save in this way? ???what do you setting in domoticz? ????

Inviato dal mio GT-I9301I utilizzando Tapatalk
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

User avatar
emme
Posts: 644
Joined: Monday 27 June 2016 11:02
Target OS: Raspberry Pi
Domoticz version: latest
Location: Milano, Italy
Contact:

Re: backup every 12 hours on qnap ftp

Post by emme » Wednesday 21 December 2016 16:24

Egregius wrote:I only backup the complete domoticz folder to a versioned system with rsync.
I'm curious too....
I tried an FTP copy but there are a lot of locked files (the db for example :P)
That's one small step for a programmer, one giant leap for me!
- - - - -
Raspberry Pi3 + AeonStick G5 + RFXCOMM433 + iRTrans Server + NodeJS
There's no stress in the mess!!

User avatar
Egregius
Posts: 2446
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 17:19

rsync doesn't seem to have issues with that.

Code: Select all

SOURCE="/volume1/@appstore/domoticz/"
DESTINATION="/volume1/homes/guy/backup/domoticz"
LOGFILE=/volume1/web/secure/backups/logs/domoticz.$NOW.txt
echo ------------------------------------ START domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
rsync -aP --exclude-from '/volume1/homes/guy/backup/excludedfiles.txt' --stats --delete-after --ignore-errors --links --link-dest="$DESTINATION/__prev/" "$SOURCE" "$DESTINATION/$NOW" | tee -a $LOGFILE
rm -f "$DESTINATION/__prev"
ln -s "$NOW" "$DESTINATION/__prev"
echo ------------------------------------ END  domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
Creates a folder in DESTINATION for each day the script has run.
Folder size is only the updated files as the rest are symlinks.

I do have an exclusion list wich hold this:

Code: Select all

var/domoticz.db-shm
var/domoticz.db-wal
var/domoticz.log
var/domoticz.pid
var/ozwcp.poll.XXXXXX.xml
var/ozwcp.topo.XXXXXX
var/zwscene.xml
pi/domoticz/domoticz.db-shm
pi/domoticz/domoticz.db-wal
pi/domoticz/domoticz.log
pi/domoticz/Config/OZW_Log.txt
pi/domoticz/Config/zwscene.xml
2 paths in it because I also backup a Domoticz Pi from a friend

User avatar
sincze
Posts: 729
Joined: Monday 02 June 2014 22:46
Target OS: Linux
Domoticz version: 3.8153
Location: Netherlands
Contact:

Re: backup every 12 hours on qnap ftp

Post by sincze » Wednesday 21 December 2016 18:48

I can confirm rsync is working.... nicely
or you can run a .sh ...
- stop domoticz.
- backup
- start domoticz.

:D
Using Pass2php since 2016-12
LAN: RFLink, P1-Port, OTGW, MySensors
USB: RFXCom, ZWave
WIFI: Mi-light Wifi-Bridge, Sonoff, ESP8266, Xiaomi Gateway
Solar: Omnik Inverter, PVOutput
Video: Kodi clients with Harmony HUB
Sensors: You name it I probably got 1.

User avatar
Egregius
Posts: 2446
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 19:21

No need to stop domoticz, backup is complete with rsync without stopping it.

SweetPants
Posts: 1622
Joined: Friday 12 July 2013 21:24
Target OS: Linux
Domoticz version: V3.8791
Location: The Netherlands
Contact:

Re: backup every 12 hours on qnap ftp

Post by SweetPants » Wednesday 21 December 2016 19:48

Egregius wrote:No need to stop domoticz, backup is complete with rsync without stopping it.
isn't that dangerous with an open database?
Ubuntu 16.04.3 LTS on Intel NUC, MySensors, Pro Mini 3.3 with RFM69, ESP8266 (SDK2.1.0), Sonoff, RFLink, RFXcom.

User avatar
Egregius
Posts: 2446
Joined: Thursday 09 April 2015 12:19
Target OS: Linux
Domoticz version: Beta
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 19:59

I restored at least 10 times without issues. And IF one got corrupt I'l pick one from the day before :D

guantolento
Posts: 172
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

Re: backup every 12 hours on qnap ftp

Post by guantolento » Saturday 24 December 2016 19:56

Merry christmas,
actually i take this action:
- disable the automatic backup from the web gui of domoticz
- create a bash script to backup every day at 13:00 "domoticz" folder
- create a bash script to backup every day at 13:30 "script" folder
- create a bash script to backup every day at 14:00 domoticz database
This script save to an ftp QNAP-GL01, the datastore of this is on 2 hard disk in raid 1 (mirror). The datastore of QNAP-GL01 is replicated on QNAP-GL02 by a process rsync every 12 hours. Someone can tell me if i need to save another folder from the RPI2 ???? thanks a lot.
Last edited by guantolento on Saturday 24 December 2016 19:59, edited 1 time in total.
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

guantolento
Posts: 172
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

Re: backup every 12 hours on qnap ftp

Post by guantolento » Saturday 24 December 2016 19:57

emme wrote:
Egregius wrote:I only backup the complete domoticz folder to a versioned system with rsync.
I'm curious too....
I tried an FTP copy but there are a lot of locked files (the db for example :P)
i don't have this problem, i save to my FTP
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

litfiba
Posts: 7
Joined: Wednesday 07 December 2016 19:48
Target OS: Linux
Domoticz version: 3.7521
Contact:

Re: backup every 12 hours on qnap ftp

Post by litfiba » Thursday 29 December 2016 15:27

Hi guys,

A simple script I use to bachup DB or entire Domoticz folder.. Daily, weekly, Monthly.. I keep only last three backup of each..

3 3 * * * /home/pi/domoticz/scripts/script_backup.sh J DB
3 4 * * 0 /home/pi/domoticz/scripts/script_backup.sh S DB
3 5 1 * * /home/pi/domoticz/scripts/script_backup.sh M DB
3 1 * * 1 /home/pi/domoticz/scripts/script_backup.sh S SYSTEM
3 2 2 * * /home/pi/domoticz/scripts/script_backup.sh M SYSTEM

#!/bin/bash
# LOCAL/FTP PARAMETERS
SERVER="192.168.0.xxx" # IP of Synology NAS, used for ftp
USERNAME="xxx" # FTP username of Network disk used for ftp
PASSWORD="xxx" # FTP password of Network disk used for ftp
DESTDIRNAS="/Public/domoticz/" # Path to your NAS backup folder
DOMO_IP="192.168.0.yyy" # Domoticz IP
DOMO_PORT="8080" # Domoticz port
DOMO_USER="yyy"
DOMO_PASS="yyy"
### END OF USER CONFIGURABLE PARAMETERS
param=$1 # 1st parameter will be for example 'J' for day, 'S' for week or 'M' for month
mode=$2 # 2nd parameter could be 'DB' for DB only backup or 'SYSTEM' for entire domoticz bacup
BACKUPFILE=$mode"_"$param"_1"
BACKUPFILEGZ="$BACKUPFILE".gz
##delete last - rename hold one
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS" -X "DELE "$mode"_"$param"_3.gz"
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER" -Q "RNFR $DESTDIRNAS/"$mode"_"$param"_2.gz" -Q "RNTO $DESTDIRNAS/"$mode"_"$param"_3.gz"
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER" -Q "RNFR $DESTDIRNAS/"$mode"_"$param"_1.gz" -Q "RNTO $DESTDIRNAS/"$mode"_"$param"_2.gz"
### Create backup and ZIP it
#service domoticz.sh stop # not needed
case "$mode" in "DB") # case of DB only backup
/usr/bin/curl -s http://$DOMO_USER:[email protected]$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE
#service domoticz.sh start # not needed
gzip -9 /tmp/$BACKUPFILE
### Send to Network disk through FTP
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILEGZ" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
### Remove temp backup file
/bin/rm /tmp/$BACKUPFILEGZ
;;
"SYSTEM") # case of entire domoticz backup
tar -zcvf /tmp/$BACKUPFILE.tar.gz /home/pi/domoticz/
#service domoticz.sh start # not needed
### Send to Network disk through FTP
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILE.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
### Remove temp backup file
/bin/rm /tmp/$BACKUPFILE.tar.gz
;;
esac
### Done!

jammoju
Posts: 1
Joined: Sunday 23 October 2016 10:39
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: backup every 12 hours on qnap ftp

Post by jammoju » Friday 27 January 2017 17:24

Hi litfiba, (or anyone else who can help)

I was hoping you could help and give some further explination as to how to get your script working for me.

I am very new to this so sorry if its a dumb question. I am having trouble trying to automate the backup via FTP to my QNAP NAS.

This is what i get when i try to run the script

* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250-%s DISK FREE SPACE AT THIS DIRECTORY IS ONLY 838,656 Bytes.
< 250 CWD command successful.
> CWD Domoticz
< 250-%s Disk free space at this directory is 906,605,604 KB.
< 250 CWD command successful.
> EPSV
* Connect data stream passively
< 500 EPSV not understood.
* Failed EPSV attempt. Disabling EPSV
> PASV
< 227 Entering Passive Mode (192,168,1,3,4,132).
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connecting to 192.168.1.3 (192.168.1.3) port 1156
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
> TYPE I
< 200 Type set to I.
> SIZE Backup
< 550 Backup: not a regular file.
> RETR Backup
< 550 Backup: Not a regular file
* RETR response: 550
* Remembering we are in dir "/Domoticz/"
* Connection #0 to host 192.168.1.3 left intact
curl: (78) RETR response: 550
* Rebuilt URL to: ftp://192.168.1.3/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> RNFR /Domoticz/Backup/DB_J_2.gz
* ftp_perform ends with SECONDARY: 0
< 550 /Domoticz/Backup/DB_J_2.gz: No such file or directory
* QUOT command failed with 550
* Closing connection 0
curl: (21) QUOT command failed with 550
* Rebuilt URL to: ftp://192.168.1.3/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> RNFR /Domoticz/Backup/DB_J_1.gz
* ftp_perform ends with SECONDARY: 0
< 550 /Domoticz/Backup/DB_J_1.gz: No such file or directory
* QUOT command failed with 550
* Closing connection 0
curl: (21) QUOT command failed with 550
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250-%s DISK FREE SPACE AT THIS DIRECTORY IS ONLY 838,656 Bytes.
< 250 CWD command successful.
> CWD Domoticz
< 250-%s Disk free space at this directory is 906,605,604 KB.
< 250 CWD command successful.
> PASV
* Connect data stream passively
< 227 Entering Passive Mode (192,168,1,3,4,133).
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connecting to 192.168.1.3 (192.168.1.3) port 1157
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
> TYPE I
< 200 Type set to I.
> STOR Backup
< 550 Backup: Not a regular file
* Failed FTP upload: 550
* Remembering we are in dir "/Domoticz/"
* Uploaded unaligned file size (0 out of 472 bytes)
* Connection #0 to host 192.168.1.3 left intact

As I said im very new to this but im trying to learn so any help is greatly appreciated.

Thank you...

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests