backup every 12 hours on qnap ftp

On various Hardware and OS systems: pi / windows / routers / nas, etc
Post Reply
guantolento
Posts: 164
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

backup every 12 hours on qnap ftp

Post by guantolento » Wednesday 21 December 2016 1:45

good evenig. actually i use the domoticz on my rpi2, i have some script LUA made with the built in domoticz and some script in BASH in another folder.
i want to create a script to save on an external ftp a full backup. in the wiki i can see the procedure to create the script, but i don't understand what folder i need to backup. What do you recommend ? a full SD backup or some single folder ? thanks a lot.
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

User avatar
Egregius
Posts: 2378
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 2:29

I only backup the complete domoticz folder to a versioned system with rsync.

guantolento
Posts: 164
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

Re: backup every 12 hours on qnap ftp

Post by guantolento » Wednesday 21 December 2016 16:04

How you can save in this way? ???what do you setting in domoticz? ????

Inviato dal mio GT-I9301I utilizzando Tapatalk
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

User avatar
emme
Posts: 507
Joined: Monday 27 June 2016 11:02
Target OS: Raspberry Pi
Domoticz version: latest
Location: Milano, Italy
Contact:

Re: backup every 12 hours on qnap ftp

Post by emme » Wednesday 21 December 2016 16:24

Egregius wrote:I only backup the complete domoticz folder to a versioned system with rsync.
I'm curious too....
I tried an FTP copy but there are a lot of locked files (the db for example :P)
That's one small step for a programmer, one giant leap for me!
- - - - -
Raspberry Pi3 + AeonStick G5 + RFXCOMM433 + iRTrans Server + NodeJS
There's no stress in the mess!!

User avatar
Egregius
Posts: 2378
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 17:19

rsync doesn't seem to have issues with that.

Code: Select all

SOURCE="/volume1/@appstore/domoticz/"
DESTINATION="/volume1/homes/guy/backup/domoticz"
LOGFILE=/volume1/web/secure/backups/logs/domoticz.$NOW.txt
echo ------------------------------------ START domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
rsync -aP --exclude-from '/volume1/homes/guy/backup/excludedfiles.txt' --stats --delete-after --ignore-errors --links --link-dest="$DESTINATION/__prev/" "$SOURCE" "$DESTINATION/$NOW" | tee -a $LOGFILE
rm -f "$DESTINATION/__prev"
ln -s "$NOW" "$DESTINATION/__prev"
echo ------------------------------------ END  domoticz -- $(date +"%Y-%m-%d %H:%M:%S") UTC+2 | tee -a $LOGFILE
Creates a folder in DESTINATION for each day the script has run.
Folder size is only the updated files as the rest are symlinks.

I do have an exclusion list wich hold this:

Code: Select all

var/domoticz.db-shm
var/domoticz.db-wal
var/domoticz.log
var/domoticz.pid
var/ozwcp.poll.XXXXXX.xml
var/ozwcp.topo.XXXXXX
var/zwscene.xml
pi/domoticz/domoticz.db-shm
pi/domoticz/domoticz.db-wal
pi/domoticz/domoticz.log
pi/domoticz/Config/OZW_Log.txt
pi/domoticz/Config/zwscene.xml
2 paths in it because I also backup a Domoticz Pi from a friend

User avatar
sincze
Posts: 684
Joined: Monday 02 June 2014 22:46
Target OS: Linux
Domoticz version: 3.7546
Location: Netherlands
Contact:

Re: backup every 12 hours on qnap ftp

Post by sincze » Wednesday 21 December 2016 18:48

I can confirm rsync is working.... nicely
or you can run a .sh ...
- stop domoticz.
- backup
- start domoticz.

:D
Running Domoticz since 2013 with Cubietruck
LAN: RFLink, P1, OTGW, MySensors
USB: RFXCom, ZWave
WIFI: Mi-light Wifi-Bridge, Sonoff, ESP8266
Solar: Omnik Inverter, PVOutput
Video: Kodi clients with Harmony HUB
Sensors: You name it I probably got 1.

User avatar
Egregius
Posts: 2378
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 19:21

No need to stop domoticz, backup is complete with rsync without stopping it.

SweetPants
Posts: 1477
Joined: Friday 12 July 2013 21:24
Target OS: Linux
Domoticz version: V3.8394
Location: The Netherlands
Contact:

Re: backup every 12 hours on qnap ftp

Post by SweetPants » Wednesday 21 December 2016 19:48

Egregius wrote:No need to stop domoticz, backup is complete with rsync without stopping it.
isn't that dangerous with an open database?

User avatar
Egregius
Posts: 2378
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: backup every 12 hours on qnap ftp

Post by Egregius » Wednesday 21 December 2016 19:59

I restored at least 10 times without issues. And IF one got corrupt I'l pick one from the day before :D

guantolento
Posts: 164
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

Re: backup every 12 hours on qnap ftp

Post by guantolento » Saturday 24 December 2016 19:56

Merry christmas,
actually i take this action:
- disable the automatic backup from the web gui of domoticz
- create a bash script to backup every day at 13:00 "domoticz" folder
- create a bash script to backup every day at 13:30 "script" folder
- create a bash script to backup every day at 14:00 domoticz database
This script save to an ftp QNAP-GL01, the datastore of this is on 2 hard disk in raid 1 (mirror). The datastore of QNAP-GL01 is replicated on QNAP-GL02 by a process rsync every 12 hours. Someone can tell me if i need to save another folder from the RPI2 ???? thanks a lot.
Last edited by guantolento on Saturday 24 December 2016 19:59, edited 1 time in total.
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

guantolento
Posts: 164
Joined: Saturday 01 October 2016 14:48
Target OS: Raspberry Pi
Domoticz version: LastBeta
Location: Italy BG
Contact:

Re: backup every 12 hours on qnap ftp

Post by guantolento » Saturday 24 December 2016 19:57

emme wrote:
Egregius wrote:I only backup the complete domoticz folder to a versioned system with rsync.
I'm curious too....
I tried an FTP copy but there are a lot of locked files (the db for example :P)
i don't have this problem, i save to my FTP
Master (pi3) - MB sensors + Dummy + Sysfs gpio + 1 NodOn MSP-3 + 1 Aeotec DSB28 Energy Meter + 3 Danfoss RS Room + 7 Danfoss Z Thermostat + 1 Swiid Inter + 9 FGS222
Slave01 (pi2) - MB sensors + Dummy + Weather Underground + 2 FGS223 + 1 FGWPE/F

litfiba
Posts: 7
Joined: Wednesday 07 December 2016 19:48
Target OS: Linux
Domoticz version: 3.7521
Contact:

Re: backup every 12 hours on qnap ftp

Post by litfiba » Thursday 29 December 2016 15:27

Hi guys,

A simple script I use to bachup DB or entire Domoticz folder.. Daily, weekly, Monthly.. I keep only last three backup of each..

3 3 * * * /home/pi/domoticz/scripts/script_backup.sh J DB
3 4 * * 0 /home/pi/domoticz/scripts/script_backup.sh S DB
3 5 1 * * /home/pi/domoticz/scripts/script_backup.sh M DB
3 1 * * 1 /home/pi/domoticz/scripts/script_backup.sh S SYSTEM
3 2 2 * * /home/pi/domoticz/scripts/script_backup.sh M SYSTEM

#!/bin/bash
# LOCAL/FTP PARAMETERS
SERVER="192.168.0.xxx" # IP of Synology NAS, used for ftp
USERNAME="xxx" # FTP username of Network disk used for ftp
PASSWORD="xxx" # FTP password of Network disk used for ftp
DESTDIRNAS="/Public/domoticz/" # Path to your NAS backup folder
DOMO_IP="192.168.0.yyy" # Domoticz IP
DOMO_PORT="8080" # Domoticz port
DOMO_USER="yyy"
DOMO_PASS="yyy"
### END OF USER CONFIGURABLE PARAMETERS
param=$1 # 1st parameter will be for example 'J' for day, 'S' for week or 'M' for month
mode=$2 # 2nd parameter could be 'DB' for DB only backup or 'SYSTEM' for entire domoticz bacup
BACKUPFILE=$mode"_"$param"_1"
BACKUPFILEGZ="$BACKUPFILE".gz
##delete last - rename hold one
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS" -X "DELE "$mode"_"$param"_3.gz"
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER" -Q "RNFR $DESTDIRNAS/"$mode"_"$param"_2.gz" -Q "RNTO $DESTDIRNAS/"$mode"_"$param"_3.gz"
curl -v -u"$USERNAME:$PASSWORD" "ftp://$SERVER" -Q "RNFR $DESTDIRNAS/"$mode"_"$param"_1.gz" -Q "RNTO $DESTDIRNAS/"$mode"_"$param"_2.gz"
### Create backup and ZIP it
#service domoticz.sh stop # not needed
case "$mode" in "DB") # case of DB only backup
/usr/bin/curl -s http://$DOMO_USER:[email protected]$DOMO_IP:$DOMO_PORT/backupdatabase.php > /tmp/$BACKUPFILE
#service domoticz.sh start # not needed
gzip -9 /tmp/$BACKUPFILE
### Send to Network disk through FTP
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILEGZ" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
### Remove temp backup file
/bin/rm /tmp/$BACKUPFILEGZ
;;
"SYSTEM") # case of entire domoticz backup
tar -zcvf /tmp/$BACKUPFILE.tar.gz /home/pi/domoticz/
#service domoticz.sh start # not needed
### Send to Network disk through FTP
curl -s --disable-epsv -v -T"/tmp/$BACKUPFILE.tar.gz" -u"$USERNAME:$PASSWORD" "ftp://$SERVER/$DESTDIRNAS"
### Remove temp backup file
/bin/rm /tmp/$BACKUPFILE.tar.gz
;;
esac
### Done!

jammoju
Posts: 1
Joined: Sunday 23 October 2016 10:39
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: backup every 12 hours on qnap ftp

Post by jammoju » Friday 27 January 2017 17:24

Hi litfiba, (or anyone else who can help)

I was hoping you could help and give some further explination as to how to get your script working for me.

I am very new to this so sorry if its a dumb question. I am having trouble trying to automate the backup via FTP to my QNAP NAS.

This is what i get when i try to run the script

* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250-%s DISK FREE SPACE AT THIS DIRECTORY IS ONLY 838,656 Bytes.
< 250 CWD command successful.
> CWD Domoticz
< 250-%s Disk free space at this directory is 906,605,604 KB.
< 250 CWD command successful.
> EPSV
* Connect data stream passively
< 500 EPSV not understood.
* Failed EPSV attempt. Disabling EPSV
> PASV
< 227 Entering Passive Mode (192,168,1,3,4,132).
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connecting to 192.168.1.3 (192.168.1.3) port 1156
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
> TYPE I
< 200 Type set to I.
> SIZE Backup
< 550 Backup: not a regular file.
> RETR Backup
< 550 Backup: Not a regular file
* RETR response: 550
* Remembering we are in dir "/Domoticz/"
* Connection #0 to host 192.168.1.3 left intact
curl: (78) RETR response: 550
* Rebuilt URL to: ftp://192.168.1.3/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> RNFR /Domoticz/Backup/DB_J_2.gz
* ftp_perform ends with SECONDARY: 0
< 550 /Domoticz/Backup/DB_J_2.gz: No such file or directory
* QUOT command failed with 550
* Closing connection 0
curl: (21) QUOT command failed with 550
* Rebuilt URL to: ftp://192.168.1.3/
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> RNFR /Domoticz/Backup/DB_J_1.gz
* ftp_perform ends with SECONDARY: 0
< 550 /Domoticz/Backup/DB_J_1.gz: No such file or directory
* QUOT command failed with 550
* Closing connection 0
curl: (21) QUOT command failed with 550
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
< 220 NASFTPD 3.x Server [TS-101]
> USER julian
< 331 Password required for julian.
> PASS MY*PASSWORD
< 230 User julian logged in.
> PWD
< 257 "/" is current directory.
* Entry path is '/'
> CWD /
* ftp_perform ends with SECONDARY: 0
< 250-%s DISK FREE SPACE AT THIS DIRECTORY IS ONLY 838,656 Bytes.
< 250 CWD command successful.
> CWD Domoticz
< 250-%s Disk free space at this directory is 906,605,604 KB.
< 250 CWD command successful.
> PASV
* Connect data stream passively
< 227 Entering Passive Mode (192,168,1,3,4,133).
* Hostname was NOT found in DNS cache
* Trying 192.168.1.3...
* Connecting to 192.168.1.3 (192.168.1.3) port 1157
* Connected to 192.168.1.3 (192.168.1.3) port 21 (#0)
> TYPE I
< 200 Type set to I.
> STOR Backup
< 550 Backup: Not a regular file
* Failed FTP upload: 550
* Remembering we are in dir "/Domoticz/"
* Uploaded unaligned file size (0 out of 472 bytes)
* Connection #0 to host 192.168.1.3 left intact

As I said im very new to this but im trying to learn so any help is greatly appreciated.

Thank you...

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests