Read only Raspberry Pi OS to prevent corruption Topic is solved

Use this forum to discuss possible implementation of a new feature before opening a ticket.
A developer shall edit the topic title with "[xxx]" where xxx is the id of the accompanying tracker id.
Duplicate posts about the same id. +1 posts are not allowed.
oliviers
Posts: 78
Joined: Wednesday 16 September 2015 22:10
Target OS: Raspberry Pi
Domoticz version: Stable
Location: Paris area, France
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by oliviers » Monday 10 July 2017 15:54

Hello,
Like Egregius says, no big deal for me if the SD card gets corrupted after 2 or 3 years ... I would just swap it. Only 5-6€ ....
Oliviers
Paris area
Raspberry Pi 2B - RFXComm 433 - IrTrans - Zwave

User avatar
Egregius
Posts: 2387
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by Egregius » Monday 10 July 2017 19:07

dervogt wrote:I have migrated my root filesystem to an external USB drive with 500GB once my setup was up and running. This avoids the issues with symlinks or other small tricks you can do to "save" your SDCard. As the BOOT filesystem practically is read only and once the pi has booted into the OS, the SDCard practically is out of the game.

I have not gone the extra step of switching over to also boot from the USB drive, I personally am happy with the SDCard boot and then having my external HDD takeover the rest.
Your harddrive probably consumes more than your pi :?

dervogt
Posts: 34
Joined: Thursday 30 June 2016 9:00
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Amsterdam
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by dervogt » Tuesday 11 July 2017 15:23

Egregius wrote: Your harddrive probably consumes more than your pi :?
Probably this is right, but coming from an Octacore AMD HTPC with a S-ATA RAID, two DVB-S cards, major PCIx VGA and another digital PCIx SoundCard, this still is an energetic improvement :D

nigels0
Posts: 208
Joined: Thursday 23 January 2014 13:43
Target OS: Raspberry Pi
Domoticz version: 2.2657
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by nigels0 » Tuesday 11 July 2017 17:34

Egregius wrote:Booting from NAS is rather pointless, than you could also run domoticz directly on the nas. Running on my nas is not a good option here as it consumes 65W, calculate the cost of that... Besides that I also tried RPi with NFS on nas but that gave a lot more problems.
Logread wrote:I looked into this a while ago, but not long enough (1 year history so far) to prove this makes a real difference... I wrote a wiki page on what I did:
http://domoticz.com/wiki/Moving_Log_and ... _RAM_Drive

Ideally for zwave users, some of the zwave config files could also be moved (e.g. domoticz\Config\zwcfg_0x????????.xml is written rather frequently) but I did not yet venture into this (not sure a symlink would do the trick, but worth trying... otherwise may need to tweak the source code of openzwave or domoticz and this is probably not worth the effort).
I do something similar:

/etc/fstab: You may easily set a larger memory size, memory will only be occupied depending on the file sizes.

Code: Select all

tmpfs   /temp        tmpfs   defaults,noatime,nosuid,mode=777,size=300m      0       0
tmpfs   /var/log        tmpfs   defaults,noatime,nosuid,mode=777,size=300m      0       0
sudo crontab -e

Code: Select all

@reboot /home/pi/atstartup.sh >/dev/null 2>&1
0 * * * * /home/pi/cronhourly.sh >/dev/null 2>&1
1 0 * * * /home/pi/crondaily.sh >/dev/null 2>&1
2 0 * * 1 /home/pi/cronweekly.sh >/dev/null 2>&1
/home/pi/atstartup.sh: make sure some folders are writable and backups are restored to tmpfs drive

Code: Select all

#!/bin/bash
echo "performance" | tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
chmod 777 /var/log
touch /var/log/domoticz.log
chmod 666 /var/log/domoticz.log
cp /home/pi/zwcfg_0xe9238f6e.xml /var/log/zwcfg_0xe9238f6e.xml
cp /home/pi/domoticz.db /var/log/domoticz.db
cp /home/pi/options.xml /home/pi/domoticz/Config/options.xml
ln -sf /var/log/domoticz.db /home/pi/domoticz
chmod 666 /var/log/zwcfg_0xe9238f6e.xml
echo 0 | tee /sys/class/leds/led0/brightness
echo 0 | tee /sys/class/leds/led1/brightness
/opt/vc/bin/tvservice -o
sudo service domoticz.sh start
/home/pi/crondaily.sh: make a daily backup of the tmpfs files

Code: Select all

#!/bin/bash
cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
cp /var/log/domoticz.db /home/pi/domoticz.db
/home/pi/cronweekly.sh: cleanup the logfiles

Code: Select all

#!/bin/bash
sudo rm /var/log/*.gz
sudo rm /var/log/*.1
sudo rm /var/log/SBFSPOT/*
sudo truncate -s 0 /var/log/*
sudo truncate -s 0 /var/log/lighttpd/*
/home/pi/options.xml: keep some zwave settings and set /var/log as log folder

Code: Select all

<?xml version="1.0" encoding="utf-8"?>
<!-- To be effective, this file should be placed in the user data folder specified in the Options::Create method -->
<Options xmlns='http://code.google.com/p/open-zwave/'>
  <Option name="logging" value="true" />
  <Option name="UserPath" value="/var/log/" />
  <Option name="Associate" value="true" />
  <Option name="NotifyTransactions" value="false" />
  <Option name="DriverMaxAttempts" value="5" />
  <Option name="SaveConfiguration" value="true" />
  <Option name="RetryTimeout" value="5000" /><!-- Default 40000 --><!-- OK: 15000 -->
  <Option name="PerformReturnRoutes" value="true" /><!-- added -->
  <Option name="AssumeAwake" value="false" /><!-- added -->
  <Option name="SaveLogLevel" value="3" /><!-- added -->
  <Option name="QueueLogLevel" value="3" /><!-- added -->
  <Option name="DumpTriggerLevel" value="3" /><!-- added -->
  <Option name="NetworkKey" value="0x09, 0x05, 0x03, 0x04, 0x01, 0x06, 0x07, 0x01, 0x09, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0x10" />
  <Option name="RefreshAllUserCodes" value="false" />
  <Option name="ThreadTerminateTimeout" value="5000" /><!-- 5000 -->
</Options>
And finally in /etc/init.d/domoticz.sh: make a backup of the database and zwave config while stopping domoticz.

Code: Select all

do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
	cp /var/log/domoticz.db /home/pi/domoticz.db
	cp /var/log/zwcfg_0xe9238f6e.xml /home/pi/zwcfg_0xe9238f6e.xml
        return "$RETVAL"
}
I'm pretty sure that with this you avoid most of the writes to the sd card.
There's one big downside of it: if you cut the power you'll lose data of that day. In my case I don't care about that. The temperatures I care about are pushed to a MySQL database on a VPS. As long as you can send the shutdown command there's nothing to worry about.
Hi,

I Think I've got this working - I don't use z-wave, so commented anything about the zwcfg files out - but your list is missing cronhourly.sh (it's mentioned in the crontab entries)

User avatar
Egregius
Posts: 2387
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by Egregius » Tuesday 11 July 2017 22:34

I shall look into that when I get back from vacation. Out of my head I thinks there's nothing domoticz related in it. More stuff like disk usage etc.

nigels0
Posts: 208
Joined: Thursday 23 January 2014 13:43
Target OS: Raspberry Pi
Domoticz version: 2.2657
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by nigels0 » Tuesday 11 July 2017 23:37

No worries - nice set of scripts though! anything that stops the SD cards corrupting is welcome.

dressie
Posts: 108
Joined: Monday 25 May 2015 22:44
Target OS: Linux
Domoticz version: V3.8153
Location: Netherlands
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by dressie » Friday 14 July 2017 21:10

ben53252642 wrote:
EddyG wrote:24/7 is just normal for a NAS.
The better type of NAS can take care of backups/snapshots etc. So you are 100% save.
In my environment I have a Synology NAS with RAID 6 which can do LUNs, didn't know about the option to boot from a LUN.

Even found a guide on how to do it!
http://www.berryterminal.com/doku.php/s ... sing_iscsi

Combined with watchdog and monit setup, Indeed this would appear to be a very good option particularly with the ability to schedule automatic snapshots. :D
I just got a new Pi3 and used this set up. I'm testing it right now with a minimum filled Domoticz (main Domoticz is still running on a Pi2 with SD card). So far it looks good. It's fast, stable and you don't have to worry about the loss of data. I'm going to test it another few days and will move the database and the Zwave stick and 433Mhz over to the new setup. Will keep you up to date.

User avatar
Egregius
Posts: 2387
Joined: Thursday 09 April 2015 12:19
Target OS: Raspberry Pi
Domoticz version: BetaRPi2
Location: Beitem, BE
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by Egregius » Saturday 15 July 2017 6:18

nigels0 wrote:but your list is missing cronhourly.sh (it's mentioned in the crontab entries)
Here it is, as I tought it's about checking disk usage:

Code: Select all

#!/bin/bash
df -H | grep -vE '^Filesystem|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
  echo $output
  usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [ $usep -ge 80 ]; then
    msg="Running out of space \"$partition ($usep%)\" on $(hostname)"
    curl -s --data-urlencode "text=$msg" --data "silent=false" http://127.0.0.1/secure/telegram.php
  fi
	if [ $usep -ge 50 ] && [ $partition = "tmpfs" ]; then
		sudo rm /var/log/*.gz
		sudo rm /var/log/*.1
		sudo truncate -s 0 /var/log/*
		sudo truncate -s 0 /var/log/lighttpd/*
	fi
done
sudo ntpdate be.pool.ntp.org

Number8
Posts: 261
Joined: Friday 23 May 2014 7:55
Target OS: Raspberry Pi
Domoticz version: 3.5103
Location: Saint Pierre de Jards
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by Number8 » Saturday 15 July 2017 6:48

I just got a new Pi3 and used this set up. I'm testing it right now with a minimum filled Domoticz (main Domoticz is still running on a Pi2 with SD card). So far it looks good. It's fast, stable and you don't have to worry about the loss of data. I'm going to test it another few days and will move the database and the Zwave stick and 433Mhz over to the new setup. Will keep you up to date
Using wired Ethernet, Debian Lite has been installed without problem. Afterwards I tried to install using wifi and it was unsuccessful. It keeps trying to download the distribution forever. The wifi dongle is recognized by Berryterminal and connected to Internet. Once installation has been done using wired Ethernet, it is not possible to switch to wifi afterwards. I then discovered that the distribution cannot be upgraded, you have to stick to the one that has been installed. All these things considered I stopped the experiment.

dressie
Posts: 108
Joined: Monday 25 May 2015 22:44
Target OS: Linux
Domoticz version: V3.8153
Location: Netherlands
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by dressie » Saturday 15 July 2017 8:17

Number8 wrote:Using wired Ethernet, Debian Lite has been installed without problem. Afterwards I tried to install using wifi and it was unsuccessful. It keeps trying to download the distribution forever. The wifi dongle is recognized by Berryterminal and connected to Internet. Once installation has been done using wired Ethernet, it is not possible to switch to wifi afterwards. I then discovered that the distribution cannot be upgraded, you have to stick to the one that has been installed. All these things considered I stopped the experiment.
Mine will stay wired no matter what, just curious what you ment by "cannot be upgraded". Are you referring to the basic 'sudo apt-get update' and 'sudo apt-get upgrade' ?

Number8
Posts: 261
Joined: Friday 23 May 2014 7:55
Target OS: Raspberry Pi
Domoticz version: 3.5103
Location: Saint Pierre de Jards
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by Number8 » Saturday 15 July 2017 8:38

Mine will stay wired no matter what, just curious what you ment by "cannot be upgraded". Are you referring to the basic 'sudo apt-get update' and 'sudo apt-get upgrade' ?
Yes

User avatar
EddyG
Posts: 106
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8153
Location: Netherlands
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by EddyG » Saturday 15 July 2017 9:33

That is right, mine is on: Linux raspberrypi 4.9.26v7-aufs #1 SMP Tue May 9 20:14:03 CEST 2017 armv7l GNU/Linux
And sudo apt-mark showhold shows:
raspberrypi-bootloader
raspberrypi-kernel

That's because the system is using aufs (https://en.wikipedia.org/wiki/Aufs)
Regards,
Eddy

dressie
Posts: 108
Joined: Monday 25 May 2015 22:44
Target OS: Linux
Domoticz version: V3.8153
Location: Netherlands
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by dressie » Saturday 15 July 2017 10:27

So it's stable, but eventually the system will become outdated. Time for some pros and cons .

Verstuurd vanaf mijn SM-G920F met Tapatalk

Number8
Posts: 261
Joined: Friday 23 May 2014 7:55
Target OS: Raspberry Pi
Domoticz version: 3.5103
Location: Saint Pierre de Jards
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by Number8 » Saturday 15 July 2017 15:52

dressie wrote:So it's stable, but eventually the system will become outdated. Time for some pros and cons .
Indeed

User avatar
EddyG
Posts: 106
Joined: Monday 02 November 2015 6:54
Target OS: Raspberry Pi
Domoticz version: 3.8153
Location: Netherlands
Contact:

Re: Read only Raspberry Pi OS to prevent corruption

Post by EddyG » Saturday 15 July 2017 20:05

There is a lot on GitHub, so you could build your own image with new(er) system.
And I suppose that the creator will update berryboot over time.
Regards,
Eddy

Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests