Setting up overlayFS on Raspberry Pi

From Domoticz
Jump to: navigation, search

Warning: Do this only on a new SD image that can be recreated, or make sure you can mount the SD card elsewhere to revert changes.

Recent changes

Please note that overlays are now synced via an intermediate / staging directory to the underlay, i.e. /home is now synced to /home_stage before being synced to /home_org. The reason for this is that changing the underlay gives unpredictable results for the overlay view (per the design document on kernel.org).

  • Before doing so, a list of processes / services are first killed if found. By default domoticz and mosquitto (change as you please). This is to ensure valid / stable data are being synced in case you do a service saveoverlays sync. For a normal restart, these services are stopped before the sync takes place.
  • After sync, the system must be immediately rebooted without further syncs (keep root partition mounted read/write , then reboot)
  • Do not copy files directly from the overlay to any of the underlays
  • Remarks for Jessie also seem to work for Stretch

... also, the source files for setting this up are now on github (finally)

Purpose

In this page, we will describe how to implement overlayfs using the normal SD card directory tree as the bottom (persistent) layer, and as many RAM / tmpfs areas you may want as the upper layer to provide the overlay mounts. In most cases this will do:

  • /boot and the root filesystem are changed to be normally read only
  • /var and /home is changed into read/write overlay filesystems
  • An init script in /etc/init.d/saveoverlays will take care of syncing the overlay filesystems back to the bottom layer SD card upon system shutdown and restarts

Using this, you can eliminate SD card writes almost entirely. As the read-write parts are then RAM based, this will also speed up Domoticz.

Preparations

Requirements

OverlayFS is included in kernel versions 3.18 and later.

In case of errors, it is strongly recommended to be able to mount and edit the modified SD card contents via a card reader or similar on another system

Downloads

The easiest way to get the needed scripts into place is this:

cd /tmp
wget https://github.com/hansrune/domoticz-contrib/raw/master/utils/mount_overlay
sed -i 's#/bin/sh#/bin/bash#g' ./mount_overlay
wget https://github.com/hansrune/domoticz-contrib/raw/master/init.d/saveoverlays-jessie
mv saveoverlays-jessie saveoverlays
sed -i 's#/bin/sh#/bin/bash#g' ./saveoverlays
chmod a+rx saveoverlays mount_overlay
sudo cp mount_overlay /usr/local/bin
sudo cp saveoverlays /etc/init.d/
# For raspian Jessie and later, also do
sudo systemctl disable dphys-swapfile

Backup and recovery

In case of errors, it is a good idea to backup the below. You can then mount the modified SD card on another machine.

sudo cp -v /boot/cmdline.txt /boot/cmdline.txt-orig
sudo cp -v /etc/fstab /etc/fstab-orig
cd /var
sudo tar --exclude=swap -cf /var-orig.tar .

Setup procedure

Stop using swap

In case you also use the SD card for swap, this should be turned off as follows:

sudo dphys-swapfile swapoff
sudo dphys-swapfile uninstall 
sudo update-rc.d dphys-swapfile disable
# For raspian Jessie and later, also do
sudo systemctl disable dphys-swapfile

Move files you need to be able to update with a read-only root

Some files may need to be continously updated after all changes are done. For example, if you use the fake-hwclock service, the data file for that service need to be moved:

 sudo service fake-hwclock stop 
 sudo mv /etc/fake-hwclock.data /var/log/fake-hwclock.data
 sudo ln -s /var/log/fake-hwclock.data /etc/fake-hwclock.data
 sudo service fake-hwclock start
 sudo update-rc.d fake-hwclock disable
 # For raspian Jessie and later, also do
 sudo systemctl disable fake-hwclock

The fake-hwclock actions are taken care of in the saveoverlays service described later. There is no harm in leaving it as is, but it will complain during start and do nothing.

Set up fuse and mount script

Install the fuse package. We need only /sbin/mount.fuse from there. lsof is essential to find open files in case you cannot remount a filesystem back to read-only

sudo apt-get install fuse lsof

The following mount script is used in /etc/fstab to do the actual overlay mount. When the filesystem type is fuse, you can use a script like this from /usr/local/bin/mount_overlay to do the mount via mount.fuse

#!/bin/sh
DIR="$1"
[ -z "${DIR}" ] && exit 1 
if [ ! -d "${DIR}_org" ]
then
    echo "${DIR}_org does not exist" >&2
    exit 1
fi
if [ ! -d "${DIR}_rw" ]
then
    echo "${DIR}_rw does not exist" >&2
    exit 1
fi
#
# ro must be the first mount option for root .....
#
ROOT_MOUNT=$( awk '$2=="/" { print substr($4,1,2) }' /proc/mounts )
if [ "$ROOT_MOUNT" != "ro" ]; then
    /bin/mount --bind ${DIR}_org ${DIR}
else
    /bin/mount -t tmpfs ramdisk ${DIR}_rw
    /bin/mkdir ${DIR}_rw/upper
    /bin/mkdir ${DIR}_rw/work
    OPTS="-o lowerdir=${DIR}_org,upperdir=${DIR}_rw/upper,workdir=${DIR}_rw/work"
    /bin/mount -t overlay ${OPTS} overlay ${DIR}
fi

Please note that if the root filesystem is read-write in /etc/fstab, the mount will be a loopback mount instead.

Set up the saveoverlays service

This saveoverlays service (follow the link to download) will take care of saving changes to the overlays back to the underlying SD card storage (now the _org part of the overlay). It also contains some logic to check if the fake clock data should be loaded. If you are not using the fake-hwclock service, you should comment that out (two places).

The overlays are automatically detected, so you should not need to change any of that.

After copying the file to /etc/init.d/saveoverlays, enable the service as follows:

sudo update-rc.d saveoverlays  start 10 S . stop 05 0 6 . # (before Raspian jessie )
sudo systemctl enable saveoverlays.service # Raspian Jessie / systemd boot dependencies

The expected start / stop links will look something like this:

find /etc/init* /etc/rc* | grep saveover
/etc/init.d/saveoverlays
/etc/rc0.d/K08saveoverlays
/etc/rc1.d/K08saveoverlays
/etc/rc6.d/K08saveoverlays
/etc/rcS.d/S10saveoverlays

Note: If you have remounted root to be read-write, this service assumes that changes are not to be synced. Please remount back to read-only if you want changes to be synced on next reboot.

Change boot command line

Change /boot/cmdline.txt similar to this, i.e. add noswap fastboot ro. In Raspbian Stretch root=/dev/mmcblk0p2 is different and looks like: root=PARTUUID=badee776-02

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait noswap fastboot ro

Update /etc/fstab for read-only SD card

/etc/fstab need to be updated similar to this:

proc            /proc           proc    defaults    0 0
/dev/mmcblk0p1  /boot           vfat    ro          0 2
/dev/mmcblk0p2  /               ext4    ro,noatime  0 1
mount_overlay   /var            fuse    nofail,defaults 0 0
mount_overlay   /home           fuse    nofail,defaults 0 0
none            /tmp            tmpfs   defaults    0 0


/etc/fstab need to be updated similar to this for Jessie:

proc            /proc           proc    defaults    0 0
/dev/mmcblk0p1  /boot           vfat    ro          0 2
/dev/mmcblk0p2  /               ext4    ro,noatime  0 1
mount_overlay   /var            fuse    nofail,defaults,x-systemd.automount,x-systemd.requires=/var,x-systemd.device-timeout=10s 0 0
mount_overlay   /home           fuse    nofail,defaults,x-systemd.automount,x-systemd.requires=/home,x-systemd.device-timeout=10s 0 0
none            /tmp            tmpfs   defaults    0 0


Prepare the overlay directories

Services that have open files should be stopped. The following should do if you use the current Domoticz SD card image

sudo service domoticz.sh stop
sudo service nodered stop
sudo service influxdb stop
sudo service mosquitto stop
sudo service ntp stop
sudo service rsyslog stop

Then change into layered mount setups as follows:

sudo mv /home /home_org
sudo mkdir /home /home_rw 
sudo mv /var /var_org
sudo mkdir /var /var_rw

Loopback mount /home and /var before boot

This is to make sure the next reboot does not run into trouble due to moved data (rsync with delete). When root is not read-only, this will result in /var_org being loopback mounted to /var, and /home_orig loopback mounted to /home

sudo mount /home
sudo mount /var

Modify domoticz.sh service

For some reason the domoticz executable is mistaken on startup directory when in an overlay filesystem:

2015-11-08 18:09:03.803  Startup Path: /pi/domoticz/

To get around that, add the following after other DAEMON_ARGS lines in /etc/init.d/domoticz.sh

APPDIR=$( dirname $DAEMON )
DAEMON_ARGS="$DAEMON_ARGS -approot $APPDIR/ -wwwroot $APPDIR/www  -dbase $APPDIR/domoticz.db"

The trailing slash for approot is needed.

Reboot

Now reboot. Since root is not mounted read-only at this time, the saveoverlays service will not attempt to sync anything.

It is recommended to have a console attached so that any error messages can be detected, and problems dealed with.

Tests

Check your mounts

After boot, the mount command should display similar to this, with relevant changes in red

/dev/root on / type ext4 (ro,noatime,data=ordered) 
devtmpfs on /dev type devtmpfs (rw,relatime,size=234492k,nr_inodes=58623,mode=755)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=47756k,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) 
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /run/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=95500k)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
/dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
ramdisk on /var_rw type tmpfs (rw,relatime)
overlay on /var type overlay (rw,relatime,lowerdir=/var_org,upperdir=/var_rw/upper,workdir=/var_rw/work)
ramdisk on /home_rw type tmpfs (rw,relatime) 
overlay on /home type overlay (rw,relatime,lowerdir=/home_org,upperdir=/home_rw/upper,workdir=/home_rw/work)
none on /tmp type tmpfs (rw,relatime)


Check that you can update the system

For most updates, the following will do

sudo su - 
mount -o remount,rw /
mount -o remount,rw /boot
apt-get update
apt-get upgrade
service saveoverlays sync # this is just in case the next command fail . Can happen when updates restart services
mount -o remount,ro /
mount -o remount,ro /boot
init 6

Alternatively, proceed with the following script:


 #!/bin/bash
 
 # remount root rw
 mount -o remount,rw /
 
 # prapare target paths
 mkdir -p /chroot
 mkdir -p /chroot/{bin,boot,dev,etc,home,lib,opt,proc,root,run,sbin,sys,tmp,usr,var}
 
 # mount special filesystems
 mount -t proc proc /chroot/proc
 mount --rbind /sys /chroot/sys
 mount --rbind /dev /chroot/dev
 
 # bind rw directories
 for f in {home,var}; do mount --rbind /${f}_org /chroot/$f; done
 
 # bind remaining directories
 for f in {bin,boot,etc,lib,opt,root,run,sbin,tmp,usr}; do mount --rbind /$f /chroot/$f; done
 
 # chroot
 echo "Note: /boot is still mounted read-only, remount to read-write if needed."
 echo -e "\e[33mYou are now in read-write chroot. Use CTRL+D when done to exit chroot and mount read-only again.\e[39m"
 chroot /chroot /usr/bin/env PS1="(rw) \[email protected]\h:\w\$ " /bin/bash --noprofile -l
 
 # unmount mounts
 for f in /chroot/{bin,boot,dev,etc,home,lib,opt,proc,root,run,sbin,sys,tmp,usr,var}; do
 umount -l $f
 done
 
 sleep 1
 
 # remount read-only again
 echo -e "\e[32mChroot left, re-mounting read-only again.\e[39m"
 mount -o remount,ro /

If you cannot remount read-only

In case you cannot remount to read-only, you need to to a manual sync of the overlay filesystem. This can happen when updates restarts services.

sudo service saveoverlays sync
sudo reboot

Note: When the root partition is mounted read/write, a stop of the saveoverlays service does nothing.

Check that overlays are synced to SD card

These syncs are logged to /var/log/saveoverlays.log

References

There are many references for doing similar setups. The most similar one using unionfs is likely this:

Other material: