[TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

In this subforum you can show projects you have made, or you are busy with. Please create your own topic.
Post Reply
User avatar
Siewert308SW
Posts: 499
Joined: Monday 29 December 2014 16:47
Target OS: Raspberry Pi
Domoticz version: Beta
Location: The Netherlands - Zoutkamp (gn)
Contact:

[TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

Post by Siewert308SW » Saturday 26 November 2016 14:39

After 2 corrupt sd-cards i have found an alternative until MSD bootmode is final for the Pi3
At the moment MSD is only available in the branch=next firmware.
Which gave me issues as BT was going down all the time.
For me this solution works great and still able to run /root from my external HDD.
It's not my solution, as it is existing already i only gave it my twist.

Downside you still need an SD card to boot.
The advantage is that all data is written to your external HDD.
And it saves me a third upcoming corrupt sd card ;-)

Right now I use a "Seagate Expansion Portable 1TB" without USB hub.
Looks like the Pi3 with stock 5v 2.5A provides enough power to feed this disk.
I had to enable max_usb_current=1 so instead of 0.6 - 1.2A is sent through the USB port.
Which is only available on the Pi2+ and Pi3 as far as i know.

Below you'll find a description as how I configured it.
Assume that you have some expertise on Raspberry as i can't describe all the details ;-)

-------------------------------------------------- ------------

Step 1: Backing up or new SD Card stabbing
Make a backup of your existing setup.
Or in my case i used another 2gb sd card.

Step 2: Jessie download and transfer
Download Jessie Lite image in case of a headless set-up and drop it if you use Windows on your SD card using win32diskmanager.
Insert the SD card into your Pi and Pi on the boat.

Step 3: Pi Configuring
Please log in via ssh and configure the Pi first

sudo Raspi-config

- Expand filesystem and reboot

Login again and configure the rest
- Set locales
- Set Time Zone
- Set new password
- Ect ect

reboot again

Step 4: Static IP for Pi
If you want to have a static IP you want to Pi follow this step.
Otherwise, you can skip this step.

Log back in via ssh:

sudo nano /etc/dhcpcd.conf

The following static IP settings are focused on my situation.
Find out what your gateway ect and adjust if necessary.

Code: Select all

interface eth0
static ip_address=192.168.178.110/24
static routers=192.168.178.1
static domain_name_servers=8.8.8.8
Copy /Past the above setting in dhcpcd, conf
Finish with ctrl o followed by enter.
And to exit ctrl x
Reboot the Pi to let these setting take effect.

Step 5: Max USB current
Now we go to if you are going to provide your USB ports more Amps incase of a unpowered external HDD.
Note that max_usb_current=1 is only available for Pi2 + and Pi3 as far as I know
Log back in via SSH.

sudo nano /boot/config.txt

Copy /Past the following line at the bottom in the config.txt

Code: Select all

max_usb_current=1
Finish with ctrl o followed by enter.
To exit ctrl x
Reboot the Pi to let this setting take effect.

Step 6: OS update and install required packages
Now we will update your OS and already install some needed packages.
Log back in via SSH.

sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo apt-get install rsync

Optionally, if you want to use the beta kernel:

sudo apt-get install RPI update -y
sudo RPI update

installed everything? Reboot your Pi again

Step 7: Prepping your External Disk
We are going to prepare the external drive so that it can carry the /root.

In SSH enter 'sudo blkid "and take notice of what you see there.
It will be easier later on to find your just inserted external HDD.

Now insert the external HDD and enter again "sudo blkid"
And see which disk just has been added.
In my example, I take "sda" in my examples below.

We will now give the HDD a full ext4 partition.

in SSH: sudo parted /dev/sda
With the above command you start parted and all processing is now done on your external HDD

In parted do the following:

mktable msdos

Here you get a message that all data will be removed from your HDD.
Agree if you want to proceed.
--------------
Then:

mkpart primary ext4 0% 100%

It creates an ext4 partition along the entire length of your external HDD
--------------

Now you can review using the command "print"
Will be something like:

Code: Select all

Number  Start   End     Size    Type     File system  Flags
 1      1049kB  1000GB  1000GB  primary  ext4
--------------

To exit parted you enter the command "quit"
Only thing remaining is to tune the ext4 partition.
You do this by running the following command in SSH

sudo mkfs.ext4 /dev/sda1

Step 8: /root transfer to the external drive
Now we are ready to copy /root from sd-card to your external HDD partition.
In SHH, run it out below.

sudo mkdir /mnt/usbdrive
sudo mount /dev/sda1 /mnt/usbdrive/
sudo rsync -ax --progress / /mnt/usbdrive

May take some time before all the files are transferred

Step 9: Renew Certificates
Now we are going to renew the certificates on the HDD.
Otherwise you will no longer be able to log in via SSH.
In SSH, run the following commands:

cd /mnt/usbdrive
sudo mount --bind /dev dev
sudo mount --bind /sys sys
sudo mount --bind /proc proc
sudo chroot /mnt/usbdrive

rm /etc/ssh/ssh_host*
dpkg-reconfigure openssh-server
exit

sudo umount dev
sudo umount sys
sudo umount proc

Step 10: cmdline and fstab adjust
We are almost there.
We are going to adjust the boot cmdline and fstab to let your Pi know to seek on your external HDD.

Perform "sudo blkid" and locate your external HDD.
There will be something along the lines of:

Code: Select all

/dev/sda1: UUID="eda23afa-a07c-4ca5-8b00-f14b6f09095e" TYPE="ext4" PARTUUID="5905022c-01"
Remember the UUID hash and PARTUUID hash

The UUID hash is the name of your disk and PARTUUID hash your partition.
Normally, you could refer to the /root for example to /dev/sda
But we do not want to because if you ever going to use a USB stick later on it may screw everything up.
Resulting in a non booting Pi as it can't find /dev/sda because your OS has given it to your stick.
Instead, we use UUID and PARTUUID so no matter what you put into the Pi it will find your HDD using the hash

SSH in the following commands:

sudo sed -i "s,root=/dev/mmcblk0p2,root=PARTUUID=5905022c-01," /boot/cmdline.txt
sudo sed -i "s,/dev/mmcblk0p2,UUID=eda23afa-a07c-4ca5-8b00-f14b6f09095e," /etc/fstab

Now you can optionally verify cmdline.txt and fstab if the above has been applied.
Should look like the following:

- /boot/cmdline.txt

Code: Select all

dwc_otg.lpm_enable=0 console=tty1 root=PARTUUID=5905022c-01 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
- /etc/fstab

Code: Select all

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
UUID=eda23afa-a07c-4ca5-8b00-f14b6f09095e  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
Step 11: Almost done! [/ b]
Now we are ready and the external HDD contains everything from your sd-card /root.
If you've done everything properly your Pi will now start from the SD card and run from your external HDD.

SSH for now following commands.

cd ~
sudo umount /mnt/usbdrive
sudo rmdir /mnt/usbdrive
sudo reboot

If everything goes well then your Pi now reboots from the SD card and is running from your external HDD.

Step 12: Optional remove /root partition from your SD card [/ b]
You can now optionally remove /root partition from your SD card as we no longer need this.
SSH

sudo parted /dev/mmcblk0
rm 2 (2 stands for partition 2 verify it first using "print")
print
quit
sudo reboot

Step 13: Optional Domoticz install [/ b]
If want to have a fresh Domoticz installation.
Then you can SSH using "sudo curl -L install.domoticz.com | bash"
But you did above tut with your production Domoticz sd card then it should have been started already.

Last words [/ b]
It only remains me to say that hopefully someone overhere has a benefith by using this tut.
I'm still a noob in raspberry field and not so good it writing explanations. ;-)
Google was my best friend and using various found tuts on how to mount USB sticks, MSD mode to combine Pi3 to above tut.

Goodluck
Last edited by Siewert308SW on Saturday 04 November 2017 23:02, edited 2 times in total.
Setup:
1x RPi3 - PIco hv3.0A Plus / Domoticz
1x RPi3 - PIco hv3.0A Plus / PiHole
1x RFXtrx433E / Aeotec ZW090
3x FI9803P Cam
2x Youless Elec/Gas
4x KD101 detectors
a lot of KaKu/z-wave stuff

Domoticz scripts: @ GitHub
pico_installer: @ GitHub

Justintime
Posts: 319
Joined: Thursday 21 May 2015 9:08
Target OS: Raspberry Pi
Domoticz version: Beta
Contact:

Re: [TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

Post by Justintime » Sunday 18 December 2016 13:26

Thanks for the Tutorial. Ordered a 8gb SSD.
A corrupted SD card has never happened to me. But always in my mind it can.

lost
Posts: 127
Joined: Thursday 10 November 2016 10:30
Target OS: Raspberry Pi
Domoticz version:
Contact:

Re: [TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

Post by lost » Monday 19 December 2016 8:33

Another solution would be to use industrial grade SD (or PI foundation to make a on board connector for eUSB=USB keys made for embedded applications) like these:
http://www.smartm.com/salesLiterature/r ... erview.pdf

Other manufacturers maybe be virtium/micron/swissbit, they are all on the embedded storage market.

From previous document you can see a huge difference in TBW per GB capacity, between SLC and MLC based devices.

To sum-up, single level cell flash memory only store 1 state per cell = charge loaded or empty = 0/1 ; multi level ones makes uses of a comparator to be able to make difference between several levels of cell charge load, thus able to store several bits in a cell. If you have a 4 level comparator you can discriminate 00/01/10/11, thus store 2 bits in one place.

But of course, because of the erase (sector based) before write (even a single bit in a sector!) nature of flash memory, MLC in not so good in TBW: More need to erase (because more bits in a single cell), more complex management, cell wearing = more charge drain over storage time = more need to rewrite periodically in the background...

=> SLC given for 6TBW per GB capacity ; MLC for 0.2! So 30 times less!

This means that for a 16GB SD, even industrial grade, the manufacturer says lifetime writable amount of data is 96TB for SLC based devices and 3.2TB for MLC based ones.

At the moment, only setting /tmp and /var/tmp as a ram tmpfs, I have around 0.8GB written per day reported by ext4 session stats. So I would expect 4096 days for a MLC based device = 11 years.

But this is also dependant on write profiles (for very small files, this may be much less ; For big files, you can expect to reach manufacturer values).

And how do consumer grade devices compares? All are for sure MLC based and TBW are rarely published. Firmwares may also use less sophisticated wear leveling techniques. But as TBW is always related to capacity, using a 32GB device if you only need a 8GB one will multiply lifetime by 4 between 2 devices based on the same flash underlying memory technology.

Anyway, if there is a way to buy industrial grade SLC based devices designed for embedded market, that would be for sure a nice solution to this problem. Otherwise, my advice is to buy consumer grade devices from good brands and much larger than your needs.

Farnell sells swissbit products, S450 series are SLC based and S45 are MLC.
A 8GB S450 costs 108€, same price as a 64GB S45. So bit cost between SLC and MLC is multiplied by 8!

A 16GB S45 costs 39€. But underlying flash is specified for 3k erase cycles while S450 it's 100k, 33 times more! Close to the 30 found from comparing Smart devices TBW hereupper.

S45 series may indeed be a good cost/durability compromise for home use as sophisticated firmware will allow the device to wear evenly. Don't expect this from consumer grade devices that may wear very quickly on heavy used "hot spots".

See hereunder 16 and 32GB versions:
http://fr.farnell.com/swissbit/sfsd016g ... dp/2667833
http://fr.farnell.com/swissbit/sfsd032g ... dp/2667834

S46 series should also come soon: That's pSLC = MLC that can be used as SLC, specified at 20k erase cycles so fitting in between on the endurance side but using MLC chips that cost less. They should sell at the double bit cost compared to MLC/S45 when they arrive on market. So a 16GB should be in the 50/60€ price range with a nice durability.

That's the price of confidence!

User avatar
EdwinK
Posts: 921
Joined: Sunday 22 January 2017 22:46
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Rhoon
Contact:

Re: [TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

Post by EdwinK » Tuesday 14 November 2017 19:55

[deleted]
Last edited by EdwinK on Tuesday 14 November 2017 20:57, edited 1 time in total.
Running latest BETA on a Pi-3 | Toon Thermostat | RFxcom | Dashticz V2 |

User avatar
EdwinK
Posts: 921
Joined: Sunday 22 January 2017 22:46
Target OS: Raspberry Pi
Domoticz version: BETA
Location: Rhoon
Contact:

Re: [TUT] Moving OS/Domoticz to external HDD using UUID / PARTUUID (Pi2+/Pi3)

Post by EdwinK » Sunday 19 November 2017 17:23

Tried this, and it looks like all works.

But a bit hesitant about step 12. What is the root partition
Model: SD USD (sd/mmc)
Disk /dev/mmcblk0: 15.8GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
1 4194kB 48.0MB 43.8MB primary fat32 lba
2 48.2MB 15.8GB 15.8GB primary ext4
Running latest BETA on a Pi-3 | Toon Thermostat | RFxcom | Dashticz V2 |

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests