Recently, I’ve needed a bit more storage. In the past I’ve relied on Google Drive, but if you need a lot of space Google Drive becomes prohibitively expensive. The largest option available, 2 TB, runs you $100 a year at the time of writing. While Google Drive comes with a lot of features, it also comes with a lot of privacy concerns, and I need more than 2 TB anyway. Another option would be Backblaze B2 or AWS S3, but the cost is even higher. Just to set a point of comparison, 16 TB of storage would cost $960 a year with B2 and a whopping $4000 a year with S3.
Luckily in reality, the cost of storage per GB has been coming down steadily. Large hard drives are cheap to come by, and while these drives are not incredibly fast, they are much faster than the speed of my internet connection. Hard drives it is then!
While I could get a very large hard drive, it’s generally a better idea to get multiple smaller hard drives. That’s because these drives often offer a better $/GB rate, but also because it allows us to mitigate the risk of data loss. So after a bit of search, I found these “Seagate Barracuda Compute 4TB” drives. You can find them on Amazon or BestBuy.
These hard drives are available for $70 each at the time I’m writing this,and I bought 6 of them. This gets me to around $420, plus a bit more for SATA cables. Looking at Backblaze Hard Drive Stats, I think it’s fair to assume these drives will last at least 5 years. Dividing the cost by the expected lifetime, that gets me $84 per year, far below what the cloud storage costs! It’s of course not as reliable, and it requires maintenance on my end, but the difference in price is just too far to ignore.
I decided to set this all up inside my desktop computer. I have a large case so fitting all the hard drives in is not a big problem, and my motherboard does support 6 SATA drives (in addition to the NVMe that I’m booting off of). I also run Linux on my desktop computer, so I’ve got all the required software available.
For the software side of things, I decided to go with
are also other options available like ZFS (not included in the linux kernel) or
btrfs (raid-5 and raid-6 are known to be unreliable), but this was the setup I
found the most comfortable and easy to understand for me. How it works is that
mdadm combines the disks and presents it as a block device, then
formats and uses the block device the same way you use it with any regular
I was originally planning to write the steps I followed here, but in truth I just followed whatever the ArchLinux wiki told me. So I’ll just recommend you follow that as well.
The only thing I’ll warn you is that the wiki doesn’t clearly note just how long
this process takes. It took almost a week for the array to build, and until the
build is complete the array runs at a reduced performance. Be patient, and just
give it some time to finish. As a reminder, you can always check the build
Hard drives have a tendency to fail, and because RAID arrays are resilient, the failures can go unnoticed. You need to regularly check that the array is okay. Unfortunately, while there are quite a few resources online on how to set up RAID, very few of them actually talk about how to set up scrubs (full scans to look for errors) and error monitoring.
For my setup, I decided to set up systemd to check and report issues. For this, I first set up 2 timers: 1 that checks if there are any reported errors on the RAID array, and another that scrubs the RAID array. Systemd timers are 2 parts, a service file and a timer file, so here’s all the files.
[Unit] Description=Scrub the disk array After=multi-user.target OnFailureemail@example.com [Service] Type=oneshot User=root ExecStart=bash -c '/usr/bin/echo check > /sys/block/md127/md/sync_action' [Install] WantedBy=multi-user.target
[Unit] Description=Periodically scrub the array. [Timer] OnCalendar=Sat *-*-* 05:00:00 [Install] WantedBy=timers.target
The timer above is the scrub operation, it tells RAID to scan the drives for errors. It actually takes up to a couple days in my experience for the scan to complete, so I run it once a week.
[Unit] Description=Check raid array errors that were found during a scrub or normal operation and report them. After=multi-user.target OnFailurefirstname.lastname@example.org [Service] Type=oneshot ExecStart=/usr/bin/mdadm -D /dev/md127 [Install] WantedBy=multi-user.target
[Unit] Description=Periodically report any issues in the array. [Timer] OnCalendar=daily [Install] WantedBy=timers.target
And this timer above checks the RAID array status to see if there were any errors found. This timer runs much more often (once a day), because it’s instant, and also because RAID can find errors during regular operation even when you are not actively running a scan.
Another important thing here is this line in the service file:
The automated checks are of no use if I don’t know when something actually fails. Luckily, systemd can run a service when another service fails, so I’m using this to report failures to myself. Here’s what the service file looks like:
[Unit] Description=status email for %i to user [Service] Type=oneshot ExecStart=/usr/local/bin/systemd-email address %i User=root
#!/bin/sh /usr/bin/sendmail -t <<ERRMAIL To: email@example.com From: systemd <root@$HOSTNAME> Subject: Failure on $2 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 $(systemctl status --lines 100 --no-pager "$2") ERRMAIL
The service just runs this shell script, which is just a wrapper around
%i in the service is the part after
@ when you use the
service, you can see that the
OnFailure hook puts
array-report after the
which then gets passed to the email service, which then passes it on to the mail
To send emails, you also need to set up
sendmail. I decided to install
msmtp, and set it up to use my GMail
account to send me an email.
To test if the error reporting works, edit the
array-report.service and change
ExecStart line to
ExecStart=false. Then run the report service with
systemd start array-report.service, you should now get an email letting you
know that the
array-report service failed, and attaches the last 100 lines of
the service status to the email.