Thu 15 September 2011
By bcl
In Blog .
tags: Linux BackupPC LVM
I have been using BackupPC to automatically back up the systems on my LAN
for years now. It started out with a 3x250GB RAID5 as the storage pool and when
I ran out of space on that I added another disk to bring it up to about 700GB.
BackupPC does an excellent job of pooling common files together so that they
don't take up extra space. This is especially useful if you are backing up
system files on multiple systems running the same OS release.
The way it achieves this is through the use of hardlinks. A hardlink references
the inode of the original file instead of making a copy of its data. This only
works on a single filesystem since inodes are not unique between filesystems.
When you have 700GB of data backed up this results in alot of hardlinks. So
many in fact that none of the file copy tools available can deal with copying
them to a new filesystem.
BackupPC also keeps track of how full the pool is, and when it hit about
95% I started getting emails about it not backing up systems. As it expires
old backups it would drop back down to 90% but I realized it was finally time
to tackle the problem of moving the pool to a new RAID1 array. I decided to
go with RAID1 of 2T drives instead of a RAID5 because as drives get bigger the
chances of hitting an unrecoverable error increase. This means that when a 4
disk RAID5 loses a disk and you insert a new one it becomes more likely that
you will hit another error before the recovery is complete. RAID10 helps
mitigate this somewhat, using a RAID5 of mirrors, but that pretty much defeats
the 'Inexpensive' part of RAID. The article "RAID's Days May Be Numbered "
covers these issues pretty well.
I am using Linux's mdraid to manage the array. This array was originally
created while running Debian on an old single CPU 2.8GHz Celeron. Over the
years I have upgraded the hardware and even switched to using Fedora without
any problems recognizing the RAID array. I don't remember exactly why, but
it was setup as a single LVM physical volume with a single VG/LV on top of
that using ext3. These days I typically skip LVM and setup ext4 right on the
disk to eliminate any unneeded complexity. But in this case I'm glad I had
LVM setup, and will continue to use it for large blocks of extra disk.
The most common solution to moving a BackupPC pool is to use tar to pipe to
another tar process that uncompresses it on the target filesystem. tar knows
how to handle hardlinks, devices, etc. So it is a god choice for copying
things around when you can't use rsync. But when I gave this a try it ran
fairly quickly up to about 90% and then slowed way down. So slow that I could
count the number of blocks being transferred using df. According to atop and
iotop it was spending all of its time reading and very little time writing. I'm
not sure why, and didn't feel like stracing it to figure it out.
The next most common method is using dd to copy the filesystem over and then
use resize tools to grow it. This would probably have worked, but I was
concerned about it also copying UUIDs and ending up with a cloned fs and a
confused fstab. About this time I realized it was actually on LVM and that
I could use its tools to handle the move.
The LVM Howto provides some pretty good examples of how to do things with
LVM, including how to remove a disk. Section 13.5.2 covers moving things to
a new disk. Here's what I did (note, this worked for me, may not for you,
YMMV). I'd remind you to backup your data, but in this case that's exactly
what we're trying to accomplish, so...
I used parted to create a GPT disk label and partition the whole drives into
a single partition. I also set the raid flag on the partition.
I setup the new array as /dev/md3 like this:
mdadm --create /dev/md3 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1
In order to prevent mdadm from syncing the disks immediately I froze the
sync:
echo frozen > /sys/blocl/md3/md/sync_action
This may not have been needed, but seems like a good idea to get the actual
data moved over first, then let it sync.
Don't forget to shut down backuppc and unmount the existing pool. All of
these steps should be done on an unmounted system, just to be safe.
The existing RAID5 is called raid5-vol1 and it is on /dev/md0, I moved the
data over to the new array using LVM tools:
pvcreate /dev/md3
vgextend raid5-vol1 /dev/md3
pvmove /dev/md0 /dev/md3
The move took about 4 hours and has regular updates. I suggest running all
of this inside screen, just in case your ssh session/teminal/whatever goes
away.
After that is done I removed the old array from the VG:
vgreduce raid5-vol1 /dev/md0A
Now all the data is on the new array, but it is not able to take advantage
of the extra space of the larger drives. So we need to resize the lv. Run
vgdisplay and look at the 'Total PE', this is the maximum number of PE's
available. You will notice that the Alloc PE matches the size of the old
array. We are going to resize it to use all of them with:
lvresize -l xxxxxx /dev/raid5-vol1/raid5
Where xxxxxx is the Total PE number. This operation is very quick. Now you
are ready to resize the filesystem itself. I'm using ext3 so I use the e2fs
tools to operate on it, first running a filesystem check to make sure there
are no errors on the fs and then resizing it to use all the available space
on the underlying device.
e2fsck -f /dev/radi5-vol1/raid5
resize2fs -p /dev/raid5-vol1/raid5
When that finishes (took about an hour and a half for me) you can then
re-enable the array sync using:
echo check > /sys/blocl/md3/md/sync_action
And then mount your new pool and restart backuppc.
Pay careful attention to the PE number you use when resizing. It will ask you
if you make a mistake and try to shrink it (yes, I found this out when using
the -L option instead of -l). It is also easy to cut and paste the wrong number
and not grow it as much as you thought. Again, I did this, only noticing when
reviewing things for this article. I must have picked the free space number
because I still had about 700GB of PE's available. I followed the steps to
lvresize and resize2fs above and am now using the full amount of space available.
There are comments .