Problem

RAID-1 (mirroring) is a popular approach to protect the system from a harddisk failure. It is either done in hardware or software. The usual hardware solution is to buy a RAID disk controller like the popular 3ware ATA RAID controllers and then not having to deal with any software incompatibilities because the system just sees one large physical disk.

The software solution is less expensive and more flexible, but usually makes more trouble during booting. Because an important point in establishing a software RAID-1 for the system disk partitions is whether one can directly boot from the resulting mirror setup without ugly BIOS and/or boot loader tricks. Additionally IMHO it is also important that in case of a major problem with the RAID-1 software driver, one is still able to easily rescue boot from the mirror setup by treating it like a plain disk setup again.

Possibilities

For software solutions in FreeBSD 5 one can choose between the raw ata(4), the good old ccd(4), the classic vinum(4), the new GEOM Vinum, and the GEOM gmirror(8). Each one has its pros and cons:

ATA:
Although the ATA driver's RAID functionality is intended for hardware RAID controllers, it allows for creating an ATA RAID-1 on disks with any controller. This is the most raw approach, is suitable for ATA/IDE drives only and has functional restrictions.
CCD:
This is the good old Concatenated Disk Driver (CCD) which provides both Concatenation, RAID-0 (Striping) and RAID-1 (Mirroring). It was converted to use GEOM in FreeBSD 5, but it has less features than GEOM mirror AFAIK.
Classic Vinum:
This is the classic Vinum which provides all major RAID levels including RAID-1 and additionally is a full-scale volume manager. Although still existing in FreeBSD 5, is is not using GEOM and especially panics the system if automatically enabled during the system boot. It is no longer feasible to use in FreeBSD 5 IMHO, especially not for mirroring the system disks.
GEOM Vinum:
This is the new GEOM Vinum, a functional port and code rewrite of Vinum as a GEOM class. It already provides lots of functionality but is still undocumented and hence IMHO still not really ready for production use in FreeBSD 5.
GEOM Mirror:
This is the GEOM Mirror, a dedicated GEOM class for establishing a RAID-1. It is a full-featured from-scratch implementation and IMHO fully ready for production use, at least in FreeBSD 5.3-STABLE as of January 2005.

I currently prefer GEOM mirror for establishing such a RAID-1 solution for the system disk partitions because it both provides the possibility to directly boot from the mirror without tricks and is a really clean and modern solution. Additionally, GEOM based solutions can be stacked in an arbitrary way, i.e., the resulting GEOM mirror is just another GEOM "provider" on which one can base any other GEOM "consumers", including GEOM striping (gstripe) or GEOM based disk encryption (gbde).

Solution

The FreeBSD 5 geom(4) framework is an excellent abstraction layer for disk I/O. The GEOM mirror class is a software RAID-1 implementation which can be used even for establishing a mirror for the system disk partitions. GEOM mirror is very flexible and actually can be established for any GEOM "providers", including whole disks or just particular slices on a disk. Unfortunately each approach has again pros and cons and hence should be carefully understood before using one of them:

GEOM mirror on whole disk:
This is straight-forward and obvious and results in having the whole disk mirrored, including its MBR. Unfortunately is works only if the two disks are really equal in size or at least requires that the mirror is established on the smaller disk first and that space is lost on the second disk. Finally, in case of a replacement disk (which usually is a lot larger because available disks expand and expand in size without actually getting a lot more expensive), one completely waste disk space.
GEOM mirror on slices of a disk:
This is a little bit less obvious, but provides a lot more flexibility and especially allows the usage of disks of different size plus the possibility to still use (either for a second mirror or as plain partitions) the unused space on the disks.

GEOM mirror Approach 1: Whole Disk, Acceptable, Less Flexible

The following is a step-by-step command list for remotely converting a production FreeBSD 5.3-STABLE system from a single-disk (ad0) to a two-disk (ad0 & ad1) GEOM mirror (gm0) based setup without the need for console access or Fixit/LiveFS CDROM. The assumption is that the two disks are EQUAL in size or that the second disk is a little bit SMALLER than the first one.

# make sure the second disk is treated as a really fresh one
# (not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad1 bs=512 count=79

# place a GEOM mirror label onto second disk
# (actually on the last block of the disk)
gmirror label -v -n -b round-robin gm0 /dev/ad1

# activate GEOM mirror kernel layer
# (makes the /dev/mirror/gm0 device available)
gmirror load

# place a PC MBR onto the second disk
# (with a single FreeBSD slice /dev/mirror/gm0s1 covering the whole disk)
fdisk -v -B -I /dev/mirror/gm0

# place a BSD disklabel onto /dev/mirror/gm0s1
# (ATTENTION: in FreeBSD 5-STABLE before 14-Jan-2005 the
# /dev/mirror/gm0s1 device has to be specified as just "mirror/gm0s1" or
# the bsdlabel(8) will use the incorrect GEOM name "gm0s1" instead!)
# (NOTICE: figure out what partitions you want with "bsdlabel /dev/ad0" before)
# (NOTICE: start "a" partition at offset 16, "c" partition at offset 0)
bsdlabel -w -B /dev/mirror/gm0s1 # initialize
bsdlabel -e /dev/mirror/gm0s1    # create custom partitions

# manually copy filesystem data from first to to second disk
# (same procedure for partitions "g", etc)
newfs -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
dump -L -0 -f- / | (cd /mnt; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1d
mount /dev/mirror/gm0s1d /mnt/var
dump -L -0 -f- /var | (cd /mnt/var; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1e
mount /dev/mirror/gm0s1e /mnt/usr
dump -L -0 -f- /usr | (cd /mnt/usr; restore -r -v -f-)

# adjust new system configuration for GEOM mirror based setup
cp -p /mnt/etc/fstab /mnt/etc/fstab.orig
sed -e 's/dev\/ad0/dev\/mirror\/gm0/g' </mnt/etc/fstab.orig >/mnt/etc/fstab
echo 'swapoff="YES"' >>/mnt/etc/rc.conf # for 5.3-RELEASE only
echo 'geom_mirror_load="YES"' >>/mnt/boot/loader.conf

# instruct boot stage 2 loader on first disk to boot
# with the boot stage 3 loader from the second disk
# (mainly because BIOS might not allow easy booting from second ATA disk
# or at least requires manual intervention on the console)
echo "1:ad(1,a)/boot/loader" >/boot.config

# reboot system
# (for running system with GEOM mirror on second disk)
shutdown -r now

# make sure the first disk is treated as a really fresh one
# (also not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad0 bs=512 count=79

# switch GEOM mirror to auto-synchronization and add first disk
# (first disk is now immediately synchronized with the second disk content)
gmirror configure -a gm0
gmirror insert gm0 /dev/ad0

# wait for the GEOM mirror synchronization to complete
sh -c 'while [ ".`gmirror list | grep SYNCHRONIZING`" != . ]; do sleep 1; done'

# reboot into the final two-disk GEOM mirror setup
# (now actually boots with the MBR and boot stages on first disk
# as it was synchronized from second disk)
shutdown -r now

GEOM mirror Approach 2: Single Slice, Preferred, More Flexible

The following is a step-by-step command list for remotely converting a production FreeBSD 5.3-STABLE system from a single-disk/single-slice (ad0s1) to a two-disk/single-slice (ad0s1 & ad1s1) GEOM mirror (gm0s1) based setup without the need for console access or Fixit/LiveFS CDROM:

# make sure the second disk is treated as a really fresh one
# (not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad1 bs=512 count=79

# place a PC MBR onto the second disk
# (with a single FreeBSD slice /dev/mirror/gm0s1 as large as the /dev/ad0s1)
# either automatically if sizes fit...
fdisk -v -B -I /dev/ad1
# ...or manually to make sure the sizes fit:
size=`fdisk ad0 | grep ', size ' | head -1 | sed -e 's;^.*size \([0-9]*\).*$;\1;'`
# ...and reduce the size by one block because ad0 and ad0s1 else would
# share the same last sector which could lead to ad0 be recognized as
# the GEOM provider instead of ad0s1. Alternatively, you can keep ad0
# and ad0s1 of the same size and hard-code the ad0s1 GEOM provider by
# adding the -h option to the "gmirror label" command below.
size=`expr $size - 1`
(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f- -i /dev/ad1

# place a GEOM mirror label onto first slice of second disk
# (actually on the last block of the disk slice)
gmirror label -v -n -b round-robin gm0s1 /dev/ad1s1

# activate GEOM mirror kernel layer
# (makes the /dev/mirror/gm0s1 device available)
gmirror load

# place a BSD disklabel onto /dev/mirror/gm0s1
# (ATTENTION: in FreeBSD 5-STABLE before 14-Jan-2005 the
# /dev/mirror/gm0s1 device has to be specified as just "mirror/gm0s1" or
# the bsdlabel(8) will use the incorrect GEOM name "gm0s1" instead!)
# (NOTICE: figure out what partitions you want with "bsdlabel /dev/ad0" before)
# (NOTICE: start "a" partition at offset 16, "c" partition at offset 0)
bsdlabel -w -B /dev/mirror/gm0s1 # initialize
bsdlabel -e /dev/mirror/gm0s1    # create custom partitions

# manually copy filesystem data from first to to second disk
# (same procedure for partitions "g", etc)
newfs -U /dev/mirror/gm0s1a
mount /dev/mirror/gm0s1a /mnt
dump -L -0 -f- / | (cd /mnt; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1d
mount /dev/mirror/gm0s1d /mnt/var
dump -L -0 -f- /var | (cd /mnt/var; restore -r -v -f-)
newfs -U /dev/mirror/gm0s1e
mount /dev/mirror/gm0s1e /mnt/usr
dump -L -0 -f- /usr | (cd /mnt/usr; restore -r -v -f-)

# adjust new system configuration for GEOM mirror based setup
cp -p /mnt/etc/fstab /mnt/etc/fstab.orig
sed -e 's/dev\/ad0s1/dev\/mirror\/gm0s1/g' </mnt/etc/fstab.orig >/mnt/etc/fstab
echo 'swapoff="YES"' >>/mnt/etc/rc.conf # for 5.3-RELEASE only
echo 'geom_mirror_load="YES"' >>/mnt/boot/loader.conf

# instruct boot stage 2 loader on first disk to boot
# with the boot stage 3 loader from the second disk
# (mainly because BIOS might not allow easy booting from second ATA disk
# or at least requires manual intervention on the console)
echo "1:ad(1,a)/boot/loader" >/boot.config

# reboot system
# (for running system with GEOM mirror on second disk)
shutdown -r now

# make sure the first disk is treated as a really fresh one
# (also not really necessary, but makes procedure more deterministically ;-)
dd if=/dev/zero of=/dev/ad0 bs=512 count=79

# place a new PC MBR onto the first disk
# (with a single FreeBSD slice /dev/ad0s1 _exactly_ as large as the /dev/ad1s1)
size=`fdisk ad1 | grep ', size ' | head -1 | sed -e 's;^.*size \([0-9]*\).*$;\1;'`
(echo "p 1 165 63 $size"; echo "a 1") | fdisk -v -B -f- -i /dev/ad0

# switch GEOM mirror to auto-synchronization and add first disk
# (first disk is now immediately synchronized with the second disk content)
gmirror configure -a gm0s1
gmirror insert gm0s1 /dev/ad0s1

# wait for the GEOM mirror synchronization to complete
sh -c 'while [ ".`gmirror list | grep SYNCHRONIZING`" != . ]; do sleep 1; done'

# reboot into the final two-disk GEOM mirror setup
# (now actually boots with the MBR and boot stages on first disk
# as it was synchronized from second disk)
shutdown -r now

References

See the following URLs for further details: