There's not much documentation out there on how to setup OpenBSD on a simple software RAID. There's lots of documentation on how to perform obscure RAID/OpenBSD tricks. After muddling my way through the process, i've decided to document it... and document it in a way that is useful for those (like me) who need it thoroughly explained. So here it is: A step-by-step guide to installing OpenBSD on a software RAID using the built in RAIDFrame software.
These instructions will lead you through an install of a software RAID1 (mirroring) using an x86 machine and two IDE/ATA hard drives. The current OpenBSD release at the time of this writing is 3.6. I'll try to make some comments to help those trying to make a RAID5. If you're using non-IDE drives, you'll have to substitute your devices (e.g. sd0 and sd1 for SCSI) for wd0 and wd1.
This is the basic gist of what we'll be doing:
First we'll go to www.OpenBSD.org and get an install disk image from our favorite mirror listed here. Select your mirror, then navigate to: 3.6 (or the most recent release) -> i386, and download cd36.iso (this will become cd##.iso depending on the release).
Use the iso image to burn an install CD. (You could alternatively get the floppy install image... but who uses floppies? do you even have a floppy drive? i don't!) Then, put the CD in your machine and boot!
Here's the install process:
(I)nstall, (U)pgrade or (S)hell? i (enter)
Specify terminal type: [vt220] (enter)
Do you wish to select a keyboard encoding table? [no] (enter)
Proceed with install? [no] y (enter)
Available disks are: wd0 wd1.
Which one is the root disk? (or done) [wd0] (enter)
Do you want to use *all* of wd0 for OpenBSD? [no] y (enter)
Initial label editor (enter '?' for help at any prompt)
> z (enter)
> a a (enter)
offset: [63] (enter)
size: [#########] 256m (enter)
FS type: [4.2BSD] (enter)
mount point: [none] / (enter)
> a b (enter)
offset: [########] (enter)
size: [#########] 64m (enter)
FS type: [swap] (enter)
> a d (enter)
offset: [########] (enter)
size: [#########] (enter)
FS type: [4.2BSD] RAID (enter)
mount point: [none] (enter)
> q (enter)
Write new label?: [y] (enter)
Available disks are: wd1.
Which one do you wish to initialize? (or 'done') [done] (enter)
The next step *DESTROYS* all existing data on these partitions!
Are you really sure that you're ready to proceed? [no] y (enter)
Enter system hostname (short form, e.g. 'foo'): foo (enter) (enter your hostname)
Configure the network? [yes] (enter)
Available interfaces are: fxp0
Which one do you wish to initialize? [fxp0] (enter)
Symbolic (host) name for fxp0? [foo] (enter)
Do you want to change the media options? [no] (enter)
IPv4 address for fxp0? (or 'none' or 'dhcp') dhcp (enter)
DNS domain name? (e.g. 'bar.com') [my.domain] bar.com (enter) (enter your domain)
DNS nameserver? (IP address or 'none') [###.###.###.###] (enter)
Use the nameserver now? [yes] (enter)
Default IPv4 route? (IPv4 address, 'dhcp' or 'none') [dhcp] (enter)
Edit hosts with ed? [no] (enter)
Do you want to do any manual network configuration? [no] (enter)
Password for root account? (will not echo) ******** (enter)
Password for root account? (again) ******** (enter)
Where are the install sets? (or 'done') f (enter)
HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] (enter)
Display the list of known ftp servers? [yes] (enter)
Server? (IP address, hostname, list#, 'done' or '?') 62 (enter) (pick your favorite server. boulder is closest to me.)
Server? (IP address, hostname, list#, 'done' or '?') [ftp3.usa.openbsd.org] (enter)
Does the server support passive mode ftp? [yes] (enter)
Server directory? [pub/OpenBSD/3.6/i386] (enter)
Login? [anonymous] (enter)
File name? (or 'done') [bsd.mp] -game36.tgz (enter) (i'm building a serious machine - no games)
File name? (or 'done') [bsd.mp] done (enter)
Ready to install sets? [yes] (enter)
Install files will begin downloading and when they're finished, you'll see:
Where are the install sets? (or 'done') done (enter)
Start sshd(8) by default? [yes] (enter)
Do you expect to run the X Window System? [yes] n (enter) (i'm building a web server, so i dont plan to run X)
Change the default console to com0? [no] (enter)
What timezone are you in? ('?' for list) [Canada/Mountain] US (enter) (go USA!)
What sub-timezone of 'US' are you in? ('?' for list) Mountain (enter) (yay colorado!)
CONGRATULATIONS! Your OpenBSD install has been successfully completed!
yada yada...
# halt (enter)
Remove your install CD and press any key to reboot
Login to your new OpenBSD system as root. First we'll intialize our second disk (wd1), then we'll mount it. This will give us space to download the kernel source and recompile.
foo# disklabel -E wd1 (enter)
Initial label editor (enter '?' for help at any prompt)
> z (enter)
> a a (enter)
offset: [63] (enter)
size: [###########] (enter)
FS type: [4.2BSD] (enter)
> q (enter)
Write new label? [y] (enter)
foo# newfs /dev/wd1a (enter) (I had overlooked this step. Thanks to Aaron Sue for pointing this out.)
foo# mount /dev/wd1a /mnt (enter)
foo# cd /mnt (enter)
My prefered method of obtaining the kernel source is by lynx/http. To do this, you'll do the following:
foo# lynx www.openbsd.org/ftp.html#http (enter)
Use your arrow keys to navigate to your favorite server (i use the USA -West Lafayette, IN server... gotta stay true to my alma mater. go boilers!). The next page will display a file listing. Navigate to 3.6 (or current release). Then navigate to sys.tar.gz and download it. sys.tar.gz contains the kernel source and all the necessary bits to compile a new one. (note: don't get src.tar.gz - it contains everything, except the kernel source!)
You'll need to extract the source and edit the config file.
foo# tar xzvf sys.tar.gz (enter)
foo# cd sys/arch/i386/conf (enter)
Use your favorite editor (mine's Vi) to edit the file GENERIC. Add a # at the beginning of lines to comment them out (and vice-versa to uncomment lines).
I comment out the following lines because i've got a machine newer than a pentium 1:
#option I386_CPU
#option I486_CPU
#option I586_CPU
I comment out this line because my machine (and nearly recently made computer) has a math co-processor:
#option GPL_MATH_EMULATE
I comment out this line because, as i said before, i'm not planning to run X Windows:
#option APERTURE
I uncomment this line because it provides better speed:
option DUMMY_NOPS
MOST IMPORTANTLY, i uncomment this line to include RAID:
pseudo-device raid 4
Beneath the 'pseudo-device raid' line add this line which will allow us to use the RAID autoconfig:
option RAID_AUTOCONFIG
foo# config GENERIC (enter)
foo# cd ../compile/GENERIC (enter)
foo# make depend (enter)
foo# make (enter)
Backup your current kernel (you can never be too safe):
foo# mv /bsd /bsd.old (enter)
Install your new kernel (yay!):
foo# mv bsd /bsd (enter)
Clean up after yourself:
foo# cd /mnt (enter)
foo# rm -rf * (enter)
Reboot with your new kernel!
foo# halt (enter)
First we'll need to match our second drive (wd1) to our first drive (wd0).
foo# disklabel wd0 > disklabel.wd1 (enter)
foo# fdisk -i wd1 (enter) This initializes the MBR for our second disk (suggested by Mark).
foo# disklabel -R -r wd1 disklabel.wd1 (enter)
Next we'll need to create our RAID config file. Create the file raid0.conf in the /etc folder. Here's what we'll put in the file:
START array
1 2 0
START disks
/dev/wd0d
/dev/wd1d
START layout
128 1 1 1
START queue
fifo 100
What's this mean? START array begins our setup of the RAID array. We're setting this to 1 raid array made up of 2 disks and 0 spare disks. If we wanted to make a raid5 with 4 disks, the numbers would be "1 4 0" (note that 4 disks is the maximum allowed). START disks begins our setup of which specific disks to include. Here we've included partition 'd' of wd0 and wd1 (you'll remember that during our intial setup we specified a FS Type 'RAID' for these partitions. START layout begins our setup of the RAID disk. we have 128 sectors per stripe unit (or the interleave factor in blocks), 1 stripe unit per parity unit, 1 stripe unit per reconstruction unit, and a raid level of 1 (if we were building a raid5, this would be '5'). START queue begins our queue setup. this is specified as fifo or First-In, First-Out and limited to 100 requests.
Next we'll copy our current system (on our first disk, wd0) to our second disk. This way if our first disk fails, we can still boot our raid off our second disk.
foo# mount /dev/wd1a /mnt (enter)
foo# cd /mnt (enter)
foo# dump -0f - / | restore -rf - (enter) Note: Vijay commented that he encountered problems with symbolic links not copying properly (perhaps due to having installed X-Windows). Please email me (jyee at argon18 dot com) if you experience similar problems... i'll try to track down a solution.
You may recieve some 'ohci0: 1 scheduling overruns' errors. You can ignore these. I've read that these have something to do with the USB bus and don't harm anything.
foo# cd / (enter)
foo# /usr/mdec/installboot -v /mnt/boot /usr/mdec/biosboot wd1 (enter) Note: Pawel added this bit to make the secondary disk bootable. In the few tests I've run, sometimes the system booted without this, most times it didn't. i assumed it didnt because of raid corruption, but in retrospect, it was probably because i'd omitted this step.
foo# umount /mnt (enter)
foo# raidctl -C /etc/raid0.conf raid0 (enter)
Be sure that you use a capital "-C", lowercase "-c" will produce a non-functioning RAID! You will receive a number of error messages, including "number of rows/columns do not match" and fatal error messages. Do not worry!
foo# raidctl -I 100 raid0 (enter)
You will receive an error "raid0: no disk label", but don't worry, we'll give it a label later.
foo# raidctl -iv raid0 (enter)
Your raid should now begin parity re-write. You will receive more disk label errors and possibly some ohci0 errors, but again, don't worry we'll get to this shortly.
We finally deal with the raid0 disk label issue. In this step we're partitioning the raid0 disk as we would a single disk install of OpenBSD. I'm building a webserver so i'm going to build this with a big /var so I can store a lot of logs, a big /usr so I can compile lots of stuff, and a big /home so my users can make *huge* sites (or I can have lots of users with small sites). You'll have to adjust your partition sizes to suit your needs.
foo# disklabel -E raid0 (enter)
Initial label editor (enter '?' for help at any prompt)
> a a (enter) (/)
offset: [0] (enter)
size: [#########] 200m (enter)
FS type: [4.2BSD] (enter)
> a b (enter) (swap)
offset: [######] (enter)
size: [#########] 256m (enter)
FS type: [swap] (enter)
> a d (enter) (/tmp)
offset: [#######] (enter)
size: [#########] 256m (enter)
FS type: [4.2BSD] (enter)
> a e (enter) (/var)
offset: [########] (enter)
size: [#########] 3g (enter)
FS type: [4.2BSD] (enter)
> a f (enter) (/usr)
offset: [########] (enter)
size: [#########] 4g (enter)
FS type: [4.2BSD] (enter)
> a g (enter) (/home)
offset: [########] (enter)
size: [#########] (enter)
FS type: [4.2BSD] (enter)
> q (enter)
Write new label? [y] (enter)
Note that for each of these commands, there are two r's in the device name. (rraid0a, NOT raid0a) Also note that we skip /dev/rraid0b... this is our swap partition. The swap partition will not work if you newfs it.
foo# newfs /dev/rraid0a (enter)
foo# newfs /dev/rraid0d (enter)
foo# newfs /dev/rraid0e (enter)
foo# newfs /dev/rraid0f (enter)
foo# newfs /dev/rraid0g (enter)
First we'll need to mount our raid partitions, then we'll use the same 'dump | restore' command we used earlier to match our disks.
foo# mount /dev/raid0a /mnt (enter)
foo# cd /mnt (enter)
foo# mkdir tmp (enter)
foo# mkdir var (enter)
foo# mkdir usr (enter)
foo# mkdir home (enter)
foo# mount /dev/raid0d /mnt/tmp (enter)
foo# mount /dev/raid0e /mnt/var (enter)
foo# mount /dev/raid0f /mnt/usr (enter)
foo# mount /dev/raid0g /mnt/home (enter)
foo# dump -0f - / | restore -rf - (enter)
foo# raidctl -A root raid0 (enter)
We need to modify our fstab to reflect our new mount points. To do this we'll need to edit /mnt/etc/fstab to look like this:
/dev/raid0a / ffs rw 1 1
/dev/raid0b none swap sw 0 0
/dev/raid0d /tmp ffs rw,nodev,nosuid,softdep 1 2
/dev/raid0e /var ffs rw,nodev,nosuid,softdep 1 2
/dev/raid0f /usr ffs rw,nodev,softdep 1 2
/dev/raid0g /home ffs rw,nodev,nosuid,softdep 1 2
That's it! Let's cross our fingers and reboot:
foo# halt (enter)
During the boot process, you should see:
/dev/rraid0a: file system is clean; not checking
for each of your raid partitions.
We can check on our raid array by running the following command:
foo# raidctl -s raid0 (enter)
The reply should be something like:
raid0 Compenents:
/dev/wd0d: optimal
/dev/wd1d: optimal
No Spares.
Parity status: clean
Reconstruction is 100% complete.
Parity Re-write is 100% complete.
Copyback is 100% complete.
Congratulations, you now have a RAIDed OpenBSD system!
I've received a few questions concerning what to do if the raid fails. There's actually 2 answers to this question... and it depends on whether you experience a complete device failure or a dirty failure. By "dirty failure" i mean a situation in which the drive is known to be good, but is reported to be failed.
In the event of a dirty failure, the following command will rebuild the raid array:
foo# raidctl -R /dev/wd0d raid0 (or /dev/wd1d... or whatever disk has soiled itself).
Note that the -R flag will fail a drive (if it is not already failed) and immediately rebuild the array. This is can be useful to test your array.
In the event of a device failure, you'll need to replace your drive (duh!). Next you'll need to:
1. re-match the drives (see '3. Setup a RAID Disk: Matching drives')
2. foo# newfs /dev/wd1a (or wd0a if that's the drive that failed)
3. copy the boot partition to your new drive (see '3. Setup a RAID Disk: Copying boot to drive 2')
From this point you should be able to run the raidctl -R /dev/wd1d raid0 mentioned above.
Anyway, that's how it works in theory. When misfortune strikes me down and i have a device failure, I'll try to write a step by step guide for recovery. Until then, I hope that's enough information to help you through.
OpenBSD 3.7 has been released and Ted Bardusch sent in a great question about upgrading from 3.6 to 3.7. He pointed out the two obvious choices: 1. wipe your disks and do a fresh install of 3.7 (the instructions above still apply to 3.7) and 2. download sources for everything, recompile, and replace.
There is a third option - and it's much easier. First we'll reference the OpenBSD 3.6 to 3.7 upgrade guide (or whatever the current upgrade guide is at the time you're reading this). You should read through the guide before upgrading... but if you're too lazy, here's a 4 point summary:
1. download the new 3.7 install files
2. install the new generic 3.7 kernel and firmware blobs
3. reboot on the generic 3.7 kernel
4. install the "userland application" file sets (i.e. install everything else in the 3.7 install)
If we did this, we'd run into trouble when we reboot (#3), because the generic kernel does not include RAID support. I'm assuming that you are running your OpenBSD 3.6 RAID system... so here's what we need to do:
1. Follow the OpenBSD upgrade guide and STOP when you "Place install files in a "good" location."
2. Download the OpenBSD 3.7 kernel source
3. Perform the actions in part 2 ("Recompile RAID Into the Kernel") section 4 ("Configuiring the kernel") and 5 ("Compiling the kernel"). This will install the 3.7 kernel. Note: you will not need to perform the "Clean up after yourself" step... since we're not using /mnt to juggle space.
4. reboot on your RAID'ed 3.7 kernel
5. Continue following the OpenBSD upgrade guide with the step "Install new /etc/firmware files"
This will get your main RAID system onto OpenBSD 3.7, however your initial boot partitions (/dev/wd0a and /dev/wd1a) will still be running OpenBSD 3.6. If you are lazy, you can leave them alone. If you want to upgrade them, you can do the following:
1. mount the /dev/wd0a partition (foo# mount /dev/wd0a /mnt)
2. copy the 3.7 RAID kernel to the mounted partition (foo# cp -R /bsd/* /mnt/bsd/)
3. Follow the OpenBSD upgrade guide (skip the "Reboot on the new kernel" step. For step "Install new userland applications", change "cd /" to "cd /mnt")
4. unmount /dev/wd0a and repeat with /dev/wd1a
That should do it.
©2004 Jason Yee. jyee (at) argon18 (dot) com.