This morning I figured out how to kick off a rebuild for the raid array from inside linux. The key to all interactions with the 3ware raid controller is the tw_cli command which I pulled from the cd, but it is also available on 3ware.com.
I started this process a few days ago by removing and then reinserting the same drive back into the same slot.
I ran tw_cli /c4/u0 show and saw that the array was degraded. I figured I just needed to tell the controller to rescan or rebuild and it would rebuild the degraded drive. The thing I didn't understand was that according to page 63 of the cli documentation I needed to first run tw_cli /c4/p1 export to destroy the DCB (Disk Configuration Block) on the disk in position 1. Then I ran tw_cli /c4 rescan to detect the unconfigured/new drive. And finally tw_cli /c4/u0 start rebuild disk=1 to rebuild the array with the new disk in position 1. My guess is that if I was replacing a failed drive with a brand new or unconfigured drive I would just need to do run the following after inserting the new drive:
tw_cli /c4 rescan
tw_cli /c4/u0 start rebuild disk=1
If you want to check on the status of your rebuild you can run tw_cli /c4/u0 show
I tried to get ide hot swap to work so that I could install a new drive without rebooting the system. Unfortunately the current software hotswap is little more than a hack that does not work with our hardware.
My conclusion is that the best way to add ad-hoc storage to a running system is via usb or firewire.
Another avenue that I have not explored is hot swapping SATA drives in the standalone SATA enclosure.
Adino & Eleazar are now setup in their most basic operating mode. Adino has samba installed with a couple of drive shares, which allows people to copy files onto him. I setup rsync to copy the files from himself to eleazar. Here is the cron script which runs every 15 minutes:
#!/bin/bash
command="rsync -vv --compress --stats --archive --delete /home/ eleazar:/home"
# We only want 1 rsync process at a time,
# so exit if one is already running
#
rsync_count=`ps aux|grep "$command"|wc -l`
if [ $rsync_count -gt 0 ]; then
echo "rsync already running. exiting."
exit 2
fi
source /root/.keychain/`hostname`-sh
$command >> /var/log/rsync
The log file /var/log/rsync is rotated and compressed by the logrotate daemon.
It is very important for the rsync functionality to work that you logon to each box manually after rebooting. I setup ssh to work with keychain which keeps a copy of the ssh authentication key in memory, but it only does this after you logon to the box and enter the password to de-encrypt the key. The key will then be loaded into memory in it's de-encrypted form and be available for rsync to use to authenticate itself to eleazar.