Yesterday, I called the Intel hotline and after 1h of very very good support (seriously, I was surprised!), I got a ticket ID and some instructions. The broken Intel SASUC8i will be picked up on Friday via DHL Express. I will receive a replacement Intel SASUC8i in a few days. Oh! The controller in the main server is a new one I bought back in September – so we’re talking about 2 different.

As long-term readers, you know about my little external cases – these Onnto DataTales. 4 bays per box with USB2, FW400/800 and eSATA. They became unreliable in the last months – kicking out some of the hard disks randomly, so I decided it’s enough. Additionally, I hate to split things – some stuff was on the box, the other stuff on the other one. D’oh. No. I wanted a better, more reliable and bigger solution.

I checked my hardware stock in the basement and found all stuff I need to create a backup server. Surfing a bit too much Data Storage Systems of the [H]ard|Forum, I read so much about ZFS, the new Illumos Kernel and napp-it, that I decided to get back to ZFS for the backup server.

After plumbing the PC parts to a server for a few hours, I had a running system with these specs:

I installed OpenIndiana (151a with Illumos Kernel) and created a RAIDz(1) pool of the 8x 1TB HDs which was online and usable after a few seconds – the joy of ZFS Of course, I wanted to have a nice GUI, even if I like to work on terminals. My choice is the awesome & free napp-it.

The server is currently getting loaded via rsync from the main server – because rsync via SSH was very slow (~20-30MB/s on the Gbit LAN), I now use an NFS share that is directly mounted on the main filer. It will take some time to backup all our stuff to the server – there are meanwhile ~5TB of important (well – more or less ) stored. With the new server, ~6TB usable backup storage, ECC RAM & ZFS, I’ve a very good and secure feeling.

The main server currently runs a pure Debian system, which I like, but maybe I will put ESXi back on again once I received the replacement controller from Intel, create a hardware RAID1 of my 2x 640GB HDs, install ESXi and an OpenIndiana VM onto this array. This setup is called the “All-In-One-Box” and my SuperMicro board is ready for this, supporting PCI(e)-PassThrough very well. But more on this later

PS. The gateway is nearly done, unfortunately, I couldn’t use the HD inside this box, because of heat – but it works fine externally – USB2 is USB2. More on this, also later – so please stay tuned!

This week, my OpenIndiana presented me this message:

 pool: storage
 state: UNAVAIL
status: One or more devices could not be used because the label is missing
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from
        a backup source.
   see: http://www.sun.com/msg/ZFS-8000-5E
 scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     UNAVAIL      0     0     0  insufficient replicas
          raidz2-0  UNAVAIL      0     0     0  insufficient replicas
            c1t0d0  FAULTED      0     0     0  corrupted data
            c1t1d0  FAULTED      0     0     0  corrupted data
            c1t2d0  FAULTED      0     0     0  corrupted data
            c1t3d0  FAULTED      0     0     0  corrupted data
            c1t4d0  ONLINE       0     0     0
            c1t5d0  ONLINE       0     0     0
            c1t6d0  ONLINE       0     0     0
            c1t7d0  ONLINE       0     0     0
            c2t1d0  ONLINE       0     0     0
            c2t2d0  ONLINE       0     0     0

Hmm.. had no clue what’s going on. I’ve seen the speed of data transfer and accessing the the files via AFP went slower in the last days, I guess this was an indicator for what happend. Luckily, when moving from Nexenta to OpenIndiana, I had done backups on 2 HDs in my Mac Pro – those were still available, and so I lost nearly no data. Luckily! I was a bit shocked by this super-secure filesystem and so I took a look on Google and look what I found: Dateiserver Uni Bielefeld. This text is German, sorry – it describes that their 100TB storage, based on ZFS crashed 2x and they now switched back to Debian and ext4. Interesting!

While being more a fan of Debian than of Solaris (and knowing it much better as I use it as development platform for ctrltweet every day), I decided to stop messing around with a system and filesystem that seems to be done for bigger arrays than my home server usage. So I went with Debian, kicked out the 1.5TB HDs and installed some 1TB HDs.

The system looks mostly the same, but for completeness – here’s a hardware overview:
- AMD Athlon 245e (2.9GHz, DualCore, “energy efficient”)
- 6x Hitachi 1TB 24/7 drives
- 5x WD10EADS 1TB “Green” drives
- 1x WD1001FALS 1TB “Black” drive
- 16GB ECC RAM
- System (Debian Squeeze) installed on an Intel 80GB SSD
- Intel PCIe x1 GBit Ethernet card (CT)
- Intel SASUC8i (LSI) controller with IT Firmware
- old ATI PCI graphics card which is fine for console and up to 800×600

Everything is cooled by fans and the temperatures of the HDs are between 25 and 28°C under load.

Creating the RAID10 took a while – 8h’s to be exact and I’ve postponed ANY action on it until it’s done – heard that this should work out better. Later in the evening, it was finished and I formatted the “md0″ with ext4. Very nice and fast (and meanwhile stable) filesystem. As result I have 6TB (or 5.4TB) available for my data.

I went the RAID10 road because of 3 reasons:
- Security
- Performance
- have just 3.2TB of data at the moment and grows not THAT fast

Backup is done to one of the Onnto DataTales that I still have NOT sold. I first had some issues with an eSATA PCIe x1 card and it was terrible slow. For another project I wanted to do in the past, I had 2x SATA to eSATA cables laying around in the basement. Hmm… I had one port unused left on the board, so I connected it, opened up a “COM” port hole in the case and moved it to the outside. The Onnto box now stays on top of the case, having 4x 1.5TB in RAID5 (4.5TB available) and connected directly to the mainboard. The performance is VERY good. hdparm -tT get’s 180MB/s YEY! That’s fine for a nightly rsync backup.

More on this later Still rsyncing my stuff over to the server from my Mac Pro.

Last week, I managed to install VirtualBox on this machine. The reason: I wanted to virtualize my VMs (and a real server) on this machine. Read my last 2 posts about that.

Today, I’ve got some RAM to upgrade Karinka. Previously, I had 6GB ECC RAM (DDR3-1333) installed, but ZFS + VirtualMachines boosted the RAM usage up to 97% and it started swapping to the syspool (IDE HD). I then bought 16GB ECC RAM (2x Kits of 8GB (2x4GB each)).

Here are some photos:

Before: 2x2GB ECC RAM + 2x1GB ECC RAM

Removed the old memory

Installed 4x4GB ECC RAM

Booting it up and… hui… everything as hoped

saidar -c -d 1 shows this in Nexenta

Karinka now runs with 6GB of RAM, which was nice fast for ZFS only. Today in the late afternoon, I decided to give VirtualBox a try. I currently run a Virtual Machine for my development stuff on my Mac mini server using VMware Fusion and a real intranet server in the rack. That worked fine and I have nothing negative to say about that. But wouldn’t it make sense to have the VMs also running on Karinka? I mean – ZFS is a reason alone. Don’t have to run those 2 machines 24/7 + the ZFS server is also a good one. So I googled a bit and quickly found a manual for NexentaStor, not Core or NCP, but I checked very single package with “apt-search cache ” and all were found.

You will find this awesome guide over here.

Installation of X forwarding to the Mac
While the installation was pretty easy, I needed Jacek’s help to get the X forwarding to work on my Mac Pro. In the end, it was pretty easy to get it to work, if you know what you need to do. So I tell you

On your Server:

$ export $DISPLAY="localhost:10.0"

# apt-get install xorg

On your Mac:
Install X11 from the system DVD. Reboot your system! This is mandatory!
In the terminal, you should receive a similar output:

kurumi:~ dieta$ echo $DISPLAY
/tmp/launch-qXH2MK/org.x:0

Now, type:

$ ssh -X [serverip/name]
[...]
# VirtualBox &

VirtualBox and basic configuration of a new VM
This should push VirtualBox after 2-3 seconds on your Mac screen.

Now you can go on and create a VirtualBox VM. Put the ISOs on a reachable path on your server – oh – and maybe run it NOT as root but as your user!

Of course, closing VirtualBox or the VM window would STOP your VM from working, so we can just use it for basic configuration, but that’s fine. VirtualBox has some very nice CLI tools available. Once you’ve installed your fresh Debian/Windows/Solaris… whatever – you can shut it down.

Go back to the terminal and type:

$ VBoxHeadless -s "[VMName]" &

It is very important to be aware of the case sensivity! Also the “&” at the end of line is important. It will put your task to the background, so it will not close down when you leave the terminal.

Importing a VMware Fusion VM to VirtualBox
Believe me guys, this costed me some more grey hairs So – here we go.

First – open up your VMware Fusion with your VM, create a NEW Snapshot (sorry, you can’t take them with you) and DELETE the old ones!). If you won’t do this, the oldest Snapshot will appear on VirtualBox and you can’t do anything against it.

Done? Great! Now do a right click on your VMware Fusion file and show the package contents, enter the folder and copy all files. Open the folder on your server (doesn’t matter if that’s SMB, NFS, FTP, SCP or AFP ) and create a new folder with the name of the virtual machine. Paste the files into.

I was using my Debian development platform (copy of course!) and copied it. There are some things you should know. VirtualBox loves IDE – so every imported and fresh created virtual disk will be IDE. VMware Fusion loves SATA. So you see – if you run Debian – it won’t boot because it looks for /dev/sda1 and /dev/hda1 exist

But – step by step.

When you’ve copied the files over to your Nexenta server, click on “File” -> “Virtual Media Manager”.

Click the “Add” icon and select the main vmdk file in the directory of your virtual machine. In my case it was dev.vmdk, as the name of my VM is Dev. Hit “Ok”.

Next, we need to create a BLANK VM to import the virtual harddisk into. Click the “New” button, then “Next” and enter the name of the new/imported VM. Be careful! Case sensitive! I use “Dev”, “Linux”, “Debian” (32bit) for my Development VM. And hit “Next”.

256MB is fine for my Dev Machine, so “Next”.

Then I select the “Use existing hard disk” radio and select my “dev.vmdk”.

On the last screen accept all settings. Done! Now we need to edit some settings.

First of all, we need to move from IDE to SATA. Click your imported VM and then the button “Settings”, then on “Storage”. Hit the 3rd icon (white with green plus) on the bottom of the box and select “Add SATA controller”. Click the SATA controller and hit the first icon (data with green plus) and click it until the “dev.vmdk” appears. Delete the not used datasets, also, you can remove the Floppy and CD/DVD drive if you don’t need them. On my site it looks like that:

I have also removed USB2 and Audio support. I’ve set the network to “bridged” and the graphic card memory to 16MB. That’s why I get the message “Non-optimal settings detected”, but that’s fine. Hit “Ok” and you’re done!

Now we need to launch the VM from the terminal, but first, we need to make one more settings to the VM itself.

If I would run it now, I would receive this error message:

dieta@karinka:~$ nohup VBoxHeadless -s "Dev" &
Sun VirtualBox Headless Interface 3.1.6
(C) 2008-2010 Sun Microsystems, Inc.
All rights reserved.

VRDP server failed to start.
VRDP server is inactive.

We will have to set the the vrdpport. Shut down the now running VM:

$ VBoxManage controlvm "Dev" poweroff

Now the setting:

$ VBoxManage modifyvm "Dev" --vrdpport 3390

(!) The port number has to grow! If you run 3 machines it should look like:
VM1: 3389
VM2: 3390
VM3: 3391
etc.

And finally launch the VM:

$ nohup VBoxHeadless -s "Dev" &

Press Enter afterwards to get back into the foreground, the VM is running in the background. In my case, I had setup a static IP and after 30 seconds, I was able to reach the VM with my browser and my terminal via ssh & ping.

dieta@dev:~$ uptime
 23:45:41 up 1 min,  1 user,  load average: 0.20, 0.08, 0.03
dieta@dev:~$ uname -a
Linux dev 2.6.26-2-686 #1 SMP Thu Sep 16 19:35:51 UTC 2010 i686 GNU/Linux

Hope you enjoyed this! Until this morning, I was not sure if I could run ANY virtualization on this machine but I have never expected that it will work so great!

I’ve ordered 16GB ECC RAM today, ZFS loves memory and VMs also do That’s just a little upgrade – because nothing big will change before 2012.

ZFS offers the option to do snapshots of your current data. This counts per pool. It’s 100% incremental and a very space-saving option to do backups! Please note that it will NOT 100% replace a backup – if you’re running a RAIDz1 and 2 drives die the same time, all snapshots are also lost, but it’s fine to revert back accidentally deleted files.

The regular syntax to do a snapshot looks like that

zfs snapshot backup/test@13012010

zfs snapshot is the command with the parameter
@ defines the snapshot
13012010 is an example name that makes most sense in my mind. You can use anything else like “project_test”, too.

My script is (like gesund) made to be placed into the crontab. Here’s the code

#!/bin/bash
Dat=$(date +%d%m%Y)
echo Creating snapshot of pool $1
zfs snapshot $1@$Dat

As I mentioned in the intro – pretty small one

Usage

./snapshot backup/test

This will create a snapshot of the used pool with the current date at the end. It’s meant to be used 1x per day.

On the crontab, I’ve added it like this

0 11 * * * /root/snapshot storage/privat
2 11 * * * /root/snapshot storage/firmendaten
4 11 * * * /root/snapshot mediapool

It will run every day at 11am, doing a snapshot one after one for every pool.

Of course, you can make it a bit more flexible placing it into the crontab like this

0 11 * * * /root/snapshot storage/privat && /root/snapshot storage/firmendaten && /root/snapshot mediapool

Don’t forget to set the shell script to chmod a+x to run it

Oh! Nearly forgot – you want to see your snapshots listed? No problem!

zfs list -t snapshot
NAME                             USED  AVAIL  REFER  MOUNTPOINT
storage/test@project_test           0      -  43.4K  -

This will contain ALL snapshots you do.

New features
- Logging
Everytime you run the script, it will place the follwing line into the log file: /var/log/gesund.log

Example

Checking ZFS pool 'storage' on 12.01.2011 at 09:00:00
Everything fine on 'storage'!
-----------------------------------------------------

- Testmode
Why to manipulate the script everytime you want to see if the script works fine when your zpool is not “healthy” anymore.

Usage

./gesund.sh backup -test

It is important that you use the parameter “-test” as 2nd one and that you add an existing zpool name.

prowl requirements
Maybe you see prowl.pl complaining like this:

root@karinka:~# perl prowl.pl -apikey=xxxx -notification="FEHLER BEIM BACKUP" -event=BACKUP -application=KARINKA -priority=2
Can't locate LWP/UserAgent.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at prowl.pl line 33.
BEGIN failed--compilation aborted at prowl.pl line 33.

Then you need to install libwww-perl:

apt-get install libwww-perl

and Crypt:SSL.eay, too:

perl -MCPAN -e 'install Crypt::SSLeay'

Many thanks to @jck_d for explaining how to install this and to @MaximDim for highlighting that this also could happen on other systems

Download
Download 3KB, .tar.gz

or

wget http://klein2.de/files/gesund-02.tar.gz

Hopy you enjoy this updated script Please also refer to my previous post to read about why I did this script and how it may help you

PS. My crontab now scans ALL pools every 3h’s – this helps to see WHEN something went wrong.

What does this script do?
gesund.sh grabs the result of the command zpool status -x . If it says “healthy” – fine, but if not it sends a message to your iOS device complaining that something is not Ok.

Can I put it into my crontab?
Yes! It’s made for this

What do I need?
- prowl for your iOS device (2,39€ one time, sending and receiving messages just costs you data transfers if you’re out of a WiFi spot)
- a (free) prowl API key
- root access to your ZFS server (I’ve coded this script for Nexenta – if you want to use it on other systems, you MAYBE need to change some stuff)
- installed perl on your ZFS server
- the following script

Download
Download 2KB gesund-01.tar.gz archive

or

wget http://klein2.de/files/gesund-01.tar.gz
tar -xvfz gesund-01.tar.gz

Customize the script (needed)
Open the gesund-01.sh in your favorite editor (I personally use nano, a lot love vi.. whatever ).

In line 25, replace with your own prowl API KEY. You should also replace the event and application name so that it fits your needs.

If you want to test the script first (what I would recommend, also do this for line 19 and remove the # in front of it.

Once this is done, you have to set the right file rights.

chmod a+x gesund-01.sh
chmod a+x prowl.pl

Test
Look for your pool names typing zfs list.

./gesund-01.sh backup

If everything was set up fine and you’ve removed the # in line 19 and customized it, you should receive a message telling you that (hopefully) everything is fine.

On screen you should see:

Everything fine!

or

ZFS pool error in pool [poolname]

Extended testing
In line 10, replace “healthy” with anything that the command zpool status -x does NOT contain. This will give you the error message.

crontab
It makes a lot sense to use it in your crontab, so this check will be done often enough – it’s perfectly timed after a scrubbing.

Example

0 10 * * * /root/gesund-01.sh backup && /root/gesund.sh mediapool

As you can see, my gesund-01 script runs every day at 10am. I haven’t placed it a night – because I sleep that time. You can easily attach more pools to check by simply adding a && between both commands.

I have the files placed in my /root folder – of course you can place it where ever you want and the cronjob has access to! For the Linux users reading this – on Solaris (and that’s the Nexenta kernel), you don’t need to add a username in front of the command inside the crontab.

Hope you guys will find it useful!

- Mac Pro “Kurumi” in the office
- iMac “Indy” in the office
- MacBook “Dawn” – everywhere
- MacBook “Hammy” – everywhere
- Mac mini “Belldandy” in the living room
- Mac mini “Saki” in the office

As both Mac minis are “just” used to playback videos (from the Nexenta server), TV (Sat) & Radio (Sat), I don’t think a backup makes any sense. So – still 4 Macs to backup left. On Debian it was a hassle to create new partitions now and oh – the backup pool (4x 1.5TB RAIDz1) was already created and data is stored onto.

Luckily, ZFS is a fantastic filesystem. I simply create new pools below the mainpool.

zfs create backup/tm-kurumi

Done! This took just a few seconds. Now I want that Time Machine does not fill up the complete space of the backup pool, so I set a quota. 500GB sounds reasonable, since I save most of my stuff directly to my server and my Mac Pro’s OS drive is a 240GB SSD.

zfs set quota=500G backup/tm-kurumi

Let’s see if it works:

root@karinka:/var/spool/cron/crontabs# zfs list
NAME                     USED  AVAIL  REFER  MOUNTPOINT
backup                  1.15T  2.85T   578G  /backup
backup/tm-kurumi         65K   500G   65K  /backup/tm-kurumi

Looks good! The next step is to assign the correct rights to this volume, so that the user can really use it when logging on to it via AFP and later using Time Machine.

chmod 770 -R /backup/tm-kurumi
chown -R dieta /backup/tm-kurumi

“dieta” is my username to access the AFP network.

Last step on the UNIX side is to make this volume “supported”. In the past a lot people were working with the following command on the CLIENT

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

but this is a bad idea – you need to redo this on every single client. There’s an easier option (especially from admins point of view) to make a volume supported from the server’s side.

touch /backup/tm-kurumi/.com.apple.timemachine.supported

That’s all you need to do (sure, per volume, but it saves you to touch others Macs from the terminal). Now this is a fully supported Time Machine volume Oh! It seems like this also works for Debian & other systems and not only Nexenta/OpenSolaris, so give it a try!

To share it, we need to add a single line (per Time Machine) to the /etc/netatalk/AppleVolumes.default:

/backup/tm-kurumi       "Time Machine Kurumi"   allow:dieta cnidscheme:dbd options:tm

Maybe you want to restart netatalk now using

/etc/init.d/netatalk restart

But I found out, that’s it’s not needed. AFP loads the netatalk config files when you log on to the server via Finder.

At last, connect to your server via AFP and log in to the new volume and them tell Time Machine to use this volume. Here’s a screenshot how it looks when the backup is done

Hope you enjoyed this post and have learned some stuff!

By the way: did it for all 4 Macs and this looks like this on Nexenta:

I like the little tool hddtemp, but unfortunately it’s not available for the OpenSolaris kernel. But there’s a maybe better alternative: smartmonctl. I’ve got a link last night to this Simon’s blog who wrote a lot about ZFS & OpenSolaris. smartmontools seems not to be maintained for OpenSolaris/Solaris anymore, BUT in this case, I found it via apt-get in Nexenta.

So here’s how to install it:

apt-get install smartmontools

That was easy, but to get some information out of your HDs you need a command, or even better, a script that scans all your HDs and displays the temps. Simon was so nice and did one, I just had to change some things to make it work on Nexenta.

#!/usr/bin/bash
# is there a way to populate the array by querying the zfs pool directly?

drives=(c2d0 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c3t0d0 c3t1d0 c3t2d0 c3t3d0)
temp=""
drivetemp () {
  drive=$1
#  printf "drive passed in = %s\n\n" $drive
  temp=`/usr/sbin/smartctl -a -d scsi /dev/rdsk/$drive | \
       grep Current | awk '{print $4 $5}'`
  return
}
drives_count=${#drives[@]}
index=0
while [ "$index" -lt "$drives_count" ]
do
  drivetemp ${drives[$index]}
  printf "Drive %s temp is %s\n" ${drives[$index]} $temp
  let "index = $index + 1"
done

This is my config, but of course you need to add your own drives. In the above script you can see them in italic. To see your drive names, simply type:

format

and press Ctrl+C to get out of the command. You should see a list like that:

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 ATA-Hitachi HDS72101-A39C-931.51GB
          /pci@0,0/pci1458,b002@11/disk@0,0
       1. c0t1d0 ATA-Hitachi HDS72101-A3EA-931.51GB
          /pci@0,0/pci1458,b002@11/disk@1,0
       2. c0t2d0 ATA-Hitachi HDS72101-A3EA-931.51GB
          /pci@0,0/pci1458,b002@11/disk@2,0
       3. c0t3d0 ATA-Hitachi HDS72101-A3EA-931.51GB
          /pci@0,0/pci1458,b002@11/disk@3,0
       4. c0t4d0 ATA-Hitachi HDS72101-A39C-931.51GB
          /pci@0,0/pci1458,b002@11/disk@4,0
       5. c0t5d0 ATA-Hitachi HDS72101-A3EA-931.51GB
          /pci@0,0/pci1458,b002@11/disk@5,0
       6. c2d0 DEFAULT cyl 19454 alt 2 hd 255 sec 63
          /pci@0,0/pci-ide@14,1/ide@0/cmdk@0,0
       7. c3t0d0 ATA-WDC WD10EADS-65L-1A01-931.51GB
          /pci@0,0/pci1002,5978@2/pci1000,3140@0/sd@0,0
       8. c3t1d0 ATA-WDC WD10EADS-65L-1A01-931.51GB
          /pci@0,0/pci1002,5978@2/pci1000,3140@0/sd@1,0
       9. c3t2d0 ATA-WDC WD10EADS-65L-1A01-931.51GB
          /pci@0,0/pci1002,5978@2/pci1000,3140@0/sd@2,0
      10. c3t3d0 ATA-WDC WD10EADS-00L-1A01-931.51GB
          /pci@0,0/pci1002,5978@2/pci1000,3140@0/sd@3,0
Specify disk (enter its number): 

Now simply note your drive names and put them into the script – done! At least, make the script accessable:

chmod a+x hddtemp.sh

Running this could give you an output like this:

root@karinka:~# ./hddtemp.sh
Drive c2d0 temp is
Drive c0t0d0 temp is
Drive c0t1d0 temp is
Drive c0t2d0 temp is
Drive c0t3d0 temp is
Drive c0t4d0 temp is
Drive c0t5d0 temp is
Drive c3t0d0 temp is 27C
Drive c3t1d0 temp is 26C
Drive c3t2d0 temp is 26C
Drive c3t3d0 temp is 26C

If you don’t have temperatures for some HDs check if you typed the HD names correctly and check if your mainboard has SMART enabled. Without that, you won’t get any results. In my case – SMART only works for the 4 drives connected to my LSI SAS controller, because I forgot to enable it for the onboard controllers.

PS. If you’re wondering why I write something about Nexenta, even if I wrote that my Debian SW RAID5 works fine in my last post, well – it was because I found some corrupt files a few days ago and want to avoid having them in future. On private data I would say: doesn’t matter that much (ok noone wants broken photos ), but especially with my company data, I’m very careful. Still – Nexenta is based on Debian which makes it easy for me to use it. I also tried FreeBSD 8.1, ZFSguru, OpenIndiana and Solaris 11 Express in VMs for some days, but in the end, Nexenta is the distribution with the best read/write performance and best to handle. I know it’s based on “hardy-unstable”, Nexenta Core 3.0.1 itself is STABLE. Don’t ask me why. The ZFS – which is the main part of the filer is the same as in OpenSolaris. 4 more drives will be used starting today: 4x 1.5TB which drives to storage to 16TB + 160GB for the system on a seperate drive.

What made me nuts was performing backups. I use my rsync script I’ve shown earlier. Doing backups from one DataTale to another one was as slow as 5MB/s. Imagine that a 3TB backup takes a bit too long.

A solution must be found. Stripping down the data I want to backup to only company data, photos & documents helps, “just” 750GB instead 3TB. The rest is replaceable – mostly our DVD collection that I’m ripping to make it available to the media boxes / Mac minis around the house from a central place.

Now you may think: Why don’t simply get a RAID1 external device and connect it to a Mac Mini? This would be a half solution, yes, but I also want my other data in a secure way.

In the past I was using OpenSolaris for a bunch of tests, but never used it for a long time. The reason is that I’m not a UNIX administrator and don’t have too much clue of it. Additionally, Oracle has bought Sun and stopped OpenSolaris and now starts selling Solaris 11 Express for a fancy price of “just” 1.000$ per year. *lol* Well, on the one hand, as mentioned before, it’s not my OS and on the other hand, that would be a bit too expensive for SoHo usage I admit, I really think that ZFS is the way best filesystem and RAID management you can get for a server, so I tried Nexenta a few weeks ago, but had a lot trouble with my Onnto boxes via eSATA. In the end, the fact that Nexenta Core 3.0.1 is based on an unstable Ubuntu version (hardy-unstable), while Ubuntu itself is based on Debian-testing/unstable, was a bit too much “testing” for my taste.

Another OS that I installed and played with was OpenIndiana, the free successor of OpenSolaris. It’s currently available as a developer preview (b148). I installed it and while the setup itself looked a bit buggy and I finally was able to login, I first couldn’t find an option to set my dynamic assigned local IP to a static one. I looked on Google for more than half an hour, but maybe I’m a bit blind – didn’t found a solution. Most keys are not assigned so when hitting “Pic up” returns in some strange characters on the screen. Even in an editor like nano – very annoying. But they say: not for production use, so that’s fine and also this is not my flavor of an OS, but I gave it a short run.

There were just 2 ZFS distributions left that I would eventually gonna use: FreeNAS and FreeBSD.

While FreeNAS is pretty old and the ZFS version is from the mid-age, FreeBSD also comes with a pretty old ZFS version. v14 in FreeBSD 8.1 and v15 in the not yet released FreeBSD 8.2. There’s a LiveCD from sub.mesa (recent writer and fighter for ZFS in the Harforum ). I haven’t tested one of them, because the only OS in this area I know a bit more is Debian. Too bad Nexenta is unstable, it’s Debian – in some way.

So I ended up after 4h’s of playing around with a fresh installation of Debian 5.0.7 on the server. Luckily I haven’t touched the software RAID harddisks inside (6x Hitachi 24/7 drives á 1TB), just installed the systems on the 160GB IDE WD HD, so it was pretty easy to import the software RAID5 with mdadm. Yeah – I know, software RAID, if it’s not ZFS has it issues and yes, I would love to use ZFS, but there’s a line I don’t want to cross anymore – I simply want a stable system that works.

Setting up all services was quickly done, netatalk (AFP) and samba (SMB/CIFS) was installed and configured within a few minutes.

But there’s one thing that I don’t like about this setup and to be honest, I think that’s more an issue that you can’t fix. If I open up a directory with some more folders inside (in this case our photo directory with 162 folders), it takes ages until Finder runs smooth. But even worse, if you don’t wait, you can easily crash Finder. This is a bit disappointing. In my mind the only option to make this a bit quicker and more responsive is to turn off the file preview (photos, videos).

For a good performance, I tried AFP and SMB and copied huge single files. Of course, a folder with a lot small files which are together of the same size are way slower.

For the following tests, I’ve copied files from the software RAID5 to my internal WD Caviar Black HD in my Mac Pro via GBit ethernet.

Copying files from the software RAID5
AFP: 102MB/s peak, ~98-100MB/s average
SMB: 50MB/s peak, 30-35MB/s average

Writing files to the software RAID5
AFP: 102MB/s peak, ~80MB/s average
SMB: 60MB/s peak, ~20-25MB/s average

By the way: deleting a folder or a big file freezes the Finder window until the job is done – also a bad thing. Honestly, I don’t remember OS X Server AFPs or OS X Client share behave like this.

Comments and ideas are welcome as always (spam not ).