What made me nuts was performing backups. I use my rsync script I’ve shown earlier. Doing backups from one DataTale to another one was as slow as 5MB/s. Imagine that a 3TB backup takes a bit too long.

A solution must be found. Stripping down the data I want to backup to only company data, photos & documents helps, “just” 750GB instead 3TB. The rest is replaceable – mostly our DVD collection that I’m ripping to make it available to the media boxes / Mac minis around the house from a central place.

Now you may think: Why don’t simply get a RAID1 external device and connect it to a Mac Mini? This would be a half solution, yes, but I also want my other data in a secure way.

In the past I was using OpenSolaris for a bunch of tests, but never used it for a long time. The reason is that I’m not a UNIX administrator and don’t have too much clue of it. Additionally, Oracle has bought Sun and stopped OpenSolaris and now starts selling Solaris 11 Express for a fancy price of “just” 1.000$ per year. *lol* Well, on the one hand, as mentioned before, it’s not my OS and on the other hand, that would be a bit too expensive for SoHo usage I admit, I really think that ZFS is the way best filesystem and RAID management you can get for a server, so I tried Nexenta a few weeks ago, but had a lot trouble with my Onnto boxes via eSATA. In the end, the fact that Nexenta Core 3.0.1 is based on an unstable Ubuntu version (hardy-unstable), while Ubuntu itself is based on Debian-testing/unstable, was a bit too much “testing” for my taste.

Another OS that I installed and played with was OpenIndiana, the free successor of OpenSolaris. It’s currently available as a developer preview (b148). I installed it and while the setup itself looked a bit buggy and I finally was able to login, I first couldn’t find an option to set my dynamic assigned local IP to a static one. I looked on Google for more than half an hour, but maybe I’m a bit blind – didn’t found a solution. Most keys are not assigned so when hitting “Pic up” returns in some strange characters on the screen. Even in an editor like nano – very annoying. But they say: not for production use, so that’s fine and also this is not my flavor of an OS, but I gave it a short run.

There were just 2 ZFS distributions left that I would eventually gonna use: FreeNAS and FreeBSD.

While FreeNAS is pretty old and the ZFS version is from the mid-age, FreeBSD also comes with a pretty old ZFS version. v14 in FreeBSD 8.1 and v15 in the not yet released FreeBSD 8.2. There’s a LiveCD from sub.mesa (recent writer and fighter for ZFS in the Harforum ). I haven’t tested one of them, because the only OS in this area I know a bit more is Debian. Too bad Nexenta is unstable, it’s Debian – in some way.

So I ended up after 4h’s of playing around with a fresh installation of Debian 5.0.7 on the server. Luckily I haven’t touched the software RAID harddisks inside (6x Hitachi 24/7 drives á 1TB), just installed the systems on the 160GB IDE WD HD, so it was pretty easy to import the software RAID5 with mdadm. Yeah – I know, software RAID, if it’s not ZFS has it issues and yes, I would love to use ZFS, but there’s a line I don’t want to cross anymore – I simply want a stable system that works.

Setting up all services was quickly done, netatalk (AFP) and samba (SMB/CIFS) was installed and configured within a few minutes.

But there’s one thing that I don’t like about this setup and to be honest, I think that’s more an issue that you can’t fix. If I open up a directory with some more folders inside (in this case our photo directory with 162 folders), it takes ages until Finder runs smooth. But even worse, if you don’t wait, you can easily crash Finder. This is a bit disappointing. In my mind the only option to make this a bit quicker and more responsive is to turn off the file preview (photos, videos).

For a good performance, I tried AFP and SMB and copied huge single files. Of course, a folder with a lot small files which are together of the same size are way slower.

For the following tests, I’ve copied files from the software RAID5 to my internal WD Caviar Black HD in my Mac Pro via GBit ethernet.

Copying files from the software RAID5
AFP: 102MB/s peak, ~98-100MB/s average
SMB: 50MB/s peak, 30-35MB/s average

Writing files to the software RAID5
AFP: 102MB/s peak, ~80MB/s average
SMB: 60MB/s peak, ~20-25MB/s average

By the way: deleting a folder or a big file freezes the Finder window until the job is done – also a bad thing. Honestly, I don’t remember OS X Server AFPs or OS X Client share behave like this.

Comments and ideas are welcome as always (spam not ).

Imagine the system you know as Belldandy already, running OpenSolaris. The main system is installed on an IDE 160GB (WD / new) HD. There is an Intel Desktop CT 1000 + a Sil3132 chipped eSATA controller stored in both PCIe slots (x1 & x16). Now I would add the following HDs.

-640 Hitachi SATA-II
-640 WD SATA-II
-640 WD SATA-II
-640 WD SATA-II

2x Mirror, striped – Highest Security zpool – usable: 1280/1100GB

1000 Seagate SATA-II
1000 Seagate SATA-II

Mirror II – usable: 1000/990GB around about.

4000 STORAGE eSATA

Main Storage for very big files (Onnto DataTale “STORAGE”) usable: 2.7TB

4000 BACKUP eSATA

Backup of “Storage” (Onnto DataTale “BACKUP”) usable: 2.7TB

-160 Boot IDE

Simply the OpenSolaris on it. Usable: 140GB

…………….

12.700GB overall, usable after mirroring & striping: 7.630GB

Yeah, freaky plan and MAYBE a step back, but I still think that maybe ZFS (and even if I run the STORAGE & BACKUP in an external hardware RAID5) is the very best option.

For a secure and reliable storage that would feed my needs perfectly, I would say: Ok – I will NOT use Exchange. I will find another way to accept invitations, sure – easiest would be to run Outlook in a Parallels VM and simply accept it there. Exchange 2010 is pretty good, but I also have concerns to let it run when I’m out of the house for some days. That’s maybe my main painpoint.

So what do you guys think? Is it worth to retry it a last time? I’ve VERY compatible hardware here incl. meanwhile 6GB of ECC RAM for this box.

Would be great to get a feedback from you!

Ciao
Dennis

See for reference: About my project Belldandy

PS. Why I don’t think about using Debian? Well, I don’t see a very reliable filesystem there that I would like to use. ext3 is too slow for large files, xfs seems also not be the fastest one and ReiserFS is in jail

What is Belldandy?

Belldandy is a name of a fictional person, used in the Manga “Oh! My goddess“, one of my favorite comics. Reading this post, you will see that I use more japanese comic names for virtual machines.

But what is YOUR Belldandy?

Well, Belldandy is the name of my newest server. I write “newest”, because over the years I tried so much different systems and types of servers, that I’m really happy to announce a very stable server here.

What kind of server is it?

It’s a self-build one, but I used hardware parts that where recommended from Constantin Gonzalez from Sun. I’m using my old but very reliable TAGAN 450W power supply, an ASUS M4A78LT-M AMD mainboard which supports ECC memory and 6GB of ECC RAM. The OS harddisk is a new 160GB IDE HD. It’s not a 19″ server – it’s a regular Midi tower (MESH series by Chieftec).

Why IDE? Isn’t this a rather old standard? What about SATA?

I’m using all 6 onboard SATA ports for the zpool. IDE may be an old standard, but it works fine for the OS itself. With 6GB of RAM I don’t think too much about HD IOs on this HD. It’s more important, that the zpools are bound to a fast bus. I simply found it a waste of an port using a SATA HD for the system.

So you’ve 6 HDs inside the server?

Yes! I’ve installed 6x 1TB WD “Green” HDs into the server. Those are connected to the zpool “storage” which is the main storage of the server.

What is that “zpool”?

zpool is pool of HDs you’ve virtually stacked into one large virtual HD. It’s widely comparable to RAID, but it offers a lot advantages compared to a traditional RAID. I’m using 3x Mirror drives. Each mirror contains 2 HDs with the capacity of 1TB each. Together, because they are internally mirrored, I’ve 50% of the capacity. If you want to learn more about zpool and ZFS, I could highly recommend googling for it – there are tons of interesting and funny material out there.

Why do you mirror? Are your HDs so bad?

No It’s a thing of security for my data. Harddisks are not made forever and the fail. The question is just: When. Having a mirror protects my data from getting lost when one or more HDs are crashing the same time. Of course, if it’s a series issue, maybe I’m also not secured against this – 3 HDs where bought at the same day, but I think it’s a very way which let’s me sleep deep.

Which OS is that? Linux?

No, I’m running OpenSolaris which is something between Linux and UNIX, I would think it contains the best of both worlds. It’s mainly based on Sun’s Solaris but with a GNome environment. It’s a great system, but I’m also rather new to it, so I need to learn a lot about it. It’s amazingly pretty easy to create storages and zpools.

How do you connect to your server from your clients?

I’ve installed netatalk on the OpenSolaris machine to be able to use it with our Macs. It’s pretty much forward. A bad idea is to use SMB and Macs, there seems to be a bug inside the Finder’s SMB implementation. Anyway – it’s cool to have AFP running (netatalk).

Of course, I could always use the SCP protocol to connect to the server if I need to, but that’s more useful for the administration than for normal usage.

Isn’t it loud? Have you stored it in your office?

It’s really too loud as I’ve installed 4 very efficient fans into the server. I had it for a couple of days in our office and it was not very comfortable, so we decided to putit into the basement. I honestly prefer a quiet office (even music sometimes annoys me and I’m working mostly in silence).

You have some external boxes on that desk – what are they for?

The bigger box is a great box – it is my Onnto DataTale. It allows me to put 4 HDs into and comes with a hardware RAID. I’ve currently installed 4x 640GB WD Blue HDs and connected it via eSATA to the server to store temporary data onto. It gives me 1.8TB of usable storage.

What are these other 4 boxes for?

That’s (from left to right) a Seagate Desktop drive, a MacPower enclosure with another 1TB Seagate HD inside and 2 Hitachi external HDs. They are all connected via a USB2-Hub to the server and are set up as an RAIDz1 backup zpool. It gives me 2.7TB of available storage for backing up our data. RAIDz1 is more or less comparable with RAID5.

How do you connect the server to the internet, or is it an offline server for local use only?

It’s connected directly to the internet and out 2 DSL lines. OpenSolaris itself is connected to the Vodafone line which offers a fix IP and NO disconnect every 24h’s, so it’s a real and cheap dedicated line. It gives me 6MBit/s download and 640KBit/s upload. So or so, I don’t run any services on the host system itself for protection. The other line is from the Deutsche Telekom and allows us to surf with 16MBit/s (1MBit/s upstream). This is mostly used for the clients only.

Do you run any virtual machine on the server?

Yes, indeed. I run 2x Debian (“Urd” & “Mila”) (one as development machine for my web design & development, the other as media streaming device (namely Firefly iTunes server or “mt-daapd)). I also run a Windows Server (“Miyuki”) on the server as mailserver. It’s a Windows Server 2008 r2 x64 with an Exchange 2010 installed on top of it. If Microsoft creates a great product, it’s in my mind Exchange. And I say that as Mac user

Which software do you use to virtualize this stuff?

Sun gives away their virtualization software VirtualBox for free – and of course it runs fantastic on their own OpenSolaris. I’m very happy that it runs so well.

On my clients we run Parallels on my Mac Pro for Windows virtualization and my wife is running a VMWare Fusion on her MacBook for Windows 7.

Do you have a firewall installed in your rack?

Yes and no, I’ve a IBM NetVista mounted inside the rack. It’s a very small and slow machine – a Thin Client – but it’s fantastic to run IPCop on. Currently, because it was a long road to get to this very satisfying server, it’s switched off, but I will turn it on again soon, and even if it’s for local proxy, free DNS servers and local DNS (don’t you guys also hate to type IPs for LAN servers? ).

How much HD capacity/usable capacity have you attached to Belldandy?

- 6x 1 TB internal
- 1x 160 GB internal
- 4x 640 GB external
- 4x 1 TB external

Makes around about 12 TB of storage, I can use up to 7,8 TB for real storage. There’s a lot of overhead for security, but I think it’s really worth it. ZFS as filesystem itself is very secure, but in this combination, it’s fantastic and I’m really looking forward to have a permanent server. In case I want to try something else, I will now simply create a new VM to play with instead of killing the server itself.

Do you run Linux anywhere?

I do! “Obelix” – my dedicated webserver runs Debian Lenny, the 2 virtual machines are also both Debian and IPCop is also based on Linux. I would say, I’m using a good mix of the different operating systems.

Many thanks for reading this post! Hope you enjoyed it as much as I enjoyed typing it

Ciao
Dennis 

I’ve found this entry by crawling through sale e pepe which is another fascinating blog about OpenSolaris as (file)server and Macs as clients.

Ciao
Dennis 

First, login with your regular user. We’re gonna use pfexec here.

We need to install some stuff to be able to compile the database & netatalk itself.

# pfexec pkg install SUNWgcc SUNWgmake SUNWxorg-headers SUNWGtk SUNWsfwhea

Once this is done, we need to download the database. You can download it using wget.

# wget http://download.oracle.com/berkeley-db/db-4.4.20.tar.gz

This will – depending on your internet connection – take a moment. / In case you don’t want to download the file (or can’t for any reason) on your server, you should download the file to your workstation and kick it into your $HOME directory using SCP.

Now, let’s create a source folder where we can extract our files into.

# mkdir -p $HOME/src
# pfexec crle -u -l /usr/local/lib

The second command is used to install the stuff into the right place.

Now it’s time to extract the files.

# cd $HOME/src
# gzcat $HOME/db-4.2.52.tar.gz | tar xf -

Great, let’s create a new folder…

# mkdir -p $HOME/src/db-build
# cd $HOME/src/db-build 

…and configure the installation…

# ../db-4.2.52/dist/configure –prefix=/usr/local

You will notice a lot of text scrolling on your terminal. Once this is done, compile the installation.

# make

Oh dear! Even more text At the end of this process we need to install it.

# pfexec make install

That was a quick one, right? Congratulation! Your database is now installed.
Let’s move on the the netatalk itself.
First, we need to get the stuff, download it again using wget (or on your workstation and transfer it via SCP). I’ll grab this one, but maybe at a later time, you want to get a more current version from here: http://sourceforge.net/projects/netatalk/files/netatalk/

# cd $HOME
# wget http://sourceforge.net/projects/netatalk/files/netatalk/2.0.5/netatalk-2.0.5….

Join the src folder again… and extract the netatalk files.

# cd src
# bzcat $HOME/netatalk-2.0.5.tar.bz2 | tar xf -

The last thing we need to do is to edit some files. I would suggest to use nano as editor. Of course you can use vi, too

 

Save your file – if your editor complains, be sure that you’ve used the pfexec command

I hope you enjoyed this little tutorial and are now able to use your OpenSolaris server with your Mac clients in a for us “more natural way”

P.S. You need to connect your Mac clients by hand to your server, as we haven’t installed the Avahi daemon (yet), so click the Finder icon on your desktop and press CMD+K. Type afp://<SERVER-IP> and you should be ready to use it

Before someone asks – I’m a big fan of Bon Jovi and I’ve ALL albums original.

• SFTP or FTP (works – I’ve tested it) – uncomfortable but ok for basic restoring to the zpool
• rsync via network (not to the SMB mount!) – ok for backing up, but not for live working with the data on the server
• Netatalk – not sure where and if it exist for OpenSolaris, but maybe the best option

So currently, if I can’t get Netatalk to work on OpenSolaris, as much as I want to use it, I’ll have to remove it It makes no sense to access all stuff ONLY via SFTP. SMB fails here.

I’ll keep you updated on this!

Ciao
Dennis

I started over by installing Win Server 2008 R2 Eval onto the AMD server to see what happend if I copy these maybe (or maybe not) corrupted files to a NTFS partition. For my surprise – it worked. It worked like a chame. Every single bit was transferred without any issues.

So what’s the problem? I’m not sure and as mentioned before, I’m not an OpenSolaris expert, but for me it seems like the ZFS denies to copy a corrupt file. That’s just an idea, I didn’t say that this is the reason. I mean, sure – it makes no sense to copy over corrupted files, but neither Finder nor the Terminal nor muCommander have asked my if I want to skip this (maybe) corrupt file(s).

I’m currently stuck here and the weekend is over (oh great, and I wanted to have this stuff finished before I return to work on Tuesday).

If you’ve ANY ideas, you’re more than welcome to leave your comment here!

Many thanks!

Ciao
Dennis

Tool on top: Finder, Tool below: mucommander.

What you see are 2 independent tools that I tried to copy my “photos” folder from the backup HD (external) to the ZFS volume using Samba.

First test, as described last night, was using my “root” user and moving the stuff with the “cp” command. Even as the both windows above, it simply stopped working. Ctrl+C? No escape – needed to reboot the machine.

But now comes the very strange part! I copied ALL files successful from the external HD to the internal RAID10 in my Mac Pro. No issues! All (tested, of course I didn’t tested 190GB of files) were working fine.

I’ve no idea what to do now. I’ve that bad feeling, that it depends on the SMB implementation. Maybe I could try it using a FTP server on the OpenSolaris machine, but this is no real solution.

Any ideas?

Ciao
Dennis

But, it seems as there is a version of Firefly for OpenSolaris, but I prefer the installation of Firefly inside a Debian VM. It was pretty easy to set up, downloading Sun’s VirtuaBox (free!), Installation within a few minutes and in the end, setting up the virtual Debian. This all was done in less than 1h.

To access the files on OpenSolaris, I simply installed the Samba client for Debian, and mounted the share within the /etc/fstab with the following command:

//192.168.0.50/share /media/share smbfs defaults,credentials=/etc/recurring.smbpass 00

“Urd”, another name from my fav. Mangas is shown up thanks to the Avahi daemon running on the Debian VM.

Ciao
Dennis

So the first step is done. The main pool is created. Now it’s time to allow users to access the Storage via Sun’s own Samba implementation which is built into OpenSolaris. Yey! Looks like I’m fallen in love with the ORIGINAL system

More soon

Ciao
Dennis