I was running my first VMs on the NFS share from the half-virtualized Debian Fileserver. That was working well – but could be faster. So I decided to simply plug my 3x 640GB HD SFF-8087 from the HP SAS Expander to the RAIDable other Intel/LSI controller and create a RAID10e of it. RAID10e is a funny RAID that allows you to use also 3,5,7 etc. HDs as a Mirror. As result, I’ve got a ~900GB RAID10e storage that is luckily recognized by ESXi (the controller itself is supported). This is a very fast storage for my VMs. I still use my 2.5″ drives in the server to store the ISOs on. Of course, I could do this also on the NFS share, but to have a copy of them is a good idea.

As the big storage on the half-virtualized fileserver is about 10TB, I don’t really care to create a TimeMachine space there. There’s really enough free space and doing this is on my very long “ToDo” list. At the moment, I’m restoring my data from the backup server back to the fileserver using “curlftpfs” – that is a very nice FUSE add-on for Linux which allows you to rsync files from a FTP share (you mount it like a drive, a Samba share…). I now receive the files at ~120MB/s.

The backup server will receive a new network card – I found a Dual Port Intel PCI-X card (HP labeled) for just 9€. The board does not have a full PCI-X slot, but I was told that it will work, even if it’s a regular PCI slot. At this price, I bought 2 of these cards – one for the gateway – that would allow me to create a nice pfsense router from a Futro (I will get a second one tomorrow ) – so one for gateway + one for router. I just hope the card is not too long – otherwise I need to cut the case a bit…

What you see here is far from “finished”. The HP ProCurve for example is not yet installed and also the Gateway is still on my ToDo list, but you get the idea.

The big black case is the ESXi server, the smaller silver behind that is the backup server. The rack contains the UPS, the 1U SuperMicro server and the electric wiring – and needs to get the final finish! Also, there’s a Compaq 4 Port KVM switch the CRT & keyboard are connected to that I’ve got for free from Christian.

Speaking of mess – this is the inside of the ESXi server – so much cables running through the box. But I’m sure you guys are curious to see the inside.

From top to bottom:
- LSI Controller (IT Firmware) -> connects to the HP SAS Expander via SFF-8087 cable
- LSI Controller (IR Firmware/Intel brand) -> connects the 3x 640GB via “RAID10E/IME” and is the home for the VMs as by today as the NFS was a bit to slow for my taste – or: this option is a lot faster
- HP SAS Expander -> the beast which connects up to 32 drives
- Intel Pro 1000 CT PCIe x1 NIC

The red light on the bottom right is the USB stick which is shown here. The ESXi software itself is stored there.

I’ve successfully installed my Debian Fileserver with the following steps:
- Created the first – RAID6: 6x 1.5TB /dev/md0
- Created the second – RAID6: 6x 1TB /dev/md1
- Created the third – RAID5: 3x 640GB /dev/md2 (Time Machines)
- Installed lvm2 and created a LVM volume from /dev/md0 & /dev/md1

/dev/mapper/LVM-LVM     10T    23G    10T   1% /storage
/dev/md2               1,3T   5,3M   1,3T   1% /timemachines

- Mounted the LVM into /storage
- Mounted /dev/md2 into /timemachines
- Put both into the /etc/fstab
- Installed NFS-kernel-server
- Created the directory /storage/esxi
- Gave the rights nobody:nogroup to this directory
- Shared it via exports -a
- Connected the NFS volume to ESXi

- Upload the Windows Server 2008 R2 EN ISO into the DataStore A (2.5″ 160GB)
- Installed Windows Server 2008 R2 EN on the NFS share
- Updated the Win Server
- Run benchmark (see video below)

Here is a benchmarks done with dd on the Fileserver with no running VMs:

root@risu:/storage# time dd if=/dev/zero of=/storage/bench bs=1024 count=10000000
10000000+0 Datensätze ein
10000000+0 Datensätze aus
10240000000 Bytes (10 GB) kopiert, 88,0244 s, 116 MB/s

real	1m28.090s
user	0m0.616s
sys	0m9.841s

What confuses me; why do I reach higher speed over the 10Gbps VMXNET 3 network to the VM than locally? Does that make sense to anyone?

I dropped ZFS (again), because in a bunch of tests, I had some issues with SMB for example and also with NFS shares. I need to get this “pig to fly” – so I stopped digging around and decided to use the technology that worked very well for me in the last few months: mdadm.

The fileserver is half-virtualized. It is known that you should NOT virtualize a fileserver. I fully agree! I just virtualize the underlaying Debian, the rest is on “PCI passthrough”. This great technology allows me to pop the HDs & the cards out of the box in case of an emergency (or boot from a Debian CD/Stick or HD) and access all the data. The mdadm works with the REAL hard drives. In this layer, nothing is virtualized anymore. That’s in my mind the only way to use a fileserver on ESXi. The idea is based on the ZFS/ESXi All-In-One box I mentioned in a previous post.

However, the hard disks are connected to the HP SAS Expander, which is connected to the Intel SASUC8i (a LSI 1068 chip controller with LSI “IT” firmware). I use a single SFF-8087 cable for that. This draws the connection speed to SATA150, but you don’t recognize that. Why? The RAID combines the speed I think I loose a max. of 5%.

The LSI controller is “passed through” to the Debian VM which lists the HDs as if they were connected to a regular OS.

Inside Debian:

I will install NFS server on the Debian VM. VMware tools are already installed, enabling the 10Gbps VMXNET 3 driver. ESXi will mount the NFS share where all the other will be placed on. And that is what I will do now

More on this – soon!

PS. The backup server is now also running Debian – because of the previously mentioned issues with OpenIndiana/SMB & NFS.

Read more on Benchmarks of the ESXi contruction

More photos on flickr!

This rack is (in my mind) supersexy (for a rack ) and I couldn’t await to install my stuff inside. So I discovered for a not so short period of the day how the hell to put the rails from both DELL servers inside. In the end, I had to put the older PowerEdge 1650 onto the base and attach the rails for it to the very first unit. The reason? Well, the rails are a bit too long, so I could not connect the back ends of to the back supports. Putting it to the bottom allows the 1650 to gently slide above the end of the rack and with the width of the server, the rails are nice in place.

The R210 was more or less easier to install. I had to turn both ends by 180° (like it was mentioned in the instructions). It does not really fit to the front side, but it’s ok – and fits within the rack so that I’m able to close the glass door.

After putting the rack to the basement with help of my wife (isn’t it great to have such a flexible wife!? ), I put both servers inside and connected the R210 with the eSATA drive, 2x GBit network and of course power, keyboard & VGA display. The server starts as hoped

Here’s a screenshot of the server after booting up all 4 current VMs:

Both – R210 & Onnto DataTale are connected to my smaller APC UPS. I hope it can get VMware ESXi to work with it, so that it automatically shuts down/holds all VMs and shut itself down if there’s a power outage.

This is just the first step. All my 19″ equipment will move to the new rack over the next few weeks, but earlier (planned for Saturday), the old PowerEdge 1650 will be the host for Endian Firewall which allows me use both DSL lines the same time and split ports to the different lines. The reason is quit simply: If I want to send for example a 10MB file to a client, I don’t want that Exchange fills the complete 1MBit/s upstream of the main DSL line. This line is also used for surfing etc., the 2nd DSL line (Vodafone) with it’s static IP and non-disconnect should so this for me. Also, I don’t want to have speed issues in case a user access the OWA of Exchange.

As always: More to come, here!

Ciao
Dennis 

So, I saw me looking around for a stable replacement for this, especially because I gave this new AMD machine to my mother to run Win 7 on it, because her old AMD64 was really really End Of Live .

I found a very nice and relatively cheap server. First, I wanted to get a DELL PowerEdge T110. A tower based one. But then I though: Why not put it into your rack? You’ve one and also another 19″ DELL server (a PowerEdge 1650). So I my choice was, after some phone calls while staying for 2 days at the EuropaPark in Rust with the DELL team, a PowerEdge R210 19″ server. (Believe me, it was pure fun to sit in this monorail driving through the park and discussing with the nice sales guy from DELL about the best configuration for what I had in mind).

What I had in mind was a VMware ESXi server. Yeah, I know, I complained a lot about VMware Fusion on Mac, but VMware ESXi seems to be a very different story. Some of you may wonder how I want to manage that, now that I’ve no “PC” back here. Well, running a Win XP VM is easily done and I use one for my regular work in Office 2003 or Office 2007 (depends on the client), so I simply installed the VMware Sphere Client there and what should I say? It works fantastic (on my Mac Pro!). Yes, I moved the Mac Pro back to the office after overhauling the office last week, but that’s a complete different story (Photos are available here by the way).

I’m currently sitting here in front of my cool new GALANT desk, grin broadly like an idiot while running my Win XP SP3 VM on the left 22″ display, typing this and doing the main stuff on the 30″ in front of me while having chats, twitter and the new surveillance (AXIS) cam on the other 22″ on the right. I must say, I’d missed the two 22″ the last few weeks when I was working with the MacBook only. Can’t help myself – I’m simply addicted to screen real estate.

Anyway – to come back to the topic; Anyanka is the name of the VMware ESXi / DELL PowerEdge R210 server. Why? Well, I ran out of names, and yes, it’s chosen from a character of “Buffy – the vampire slayer”

The data of the server:
- DELL PowerEdge R210 Rack Case (up to 2x internal HDs) / 1U
- Intel Xeon X3450 Quad Core CPU (+ HT = 8 Cores – yey!)
- 8GB DDR3-ECC-Regged 1333MHz RAM (4x 2GB Modules / UDIMM (ugh!))
- 160GB SATA HD (which is disconnected until I need it – any day – or not…
- 16x DVD-ROM
- iDRAC6 Embedded BMC (not sure if I could/want to use it and how it works.. but… it’s there)
- NO OS (yes, there’s a license sticker on the top of the case, I just say “THANK YOU DELL ”)
- 250W Power Supply
- 2/4 Post Static Rack Rails

As you can see, it’s a very robust system. The next Xeon in the house which is clocked at the same speed as the Mac Pro CPU. It should have more than enough power to run my few VMs.

Enough the words for the moment, I know you all want to see photos (if you haven’t seen my flickr photos yet), so – here they are

As a design oriented person, I HAD TO order this great looking bezel for the front, I’m sure it will look awesome when it’s in the rack!

Memory & the powerful Intel Xeon X3450 CPU.

An overview above the components built in.

A very good idea, if you want to run a VMware ESXi, IPCop, Endian or a FreeNAS on this machine: the internal USB ports to attach a USB stick and install the OS onto it. I use a 8GB stick to run VMware ESXi on it.

You can see the FULL SET of photos here on flickr.

Maybe you’re wondering why I don’t post photos from the server inside the rack. That’s a not sooo funny story. When I said: Ok, I want to get a 19″ 1U server, I also ordered rails for my older DELL PowerEdge 1650 which will be reused as Endian or IPCop soon. I’ve received the rails and found out, that my rack is NOT for servers. I’ve never noticed that there are no support on the back of the rack, because until today, everything worked fine with the front supports only. Hmm.. so I went to eBay, looking for a small rack and I found one. 15U for only 180 Euros incl. shipping from Poland to here. I’m still waiting for a lifesign and I must admit, that’s this deal seems not to be the cleverest idea I’d ever had, since I paid with PayPal and for 1 week now, never have heard anything from that dealer. I’ve send meanwhile 3 eMails to them and never got a reply. When I don’t get the rack or ANY information, I’ll turn to eBay and cancel it :/

The other question that possibly will come up is: Where do I store my VMs?

That question is easily answered: I’m using one of my Onnto DataTales! Funny enough, this is one of the first servers supporting eSATA! After fiddling around for some hours, my friend Merlin had the idea to put the external RAID to RAID10 (well it’s RAID0+1), to get down to 2TB. The issue is, that the free version of VMware ESXi does NOT recognize HDs greater than 2TB in this version. When it was set to 2TB it worked.

Here’s a photo of the back ports of the R210 I like it!

So that’s it for today! Hope I can show you some more photos of the server built into the a new rack, soon!

Ciao
Dennis

PS. Before I forgot: DELL says the server consumes 107 Watts on Full Load, but only 37 Watts on idle. I’ll check that later and post my results.