What you see here is far from “finished”. The HP ProCurve for example is not yet installed and also the Gateway is still on my ToDo list, but you get the idea.

The big black case is the ESXi server, the smaller silver behind that is the backup server. The rack contains the UPS, the 1U SuperMicro server and the electric wiring – and needs to get the final finish! Also, there’s a Compaq 4 Port KVM switch the CRT & keyboard are connected to that I’ve got for free from Christian.

Speaking of mess – this is the inside of the ESXi server – so much cables running through the box. But I’m sure you guys are curious to see the inside.

From top to bottom:
- LSI Controller (IT Firmware) -> connects to the HP SAS Expander via SFF-8087 cable
- LSI Controller (IR Firmware/Intel brand) -> connects the 3x 640GB via “RAID10E/IME” and is the home for the VMs as by today as the NFS was a bit to slow for my taste – or: this option is a lot faster
- HP SAS Expander -> the beast which connects up to 32 drives
- Intel Pro 1000 CT PCIe x1 NIC

The red light on the bottom right is the USB stick which is shown here. The ESXi software itself is stored there.

I’ve successfully installed my Debian Fileserver with the following steps:
- Created the first – RAID6: 6x 1.5TB /dev/md0
- Created the second – RAID6: 6x 1TB /dev/md1
- Created the third – RAID5: 3x 640GB /dev/md2 (Time Machines)
- Installed lvm2 and created a LVM volume from /dev/md0 & /dev/md1

/dev/mapper/LVM-LVM     10T    23G    10T   1% /storage
/dev/md2               1,3T   5,3M   1,3T   1% /timemachines

- Mounted the LVM into /storage
- Mounted /dev/md2 into /timemachines
- Put both into the /etc/fstab
- Installed NFS-kernel-server
- Created the directory /storage/esxi
- Gave the rights nobody:nogroup to this directory
- Shared it via exports -a
- Connected the NFS volume to ESXi

- Upload the Windows Server 2008 R2 EN ISO into the DataStore A (2.5″ 160GB)
- Installed Windows Server 2008 R2 EN on the NFS share
- Updated the Win Server
- Run benchmark (see video below)

Here is a benchmarks done with dd on the Fileserver with no running VMs:

root@risu:/storage# time dd if=/dev/zero of=/storage/bench bs=1024 count=10000000
10000000+0 Datensätze ein
10000000+0 Datensätze aus
10240000000 Bytes (10 GB) kopiert, 88,0244 s, 116 MB/s

real	1m28.090s
user	0m0.616s
sys	0m9.841s

What confuses me; why do I reach higher speed over the 10Gbps VMXNET 3 network to the VM than locally? Does that make sense to anyone?

I used the following hardware for KARINKA (somehow the font looks like an IKEA furniture, right? )

CASE		Yeong Yang YY-0221
CPU			AMD Athlon X2 245e (45W)
BOARD		Gigabyte GA-MA770T-UD3
RAM			4x 4GB Kingston ECC RAM
NICs		2x Intel Desktop CT 1000 PCIe x1
CONTROLLER	Intel SASUC8i/LSI flashed to IT firmware
GRAPHIC		old ATI Rage 3D PCI
SYS HD		WD 160GB Caviar Blue IDE
RAID HDs	6x Hitachi 1TB DeskStar SATA
			1x WD 1TB Caviar Green SATA
			1x WD 1TB Caviar Black SATA
			3x Samsung 1.5TB HD154UI SATA
			3x WD 1.5TB Caviar Green SATA
OS			Debian Squeeze

And here some data of the HDD temps (while running in the basement).

root@karinka:~# temps

Temperatures of RAID Array Mediapool
---------------------------------------
/dev/sdb: SAMSUNG HD154UI: 26°C
/dev/sdc: SAMSUNG HD154UI: 27°C
/dev/sdd: SAMSUNG HD154UI: 30°C
/dev/sde: WDC WD15EARS-00Z5B1: 36°C
/dev/sdf: WDC WD15EARS-00Z5B1: 36°C
/dev/sdg: WDC WD15EARS-00Z5B1: 38°C

Temperatures of RAID Array Storagepool
---------------------------------------
/dev/sdh: Hitachi HDS721010CLA332: 30°C
/dev/sdi: Hitachi HDS721010CLA332: 31°C
/dev/sdj: Hitachi HDS721010CLA332: 31°C
/dev/sdk: Hitachi HDS721010CLA332: 30°C
/dev/sdl: Hitachi HDS721010CLA332: 31°C
/dev/sdm: Hitachi HDS721010CLA332: 32°C
/dev/sdn: WDC WD10EADS-00M2B0: 32°C
/dev/sdo: WDC WD1001FALS-41Y6A0: 35°C

Temperature of the system disk
---------------------------------------
/dev/sda: WDC WD1600AAJB-00J3A0: 36°C

I think those are quiet nice temperatures

PS. I’m now using the plugin “RSS Image Feed” – hopefully it will resize the included images to 200px width…

My daily route to work would be a 35km trip from Moers to their office in Düsseldorf. Not that far, good highway connection but the highway itself is the A57. Around the afternoon/evening you have traffic there. Daily. Like a clock work. Also, thinking of the environment, it saved me A LOT of fuel. A simple calculation shows: 30€ for a 2nd line vs. 1400km/month = 80€ fuel costs. Not calculating wasted hours and hours in the car inside awful jams.

So I ordered a 2nd line from Arcor (today: Vodafone). 6M down and 640K up. Not really fast, but much better than the Telekom line those days. I ordered it as a business customer to benefit from their static IPs. Vodafone lines are NOT disconnecting every 24h’s – so – great deal for 30€ per month.

Maybe I should highlight, that Telekom came up with ADSL2+ a few weeks later, providing 16M down and 1M up. Urgs. Ok… the contract says: 24months. It’s a pretty stable and relatively cheap line, so I still have it. Today, I work on my own as a Freelancer and I’m more than happy to be able to switch if one line is has a bad day (which does not happens often, but I like redundancy ).

So here’s how I’ve installed it:

DSL Line 1: Telekom 16M / 1M -> FritzBox A -> 10.0.0.a / 255.255.0.0
DSL Line 2: Vodafone 6M / 640K -> FritzBox B -> 10.0.0.b / 255.255.0.0

I run a small OS X Server on a Mac mini which provides DNS & DHPC services to the LAN. MAC addresses are assigned to IPs and DNS & Gateway information.

All “server-related” computers (both Mac minis at the moment) are bound to the Telekom line for better downstream (for example Apple Update Service etc.), while all Clients are bound to the Vodafone line – 6M is great for surfing. Maybe I will implement more stuff like a Linux (VM) based Proxy (squid) to speed it up a bit more, soon.

For example: My MacBook get’s this information from the DHCP server:

IP: 10.0.x.y
Subnet: 255.255.0.0
Router/Gateway: 10.0.0.b
DNS: 10.0.x.z (OS X Server)

When I want to switch to the Telekom line and I need it for more than just a little check (in that case I do a remote connection to the OS X server and test it there), I switch to manual mode and enter something like this:

IP: 10.0.x.a
Subnet: 255.255.0.0
Router/Gateway: 10.0.0.a
DNS: 10.0.0.a

And voilá – I’m on the other line, able to do faster downloads if I download bigger stuff (like yesterday’s iOS Updates) or simply check if I site is also down here.

Once I’ve found some time to complete my infrastructure, I will write a longer blog post about it. This is more an answer for @jck_d.

Why I bought another server (I’ve just sold my DELL PowerEdge R210 a few months ago)? Well, it’s much slower than the DELL was and I bought to be our new firewall. The DELL was simply too expensive to use it for that job

I will install Endian, an extended version of IPCop onto this machine (I will name it “Tortuga”). The two onboard NICs will be used for “red1″ and “red2″. “red” is the interface name for external connection to IPCops (and Endians as it seems). Vodafone has just upgraded my DSL line from 6M to 16M and Telekom will switch me from “T-Home Entertain” to a regular 16M line, which will give me additional 2M of speed which was wasted for two years on not used IPTV.

The other the NIC ports will be used for the “green” network (regular LAN) and an “orange” one (DMZ). Endian allows me to route the traffic through both lines, so for example I will use Vodafone for Port 80, because of their much better routing to my webserver.

If you’re wondering – yes – I’ve returned the Linksys/Cisco RV042 router. It was simply not what I had in mind. For the money I’ve got back, I bought 2x 32GB OCZ SSDs for the Mac Minis which was a GREAT idea, they are now flying instead of playing the network-snails. I can highly recommend to install SSDs into Mac Minis. 32GB is not much, but as ALL stuff is stored on my Onnto DataTale-boxes, I’m fine with that.

Luckily, I’ve got the server with rails, so it will fit into my rack in the basement.

Speaking of the basement. It’s a bit messy inside (again) and I will use the weekend to clean it up and will also redo the cables behind the rack. Because I like it organized, I’ve got 5x 1U accessories for the rack – one blend with a whole, so I can route the cables through it and 4x blends I will use to give the cable mess a structure.

Hopefully I will receive the additional ordered patchcables by Saturday to start doing the overhaul. Believe it or not – the shop really had magenta cables Perfect for my Telekom DSL line!

With moving along to a new (transparent) proxy, content filtering, DNS & a great DHCP with binding MACs to IPs, I’m also changing the subnet of the LAN to 255.255.0.0.

I’m thinking of using the ranges like that:
10.0.0.x = Network devices & Servers (Firewall)
10.0.1.x = DMZ (just one Mac Mini there, no switch needed – yet, rest via VMs)
10.0.33.x = Clients
10.0.35.x = Neighbor clients

A good reason to start using a proxy with content filtering is that our neighbors are using our network via WiFi and they’ve a kid of 6 years who will start using the internet sooner or later. I just want to give them a protection and honestly, I’m also not sad to not see spam, commercial banners etc. anymore.

More about that when I’m done with the overhauling. Yeah – expect photos

Here’s my “evil secret plan”:

You may ask: WHY?

That’s a good question. First of all, I must say that I like the hardware of the DELL server. It was fast, pretty cheap for a server of this class, BUT it contains one thing that bugged me: I had to run Windows Server or Linux or UNIX. That’s ALL not very compatible with our clients, which are (as you can see in the diagram) all Macs. While fighting with the server and testing out different things like VMware ESXi, Linux and again even OpenSolaris, I was everytime overwhelmed. Also, it took too much of my time that I urgently need for my company stuff (means: working for my clients instead configuring the LAN).

With these things in mind, I decided that the idea from Danny Choo (scroll down until you see the Mac Minis ) is the best, also for me. Simply getting 2 used Mac Minis, attaching some external storage and having fun with an easy OS.

No way would I want to create a “REAL” server with Mailserver, iCal-Server etc. on my own, but for making the sharing easy, I used Mac OS X Server 10.6 that I had laying around for the “Business” machine. The “private” machine is running on a regular Mac OS X Snow Leopard 10.6.(4).

I’m still a bit shocked how well the machines are working. Even if the stuff is +1year old, they are behaving very well and are doing their jobs better than expected.

Of course, everything has to fit into the new rack. Here are some photos of how I’ve managed to get everything in. It’s a great rack and it was really fun to put the stuff into the rack. Yeah I know, it’s not really meant to sit in a 19″ rack, but that’s ok

I’m very sure you’d like to see more photos, so don’t hesitate and check my flickr photos: http://dkle.in/rackjuly

Ciao
Dennis 

More photos on flickr!

This rack is (in my mind) supersexy (for a rack ) and I couldn’t await to install my stuff inside. So I discovered for a not so short period of the day how the hell to put the rails from both DELL servers inside. In the end, I had to put the older PowerEdge 1650 onto the base and attach the rails for it to the very first unit. The reason? Well, the rails are a bit too long, so I could not connect the back ends of to the back supports. Putting it to the bottom allows the 1650 to gently slide above the end of the rack and with the width of the server, the rails are nice in place.

The R210 was more or less easier to install. I had to turn both ends by 180° (like it was mentioned in the instructions). It does not really fit to the front side, but it’s ok – and fits within the rack so that I’m able to close the glass door.

After putting the rack to the basement with help of my wife (isn’t it great to have such a flexible wife!? ), I put both servers inside and connected the R210 with the eSATA drive, 2x GBit network and of course power, keyboard & VGA display. The server starts as hoped

Here’s a screenshot of the server after booting up all 4 current VMs:

Both – R210 & Onnto DataTale are connected to my smaller APC UPS. I hope it can get VMware ESXi to work with it, so that it automatically shuts down/holds all VMs and shut itself down if there’s a power outage.

This is just the first step. All my 19″ equipment will move to the new rack over the next few weeks, but earlier (planned for Saturday), the old PowerEdge 1650 will be the host for Endian Firewall which allows me use both DSL lines the same time and split ports to the different lines. The reason is quit simply: If I want to send for example a 10MB file to a client, I don’t want that Exchange fills the complete 1MBit/s upstream of the main DSL line. This line is also used for surfing etc., the 2nd DSL line (Vodafone) with it’s static IP and non-disconnect should so this for me. Also, I don’t want to have speed issues in case a user access the OWA of Exchange.

As always: More to come, here!

Ciao
Dennis 

As recent blog reader you know that I’ve 2 Onnto DataTale RAID5 boxes which are connected to my Mac Pro via eSATA (PCIe x1 card). Well, as much as I appreciate the stability and the good performance, I hate the noise they create.

Turning off the Mac Pro and not using it is no option – in my mind that would be a waste of money. Also selling it is no option. I’ve gone through this process times 3x already (bought G5, sold it, bought used G5, sold it a few months later, bought the Mac Pro 2008 – and sold it 3/4 year later). Why? No idea I was switching between Windows & Mac OS X too often. Both systems have it ups & downs. While appreciate the openess of Windows (I know how crazy that sounds, I’m talking about expansion options), I really enjoy the workflow, the look & feel and of course the design of Macs.

The reason to use the MacBook only in my home office were:

• I don’t NEED the power on my desktop
• With the RAID boxes, it was too noisy

So, I’d put the 80GB Intel Postville G2 SSD from my Mac Pro back into the MacBook, bought the Mini DisplayPort to DualLink DVI cable (as posted earlier) and installed Snow Leopard from scratch. I’m still very happy with the performance and while using it for the last 2 weeks often enough I asked myself why I had bought the Mac Pro and if I should sell it. The answer on this question was, that I love to work with different systems: OSX, Windows Server, Linux, FreeBSD, OpenSolaris – I’ll try a lot and I see a lot pros in using them mixed. Because of that I’d bought 12GB of RAM for the Mac Pro.

Funny enough, my 3D tool and Photoshop are more than fast enough on my MacBook (with 4GB or RAM) – I just need some decent speed if I want to render something, if it’s a cut movie, a 3D still or animation.

Yesterday, I decided to put the Mac Pro into the basement. Attaching the RAID boxes there is a great way for still being able to access them via the GBit house network and also having the needed silence in my home office. I admit, I love silence. Currently, typing this, I just hear the little fountain in the garden. The MacBook is superquiet, not even a click from the HD since it’s a SSD now. For my satisfaction, the 30″ display offers me great screen real estate. Sure, it’s not what I had before nor is it more than 1x 30″ – but it’s very ok and I’m happy with this compromise.

The Mac Pro is running in the basement, but just using it as fileserver would be a huge mistake and a waste of power. I gave him a real job, serving me a Mac OS X Server in a VMWare Fusion session side by side with a Windows Server 2008 R2 x64 session that has an installed and full useable Exchange 2010 server. It’s really amazing what the Xeon says about it – the cores still have not much to do, yeah it sucks RAM. OS X Server got 2GB and the Win Server 4GB, so I still have 6GB left. I’m also running a Parallels version 5 on the Mac Pro for my previously “desktop” Windows 7 (Home Premium) which I need for some Office 2007 stuff.

For rendering, Apple’s QMaster will do the job, but I think I’ve to dig into the documentation to see how this works. I must say that I’m very happy with OS X Client as base OS for this kind of “server”. I quickly connect to it using the VNC/Screensharing to control it if I don’t use ssh. The VMs are running quiet fast and so I use VNC/Screensharing for the OS X Server and Microsoft’s Remotedesktop for Mac to connect to the Win server.

Using the Mac Pro as fileserver also has some (in my mind) special pros. One is the software KNOX that I’ve just bought 2 days ago. It’s a great software to easily encrypt volumes or even complete harddisks. I’ve encrypted the external RAID boxes using this great software. Why? Because I care about my data

Another pro is that the Mac Pro delivers a LOT ports for expansion. Using the 2 eSATA ports, I could expand to FireWire 800 and USB2. The overall capacity of the system is stunning. I’ve made a text overview of the storage that I’ve connected to it. Here’s a screenshot:

Returning to the MacBook, I just can recommend, if you don’t use VMs like a devil and want to buy a new Mac, be sure not to spend too much money on a desktop machine. I did and in some way I regret that buy. I’m happy to run my stuff on it now and so it’s still the perfect solution for my usage, but most people would be fine with a MacBook or MacBook Pro in my mind. Having a silent and cool (yeah cool – the Mac Pro creates a lot of heat ) home office is worth thinking about your buy several times.

Ciao
Dennis