klein2 blog » debian

↔ The Gateway Project – Hardware + a secret

Dennis Klein — Mon, 02 Jan 2012 13:23:05 +0000

While moving all the eMail from one server to the other (my old server will be shut down by January 22nd – but this site is already on the new one), I received a package via UPS. Inside: 2x PCI-X Dual Port Intel cards. Yeah – PCI-X. I don’t have any PCI-X ports in one of my systems, but I was told it will work fine in a standard 3.3V PCI port

So I opened up the gateway box and installed it. To be honest, I never thought it would fit – I had bought the cards for the fileserver (one + 1 spare), but was too curious if it would work in the small gateway. As you can see on the following photos, it fits – and Debian lists the card as 2x Intel 82546EB ethernet ports (lspic & ifconfig assigned it to eth1 & eth2 (eth0 is the onboard NIC).

I got a second Fujitsu-Siemens Futro! Yeah! For just 20€ This one received the new 4GB CF card I bought last week and it will run Debian, too. This box will be awesome! While the first one will run as gateway in the house, this cutie will be put into the shelter.

“Why?” I see you asking. From the family, I’ve got a bit Xmas money and instead of buy expensive stuff (like lovely Animé figures (oh it’s hard to resist – believe me)), I decided to get what I’ve written on every single wish list for the last (at least) 5 years. A weather station! But not a display that you put and your desk and you’re done. No! This one will have own instruments. I’ve chosen the WH 1080, because it has great Linux support. I will use it the same way I use the internal sensors (shown in a blog post some weeks ago), a little perl script will grab the data from the sensors and print it into file(s). rrdtool and a cronjob will create graphs from this source. I can’t await to get my hands on this

However – here are 2 more photos from the “dream team”.

All photos are available in HiRes at flickr.

Benchmarks of the ESXi construction

Dennis Klein — Wed, 28 Dec 2011 16:44:28 +0000

Theory is great, but if it works as planned – that is great stuff Here are some screenshots and a video of a benchmark.

I’ve successfully installed my Debian Fileserver with the following steps:
- Created the first – RAID6: 6x 1.5TB /dev/md0
- Created the second – RAID6: 6x 1TB /dev/md1
- Created the third – RAID5: 3x 640GB /dev/md2 (Time Machines)
- Installed lvm2 and created a LVM volume from /dev/md0 & /dev/md1

/dev/mapper/LVM-LVM     10T    23G    10T   1% /storage
/dev/md2               1,3T   5,3M   1,3T   1% /timemachines

- Mounted the LVM into /storage
- Mounted /dev/md2 into /timemachines
- Put both into the /etc/fstab
- Installed NFS-kernel-server
- Created the directory /storage/esxi
- Gave the rights nobody:nogroup to this directory
- Shared it via exports -a
- Connected the NFS volume to ESXi

- Upload the Windows Server 2008 R2 EN ISO into the DataStore A (2.5″ 160GB)
- Installed Windows Server 2008 R2 EN on the NFS share
- Updated the Win Server
- Run benchmark (see video below)

Here is a benchmarks done with dd on the Fileserver with no running VMs:

root@risu:/storage# time dd if=/dev/zero of=/storage/bench bs=1024 count=10000000
10000000+0 Datensätze ein
10000000+0 Datensätze aus
10240000000 Bytes (10 GB) kopiert, 88,0244 s, 116 MB/s

real	1m28.090s
user	0m0.616s
sys	0m9.841s

What confuses me; why do I reach higher speed over the 10Gbps VMXNET 3 network to the VM than locally? Does that make sense to anyone?

Anatomy of my ESXi Server

Dennis Klein — Wed, 28 Dec 2011 10:19:49 +0000

Building that kind of ESXi server I did is a bit hard to explain. So, while waiting for my mdadm to finish the last RAID in the Debian VM, which will become the main storage, I started to create an overview of how the server works. Here’s the result:

Please click the image to open a higher resolution in a new window/tab.

I dropped ZFS (again), because in a bunch of tests, I had some issues with SMB for example and also with NFS shares. I need to get this “pig to fly” – so I stopped digging around and decided to use the technology that worked very well for me in the last few months: mdadm.

The fileserver is half-virtualized. It is known that you should NOT virtualize a fileserver. I fully agree! I just virtualize the underlaying Debian, the rest is on “PCI passthrough”. This great technology allows me to pop the HDs & the cards out of the box in case of an emergency (or boot from a Debian CD/Stick or HD) and access all the data. The mdadm works with the REAL hard drives. In this layer, nothing is virtualized anymore. That’s in my mind the only way to use a fileserver on ESXi. The idea is based on the ZFS/ESXi All-In-One box I mentioned in a previous post.

However, the hard disks are connected to the HP SAS Expander, which is connected to the Intel SASUC8i (a LSI 1068 chip controller with LSI “IT” firmware). I use a single SFF-8087 cable for that. This draws the connection speed to SATA150, but you don’t recognize that. Why? The RAID combines the speed I think I loose a max. of 5%.

The LSI controller is “passed through” to the Debian VM which lists the HDs as if they were connected to a regular OS.

Inside Debian:

I will install NFS server on the Debian VM. VMware tools are already installed, enabling the 10Gbps VMXNET 3 driver. ESXi will mount the NFS share where all the other will be placed on. And that is what I will do now

The Gateway Project – Theory

Dennis Klein — Fri, 16 Dec 2011 10:02:40 +0000

The Gateway Project is about a little box from Fujitsu, called “Futro” (S220). It just uses 15W and with that low power usage it is a great device for creating a gateway to the home network. This post is about the theory of SSH tunnels and why I use them instead of VPN.

Above you can see an overview how the gateway works. On the bottom, you see the foreign office/workstation/café – whatever. Let’s say they only allow you to connect to the ports 80 (http) & 443 (https), which is very common. This hinders you from connecting to a particular port – for example 5901 if you run a VNC server at home and want to surf on that machine instead of surfing on the proxy-servers of your employee. Sure, you can now go on and assign port 443 to the vnc server (5901) and you’re done. But I want to access more than 1 machine at home. For example: check the webcam from time to time and don’t want to put it on the internet. Makes sense, no?

This is the moment where the gateway joins the game. This little low-power device runs Linux or UNIX (in my case, until today it was running Debian, but I consider using OpenBSD in the future – but more on this in the post about the practice). The gateway listens on port 22 (SSH). Now, on the router, you have to assign port 80 or 443 to the gateway – port-forwarding as it’s called. I would recommend to use a static IP or a dynamic IP from DynDNS or so and use a CNAME in the dynamic case or an A host if you have a static address. Of course, you can simply use the static IP, but who can remind of public IP’s?

All you need to do is to connect to the gateway via SSH. I used putty on the workstation (Windows machine at the Academy) for tests. Putty is very good in tunneling – I will show you how to configure it in the next post.

Once the tunnel is set up, you can start using it. Let’s say you want to access your vnc server. Create a tunnel and point it to the IP:5901, locally assign port 5901 or 81 or whatever you like. Open your vnc client and access the vnc server on the address 127.0.0.1:5901 (or 127.0.0.1:81). This will work! No worries, you don’t need to look for ports, because you JUST use the local network (that is why you use the localhost address of 127.0.0.1).

I use the feature to access my main server at home via ssh tunnel without the need to run it 24/7. In this case, your motherboard has to have an dedicated IPMI port which allows you to access the local website and turn it on/reset the system etc pp.. This is luxurious, but of course you can also use WOL from your gateway to wake up your clients.

Some may think “why not use VPN?”
VPN does not allow you to share the local network (in your office or wherever you are) with the home network. Instead, it will tunnel EVERYTHING over the internet to your home. Bad for Intranet access

Stay tuned for the next post of “The Gateway Project” when I will write about the details of the hard- & software I use.

Screencast: How to create a graph with rrdtool & shell scripts

Dennis Klein — Tue, 01 Nov 2011 20:08:13 +0000

This video shows how to create a graph with rrdtool, shell scripts & 2 cron jobs.

Vimeo (great quality)

YouTube (not so nice quality)

QuickTime (great quality)

Sources of the files shown in the video.

load/createdb.sh

#!/bin/bash
STEP=300

rrdtool create load.rrd --step $STEP \
DS:load:GAUGE:600:0:U \
RRA:AVERAGE:0.5:1:2160 \
RRA:AVERAGE:0.5:5:2016 \
RRA:AVERAGE:0.5:15:2880 \
RRA:AVERAGE:0.5:60:8760

load/load.sh

#!/bin/bash
load=`cat /proc/loadavg | awk -F' '  '{print $1}'`
/usr/bin/rrdtool update /srv/www/statistik/load/load.rrd N:$load

create_load.sh

nice -n 19 /usr/bin/rrdtool graph /srv/www/statistik/load/loadg.png -a PNG -b 1024 --start -126900 \
-l 0 -u 10 -t "Load" --vertical-label "Current load" -w 600 -h 300 \
--color BACK#1e1816 --color FONT#ffffff --color CANVAS#1e1816 --color GRID#4E403C --color MGRID#4e403c --color FRAME#EEEEEE --color ARROW#4E403C \
--border 0  --grid-dash 1:0 --dynamic-labels --x-grid HOUR:8:DAY:1:DAY:1:43200:%A \
DEF:g1=/srv/www/statistik/load/load.rrd:load:AVERAGE \
LINE2:g1#ff3d00:"Load" \

/etc/crontab (last 2 lines)

*/2 * * * *	root	/srv/www/statistik/load/load.sh
*/2 * * * *	root	/srv/www/statistik/create_load.sh 1>/dev/null

If you are wondering why the load is that high; it’s a QuadCore Xeon processor with HyperThreading – 4 threads are 24/7 crunching SETI@home files

Graph after several hours

Stats of the basement room & server

Dennis Klein — Tue, 25 Oct 2011 20:03:36 +0000

When I look back, I always wanted to use a computer to control something that is a bit non-IT. I was always amazed seeing people soldering something together and get data from stuff like sensors or so. As mentioned before, I’m currently in an educational course to receive the LPIC-1 certificate (I already passed the 101 exam last Friday – yay!). The guy next to me is one of those guys who does this stuff and in the breaks, he described and explained in a very well manner who this works. He has lend me a dual-thermal-sensor to play with that one. It’s a 9 pin COM-plug which includes a few diodes and resistances. The cable is then connected to two thermal-sensors on long (sleeved cables). Meanwhile I got the needed material on my own to build it, but I will wait until the 102 exam is done before I create my very own one.

Check this site to see how this stuff looks like (German)

My server runs Linux, Debian Squeeze to be precisely. The tool of choice is called “digitemp” and is available in the standard repos in Debian. From this site and the mates help in the breaks, I’ve learned how to setup rrdtool, a very nice data-to-graphic tool that is also used in famous software like munin or IPCop.

Basement room temperature

The first graphic I’ve created is one to track the temperatures in the basement room. It’s kinda cold there, so I wanted to keep an eye on the temperature changes with regards to the hardware. Especially the HDs in the server (and there are 18 of them built into Risu (リス)) were a bit too cold at ~15°C. So I started to log the temperatures into a log file. One of the pros of the Linux course is that you really learn how to use the GNU tools and get everything you want from an output using a pervert long string of commands and pipes

Here’s a screenshot of the shell script that runs every 2 minutes on the server and does two things:
1st: It writes the captured temperature to the rrdtool database
2nd: It writes the captured temperature to a log file

#!/bin/bash
RAUM=`/usr/bin/digitemp -t0 -q -c/srv/www/statistik/raumtemperatur/.digitemprc | tr -s [:blank:] | cut -f7 -d" "`
/usr/bin/rrdtool update /srv/www/statistik/raumtemperatur/raumtemperatur.rrd N:$RAUM

# Ausgabe in /var/log/raumtemp.log
datum=$(date +"%F\t%T")
echo -e $datum"\t"$RAUM"°C" >> /var/log/raumtemp.log

The cronjob is pretty simple:

*/2 * * * *	root	/srv/www/statistik/raumtemperatur/raumtemperatur.sh

A second shell script reads the data from the database file and creates a PNG graphic from it.

cronjob:

*/2 * * * *	root	/srv/www/statistik/erzeugepng.sh

shell script:

nice -n 19 /usr/bin/rrdtool graph /srv/www/statistik/raumtemperatur/raumtemperaturg.png -a PNG -b 1024 --start -206900 -A \
-l 10 -u 24 -t "Raumtemperatur" --vertical-label "Grad Celsius" -w 700 -h 500 \
DEF:g1=/srv/www/statistik/raumtemperatur/raumtemperatur.rrd:raum:AVERAGE \
DEF:gmin=/srv/www/statistik/raumtemperatur/raumtemperatur.rrd:raum:MIN \
DEF:gmax=/srv/www/statistik/raumtemperatur/raumtemperatur.rrd:raum:MAX \
VDEF:g1a=g1,LAST \
VDEF:gmina=gmin,MINIMUM \
VDEF:gmaxa=gmax,MAXIMUM \
LINE2:g1#ff0000:"Raumtemperatur" \
GPRINT:g1a:"aktuell\: %5.2lf °C"

This looks a bit more complicated, right? Well, I will not explain how it works, because I guess this job is already done from the creator of rrdtool in a much better way

The result looks very nice (I have to mention that the step should better be 300 seconds instead the 60 I used here, this will eliminate the free spaces between the dots – but I need to rebuild the database for this).

HD temperatures

The next issue, that I have already mentioned above are the HD temperatures. I’ve attached a fan-control unit to the 3 fans blowing in air onto the HDs and since that day, the temperatures are a lot better. But a bit paranoid as I am, I wanted to have an eye on this, too. Grabbing those temperatures is pretty easy, thanks to hddtemp and my micro-script temp.

temp.sh (placed in /usr/sbin, so it will run for root users with a simple “temp” command)

#!/bin/bash
hddtemp /dev/sd[a-zz]

It simply calls hddtemp and captures the HDs from a to zz (in case I any day have more than a-z HDs in this server). Now all I had to do it crop the output and model it from…

/dev/sda: WDC WD6400AAKS-00A7B2: 30°C
/dev/sdb: WDC WD6400AAKS-00A7B2: 30°C
/dev/sdc: SAMSUNG HD155UI: 24°C
/dev/sdd: SAMSUNG HD155UI: 24°C
/dev/sde: SAMSUNG HD154UI: 21°C
/dev/sdf: SAMSUNG HD154UI: 20°C
/dev/sdg: Hitachi HDS721010CLA332: 25°C
/dev/sdh: Hitachi HDS721010CLA332: 25°C
/dev/sdi: Hitachi HDS722020ALA330: 30°C
/dev/sdj: SAMSUNG HD154UI: 21°C
/dev/sdk: WDC WD10EADS-00M2B0: 24°C
/dev/sdl: Hitachi HDS721010CLA332: 25°C
/dev/sdm: Hitachi HDS721010CLA332: 25°C
/dev/sdn: Hitachi HDS721010CLA332: 26°C
/dev/sdo: Hitachi HDS721010CLA332: 26°C
/dev/sdp: WDC WD10EADS-65L5B1: 25°C
/dev/sdq: WDC WD10EADS-65L5B1: 25°C
/dev/sdr: WDC WD10EADS-00L5B1: 25°C

to this…

30:30:23:24:21:20:25:25:30:21:24:25:25:26:26:25:25:25

With a bit console “magic” (known as awk), it was done in a few minutes (the much N’s were a bit confusing):

#!/bin/bash
hddtemps=`/usr/sbin/temp | awk -F' ' '{print $4}' | sed "s/°C//g" | sed -e "N;N;N;N;N;N;N;N;N;N;N;N;N;N;N;N;N;s/\n/:/g"`
/usr/bin/rrdtool update /srv/www/statistik/hddtemps/hddtemps.rrd N:$hddtemps

Voilá! This was the trick to get the data into the seconds rrdtool database on my system. The graph generator shell script is a bit longer.

nice -n 19 /usr/bin/rrdtool graph /srv/www/statistik/hddtemps/hddtempsg.png -a PNG -b 1024 --start -126900 -A \
-l 20 -u 40 -t "Festplattentemperaturen" --vertical-label "Grad Celsius" -w 700 -h 500 \
DEF:g1=/srv/www/statistik/hddtemps/hddtemps.rrd:hd1:AVERAGE \
DEF:g2=/srv/www/statistik/hddtemps/hddtemps.rrd:hd2:AVERAGE \
DEF:g3=/srv/www/statistik/hddtemps/hddtemps.rrd:hd3:AVERAGE \
DEF:g4=/srv/www/statistik/hddtemps/hddtemps.rrd:hd4:AVERAGE \
DEF:g5=/srv/www/statistik/hddtemps/hddtemps.rrd:hd5:AVERAGE \
DEF:g6=/srv/www/statistik/hddtemps/hddtemps.rrd:hd6:AVERAGE \
DEF:g7=/srv/www/statistik/hddtemps/hddtemps.rrd:hd7:AVERAGE \
DEF:g8=/srv/www/statistik/hddtemps/hddtemps.rrd:hd8:AVERAGE \
DEF:g9=/srv/www/statistik/hddtemps/hddtemps.rrd:hd9:AVERAGE \
DEF:g10=/srv/www/statistik/hddtemps/hddtemps.rrd:hd10:AVERAGE \
DEF:g11=/srv/www/statistik/hddtemps/hddtemps.rrd:hd11:AVERAGE \
DEF:g12=/srv/www/statistik/hddtemps/hddtemps.rrd:hd12:AVERAGE \
DEF:g13=/srv/www/statistik/hddtemps/hddtemps.rrd:hd13:AVERAGE \
DEF:g14=/srv/www/statistik/hddtemps/hddtemps.rrd:hd14:AVERAGE \
DEF:g15=/srv/www/statistik/hddtemps/hddtemps.rrd:hd15:AVERAGE \
DEF:g16=/srv/www/statistik/hddtemps/hddtemps.rrd:hd16:AVERAGE \
DEF:g17=/srv/www/statistik/hddtemps/hddtemps.rrd:hd17:AVERAGE \
DEF:g18=/srv/www/statistik/hddtemps/hddtemps.rrd:hd18:AVERAGE \
VDEF:g1a=g1,LAST \
VDEF:g2a=g2,LAST \
VDEF:g3a=g3,LAST \
VDEF:g4a=g4,LAST \
VDEF:g5a=g5,LAST \
VDEF:g6a=g6,LAST \
VDEF:g7a=g7,LAST \
VDEF:g8a=g8,LAST \
VDEF:g9a=g9,LAST \
VDEF:g10a=g10,LAST \
VDEF:g11a=g11,LAST \
VDEF:g12a=g12,LAST \
VDEF:g13a=g13,LAST \
VDEF:g14a=g14,LAST \
VDEF:g15a=g15,LAST \
VDEF:g16a=g16,LAST \
VDEF:g17a=g17,LAST \
VDEF:g18a=g18,LAST \
LINE2:g1#000000:"sda" \
LINE2:g2#006400:"sdb" \
LINE2:g3#8B8989:"sdc" \
LINE2:g4#BCEE68:"sdd" \
LINE2:g5#20B2AA:"sde" \
LINE2:g6#00FA9A:"sdf" \
LINE2:g7#FF8C00:"sdg" \
LINE2:g8#8B4500:"sdh" \
LINE2:g9#F08080:"sdi" \
LINE2:g10#8B7D6B:"sdj" \
LINE2:g11#FF1493:"sdk" \
LINE2:g12#8B3A62:"sdl" \
LINE2:g13#FF4500:"sdm" \
LINE2:g14#FF0000:"sdn" \
LINE2:g15#68228B:"sdo" \
LINE2:g16#6495ED:"sdp" \
LINE2:g17#00008B:"sdq" \
LINE2:g18#00F5FF:"sdr"

Yikes! But seriously, I haven’t found a shorter way to deal with the 18 HDs

Result graphic: (I’ve created it today in the 45min break – so don’t wonder that it’s not filled already)

Power usage

I was always curious how much power my server uses, so I bought a simply measurement device for the power socket.

Nice, but who wants to the basement to check every time? Not me Too lazy

Then I checked for devices with LAN/WiFi connection but couldn’t find anything nice. One day, I had the idea: I use an APC UPS to protect the server and switch (and more stuff). With the apcupds, I’m able to get the usage and I know how much watts it has max. (540W). Another simple script and I had a current power usage as a number. I think I don’t have to say that I wanted this in my stats immediately.

I used this script to create the database file:

#!/bin/bash STEP=300 rrdtool create apc.rrd --step $STEP \ DS:apc:GAUGE:600:0:U \ RRA:AVERAGE:0.5:1:2160 \ RRA:AVERAGE:0.5:5:2016 \ RRA:AVERAGE:0.5:15:2880 \ RRA:AVERAGE:0.5:60:8760

Then I cut the power usage (in %) out of the apcaccess tool:

apcaccess | head -n 12 | tail -n 1 | awk -F' ' '{print (540*($3/100)) "\n"}'

In the next step, I’ve created a whole shell script to write it directly to the database file to be able to create a graphic from that one.

#!/bin/bash prozent=`apcaccess | head -n 12 | tail -n 1 | awk -F' ' '{print (540*($3/100)) "\n"}'` /usr/bin/rrdtool update /srv/www/statistik/apc/apc.rrd N:$prozent

The generator file looks very similar to the one for the room & HD temperatures:

nice -n 19 /usr/bin/rrdtool graph /srv/www/statistik/apc/apcg.png -a PNG -b 1024 --start -126900 -A \ -l 20 -u 520 -t "USV Last" --vertical-label "Watt" -w 700 -h 400 \ DEF:g1=/srv/www/statistik/apc/apc.rrd:apc:AVERAGE AREA:g1#ff0000:"Last" \ VDEF:g1a=g1,LAST \ #LINE1:g1#ff0000:"Last" \

The result: (also created today )

Now that I know how to do this kind of stuff, it’s just a matter of time until I will create statistics for core(s) usage, network bandwidth, HD usage etc pp..

The site itself is on my intranet, but here’s a screenshot:

Ciao
Dennis

Some more details on Karinka

Dennis Klein — Fri, 03 Jun 2011 08:32:18 +0000

Some question came up when I posted the photos of Karinka a few days ago. I answered some of them in the comments, but to wrap up, here’s a post on this.

OS choice
I was using Nexenta Core, OpenIndiana and Solaris 11 Express in the last few months nearly exclusive. The reason for that is clear: ZFS. I will write more about my choice of the filesystem below, but want to cover the OS choice here. However – I’ve chosen Debian Squeeze for my fileserver for several reasons:

Access to tools
Believe it or not – this was one of the main reasons. It’s kinda hard to get temperatures of your harddisks and system itself on Solaris kernel systems, if you don’t have SUN hardware. Also, I missed the opportunities of a Debian system like using VMware server, APC UPS USB support and a familiar environment.

BackUPS
As mentioned before, the APC UPS USB support was more or less impossible to get to work using a Solaris kernel. I’ve compiled and followed a bunch of different tutorials, but they were mostly for SMART APCs and not USB. This didn’t help in case of a power outage that requiere a controlled shutdown of the system.

Temperatures
Call me a bit paranoid, but I simply love to check my HD temps from time to time. As it’s not possible with Solaris kernel based systems for MY BUILD (sure – the Thumper may show those information on Solaris ), that was one of the key arguments to use Debian.

Filesystem
Unlike before, I started using JFS as filesystem. This is a filesystem developed by IBM beginning the 90ies and still in development process (even if some people may say it’s dead). However – it is a very quick one, especially if you need to rebuild something – fsck & fscklog should be done in seconds than in a lot of minutes or hours. But most important, it’s stable and proven. ZFS should be stable, too – even more secure. That’s true, but using ZFS would hinder me using the tools I like and want to use. Btw. even BackBlaze is using JFS with mdadm for their servers, so it can’t be that wrong.

Unfortunately, Debian Squeeze still delivers the JFSutils from 2007 (wow!), so you will have to grab the newest version from their website and compile it on your own. Not nice, but not too complicated. The current one (1.1.15) is from March 2011.

Networking
Oh networking! OpenIndiana and Solaris 11 Express drove me nuts. I was nearly lost with the current versions until Constantin has sent me a good description how to configure the network on Solaris kernel based systems. It’s way more comfy using Debian for that, believe me. I run 2 Intel GBit NICs in this server, each has an own IP and delivers full duplex speed to the network. It’s not (yet) bond, but this way 2 users are able to work parallel without loosing too much speed.

RAID
While ZFS delivers a very nice implementation of a RAID & LVM inside itself, you have to use a hardware RAID controller or use the software RAID mdadm to setup a redundant filesystem. As I have my Intel SASUC8i with the LSI IT firmware still running in Karinka, I use mdadm to create a RAID5 of 8x1TB HDs (storagepool). I use the 6x onboard SATA-II ports for the other pool: storagepool which is built from 6×1.5TB HDs – this is also RAID5. Btw. the system itself is running on a seperate 160GB IDE HD.

Here are some benchmarks:
Mediapool

root@karinka:/home/dieta# hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 3802 MB in 2.00 seconds = 1901.12 MB/sec Timing buffered disk reads: 888 MB in 3.10 seconds = 286.13 MB/sec

Storagepool

root@karinka:/home/dieta# hdparm -tT /dev/md1 /dev/md1: Timing cached reads: 3782 MB in 2.00 seconds = 1890.86 MB/sec Timing buffered disk reads: 2222 MB in 3.00 seconds = 740.49 MB/sec

You can see that the 4k HDs (WD15EARS) in the Mediapool are also a problem for mdadm. They drop the reads a lot and even more the writes.

Power
Karinka is easily sucking up to 100W out of the power line. Not that much if you see 15 HDs running inside + 4 PCI/e cards and a nice Dual Core CPU (AMD Athlon X2 245e), but why let it run 24/7? Because it’s a server! Yeah.. true, but.. it sucks so much power! So I was looking for a nice App for my iPhone and iPad to wake up Karinka from sleep using the WOL magic package.

Shutdown is still not possible using the above show iNet WOL software Just for Macs and not Linux so far. But using iSSH on my iPhone and iPad, I’m able to login and “poweroff” manually. Not that hard.

With those procedures, I save a lot of money, that’s for sure. And of course, it’s “green” A good thing.

What you need to do to enable WOL (Wake on LAN) is to enable the option in your BIOS if it’s an onboard card, if it’s an PCIe card (like in my case), you need to set the power management that it allows the WOL signal to arrive. This will lift the power consumption above the 1W to 2-3W, but I think that’s a fair deal if you switch your server off for some or a lot of hours!

Protocols
AFP – that gave me headache, too. AFP is the Apple File Protocol. On ZFS, it was stable but superslow. Seriously – when browsing through a directory on a AFP connection it’s painful. On the other hand, AFP (enabled by the Netatalk application on both, Linux and UNIX) allows faster copies. Sounds a bit strange, right? Yes it is. Copying went up to 119MB/s what is the max. you can get out of a GBit connection.

On Debian, the directory browsing is pretty fast and neat – BUT unstable. Opening up to much parallel sessions/folders or even opening up an Aperture lib is the way to crash it. Then you suddenly see Aperture passing away and a nice box telling you that the “servername” has stopped working. Not fun!

So I decided to use AFP ONLY for Time Machine backups (which are honestly not yet set up in the current system). SMB (Samba) is a CIFS based sharing for Windows machines, but luckily, Apple supports it, too. This is fast for browsing and the max. speed you can reach with GBit from Mac to Debian is up to 60MB/s. That’s exactly the 1/2 of the AFP speed, but it’s stable.

Let’s be realistic: I’ve copied my data (currently there are… oh wait:

/dev/md0 7,6T 3,6T 4,0T 48% /mediapool /dev/md1 7,1T 612G 6,4T 9% /storage

some TBytes in use ) several times from A to B and back to A and then to C, D, E…. From HFS+ to ZFS to HFS+ to NTFS to ext4 to HFS+ to JFS. And that is just a reduced version of what I’ve done in the last few months to find out what works best for my scenario. Luckily, as far as I can say, no file is yet corrupt. Hopefully it stays this way even with a non-corruption-detect FS like ZFS.

For a short period of time (2 weeks ago), I had still running OpenIndiana and shared an iSCSI volume to my Mac mini which the shared this volume (HFS+ formatted) to the users – sounds nice, but the Mac mini just has 1 NIC port (GBit), which made it kinda slow Too bad

So that was a quick overview of what happens here. Some temperatures you want to see? Sure!

/dev/sda: WDC WD1600AAJB-00J3A0: 29°C /dev/sdb: SAMSUNG HD154UI: 20°C /dev/sdc: SAMSUNG HD154UI: 21°C /dev/sdd: SAMSUNG HD154UI: 22°C /dev/sde: WDC WD15EARS-00Z5B1: 26°C /dev/sdf: WDC WD15EARS-00Z5B1: 27°C /dev/sdg: WDC WD15EARS-00Z5B1: 26°C /dev/sdh: Hitachi HDS721010CLA332: 26°C /dev/sdi: Hitachi HDS721010CLA332: 27°C /dev/sdj: Hitachi HDS721010CLA332: 27°C /dev/sdk: Hitachi HDS721010CLA332: 26°C /dev/sdl: Hitachi HDS721010CLA332: 27°C /dev/sdm: WDC WD10EADS-00M2B0: 27°C /dev/sdn: WDC WD1001FALS-41Y6A0: 29°C /dev/sdo: Hitachi HDS721010CLA332: 27°C

14 of those HDs are actively cooled and controller by a fan controller (thanks to MAFRI for this gift you send me 1 1/2 years ago, today I really use it ). The IDE drive is not cooled by a fan, bit normally in the range of 28-34°C. Up to 50°C should be fine, but I like it more cooled like this.

KARINKA カリンカ Photoshoot (nerds only ;))

Dennis Klein — Mon, 23 May 2011 11:35:30 +0000

As mentioned in my last post about the Home Office 8.0a (the planning is still in progress), I couldn’t get the spare parts sold that I had used for the server before. As I don’t want to put those nice parts into a rack and leave them there and already complain about the bad performance of the Onntos attached to the Mac mini and then shared via network, I put the parts together last night. I used to installed Debian Squeeze as OS.

I used the following hardware for KARINKA (somehow the font looks like an IKEA furniture, right? )

CASE Yeong Yang YY-0221 CPU AMD Athlon X2 245e (45W) BOARD Gigabyte GA-MA770T-UD3 RAM 4x 4GB Kingston ECC RAM NICs 2x Intel Desktop CT 1000 PCIe x1 CONTROLLER Intel SASUC8i/LSI flashed to IT firmware GRAPHIC old ATI Rage 3D PCI SYS HD WD 160GB Caviar Blue IDE RAID HDs 6x Hitachi 1TB DeskStar SATA 1x WD 1TB Caviar Green SATA 1x WD 1TB Caviar Black SATA 3x Samsung 1.5TB HD154UI SATA 3x WD 1.5TB Caviar Green SATA OS Debian Squeeze

And here some data of the HDD temps (while running in the basement).

root@karinka:~# temps Temperatures of RAID Array Mediapool --------------------------------------- /dev/sdb: SAMSUNG HD154UI: 26°C /dev/sdc: SAMSUNG HD154UI: 27°C /dev/sdd: SAMSUNG HD154UI: 30°C /dev/sde: WDC WD15EARS-00Z5B1: 36°C /dev/sdf: WDC WD15EARS-00Z5B1: 36°C /dev/sdg: WDC WD15EARS-00Z5B1: 38°C Temperatures of RAID Array Storagepool --------------------------------------- /dev/sdh: Hitachi HDS721010CLA332: 30°C /dev/sdi: Hitachi HDS721010CLA332: 31°C /dev/sdj: Hitachi HDS721010CLA332: 31°C /dev/sdk: Hitachi HDS721010CLA332: 30°C /dev/sdl: Hitachi HDS721010CLA332: 31°C /dev/sdm: Hitachi HDS721010CLA332: 32°C /dev/sdn: WDC WD10EADS-00M2B0: 32°C /dev/sdo: WDC WD1001FALS-41Y6A0: 35°C Temperature of the system disk --------------------------------------- /dev/sda: WDC WD1600AAJB-00J3A0: 36°C

I think those are quiet nice temperatures

PS. I’m now using the plugin “RSS Image Feed” – hopefully it will resize the included images to 200px width…

Server back to Debian

Dennis Klein — Fri, 08 Apr 2011 09:35:34 +0000

You maybe remember my happiness with ZFS. Well, how should I say? Using ZFS as a private person has it’s drawbacks. When you run OpenSolaris, OpenIndiana, Nexenta or even (brrr) Solaris 11 Express – you mostly use it for ZFS. Sure – you can virtualize stuff using VirtualBox, but I personally still missed the “easy” things I can do with the “real” Debian for example hddtemp – as simple as this sounds!

This week, my OpenIndiana presented me this message:

pool: storage state: UNAVAIL status: One or more devices could not be used because the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. see: http://www.sun.com/msg/ZFS-8000-5E scan: none requested config: NAME STATE READ WRITE CKSUM storage UNAVAIL 0 0 0 insufficient replicas raidz2-0 UNAVAIL 0 0 0 insufficient replicas c1t0d0 FAULTED 0 0 0 corrupted data c1t1d0 FAULTED 0 0 0 corrupted data c1t2d0 FAULTED 0 0 0 corrupted data c1t3d0 FAULTED 0 0 0 corrupted data c1t4d0 ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 c1t6d0 ONLINE 0 0 0 c1t7d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t2d0 ONLINE 0 0 0

Hmm.. had no clue what’s going on. I’ve seen the speed of data transfer and accessing the the files via AFP went slower in the last days, I guess this was an indicator for what happend. Luckily, when moving from Nexenta to OpenIndiana, I had done backups on 2 HDs in my Mac Pro – those were still available, and so I lost nearly no data. Luckily! I was a bit shocked by this super-secure filesystem and so I took a look on Google and look what I found: Dateiserver Uni Bielefeld. This text is German, sorry – it describes that their 100TB storage, based on ZFS crashed 2x and they now switched back to Debian and ext4. Interesting!

While being more a fan of Debian than of Solaris (and knowing it much better as I use it as development platform for ctrltweet every day), I decided to stop messing around with a system and filesystem that seems to be done for bigger arrays than my home server usage. So I went with Debian, kicked out the 1.5TB HDs and installed some 1TB HDs.

The system looks mostly the same, but for completeness – here’s a hardware overview:
- AMD Athlon 245e (2.9GHz, DualCore, “energy efficient”)
- 6x Hitachi 1TB 24/7 drives
- 5x WD10EADS 1TB “Green” drives
- 1x WD1001FALS 1TB “Black” drive
- 16GB ECC RAM
- System (Debian Squeeze) installed on an Intel 80GB SSD
- Intel PCIe x1 GBit Ethernet card (CT)
- Intel SASUC8i (LSI) controller with IT Firmware
- old ATI PCI graphics card which is fine for console and up to 800×600

Everything is cooled by fans and the temperatures of the HDs are between 25 and 28°C under load.

Creating the RAID10 took a while – 8h’s to be exact and I’ve postponed ANY action on it until it’s done – heard that this should work out better. Later in the evening, it was finished and I formatted the “md0″ with ext4. Very nice and fast (and meanwhile stable) filesystem. As result I have 6TB (or 5.4TB) available for my data.

I went the RAID10 road because of 3 reasons:
- Security
- Performance
- have just 3.2TB of data at the moment and grows not THAT fast

Backup is done to one of the Onnto DataTales that I still have NOT sold. I first had some issues with an eSATA PCIe x1 card and it was terrible slow. For another project I wanted to do in the past, I had 2x SATA to eSATA cables laying around in the basement. Hmm… I had one port unused left on the board, so I connected it, opened up a “COM” port hole in the case and moved it to the outside. The Onnto box now stays on top of the case, having 4x 1.5TB in RAID5 (4.5TB available) and connected directly to the mainboard. The performance is VERY good. hdparm -tT get’s 180MB/s YEY! That’s fine for a nightly rsync backup.

More on this later Still rsyncing my stuff over to the server from my Mac Pro.

How to move a real Linux server to VirtualBox

Dennis Klein — Sun, 06 Feb 2011 14:10:52 +0000

With installing VirtualBox on my Nexenta server “Karinka” at the end of last week, I had the feeling that this was a start of a wonderful thing As I have described in my last blog post, it’s not that complicated to move a current virtual machine from VMware Fusion to VirtualBox. But migrating a real running machine sounds like a lot more work.

First, I found “mic2″, a tool by MeeGo that should create an image of a current running system. I was not able to find the right sources nor did I found any deb files, so I stopped this after 2 frustrating hours.

A new Google search showed up a nice thread in the german Administrator.de board, describing how to move a real machine to VirtualBox.

Before we start, you should know that this process will create a block-based image of the complete HARDDISK! This means: if you have a 1TB harddisk in your server and you have just 900GB of free space on your destination server/pool/partition, this will NOT work!

I had built in a small 160GB IDE HD in this particular machine (Intranetserver “Hotaru”) and have more than 2.7TB of free storage in the destination pool, so this didn’t created any headache for me.

What you need
- SSH access to the destination server
- a good connection (GBit is recommended) between your real and your destination server
- enough free space on your destination server/pool/partition

This SHOULD work with ANY Linux/UNIX OS

Creating the image via network
First, log into your real server that you want to virtualize. Maybe switch off all running big tasks like MySQL, Apache2… whatever. This is a tip, but honestly, I forgot and had no issues creating the image.

Because this will take a LOOONG time, I recommend to use screen for this!

Log in as root and type:

# dd if=/dev/hda | ssh username@destination-server-ip dd of=/storage/vms/hotaru.img

OK – of course you should replace /dev/hda with your systems HD like /dev/sda if it’s SATA/SAS/SCSI, enter your username of the destination directory.

Note: If you want to copy the image to your /root folder on a Solaris based OS like Nexenta, you CAN’T login as /root here. I’ve used my regular user and selected a directory the user is allowed to write to.

If you don’t run an own DNS on your LAN with host-resolution for LAN devices, type the IP of the destination server.
of should be the server’s folder your user is allowed to write to.

You will be prompted to log in into your destination server by typing your password.

Now, depending on the speed of your servers, your network connection this could take from 30mins to several hours. In my case, I let it run over night, the next morning, everything was copied and the file “hotaru.img” was on my Nexenta server in the path /storage/vms, where all other VMs are stored into, too.

Converting the .img to .vdi
To be able to use the image, you will need to convert it. This will again take some time, so consider doing this via screen, too.

$ VBoxManage convertdd hotaru.img hotaru.vdi

Importing the VM into VirtualBox
The step of importing is pretty easy – just follow my instructions of my previous post:

VirtualBox and basic configuration of a new VM
Open VirtualBox:

$ VirtualBox &

In the appearing VirtualBox on your local machine, click on “File” -> “Virtual Media Manager”.

Click the “Add” icon and select the new vdi file. In my case it was hotaru.vdi. Hit “Ok”.

Next, we need to create a BLANK VM to import the virtual harddisk into. Click the “New” button, then “Next” and enter the name of the new/imported VM. Be careful! Case sensitive!

256MB is fine for my Intranet Machine, so “Next”.

Then I select the “Use existing hard disk” radio and select my “hotaru.vdi”.

On the last screen accept all settings. Done! Now we need to edit some settings.

My real machine is a bit older (SuperMicro 1U server with IDE, 2x Intel 10/100er onboard NICs and Celeron 2.4GHz CPU). As it’s using IDE, that’s fine for VBox.

In case your machine is SATA, follow those steps:

Click your imported VM and then the button “Settings”, then on “Storage”. Hit the 3rd icon (white with green plus) on the bottom of the box and select “Add SATA controller”. Click the SATA controller and hit the first icon (data with green plus) and click it until the “dev.vmdk” appears. Delete the not used datasets, also, you can remove the Floppy and CD/DVD drive if you don’t need them. On my site it looks like that:

I have removed USB2 and Audio support. I’ve set the network to “bridged” and the graphic card memory to 16MB. That’s why I get the message “Non-optimal settings detected”, but that’s fine. Hit “Ok” and you’re done!

Now we need to launch the VM from the terminal, but first, we need to make one more settings to the VM itself.

If I would run it now, I would receive this error message:

dieta@karinka:~$ nohup VBoxHeadless -s "Hotaru" & Sun VirtualBox Headless Interface 3.1.6 (C) 2008-2010 Sun Microsystems, Inc. All rights reserved. VRDP server failed to start. VRDP server is inactive.

We will have to set the the vrdpport. Shut down the now running VM:

$ VBoxManage controlvm "Hotaru" poweroff

Now the setting:

$ VBoxManage modifyvm "Hotaru" --vrdpport 3391

(!) The port number has to grow! If you run 3 machines it should look like:
VM1: 3389
VM2: 3390
VM3: 3391
etc.

Now, it’s time check if everything works – you can launch the VMs in a window by simply starting them from the X11 window. This will pop up your VM. Please keep in mind, that this is NOT the to run them normally, as closing those windows will kill the VMs! This is fine for maintaining – what was the reason I was in need to use them that way; the AMD NIC of VBox was recognized automatically by the Debian, but set to the non-configured port “eth3″, so I had set up the IP, DNS & Gateway again – a thing of 1 minute.

Maybe to mention also, I had to run a fsck check before I was able to boot the VM and first I thought it would not work But in the end, it booted fine and it works great now.

Once you’re done with error-fixing (in case you have to), you can shut the new VM down and launch it from the terminal:

$ nohup VBoxHeadless -s "Hotaru" &

My VMs are now running fine, be sure to check them by quitting all terminals, reopen and ping. It happened to me that one was running via the terminal, because I had forgot to attach the “&” to the end, and by closing the terminal, it was killed. It’s always fine to have Apache2 servers running for a quick check.