One Server: Researching the hardware
Using my list of requirements, I set out to find the hardware for my new server. I was building this from scratch so at minimum my purchase list needed to include
- hard drive storage
- server case
- motherboard, RAM & CPU
Hard drive storage
I decided to focus first on the requirements for the fileserver side of the project. Recall that I was planning for 16TB of storage space. At the time, the largest consumer hard drives were 2 TB. I also wanted to be able to support multiple drive failures and be able to replace the drives without shutting down the system. That meant I needed at least 10 hard drives.
When researching the hardware for this server I came across a good blog post from Adaptec about real life RAID reliability. That article compared the reliability of RAID-5 and RAID-6 arrays and showed that a RAID-6 array should last 172 times longer than a RAID-5 array. Reliability was important to me on this project, so I decided to go with RAID-6. The Adaptec article only considered enterprise grade drives. I planned to build this server with consumer grade drives. Therefore, as a precaution, I chose to add two extra drives as hot spares.
A RAID-6 array with 10 drives was likely to run slow. So I did some more searching and came across RAID-60. RAID-60 combines the redundancy of RAID-6 with the speed of striping found in RAID-0. However, to get 16 TB with RAID-60, and have two hot spares, I now needed 14 hard drives. Six drives for each of the two RAID-6 arrays and two hot spares.
RAID-60 with 8 drives. Image from Wikipedia
I wanted to make sure the fileserver would run quickly so that I could stream video from it while MythTV was also recording new programs and all the virtual machines were also running. I thought running everything off one set of storage drives might be too much, so I decided to split the VM storage from the NAS storage. That meant adding additional drives. I had four 1TB drives from my previous NAS, so I decided to use them for storing the VM images.
That put the total number of hard drives needed at 18. This was shaping up to be quite a storage server! The next task was determining how to fit that many hard drives into a computer case.
Server Case
I wanted the server to be able to stay running while I replaced a failed drive, so I needed a case that accommodated hot swappable drive bays. I considered putting the drives into some six-bay external drive enclosures, but decided that would get too expensive and end up using more power then was needed. Plus, I could just see the cables getting disconnected between the external enclosures and the main CPU.
No tower-style case that I could find would hold that many drives, so I looked for rack mountable cases.  To fit 18 drives, a 4U rack mount case was needed.
Motherboard, RAM & CPU
I wanted to be able to expand this system in the future, so when choosing the motherboard I focused on server boards that supported dual CPUs. My plan was to put the system together with one CPU, and if needed, add another CPU later. I also needed to find a motherboard with multiple network interfaces, and plenty of PCI-express slots for adding RAID cards. Since reliability was important to me, I focused only on motherboards that supported ECC RAM. Form factor wasn’t a big issue for this system as it was being placed into a rack mount case with plenty of space.
For the CPU, I needed a processor that supported VT-d. VT-d processors support mapping cards plugged into PCIe slots directly into virtual machines. My goal was to create a virtual machine for the fileserver and map the RAID card directly into that VM.
Another goal of mine was to make the new server easy to administer. I didn’t want to have to find a spare keyboard, mouse, and monitor and plug them all in when there was trouble. The solution, IPMI. A motherboard with IPMI would allow me to remotely control the keyboard, mouse, video and even attach a remote DVD-ROM to perform an OS install. It is basically a built-in KVM over IP. I can even remotely reset the computer using IPMI.
Parts list
I ended up purchasing the following components for this system
- 14 x 2 TB hard drives (5 x Hitachi HDS72202, 3 x SAMSUNG HD203WI, 6 x WDC WD20EADS)
- Norco RPC-4220 4U rackmount case with 20 hot swappable drive bays
- Supermicro X8DTi-F motherboard with 3 PCIe 8x slots and IPMI
- 24Gb ECC Registered DDR3 1066 RAM
- Intel Xeon E5506 Nehalem-EP 2.14 GHz processor