ZiaNet Labs

The internet of tomorrow, today!

Goals

Build a storage server with high storage capacity.

Keep the cost down.

Capable of handling a multiple of tasks(NFS,Samba...)

We wanted it to be optimized mainly for a mail server environment, e.g. large amounts of small files.

Hardware

Part	Part Description	Quantity	Notes	Link
Motherboard	Tyan S2462UNG	1	---
Processor	AMD 1900MP	2	---
Memory	Corsair CM73SD512R-2100 512MB PC-2100 DDR ECC	2	---
RAID	3ware 7850	2	---
Hard Drives	Maxtor D540X 160GB 5400RPM	16 + 2 spare	UDMA133 IDE storage drives--2.56TB
Hard Drives	Seagate ST336737LW	1	SCSI system drive
Power Supplies	Antec PP-510D 510W ATX*	1	System power supply for Tyan S2462 only
Power Supplies	Antec PP-403X 400W ATX	2	Hard drive power supplies
Case	Chenbro SR101 Ultra	1	---	Chenbro

* We originally had an NMB 460W power supply to power the motherboard but
if failed after like a week.

Reasons we chose what we did:

Motherboard:

One of the few dual Athlon motherboards out there. Also it has
had it's burn in time in the Linux community.

Processor:

We wanted the most bang for our buck, plus the memory architecture
is advantageous.

Memory:

Was certified to run in the Tyan motherboard and have had good experience
in the past.

RAID:

Has a very well maintained Linux drivers and is one of the best performing ATA
controllers out there.

Hard drives:

The Maxtor drives because these were the largest drives available.
It would have been nice to have 7200 RPM drives but storage capacity was more
important then speed.
The Seagate was used because we had it lying around.

Power supplies:

Antec was chosen because the Tyan motherboard requires a special power supply
that Antec, among other manufacturers, makes. Since our original pick of an NMB
power supply failed within two weeks we decided on another manufacturer.
Antec was chosen for the drive PSU's because it was a high wattage power supply
for a reasonable price. Each RAID set is powered by it's own PSU.

Case:

Chenbro SR101 was used because of the high drive bay count which was needed
to mount all of the hard drives, plus the multiple power supply bays, and
the design. We really wanted the purple front doors but we could not find them.

Software

The operating system we chose was Linux, and the distribution
was RedHat 7.3. We chose Linux because...well, because it kicks ass. We picked
the RedHat distribution because it was the most up to date at the time and we were
already very familiar with it. We used the RedHat kernel build which is 2.4.18-4smp.
The use of a >=2.4.18 kernel version was essential if not required because of the
>1TB filesystems on the RAID.

Raid Setup

Both the 3ware raid cards each had 8 of the Maxtor drives on them. These drives/cards
were configured with RAID 5. The initial builds of the raid took about 4-5 hours per
card, for about a total of 8-10 hours. The two cards were then software raided with
RAID 1 (mirroring). In the final configuation we ended up with 1.1TB of mirrored free
space. So basically we have two 1.1TB hardware RAID 5 software mirrored(RAID 1).

We did investigate software striping (RAID 0) the two controllers as well, but decided
against this when we learned that >2TB filesystems were not supported in the 2.4.x
kernels. We could have made a 2TB filesystem but we would be wasting space unless we
made another partition for the 200MB left over.

Benchmarks

Bonnie++ benchmarks
Postmark benchmarks
Postmark benchmarks over NFS
IOZone benchmarks
IOZone benchmarks over NFS using 4096 block size
IOZone benchmarks over NFS using 8192 block size
IOZone benchmarks over NFS using 16384 block size

Notes:
Reiserfs filesystems were made with following command:
mkreiserfs /dev/md0

Ext3 filesystem was made with following command:
mke2fs -j -m 1 /dev/md0
All ext3 benchmarks were using journaled mode.

Bonnie++ was not used over NFS because it broke half way through the
tests on the FreeBSD boxes and I was to lazy to figure it out.

ReiserFS with tails was dropped from the tests after it became obvious
that it was not going to be competitive. This can be seen mainly from
the Postmark benchmarks and to a lesser degree with the Bonnie++
benchmarks.

The number of nfsd daemons running seemed to have minimal impact on performance.

The size of the rmem_default and rmem_max buffers seemed to have
minimal impact on performance.

Conclusions

After many many hours of benchmark torture we decided that the Ext3 filesystem was
the best solution for our application. The benchmarks show that ReiserFS is faster
then Ext3 when not using NFS but our needs require NFS. Ext3 proved to be the champion
while using NFS, though most of the time by a narrow margin. We theorize that we are
hitting a bottleneck in network bandwidth on tests where results are very similiar.
Later these bandwidth restrictions may be reduced with a gigabit network, but that will
still require some study.

Another concern that eventually led to decision to use Ext3 was the handling of bad
blocks in the ReiserFs filesystem. Information from various sources indicated that
Ext3 was more graceful at handling bad blocks then ReiserFS. We did not want to have
bring down the entire system due to a couple of bad blocks on the hard drive.

When all was said and done this box was built for under $8000 dollars. So you too could
have a very large storage system for a fraction of the cost of a commercial product.

Contacts

As of now I can be reached at: kwijibo@zianet.com

Let me know of any information that you think should be available on the web page.
Or if you see something that looks totally out of place let me know. I could have
entered the data wrong.

Our alternative design if the previous fails: