Hi all

Am just setting up a new amd64 dual opteron server for high-performance
number crunching and data storage. Key requirements are:

- floating point performance on multi-threaded codes with significant
but not huge memory footprint
- large amounts of long-term safe data storage
- high electrical efficiency & low noise/heat output
- tower case of modest physical size
- reasonable cost

Disk speed performance is not a big issue. Have assembled components
that I believe will satisfy all this:

- 2x Opteron 2354 (good compromise between power & SPEC fp_rate2006/OMP2001)
- tyan thunder 2927-E mobo (only ATX board I know of for dual opty)
- enermax 82+ psu (apparently both very quiet & very high efficiency)
- 2x WD RE-2 greenpower 750G hard disks (ditto + long warranty)
- 4x 2G ddr2 667 ECC (kingston 3.2w per stick)
- large passive heatpipe CPU heatsinks & Noctua case fan
- Antec solo ATX case (nice build & noise reduction features)

Probably more detail than needed but better more than less I guess.

I will be installing either Ubuntu 8.04 LTS or debian Lenny.

Although I am not a Linux novice - have been developing on and
admin’ing an Ubuntu amd64 box for about 5 years - I’m not an expert
either. My Linux skills are mainly self-taught and focus on floating
point maths code development and simple admin tasks such as apt-get,
rsync backup, setting up firewall/NFS/ssh.

The main issue that I am considering now is the exact setup of the
disks and file structure. I definitely need redundancy & no nasty
surprises but not speed, so software RAID 1 using mdadm would seem like
the best option from that point of view. I would like to have a
strongly encrypted /home directory if at all possible. I would also
like to have at least / & /home on separate partitions so that system
and data are separated to make changes easier in the future. Possibly
also for /var & /tmp as I have been advised that if I ever expose this
machine to the net and run ftp or a website on it the logs and other
temporary files can get large - I have no experience of this but am
trying to plan wisely in advance.

Having googled the net I have seen many articles on this but quite a
few show different ways of doing the same thing or even contradictory
info. From what I have gathered so far it seems that one of the
preferred options is:

- form a RAID 1 setup with mdadm on the two HD’s
- encrypt this whole array? dm-crypt?
- on top of the encrypted array use LVM to create the various partitions

...but I am unsure (because I have not done any of these before):

- if encrypting the _whole_ array is necessary or even a good idea
from the point of
view of safety/redundancy of data & is dm-crypt the best route?
- if this is the right order? i.e. could I use LVM on top of the RAID
array and then
just encrypt /home
- does disk encryption and software RAID 1 interact in a potentially
troublesome way?
- does LVM allow a new system to be installed without touching /home?
- what tools are available to make the whole setup process relatively
easy and error-

Another thing that I would like to try is to put /boot separately on a
little IDE->CF adaptor so that if a HD in the array fails I can always
guarantee a re-boot, as I believe there can be issues booting from the
‘other’ disk in a software RAID 1 array in that circumstance. Also
planning to use an APC UPS with the apcupsd daemon but I believe that
is all fairly straightforward to set up.

Any thoughts - particularly on the reliability, flexibility and
feasibility of the various disk/file setup options - would be much




Hopkins Research Touch the Future