I just installed a single node p575 (it has no switch but 8 cpus that
use shared memory) I'm trying to run my code that I can sucessfully run
on bassi at NERSC (a multi-node 575 with the federation switch). All
communication in the code is done via MPI which will use shared memory.
The machine has 32GB of RAM.

The code is compiled in 64-bit mode and the kernel is also 64bit
[baron@basie:3]% file phoenix
phoenix: 64-bit XCOFF executable or object module not stripped
[baron@basie:4]%

The serial version runs fine, and the size of the serial and parallel
version is nearly identical:

Parallel size:
[baron@basie:4]% size -X64 phoenix
phoenix: 5017424 + 5440344 + 815545080 + 85319 + 276 + 2300 = 826090743
[baron@basie:5]%

Serial Size:
[baron@basie:5]% cd phxexp_ser
/home/baron/basie2/phxexp_ser
[baron@basie:6]% size -X64 phoenix
phoenix: 4627664 + 4939912 + 809599048 + 80934 + 276 + 1750 = 819249584
[baron@basie:7]%

The serial code runs fine. The ulimits should be fine:

[baron@basie:23]% ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 4194304
memory(kbytes) unlimited
coredump(blocks) 2097151
nofiles(descriptors) 2000

a simple poe job runs fine:

[baron@basie:24]% poe date -procs 8
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
Fri Jan 19 14:39:53 CST 2007
[baron@basie:25]%

but the parallel jobs won't load:

+ /home2/basie/baron/phxexp/phoenix -node 1 -procs 8 -hostfile
host.list
exec(): 0509-036 Cannot load program /home2/basie/baron/phxexp/phoenix
because of the following errors:
0509-026 System error: There is not enough memory available
now.

Any clues?????

Thanks in advance