"Bill Todd" wrote in message
> The AMD Opterons on the other hand
> > are a completely different animal, they live for scaling

> Up to 4 processors, certainly. Up to 8 processors, possibly. Beyond

> no one knows yet (at least no one who is talking).

Maybe the examples don't exist yet, but looking at the architecture it
certainly seems like it should not be a problem scaling beyond 8 processors
with external solutions.

> -- each Opteron
> > controls its own memory directly and they talk to each other over
> > point-to-point links rather than shared buses.

> Only up to 8 processors. What happens beyond that point has not yet been
> disclosed, AFAIK.

At that point it would be a proprietary solution, something like Sun's
Fireplane interconnect. It doesn't have to Hypertransport through and
through. You could use HT between the processors within any single system
board, but you can connect each of the system boards with any other
interconnect you like. It's just a basic fact of life in NUMA architectures,
some connections are going to be slower than others.

> Some people question how well Opteron will scale to 8 processors in a
> glueless configuration. In particular, it uses a broadcast-invalidation
> (rather than directory-based) cache-coherence mechanism that certainly

> have difficulty scaling much beyond 8 processors and may scale less than
> linearly in the 4 - 8 processor region. We'll presumably see soon whether
> AMD just allowed that extension because it was easy to do so and because

> scales usefully even if not very linearly.

I really don't see this as a problem. This would be a problem in a shared
memory architecture like Intel's, but each Opteron directly accesses its own
assigned memory, and when it comes time to access memory located on another
Opteron, each remote Opteron has to ask permission from the local Opteron's

> There is already a Cray
> > supercomputer project for Sandia National Labs making use of 10,000
> > Opterons; that uses a special Cray proprietary crossbar connector

> > system boards, but the rest is just off-the-shelf.

> My impression is that it's not ccNUMA, however. If so, while it may well

> interesting for Cray, most operating systems would have no idea how to use
> it.

There is also a project at some US university to use a 128 Opterons in a
system. I think it was the University of Texas. Anyways we'll have to see
how they cross-connect those Opterons too.

Yousuf Khan