GCC on multiprocessor platform? - BSD

This is a discussion on GCC on multiprocessor platform? - BSD ; How to use more than one cpu (I am using bsd.mp) for building something e.g. make I was thinking something like fork process on several cpu (maybe special flags for GCC) ? -- best M...

+ Reply to Thread
Results 1 to 11 of 11

Thread: GCC on multiprocessor platform?

  1. GCC on multiprocessor platform?

    How to use more than one cpu (I am using bsd.mp) for building something
    e.g. make

    I was thinking something like fork process on several cpu (maybe special
    flags for GCC) ?

    --
    best
    M

  2. Re: GCC on multiprocessor platform?

    man make


  3. Re: GCC on multiprocessor platform?

    tedu napisał(a):
    > man make
    >


    please think first and then post...

    I know that there are options like -j, but if I set up it it won't work
    (I have 25% of CPU usage so it means that only 1 from 4 CPU's are used!)

    and look at your other replays, only "why?" "read man"

    try to help not only show man! everyone knows what is man so you did not
    discover the new world!

    Do someone know why is the problem of making only on one cpu

    I use

    make -j 5 (one more process than cpu's)

    --
    m

  4. Re: GCC on multiprocessor platform?

    On 2005-06-03, m wrote:
    > tedu napisał(a):
    >> man make
    >>

    >
    > please think first and then post...


    pot, meet the kettle.

    >
    > I know that there are options like -j, but if I set up it it won't work
    > (I have 25% of CPU usage so it means that only 1 from 4 CPU's are used!)


    just because you only have 25% usage doesn't mean only 1 cpu is being used.

    >
    > and look at your other replays, only "why?" "read man"
    >
    > try to help not only show man! everyone knows what is man so you did not
    > discover the new world!
    >
    > Do someone know why is the problem of making only on one cpu
    >
    > I use
    >
    > make -j 5 (one more process than cpu's)
    >


    I might have helped you further, but you were mean to Ted.

    --
    ratfood@food.skaterat.net
    All foods should be removed to reply

  5. Re: GCC on multiprocessor platform?

    John McGrail napisał(a):
    > On 2005-06-03, m wrote:
    >
    >>tedu napisał(a):
    >>
    >>>man make
    >>>

    >>
    >>please think first and then post...

    >
    >
    > pot, meet the kettle.
    >
    >
    >>I know that there are options like -j, but if I set up it it won't work
    >>(I have 25% of CPU usage so it means that only 1 from 4 CPU's are used!)

    >
    >
    > just because you only have 25% usage doesn't mean only 1 cpu is being used.
    >
    >
    >>and look at your other replays, only "why?" "read man"
    >>
    >>try to help not only show man! everyone knows what is man so you did not
    >>discover the new world!
    >>
    >>Do someone know why is the problem of making only on one cpu
    >>
    >>I use
    >>
    >>make -j 5 (one more process than cpu's)
    >>

    >
    >
    > I might have helped you further, but you were mean to Ted.
    >


    eot.

  6. Re: GCC on multiprocessor platform?

    m writes:

    > How to use more than one cpu (I am using bsd.mp) for building something
    > e.g. make


    Make sure that there are enough processes running to require all available
    cpus. On my two processor system, for example, it takes something
    like 8 concurrent compiles to keep both processors busy most of the time.

    Different disk/disk controllers/memory bandwidth/etc will make your
    system have different requirements.

    The number for your system can be found by trial and error.

    // marc

  7. Re: GCC on multiprocessor platform?

    Marco S Hyman napisał(a):
    > m writes:
    >
    >
    >>How to use more than one cpu (I am using bsd.mp) for building something
    >>e.g. make

    >
    >
    > Make sure that there are enough processes running to require all available
    > cpus. On my two processor system, for example, it takes something
    > like 8 concurrent compiles to keep both processors busy most of the time.
    >
    > Different disk/disk controllers/memory bandwidth/etc will make your
    > system have different requirements.
    >
    > The number for your system can be found by trial and error.
    >
    > // marc


    Thanks

    you where right, I used before one more process than cpu (like I read
    somewhere in gentoo faq) but now I set up more and CPU usage grows up

    --
    best
    m

  8. Re: GCC on multiprocessor platform?

    In article , m wrote:
    >Marco S Hyman napisał(a):
    >> m writes:
    >>>How to use more than one cpu (I am using bsd.mp) for building something
    >>>e.g. make

    >>
    >> Make sure that there are enough processes running to require all available
    >> cpus. On my two processor system, for example, it takes something
    >> like 8 concurrent compiles to keep both processors busy most of the time.
    >>

    >you where right, I used before one more process than cpu (like I read
    >somewhere in gentoo faq) but now I set up more and CPU usage grows up


    We tested this extensively under HP-UX once. We found that the optimum was
    8 compilation processes per CPU (we only tested powers of two, and above 4
    processes per CPU there was little difference). Meaning on an 8-way
    machine, you had to use -j64, and there are very few source trees where this
    can be put to full use. The answer depends somewhat on the IO system, the
    total amount of memory, and the ratio of memory bandwidth to CPU speed. It
    makes a huge difference if the complete source tree is already in cache when
    you start the compilation (for example it was just untared); then the
    bottleneck is not the incoming IO system, and you need fewer compilation
    processes to saturate the CPU. A lot of memory (many gigs) and a large TLB
    is a must for this kind of game. The answer on an x86 with IDE disks will
    probably be somewhat different from the answer for a high-end RISC machine
    with fibre channel and a high-end RAID subsystem. But -j5 on a 4-way
    machine is way too low.

    --
    The address in the header is invalid for obvious reasons. Please
    reconstruct the address from the information below (look for _).
    Ralph Becker-Szendy _firstname_@lr_dot_los-gatos_dot_ca.us

  9. Re: GCC on multiprocessor platform?

    On 04/06/2005 1:08 AM, _firstname_@lr_dot_los-gatos_dot_ca.us wrote:
    > In article , m wrote:
    >
    >>Marco S Hyman napisał(a):
    >>
    >>>m writes:
    >>>
    >>>>How to use more than one cpu (I am using bsd.mp) for building something
    >>>>e.g. make
    >>>
    >>>Make sure that there are enough processes running to require all available
    >>>cpus. On my two processor system, for example, it takes something
    >>>like 8 concurrent compiles to keep both processors busy most of the time.
    >>>

    >>
    >>you where right, I used before one more process than cpu (like I read
    >>somewhere in gentoo faq) but now I set up more and CPU usage grows up

    >
    >
    > We tested this extensively under HP-UX once. We found that the optimum was
    > 8 compilation processes per CPU (we only tested powers of two, and above 4
    > processes per CPU there was little difference). Meaning on an 8-way
    > machine, you had to use -j64, and there are very few source trees where this
    > can be put to full use. The answer depends somewhat on the IO system, the
    > total amount of memory, and the ratio of memory bandwidth to CPU speed. It
    > makes a huge difference if the complete source tree is already in cache when
    > you start the compilation (for example it was just untared); then the
    > bottleneck is not the incoming IO system, and you need fewer compilation
    > processes to saturate the CPU. A lot of memory (many gigs) and a large TLB
    > is a must for this kind of game. The answer on an x86 with IDE disks will
    > probably be somewhat different from the answer for a high-end RISC machine
    > with fibre channel and a high-end RAID subsystem. But -j5 on a 4-way
    > machine is way too low.
    >

    Sounds reasonable. I think most of us just read (and accepted!) the
    Linux build docs which has always said the arg to -j is the number of
    CPUs +1. It sounds a bit like one of those ad hoc settings that became
    the defacto standard over time, like "swap should be 2.5 times your
    physical memory."

    Anyway, -j3 on a two-way box was plenty fast enough to build a Linux
    kernel, which is the only place I ever used this argument. Anything
    larger just had a large number of make threads waiting for other stuff
    to finish (or crashing out on missing dependencies.)

  10. Re: GCC on multiprocessor platform?

    Begin
    On 2005-06-06, clvrmnky wrote:
    > On 04/06/2005 1:08 AM, _firstname_@lr_dot_los-gatos_dot_ca.us wrote:
    >>
    >> We tested this extensively under HP-UX once. [...]

    >
    > Sounds reasonable. I think most of us just read (and accepted!) the
    > Linux build docs which has always said the arg to -j is the number of
    > CPUs +1.


    I don't think so. Or at least, I think by now that at least here people
    should know better, and people who just accept anything linux people say
    or write deserve whatever they get. :-p

    Besides, with a bit of varying the argument to -j you can get an idea
    pretty quickly what doesn't work and what works better.


    > It sounds a bit like one of those ad hoc settings that became
    > the defacto standard over time, like "swap should be 2.5 times your
    > physical memory."


    Uhm, actually. There used to be Really Good Reasons why swap HAD to be
    at least twice the available memory to be of any use at all. Twice and
    a bit is still a good rule of thumb, but I am certainly not afraid to
    add more, much much more if I have to. With that I subsequently broke
    a linux box because it couldn't handle that much swap. Oh irony.


    > Anyway, -j3 on a two-way box was plenty fast enough to build a Linux
    > kernel, which is the only place I ever used this argument. Anything
    > larger just had a large number of make threads waiting for other stuff
    > to finish (or crashing out on missing dependencies.)


    Heh. I'd guess that was an older kernel with a vm that could use some
    work. I'm by no means a kernel hacker but a lot of this stuff depends
    heavily on the system (hardware and software); where HPUX at least used
    to run on reasonably well engineered hardware, its feel may have left
    something to be desired but I'd certainly expect its vm and smp support
    to be industrial strenght. I'd not expect that of linux, at least older
    kernels, and I don't expect much from peecee hardware either, where
    the performance numbers drive marketdroids more than the cpu. And then
    there's the rest of the board.


    --
    j p d (at) d s b (dot) t u d e l f t (dot) n l .

  11. Re: GCC on multiprocessor platform?

    _firstname_@lr_dot_los-gatos_dot_ca.us wrote:
    > In article , m wrote:
    >>Marco S Hyman napisa?(a):
    >>> m writes:
    >>>>How to use more than one cpu (I am using bsd.mp) for building something
    >>>>e.g. make
    >>> Make sure that there are enough processes running to require all available
    >>> cpus. On my two processor system, for example, it takes something
    >>> like 8 concurrent compiles to keep both processors busy most of the time.

    >>you where right, I used before one more process than cpu (like I read
    >>somewhere in gentoo faq) but now I set up more and CPU usage grows up

    > We tested this extensively under HP-UX once. We found that the optimum was
    > 8 compilation processes per CPU (we only tested powers of two, and above 4

    ....

    While I strongly advocate the "suck it and see" methodology, a little
    analysis can also be helpful.

    I guess many of us with mp systems have a cmd like "prun" ("parallel run"
    I calls it that
    creates process farms or spmd pools for mass compiles or builds.
    Or even the odd parameter search.

    The question of the amount of parallelism (i.e. number of simultaneous
    processes) that is "optimal" then comes up.

    In this case we can assume a simple model like

    T = W/n + a n

    is sufficient to predict total wall time (T) given parallelism/number
    of processes (n) and "amount of work" (W -- measured in any convenient
    units, e.g. number of like-minded jobs that need executing).

    I.e. the elapsed time for the run is proportional to the amt of
    work, inversely propertional to the number of processes sharing the work,
    but also has a term for "communication overhead", incl disk I/O
    on individ cpus or packet traffic over nets.

    The simple software ("prun") I use ramps up n during the start-up phase of the
    job, saves a few "n" and "T" values, plonks a linear regression through it
    to measure the proportionality constant "a", and then predicts and
    optimum "n" to minimise "T". Estimates than are continually updated
    each time a process exits.

    The difference between this method an the common "start n processes
    and wait for them all to finish" can be a matter of 50%, esp if there's
    a moderate spread in the runtimes of the processes.

    And you don't wanna waste 1/2 your h/w, do you?

+ Reply to Thread