man make

tedu napisa³(a):
> man make
>

please think first and then post...

I know that there are options like -j, but if I set up it it won't work
(I have 25% of CPU usage so it means that only 1 from 4 CPU's are used!)

and look at your other replays, only "why?" "read man"

try to help not only show man! everyone knows what is man so you did not
discover the new world!

Do someone know why is the problem of making only on one cpu

I use

make -j 5 (one more process than cpu's)

--
m

On 2005-06-03, m wrote:
> tedu napisa³(a):
>> man make
>>
>
> please think first and then post...

pot, meet the kettle.

>
> I know that there are options like -j, but if I set up it it won't work
> (I have 25% of CPU usage so it means that only 1 from 4 CPU's are used!)

just because you only have 25% usage doesn't mean only 1 cpu is being used.

>
> and look at your other replays, only "why?" "read man"
>
> try to help not only show man! everyone knows what is man so you did not
> discover the new world!
>
> Do someone know why is the problem of making only on one cpu
>
> I use
>
> make -j 5 (one more process than cpu's)
>

I might have helped you further, but you were mean to Ted.

--
ratfood@food.skaterat.net
All foods should be removed to reply

John McGrail napisa³(a):
> On 2005-06-03, m wrote:
>
>>tedu napisa³(a):
>>
>>>man make
>>>
>>
>>please think first and then post...
>
>
> pot, meet the kettle.
>
>
>>I know that there are options like -j, but if I set up it it won't work
>>(I have 25% of CPU usage so it means that only 1 from 4 CPU's are used!)
>
>
> just because you only have 25% usage doesn't mean only 1 cpu is being used.
>
>
>>and look at your other replays, only "why?" "read man"
>>
>>try to help not only show man! everyone knows what is man so you did not
>>discover the new world!
>>
>>Do someone know why is the problem of making only on one cpu
>>
>>I use
>>
>>make -j 5 (one more process than cpu's)
>>
>
>
> I might have helped you further, but you were mean to Ted.
>

eot.

m writes:

> How to use more than one cpu (I am using bsd.mp) for building something
> e.g. make

Make sure that there are enough processes running to require all available
cpus. On my two processor system, for example, it takes something
like 8 concurrent compiles to keep both processors busy most of the time.

Different disk/disk controllers/memory bandwidth/etc will make your
system have different requirements.

The number for your system can be found by trial and error.

// marc

Marco S Hyman napisa³(a):
> m writes:
>
>
>>How to use more than one cpu (I am using bsd.mp) for building something
>>e.g. make
>
>
> Make sure that there are enough processes running to require all available
> cpus. On my two processor system, for example, it takes something
> like 8 concurrent compiles to keep both processors busy most of the time.
>
> Different disk/disk controllers/memory bandwidth/etc will make your
> system have different requirements.
>
> The number for your system can be found by trial and error.
>
> // marc

Thanks

you where right, I used before one more process than cpu (like I read
somewhere in gentoo faq) but now I set up more and CPU usage grows up

--
best
m

In article , m wrote:
>Marco S Hyman napisa³(a):
>> m writes:
>>>How to use more than one cpu (I am using bsd.mp) for building something
>>>e.g. make
>>
>> Make sure that there are enough processes running to require all available
>> cpus. On my two processor system, for example, it takes something
>> like 8 concurrent compiles to keep both processors busy most of the time.
>>
>you where right, I used before one more process than cpu (like I read
>somewhere in gentoo faq) but now I set up more and CPU usage grows up

We tested this extensively under HP-UX once. We found that the optimum was
8 compilation processes per CPU (we only tested powers of two, and above 4
processes per CPU there was little difference). Meaning on an 8-way
machine, you had to use -j64, and there are very few source trees where this
can be put to full use. The answer depends somewhat on the IO system, the
total amount of memory, and the ratio of memory bandwidth to CPU speed. It
makes a huge difference if the complete source tree is already in cache when
you start the compilation (for example it was just untared); then the
bottleneck is not the incoming IO system, and you need fewer compilation
processes to saturate the CPU. A lot of memory (many gigs) and a large TLB
is a must for this kind of game. The answer on an x86 with IDE disks will
probably be somewhat different from the answer for a high-end RISC machine
with fibre channel and a high-end RAID subsystem. But -j5 on a 4-way
machine is way too low.

--
The address in the header is invalid for obvious reasons. Please
reconstruct the address from the information below (look for _).
Ralph Becker-Szendy _firstname_@lr_dot_los-gatos_dot_ca.us

On 04/06/2005 1:08 AM, _firstname_@lr_dot_los-gatos_dot_ca.us wrote:
> In article , m wrote:
>
>>Marco S Hyman napisa³(a):
>>
>>>m writes:
>>>
>>>>How to use more than one cpu (I am using bsd.mp) for building something
>>>>e.g. make
>>>
>>>Make sure that there are enough processes running to require all available
>>>cpus. On my two processor system, for example, it takes something
>>>like 8 concurrent compiles to keep both processors busy most of the time.
>>>
>>
>>you where right, I used before one more process than cpu (like I read
>>somewhere in gentoo faq) but now I set up more and CPU usage grows up
>
>
> We tested this extensively under HP-UX once. We found that the optimum was
> 8 compilation processes per CPU (we only tested powers of two, and above 4
> processes per CPU there was little difference). Meaning on an 8-way
> machine, you had to use -j64, and there are very few source trees where this
> can be put to full use. The answer depends somewhat on the IO system, the
> total amount of memory, and the ratio of memory bandwidth to CPU speed. It
> makes a huge difference if the complete source tree is already in cache when
> you start the compilation (for example it was just untared); then the
> bottleneck is not the incoming IO system, and you need fewer compilation
> processes to saturate the CPU. A lot of memory (many gigs) and a large TLB
> is a must for this kind of game. The answer on an x86 with IDE disks will
> probably be somewhat different from the answer for a high-end RISC machine
> with fibre channel and a high-end RAID subsystem. But -j5 on a 4-way
> machine is way too low.
>
Sounds reasonable. I think most of us just read (and accepted!) the
Linux build docs which has always said the arg to -j is the number of
CPUs +1. It sounds a bit like one of those ad hoc settings that became
the defacto standard over time, like "swap should be 2.5 times your
physical memory."

Anyway, -j3 on a two-way box was plenty fast enough to build a Linux
kernel, which is the only place I ever used this argument. Anything
larger just had a large number of make threads waiting for other stuff
to finish (or crashing out on missing dependencies.)

Begin
On 2005-06-06, clvrmnky wrote:
> On 04/06/2005 1:08 AM, _firstname_@lr_dot_los-gatos_dot_ca.us wrote:
>>
>> We tested this extensively under HP-UX once. [...]
>
> Sounds reasonable. I think most of us just read (and accepted!) the
> Linux build docs which has always said the arg to -j is the number of
> CPUs +1.

I don't think so. Or at least, I think by now that at least here people
should know better, and people who just accept anything linux people say
or write deserve whatever they get. :-p

Besides, with a bit of varying the argument to -j you can get an idea
pretty quickly what doesn't work and what works better.


> It sounds a bit like one of those ad hoc settings that became
> the defacto standard over time, like "swap should be 2.5 times your
> physical memory."

Uhm, actually. There used to be Really Good Reasons why swap HAD to be
at least twice the available memory to be of any use at all. Twice and
a bit is still a good rule of thumb, but I am certainly not afraid to
add more, much much more if I have to. With that I subsequently broke
a linux box because it couldn't handle that much swap. Oh irony.


> Anyway, -j3 on a two-way box was plenty fast enough to build a Linux
> kernel, which is the only place I ever used this argument. Anything
> larger just had a large number of make threads waiting for other stuff
> to finish (or crashing out on missing dependencies.)

Heh. I'd guess that was an older kernel with a vm that could use some
work. I'm by no means a kernel hacker but a lot of this stuff depends
heavily on the system (hardware and software); where HPUX at least used
to run on reasonably well engineered hardware, its feel may have left
something to be desired but I'd certainly expect its vm and smp support
to be industrial strenght. I'd not expect that of linux, at least older
kernels, and I don't expect much from peecee hardware either, where
the performance numbers drive marketdroids more than the cpu. And then
there's the rest of the board.


--
j p d (at) d s b (dot) t u d e l f t (dot) n l .

_firstname_@lr_dot_los-gatos_dot_ca.us wrote:
> In article , m wrote:
>>Marco S Hyman napisa?(a):
>>> m writes:
>>>>How to use more than one cpu (I am using bsd.mp) for building something
>>>>e.g. make
>>> Make sure that there are enough processes running to require all available
>>> cpus. On my two processor system, for example, it takes something
>>> like 8 concurrent compiles to keep both processors busy most of the time.
>>you where right, I used before one more process than cpu (like I read
>>somewhere in gentoo faq) but now I set up more and CPU usage grows up
> We tested this extensively under HP-UX once. We found that the optimum was
> 8 compilation processes per CPU (we only tested powers of two, and above 4
....

While I strongly advocate the "suck it and see" methodology, a little
analysis can also be helpful.

I guess many of us with mp systems have a cmd like "prun" ("parallel run"
I calls it that
creates process farms or spmd pools for mass compiles or builds.
Or even the odd parameter search.

The question of the amount of parallelism (i.e. number of simultaneous
processes) that is "optimal" then comes up.

In this case we can assume a simple model like

T = W/n + a n

is sufficient to predict total wall time (T) given parallelism/number
of processes (n) and "amount of work" (W -- measured in any convenient
units, e.g. number of like-minded jobs that need executing).

I.e. the elapsed time for the run is proportional to the amt of
work, inversely propertional to the number of processes sharing the work,
but also has a term for "communication overhead", incl disk I/O
on individ cpus or packet traffic over nets.

The simple software ("prun") I use ramps up n during the start-up phase of the
job, saves a few "n" and "T" values, plonks a linear regression through it
to measure the proportionality constant "a", and then predicts and
optimum "n" to minimise "T". Estimates than are continually updated
each time a process exits.

The difference between this method an the common "start n processes
and wait for them all to finish" can be a matter of 50%, esp if there's
a moderate spread in the runtimes of the processes.

And you don't wanna waste 1/2 your h/w, do you?