will system() waste memory?

"Daniel Rock" <v200742@deadcafe.de> writes:[color=blue]
> Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=green][color=darkred]
>>> But the kernel doesn't know. There are more cases fork() can be used than
>>> calling exec(). Backing store (disks) is cheap nowadays.[/color]
>>
>> a) optimistic assumption: Backing store will be available when
>> (and if) it is actually needed.
>>
>> b) pessimistic assumption: Backing store for the complete
>> address space will be needed and won't be available unless
>> reserved beforehand.
>>
>> Both assumptions can cause something to fail which would otherwise
>> have succeeded. The qualitative difference can be expressed by the
>> question: Should the system rather try and eventually fail or rather
>> refuse to avoid failing?[/color]
>
> No the system doesn't fail in the pessimistic case.
> A return value of NULL to malloc() is not a failure to the system.[/color]

The system call in question happened to be fork, not some kind of
'address space extension primitive'. And a process which cannot
accomplish some task because of lack of ressources and a process which
cannot accomplish some task because the system refused to create in in
the first place have both failed to accomplish 'some task'.

NB: "An den Sophisten kann man doch bloss intellektuelle Turnuebungen
vollziehen" (Sophists are useless except as devices for intellectual
exercise).

[...]
[color=blue][color=green]
>> And this is a policy question with no 'right' answer at this
>> level. A sensible default would (IMHO) be to usually provide 'best
>> effort' service and to allow for a policy override in situation where
>> failure cannot be tolerated, assuming a general-purpose timesharing/
>> internetworking system.[/color]
>
> Don't let the computer try to be clever. He will be wrong most of
> the time.[/color]

Correct. So don't let the computer try to be clever wrt how much VM it
may need in future. It will be wrong most of the time and
gratuitously refuse to serve its users. But this actually cuts both
ways. The 'right' strategy is the one which would not have led to the
failure that happened because the other strategy had been chosen.

Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=blue]
> NB: "An den Sophisten kann man doch bloss intellektuelle Turnuebungen
> vollziehen" (Sophists are useless except as devices for intellectual
> exercise).
>[/color]

Translation: blah blah
[color=blue][color=green]
>> Don't let the computer try to be clever. He will be wrong most of
>> the time.[/color]
>
> Correct. So don't let the computer try to be clever wrt how much VM it
> may need in future. It will be wrong most of the time and
> gratuitously refuse to serve its users.[/color]

.... Instead it will kill poor unrelated victims which tried to access a
page they malloc'd long before at the wrong time.

Yes, you are absolutely right. Don't let the OS decide. Let the adminstrator
configure the system with enough swap.

--
Daniel

"Daniel Rock" <v200742@deadcafe.de> writes:
[color=blue]
> Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=green]
>> NB: "An den Sophisten kann man doch bloss intellektuelle Turnuebungen
>> vollziehen" (Sophists are useless except as devices for intellectual
>> exercise).
>>[/color]
>
> Translation: blah blah[/color]

The translation of the text you happened to quote would be 'I strongly
suspect that you try to score a point by suitable reinterpretations of
parts of the text I had actually written. This implies that you are not
actually interested in discussing a technical phenomenon and that
taking you seriously is, except insofar pretending to would be
entertaining, somewhat of a waste of time.

[...]
[color=blue][color=green]
>> Correct. So don't let the computer try to be clever wrt how much VM it
>> may need in future. It will be wrong most of the time and
>> gratuitously refuse to serve its users.[/color]
>
> ... Instead it will kill poor unrelated victims which tried to access a
> page they malloc'd long before at the wrong time.
>
> Yes, you are absolutely right.[/color]

I am absolutely right(*): There are (for the fork case I was writing
about), two possible strategies and each of them may be the wrong
choice.

(*) At least, that would be my understanding of the issue and
you haven't added anything of any substance to that.

Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=blue]
> "Daniel Rock" <v200742@deadcafe.de> writes:
> The translation of the text you happened to quote would be 'I strongly
> suspect that you try to score a point by suitable reinterpretations of
> parts of the text I had actually written. This implies that you are not
> actually interested in discussing a technical phenomenon and that
> taking you seriously is, except insofar pretending to would be
> entertaining, somewhat of a waste of time.[/color]

Blah Blah
[color=blue]
> There are (for the fork case I was writing
> about), two possible strategies and each of them may be the wrong
> choice.[/color]

No, giving the system enough swap is never the wrong choice. But killing
unrelated processes randomly is never the right choice.

--
Daniel

On Wed, 17 Oct 2007 15:06:46 +0000 (UTC), "Daniel Rock" <v200742@deadcafe.de> wrote:[color=blue]
>Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=green]
>> There are (for the fork case I was writing about), two possible
>> strategies and each of them may be the wrong choice.[/color]
>
> No, giving the system enough swap is never the wrong choice. But killing
> unrelated processes randomly is never the right choice.[/color]

I think what Rainer means is that it may be 'tricky' at best to define
'enough' in the phrase 'enough swap'.

He doesn't disagree with your preference for non-overcommitting memory;
he is just stating that sometimes, despite all the efforts put into
properly planning ahead, it is still possible to find one's self in a
state where swap is not enough. In those cases, fork() will fail to
create a new process. You are entitled to an opinion that this is not a
*system* failure, but a failure in a single process attempting to
fork(). This is still _some_ sort of failure though -- much more
predictable, reproducible and traceable than a random OOM killer reaping
innocent processes, mind you -- but _some_ sort of failure, nonetheless :)

We have diverged very far from the original "does system() waste memory"
question. Can we agree that depending on a design decision (to
overcommit or not), there may be failure modes regardless of the initial
design choise? Flaming each other to ashes doesn't really sound like a
constructive thing to do in c.u.p :)

- Giorgos

Giorgos Keramidas <keramida@ceid.upatras.gr> writes:[color=blue]
> On Wed, 17 Oct 2007 15:06:46 +0000 (UTC), "Daniel Rock" <v200742@deadcafe.de> wrote:[color=green]
>>Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=darkred]
>>> There are (for the fork case I was writing about), two possible
>>> strategies and each of them may be the wrong choice.[/color]
>>
>> No, giving the system enough swap is never the wrong choice. But killing
>> unrelated processes randomly is never the right choice.[/color]
>
> I think what Rainer means is that it may be 'tricky' at best to define
> 'enough' in the phrase 'enough swap'.
>
> He doesn't disagree with your preference for non-overcommitting
> memory;[/color]

I wasn't writing about general memory overcommittment at all. Neither
did I write about 'system failures'. Both were just issues Mr Rock
introduced into the 'discussion'. For the sake of completeness, I
repeat the statement I made, just to show the difference again.

Assuming a sufficiently large process (larger than the amount of swap
space the system can presently provide) calls fork, there are two
possible assumptions the kernel can make about its future memory usage:

a) The process will not need all of its inherited address
space and what it actually will need will be available by the
time it is needed. This is very likely if the new process will
do an exec soon. So, let the fork succeed and hope for the
best. If this assumption is wrong, the process will fail
to complete whatever its task was because it will run out of
memory while trying to.

b) The process will need all of its inherited address space
and sufficient virtual memory will not be available in time,
hence, it must be reserved beforehand. Consequently, the fork
will fail and the process that should have been created will
fail to accomplish its task because of this.

Assuming the failure in a) is worse than the failure in b), there is
no 'correct' descision the kernel can make: Not trying ascertains a
'more harmless' failure. While trying could succeed, it could equally
well lead to a 'more dangerous failure'.

And 'failure' in this context is just supposed to mean 'the program
could not accomplish what it should have accomplished if the fork had
succeeded and sufficient memory to complete this task had been
available'.

Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=blue]
> Assuming the failure in a) is worse than the failure in b), there is
> no 'correct' descision the kernel can make:[/color]

The kernel shouldn't decide or throw dices. It's the administrator's
responsibilty to give the system enough swap space.

--
Daniel

Daniel Rock wrote:[color=blue]
> Rainer Weikusat <rweikusat@mssgmbh.com> wrote:[color=green]
>> Assuming the failure in a) is worse than the failure in b), there is
>> no 'correct' descision the kernel can make:[/color]
>
> The kernel shouldn't decide or throw dices. It's the administrator's
> responsibilty to give the system enough swap space.[/color]

So the admin is supposed to look in his crystal ball and know this user who
doesn't even have an account on the machine yet, is going to run this big
process a couple of years later and allocate more space even though the bosses
above refused the request for more disk space because it isn't needed today.

Golden California Girls <GldnCAGrls@aol.com.mil> wrote:[color=blue]
> Daniel Rock wrote:[color=green]
>> The kernel shouldn't decide or throw dices. It's the administrator's
>> responsibilty to give the system enough swap space.[/color]
>
> So the admin is supposed to look in his crystal ball and know this user who
> doesn't even have an account on the machine yet, is going to run this big
> process a couple of years later and allocate more space even though the bosses
> above refused the request for more disk space because it isn't needed today.[/color]

No. Why should an admin configure for something which isn't there yet?
It is very unlikely the machine won't have an admin in a couple of years
later.

Adding swap isn't a one way task. It can be added anytime.

--
Daniel

RW> For a program with 'simple' error handling (ie failure is always
RW> terminal), and example could look like this:
RW> [...26 line function...]
RW> Accomplishing the same using a 'unified' spawn-primitive is
RW> signifcantly more complicated.

Utter rubbish. Most of the code for doing that can be found as
example code that is included in the specification itself. It
comprises just 24 lines. Here they are:

#include <spawn.h>
#include <unistd.h>

int
start_send_cmd ( const char * name, char *const argv[], char *const
envv[] )
{
int fds[2];
if (-1 == pipe(fds))
sys_die(__func__, "pipe", errno);

posix_spawn_file_actions_t file_actions ;
posix_spawn_file_actions_init(&actions) ;
posix_spawn_file_actions_dup2(&actions, fds[0], 0) ;
posix_spawn_file_actions_close(&actions, fds[0]) ;
posix_spawn_file_actions_close(&actions, fds[1]) ;

pid_t pid ;
if (int e = posix_spawn(&pid, name, &actions, NULL, argv, envv))
sys_die(__func__, "posix_spawn", e);

posix_spawn_file_actions_destroy(&actions) ;
close(fds[0]) ;
return fds[1] ;
}