will system() waste memory? - Unix

This is a discussion on will system() waste memory? - Unix ; "Daniel Rock" writes: > Rainer Weikusat wrote: >>> But the kernel doesn't know. There are more cases fork() can be used than >>> calling exec(). Backing store (disks) is cheap nowadays. >> >> a) optimistic assumption: Backing store will be ...

+ Reply to Thread
Page 2 of 2 FirstFirst 1 2
Results 21 to 30 of 30

Thread: will system() waste memory?

  1. Re: will system() waste memory?

    "Daniel Rock" writes:
    > Rainer Weikusat wrote:
    >>> But the kernel doesn't know. There are more cases fork() can be used than
    >>> calling exec(). Backing store (disks) is cheap nowadays.

    >>
    >> a) optimistic assumption: Backing store will be available when
    >> (and if) it is actually needed.
    >>
    >> b) pessimistic assumption: Backing store for the complete
    >> address space will be needed and won't be available unless
    >> reserved beforehand.
    >>
    >> Both assumptions can cause something to fail which would otherwise
    >> have succeeded. The qualitative difference can be expressed by the
    >> question: Should the system rather try and eventually fail or rather
    >> refuse to avoid failing?

    >
    > No the system doesn't fail in the pessimistic case.
    > A return value of NULL to malloc() is not a failure to the system.


    The system call in question happened to be fork, not some kind of
    'address space extension primitive'. And a process which cannot
    accomplish some task because of lack of ressources and a process which
    cannot accomplish some task because the system refused to create in in
    the first place have both failed to accomplish 'some task'.

    NB: "An den Sophisten kann man doch bloss intellektuelle Turnuebungen
    vollziehen" (Sophists are useless except as devices for intellectual
    exercise).

    [...]

    >> And this is a policy question with no 'right' answer at this
    >> level. A sensible default would (IMHO) be to usually provide 'best
    >> effort' service and to allow for a policy override in situation where
    >> failure cannot be tolerated, assuming a general-purpose timesharing/
    >> internetworking system.

    >
    > Don't let the computer try to be clever. He will be wrong most of
    > the time.


    Correct. So don't let the computer try to be clever wrt how much VM it
    may need in future. It will be wrong most of the time and
    gratuitously refuse to serve its users. But this actually cuts both
    ways. The 'right' strategy is the one which would not have led to the
    failure that happened because the other strategy had been chosen.

  2. Re: will system() waste memory?

    Rainer Weikusat wrote:
    > NB: "An den Sophisten kann man doch bloss intellektuelle Turnuebungen
    > vollziehen" (Sophists are useless except as devices for intellectual
    > exercise).
    >


    Translation: blah blah

    >> Don't let the computer try to be clever. He will be wrong most of
    >> the time.

    >
    > Correct. So don't let the computer try to be clever wrt how much VM it
    > may need in future. It will be wrong most of the time and
    > gratuitously refuse to serve its users.


    .... Instead it will kill poor unrelated victims which tried to access a
    page they malloc'd long before at the wrong time.

    Yes, you are absolutely right. Don't let the OS decide. Let the adminstrator
    configure the system with enough swap.

    --
    Daniel


  3. Re: will system() waste memory?

    "Daniel Rock" writes:

    > Rainer Weikusat wrote:
    >> NB: "An den Sophisten kann man doch bloss intellektuelle Turnuebungen
    >> vollziehen" (Sophists are useless except as devices for intellectual
    >> exercise).
    >>

    >
    > Translation: blah blah


    The translation of the text you happened to quote would be 'I strongly
    suspect that you try to score a point by suitable reinterpretations of
    parts of the text I had actually written. This implies that you are not
    actually interested in discussing a technical phenomenon and that
    taking you seriously is, except insofar pretending to would be
    entertaining, somewhat of a waste of time.

    [...]

    >> Correct. So don't let the computer try to be clever wrt how much VM it
    >> may need in future. It will be wrong most of the time and
    >> gratuitously refuse to serve its users.

    >
    > ... Instead it will kill poor unrelated victims which tried to access a
    > page they malloc'd long before at the wrong time.
    >
    > Yes, you are absolutely right.


    I am absolutely right(*): There are (for the fork case I was writing
    about), two possible strategies and each of them may be the wrong
    choice.

    (*) At least, that would be my understanding of the issue and
    you haven't added anything of any substance to that.


  4. Re: will system() waste memory?

    Rainer Weikusat wrote:
    > "Daniel Rock" writes:
    > The translation of the text you happened to quote would be 'I strongly
    > suspect that you try to score a point by suitable reinterpretations of
    > parts of the text I had actually written. This implies that you are not
    > actually interested in discussing a technical phenomenon and that
    > taking you seriously is, except insofar pretending to would be
    > entertaining, somewhat of a waste of time.


    Blah Blah

    > There are (for the fork case I was writing
    > about), two possible strategies and each of them may be the wrong
    > choice.


    No, giving the system enough swap is never the wrong choice. But killing
    unrelated processes randomly is never the right choice.

    --
    Daniel

  5. Re: will system() waste memory?

    On Wed, 17 Oct 2007 15:06:46 +0000 (UTC), "Daniel Rock" wrote:
    >Rainer Weikusat wrote:
    >> There are (for the fork case I was writing about), two possible
    >> strategies and each of them may be the wrong choice.

    >
    > No, giving the system enough swap is never the wrong choice. But killing
    > unrelated processes randomly is never the right choice.


    I think what Rainer means is that it may be 'tricky' at best to define
    'enough' in the phrase 'enough swap'.

    He doesn't disagree with your preference for non-overcommitting memory;
    he is just stating that sometimes, despite all the efforts put into
    properly planning ahead, it is still possible to find one's self in a
    state where swap is not enough. In those cases, fork() will fail to
    create a new process. You are entitled to an opinion that this is not a
    *system* failure, but a failure in a single process attempting to
    fork(). This is still _some_ sort of failure though -- much more
    predictable, reproducible and traceable than a random OOM killer reaping
    innocent processes, mind you -- but _some_ sort of failure, nonetheless

    We have diverged very far from the original "does system() waste memory"
    question. Can we agree that depending on a design decision (to
    overcommit or not), there may be failure modes regardless of the initial
    design choise? Flaming each other to ashes doesn't really sound like a
    constructive thing to do in c.u.p

    - Giorgos


  6. Re: will system() waste memory?

    Giorgos Keramidas writes:
    > On Wed, 17 Oct 2007 15:06:46 +0000 (UTC), "Daniel Rock" wrote:
    >>Rainer Weikusat wrote:
    >>> There are (for the fork case I was writing about), two possible
    >>> strategies and each of them may be the wrong choice.

    >>
    >> No, giving the system enough swap is never the wrong choice. But killing
    >> unrelated processes randomly is never the right choice.

    >
    > I think what Rainer means is that it may be 'tricky' at best to define
    > 'enough' in the phrase 'enough swap'.
    >
    > He doesn't disagree with your preference for non-overcommitting
    > memory;


    I wasn't writing about general memory overcommittment at all. Neither
    did I write about 'system failures'. Both were just issues Mr Rock
    introduced into the 'discussion'. For the sake of completeness, I
    repeat the statement I made, just to show the difference again.

    Assuming a sufficiently large process (larger than the amount of swap
    space the system can presently provide) calls fork, there are two
    possible assumptions the kernel can make about its future memory usage:

    a) The process will not need all of its inherited address
    space and what it actually will need will be available by the
    time it is needed. This is very likely if the new process will
    do an exec soon. So, let the fork succeed and hope for the
    best. If this assumption is wrong, the process will fail
    to complete whatever its task was because it will run out of
    memory while trying to.

    b) The process will need all of its inherited address space
    and sufficient virtual memory will not be available in time,
    hence, it must be reserved beforehand. Consequently, the fork
    will fail and the process that should have been created will
    fail to accomplish its task because of this.

    Assuming the failure in a) is worse than the failure in b), there is
    no 'correct' descision the kernel can make: Not trying ascertains a
    'more harmless' failure. While trying could succeed, it could equally
    well lead to a 'more dangerous failure'.

    And 'failure' in this context is just supposed to mean 'the program
    could not accomplish what it should have accomplished if the fork had
    succeeded and sufficient memory to complete this task had been
    available'.

  7. Re: will system() waste memory?

    Rainer Weikusat wrote:
    > Assuming the failure in a) is worse than the failure in b), there is
    > no 'correct' descision the kernel can make:


    The kernel shouldn't decide or throw dices. It's the administrator's
    responsibilty to give the system enough swap space.

    --
    Daniel


  8. Re: will system() waste memory?

    Daniel Rock wrote:
    > Rainer Weikusat wrote:
    >> Assuming the failure in a) is worse than the failure in b), there is
    >> no 'correct' descision the kernel can make:

    >
    > The kernel shouldn't decide or throw dices. It's the administrator's
    > responsibilty to give the system enough swap space.


    So the admin is supposed to look in his crystal ball and know this user who
    doesn't even have an account on the machine yet, is going to run this big
    process a couple of years later and allocate more space even though the bosses
    above refused the request for more disk space because it isn't needed today.

  9. Re: will system() waste memory?

    Golden California Girls wrote:
    > Daniel Rock wrote:
    >> The kernel shouldn't decide or throw dices. It's the administrator's
    >> responsibilty to give the system enough swap space.

    >
    > So the admin is supposed to look in his crystal ball and know this user who
    > doesn't even have an account on the machine yet, is going to run this big
    > process a couple of years later and allocate more space even though the bosses
    > above refused the request for more disk space because it isn't needed today.


    No. Why should an admin configure for something which isn't there yet?
    It is very unlikely the machine won't have an admin in a couple of years
    later.

    Adding swap isn't a one way task. It can be added anytime.

    --
    Daniel

  10. will system() waste memory?

    RW> For a program with 'simple' error handling (ie failure is always
    RW> terminal), and example could look like this:
    RW> [...26 line function...]
    RW> Accomplishing the same using a 'unified' spawn-primitive is
    RW> signifcantly more complicated.

    Utter rubbish. Most of the code for doing that can be found as
    example code that is included in the specification itself. It
    comprises just 24 lines. Here they are:

    #include
    #include

    int
    start_send_cmd ( const char * name, char *const argv[], char *const
    envv[] )
    {
    int fds[2];
    if (-1 == pipe(fds))
    sys_die(__func__, "pipe", errno);

    posix_spawn_file_actions_t file_actions ;
    posix_spawn_file_actions_init(&actions) ;
    posix_spawn_file_actions_dup2(&actions, fds[0], 0) ;
    posix_spawn_file_actions_close(&actions, fds[0]) ;
    posix_spawn_file_actions_close(&actions, fds[1]) ;

    pid_t pid ;
    if (int e = posix_spawn(&pid, name, &actions, NULL, argv, envv))
    sys_die(__func__, "posix_spawn", e);

    posix_spawn_file_actions_destroy(&actions) ;
    close(fds[0]) ;
    return fds[1] ;
    }


+ Reply to Thread
Page 2 of 2 FirstFirst 1 2