will system() waste memory? - Unix

This is a discussion on will system() waste memory? - Unix ; Hi, As I know, fork() will duplicate the parent's memory, to child, if the child doesn't want to use the parent's memory, the memory is wasted, system() is a thing like this. Most call to system() is to run a ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 30

Thread: will system() waste memory?

  1. will system() waste memory?

    Hi,

    As I know, fork() will duplicate the parent's memory, to child, if the
    child doesn't want to use the parent's memory, the memory is wasted,
    system() is a thing like this.

    Most call to system() is to run a brand new program but I want to know
    if the parent run system() after allocated a large memory, will the
    memory duplicated to the systemed program which is unnecessary?

    If so, how can I do to avoid the wasted memory?

    Thanks.
    Bin


  2. Re: will system() waste memory?

    Bin Chen writes:

    > As I know, fork() will duplicate the parent's memory, to child, if the
    > child doesn't want to use the parent's memory, the memory is wasted,
    > system() is a thing like this.


    No. Whilst system might be implemented very badly, one can reasonably
    assume that it will fork and immediately exec, thus replacing the
    child's copy of the parent's process image.

    > Most call to system() is to run a brand new program but I want to know
    > if the parent run system() after allocated a large memory, will the
    > memory duplicated to the systemed program which is unnecessary?


    It may be duplicated briefly but, if it is, *nix systems are designed
    to make this efficient.

    --
    Ben.

  3. Re: will system() waste memory?

    Bin Chen writes:
    > As I know, fork() will duplicate the parent's memory, to child, if the
    > child doesn't want to use the parent's memory, the memory is wasted,


    Not exactly. fork() gives the child a replicated address space.
    Whether things are actually copied or not depends on how the VM
    subsystem is implemented, and what sorts of performance optimizations
    the designer of the system you're using has made. Typically there's
    only "some" copying going on.

    > system() is a thing like this.


    system() calls exec() -- that tears down all address space mappings in
    the process.

    > Most call to system() is to run a brand new program but I want to know
    > if the parent run system() after allocated a large memory, will the
    > memory duplicated to the systemed program which is unnecessary?


    No. exec() causes the address space to be torn down and
    reconstructed. See the exec(2) man page on your system for details.

    --
    James Carlson, Solaris Networking
    Sun Microsystems / 35 Network Drive 71.232W Vox +1 781 442 2084
    MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677

  4. Re: will system() waste memory?

    Bin Chen wrote:
    > Most call to system() is to run a brand new program but I want to know
    > if the parent run system() after allocated a large memory, will the
    > memory duplicated to the systemed program which is unnecessary?
    >
    > If so, how can I do to avoid the wasted memory?


    For this reason posix_spawn() has been introduced:

    http://www.opengroup.org/onlinepubs/...six_spawn.html

    See the discussion in the "Rationale" section.

    You can assume system() uses the optimal implementation available on your
    platform. system() might succeed in some cases where fork() + exec() would
    fail.


    --
    Daniel

  5. Re: will system() waste memory?

    "Daniel Rock" writes:
    > Bin Chen wrote:
    >> Most call to system() is to run a brand new program but I want to know
    >> if the parent run system() after allocated a large memory, will the
    >> memory duplicated to the systemed program which is unnecessary?
    >>
    >> If so, how can I do to avoid the wasted memory?

    >
    > For this reason posix_spawn() has been introduced:


    This is not correct. The rationale states that 'posix_spawn' has been
    introduced to supply a 'program execution primitive' which can be
    implemented efficiently on hardware lacking a MMU:

    The posix_spawn() function

    [...]

    been introduced to overcome the following perceived
    difficulties with fork(): the fork() function is difficult or
    impossible to implement without swapping or dynamic address
    translation.

    [...]

    POSIX needs process creation and file execution primitives
    that can be efficiently implemented without address
    translation or other MMU services.

    [...]

    Also, although they may be an efficient replacement for many
    fork()/ exec pairs, their goal is to provide useful process
    creation primitives for systems that have difficulty with
    fork(), not to provide drop-in replacements for fork()/ exec.

    > You can assume system() uses the optimal implementation available on
    > your platform.


    Since system starts a shell which executes the supplied command, it is
    certain that it is neither 'the optimal implementation' nor that it
    uses anything except fork/exec for usual shells.

    Additionally, it is equally certain that posix_spawn is less efficient
    than fork/exec except if the environment of the newly created process
    does not need to be modified before exec'ing a new program because it
    needs to run an interpreter for a declarative micro-language to
    determine and execute the intended changes, whereas code using
    fork/exec would just implement them directly. It could be more
    efficient if the OS in question supports a real vfork and the
    programmer of the code its performance is compared with did not use
    vfork despite no modifications to the environment of the newly created
    process were necessary.

    None if this matters anyhow, taking into account that UNIX(*) has been
    used for timesharing on computers that were common in the mid-to-late
    seventies with fork/exec, which apparently wasn't a problem.

  6. Re: will system() waste memory?

    Bin Chen wrote:

    > Most call to system() is to run a brand new program but I want to know
    > if the parent run system() after allocated a large memory, will the
    > memory duplicated to the systemed program which is unnecessary?


    As others have said, it's not likely to be an issue. Most modern
    systems use a copy-on-write mechanism, which means that in a fork/exec
    scenario very little memory is actually duplicated.

    One case where there could potentially be a problem is if strict memory
    accounting is enabled on linux with very tight accounting settings. If
    there is not enough memory left in the system for two copies of the
    original program, the fork() call may fail.

    Generally the memory accounting settings are loose enough (and
    individual apps are small enough) that this isn't a common problem.

    Chris

  7. Re: will system() waste memory?

    Rainer Weikusat wrote:
    > "Daniel Rock" writes:
    >> You can assume system() uses the optimal implementation available on
    >> your platform.

    >
    > Since system starts a shell which executes the supplied command, it is
    > certain that it is neither 'the optimal implementation' nor that it
    > uses anything except fork/exec for usual shells.


    There is no "unusual shell" in the sense of system().

    Why shouldn't it use something else if that is available? Solaris does use
    posix_spawn() for system():

    http://cvs.opensolaris.org/source/xr...stdio/system.c

    http://developers.sun.com/solaris/ar...ubprocess.html

    > Additionally, it is equally certain that posix_spawn is less efficient
    > than fork/exec except if the environment of the newly created process
    > does not need to be modified before exec'ing a new program because it
    > needs to run an interpreter for a declarative micro-language to
    > determine and execute the intended changes, whereas code using
    > fork/exec would just implement them directly. It could be more
    > efficient if the OS in question supports a real vfork and the
    > programmer of the code its performance is compared with did not use
    > vfork despite no modifications to the environment of the newly created
    > process were necessary.


    posix_spawn() is likely to be more efficient than fork()/exec() in most
    of the cases when implementing system(). Preparing the address space
    of the originating process and reserving the virtual memory needed for
    duplicating the address space are expensive tasks.



    --
    Daniel

  8. Re: will system() waste memory?

    On 10 15 , 11 43 , Chris Friesen wrote:
    > Bin Chen wrote:
    > > Most call to system() is to run a brand new program but I want to know
    > > if the parent run system() after allocated a large memory, will the
    > > memory duplicated to the systemed program which is unnecessary?

    >
    > As others have said, it's not likely to be an issue. Most modern
    > systems use a copy-on-write mechanism, which means that in a fork/exec
    > scenario very little memory is actually duplicated.
    >

    No, I think copy-on-write still consumpt something that is unnessary
    such as page frame structure. But if first fork than exec, it still
    will be a performance penalty, right? We do a allocate mm in child and
    exec() will free than, why not just provide a single function just NOT
    copy the mm?

    I think it is the point.
    > One case where there could potentially be a problem is if strict memory
    > accounting is enabled on linux with very tight accounting settings. If
    > there is not enough memory left in the system for two copies of the
    > original program, the fork() call may fail.
    >
    > Generally the memory accounting settings are loose enough (and
    > individual apps are small enough) that this isn't a common problem.
    >
    > Chris




  9. Re: will system() waste memory?

    On 10 15 , 11 43 , Chris Friesen wrote:
    > Bin Chen wrote:
    > > Most call to system() is to run a brand new program but I want to know
    > > if the parent run system() after allocated a large memory, will the
    > > memory duplicated to the systemed program which is unnecessary?

    >
    > As others have said, it's not likely to be an issue. Most modern
    > systems use a copy-on-write mechanism, which means that in a fork/exec
    > scenario very little memory is actually duplicated.
    >

    No, I think copy-on-write still consumpt something that is unnessary
    such as page frame structure. But if first fork than exec, it still
    will be a performance penalty, right? We do a allocate mm in child and
    exec() will free than, why not just provide a single function just NOT
    copy the mm?

    I think it is the point.
    > One case where there could potentially be a problem is if strict memory
    > accounting is enabled on linux with very tight accounting settings. If
    > there is not enough memory left in the system for two copies of the
    > original program, the fork() call may fail.
    >
    > Generally the memory accounting settings are loose enough (and
    > individual apps are small enough) that this isn't a common problem.
    >
    > Chris




  10. Re: will system() waste memory?

    On Oct 15, 5:00 am, Bin Chen wrote:

    > As I know, fork() will duplicate the parent's memory, to child, if the
    > child doesn't want to use the parent's memory, the memory is wasted,
    > system() is a thing like this.
    >
    > Most call to system() is to run a brand new program but I want to know
    > if the parent run system() after allocated a large memory, will the
    > memory duplicated to the systemed program which is unnecessary?
    >
    > If so, how can I do to avoid the wasted memory?


    Why do you care? Virtual memory is not a scarce resource.

    DS


  11. Re: will system() waste memory?

    Bin Chen wrote:

    > No, I think copy-on-write still consumpt something that is unnessary
    > such as page frame structure. But if first fork than exec, it still
    > will be a performance penalty, right? We do a allocate mm in child and
    > exec() will free than, why not just provide a single function just NOT
    > copy the mm?


    That's why vfork() was created. It does exactly that--the child runs
    using the parent's page tables, and is very limited in what it can do.
    However, there are issues with using vfork() properly. It's possible
    that on some implementations system() is implemented using vfork/exec
    internally to libc.

    On the other hand, anyone that cares about performance shouldn't be
    using system() to begin with...they should be using long-running
    processes communicating with messages or shared memory.

    Chris

  12. Re: will system() waste memory?

    David Schwartz wrote:
    > On Oct 15, 5:00 am, Bin Chen wrote:


    >>As I know, fork() will duplicate the parent's memory, to child, if the
    >>child doesn't want to use the parent's memory, the memory is wasted,
    >>system() is a thing like this.
    >>If so, how can I do to avoid the wasted memory?


    > Why do you care? Virtual memory is not a scarce resource.


    It is if strict memory accounting is enabled and the process being
    forked has allocated more than half of physical memory. However, I
    think the OP was more concerned about performance.

    Chris

  13. Re: will system() waste memory?

    On 15 Oct 2007 21:11:51 -0700, Bin Chen wrote:
    > On 10 15 , 11 43 , Chris Friesen wrote:
    >> Bin Chen wrote:
    >> > Most call to system() is to run a brand new program but I want to know
    >> > if the parent run system() after allocated a large memory, will the
    >> > memory duplicated to the systemed program which is unnecessary?

    >>
    >> As others have said, it's not likely to be an issue. Most modern
    >> systems use a copy-on-write mechanism, which means that in a fork/exec
    >> scenario very little memory is actually duplicated.

    >
    > No, I think copy-on-write still consumpt something that is unnessary
    > such as page frame structure. But if first fork than exec, it still
    > will be a performance penalty, right? We do a allocate mm in child and
    > exec() will free than, why not just provide a single function just NOT
    > copy the mm?


    FWIW, this is exactly what vfork() does :-)


  14. Re: will system() waste memory?

    "Daniel Rock" writes:
    > Rainer Weikusat wrote:
    >> "Daniel Rock" writes:
    >>> You can assume system() uses the optimal implementation available on
    >>> your platform.

    >>
    >> Since system starts a shell which executes the supplied command, it is
    >> certain that it is neither 'the optimal implementation' nor that it
    >> uses anything except fork/exec for usual shells.

    >
    > There is no "unusual shell" in the sense of system().


    There are quite a few more-or-less bourne-shell compatible shells
    available and most of them presumably use fork/exec to spawn the
    command to be executed because the code is sufficiently old.

    > Why shouldn't it use something else if that is available? Solaris does use
    > posix_spawn() for system():
    >
    > http://cvs.opensolaris.org/source/xr...stdio/system.c
    >
    > http://developers.sun.com/solaris/ar...ubprocess.html


    Executive summary: fork performance on Solaris sucks and it doesn't
    implement lazy swap allocation like 'most other systems'.

    Insofar you intended to write 'if you are developing applications
    whose address space is going to be extraordinarily large (>> 1G),
    which are specifcally targetted at Solaris and which need to execute
    external commands during their normal cause of operation, then
    posix_spawn can be used to work around technical deficiencies of the
    operating environment which may cause code working fine everywhere
    else to fail', that would have been correct. Adding the tactful
    assumption that it is ok to use vfork, ie to suspend the process
    trying to execute an external command until this command has
    started. Taking into account that vfork cannot be used reliably in
    multi-threaded applications on Solaris due to bugs in the
    implementation of the dynamic linker. Of course, this code won't
    even be portable to Solaris 9, then and it will actually perform worse
    on systems using glibc and not work at all on any of the BSDs, on
    HP-UX up to 11.22, on AIX up to 5L etc.

    A better suggestion would be to ignore specific problems of Solaris,
    insofar possible, and count on Sun fixing them sooner or later,
    because of this.

    >> Additionally, it is equally certain that posix_spawn is less efficient
    >> than fork/exec except if the environment of the newly created process
    >> does not need to be modified before exec'ing a new program because it
    >> needs to run an interpreter for a declarative micro-language to
    >> determine and execute the intended changes, whereas code using
    >> fork/exec would just implement them directly. It could be more
    >> efficient if the OS in question supports a real vfork and the
    >> programmer of the code its performance is compared with did not use
    >> vfork despite no modifications to the environment of the newly created
    >> process were necessary.

    >
    > posix_spawn() is likely to be more efficient than fork()/exec() in most
    > of the cases when implementing system().


    This doesn't really matter, because the execution of /bin/sh, followed
    by parsing a command and executing that via fork/exec will certainly
    be sufficient for kill any theoretical advantage insofar none of the
    original failure conditions do apply. Not to mention that it is
    entirely conceivable that posix_spawn is actually implemented using
    fork/exec.

  15. Re: will system() waste memory?

    Bin Chen writes:
    > On 10 15 , 11 43 , Chris Friesen wrote:
    >> Bin Chen wrote:
    >> > Most call to system() is to run a brand new program but I want to know
    >> > if the parent run system() after allocated a large memory, will the
    >> > memory duplicated to the systemed program which is unnecessary?

    >>
    >> As others have said, it's not likely to be an issue. Most modern
    >> systems use a copy-on-write mechanism, which means that in a fork/exec
    >> scenario very little memory is actually duplicated.
    >>

    > No, I think copy-on-write still consumpt something that is unnessary
    > such as page frame structure.


    It is theoretically possible to share that as well. That nobody has
    actually implemented this (in Linux) so far should be a strong hint
    that any eventual benefit es negligible.

    > But if first fork than exec, it still will be a performance penalty,
    > right? We do a allocate mm in child and exec() will free than, why
    > not just provide a single function just NOT copy the mm?


    Programs running external commands usually intend to use their
    functionality to accomplish something particular. This means 'certain
    arrangements' are necessary to be able to use the external command,
    like setting up file descriptors appropriately. For a program with
    'simple' error handling (ie failure is always terminal), and example
    could look like this:

    static int start_send_cmd(void)
    {
    int fds[2], rc;

    rc = pipe(fds);
    rc != -1 || sys_die(__func__, "pipe", errno);

    rc = fork();
    switch (rc) {
    case 0:
    dup2(*fds, STDIN_FILENO);

    close(*fds);
    close(fds[1]);

    execv(*eli_sa_send, (void *)eli_sa_send);

    sys_die(__func__, "execv", errno);

    case -1:
    sys_die(__func__, "fork", errno);
    }

    close(*fds);
    return fds[1];
    }
    [code (c) my employer and cited for educational purposes]

    Accomplishing the same using a 'unified' spawn-primitive is
    signifcantly more complicated.

  16. Re: will system() waste memory?

    Rainer Weikusat wrote:
    > "Daniel Rock" writes:
    >> There is no "unusual shell" in the sense of system().

    >
    > There are quite a few more-or-less bourne-shell compatible shells
    > available and most of them presumably use fork/exec to spawn the
    > command to be executed because the code is sufficiently old.


    What does this have to do with system()?


    > Executive summary: fork performance on Solaris sucks and it doesn't
    > implement lazy swap allocation like 'most other systems'.


    I don't want any system which does memory overcommit. I want deterministic
    system behaviour.


    > A better suggestion would be to ignore specific problems of Solaris,
    > insofar possible, and count on Sun fixing them sooner or later,
    > because of this.


    Which problems?


    > This doesn't really matter, because the execution of /bin/sh, followed
    > by parsing a command and executing that via fork/exec will certainly
    > be sufficient for kill any theoretical advantage


    And what makes this different from system()? (Hint: read the subject line)

    --
    Daniel

  17. Re: will system() waste memory?

    "Daniel Rock" writes:
    > Rainer Weikusat wrote:
    >> "Daniel Rock" writes:
    >>> There is no "unusual shell" in the sense of system().

    >>
    >> There are quite a few more-or-less bourne-shell compatible shells
    >> available and most of them presumably use fork/exec to spawn the
    >> command to be executed because the code is sufficiently old.

    >
    > What does this have to do with system()?


    This is a question which can be easily answered by using either the
    standard text or the available system documentation.

    >> Executive summary: fork performance on Solaris sucks and it doesn't
    >> implement lazy swap allocation like 'most other systems'.

    >
    > I don't want any system which does memory overcommit. I want
    > deterministic system behaviour.


    In the given context, this means that you want a system where 'large
    processes' cannot sensibly use fork/exec to execute external
    commands, despite reserving enough swap space to eventually store
    all of the contents of the inherited address space is completely
    unnecessary for this particular case.

    [...]

    >> A better suggestion would be to ignore specific problems of Solaris,
    >> insofar possible, and count on Sun fixing them sooner or later,
    >> because of this.

    >
    > Which problems?


    The mentioned self-deadlock of the dynamic linker, for instance. Or
    the (reportedly) bad fork performance.

    >> This doesn't really matter, because the execution of /bin/sh, followed
    >> by parsing a command and executing that via fork/exec will certainly
    >> be sufficient for kill any theoretical advantage

    >
    > And what makes this different from system()?


    Why would it be different from system?

  18. Re: will system() waste memory?

    Rainer Weikusat wrote:
    >> What does this have to do with system()?

    >
    > This is a question which can be easily answered by using either the
    > standard text or the available system documentation.


    Like I said.

    >> I don't want any system which does memory overcommit. I want
    >> deterministic system behaviour.

    >
    > In the given context, this means that you want a system where 'large
    > processes' cannot sensibly use fork/exec to execute external
    > commands, despite reserving enough swap space to eventually store
    > all of the contents of the inherited address space is completely
    > unnecessary for this particular case.


    But the kernel doesn't know. There are more cases fork() can be used than
    calling exec(). Backing store (disks) is cheap nowadays.


    >> And what makes this different from system()?

    >
    > Why would it be different from system?


    That's what I'm asking you.

    --
    Daniel

  19. Re: will system() waste memory?

    "Daniel Rock" writes:
    > Rainer Weikusat wrote:
    >> In the given context, this means that you want a system where 'large
    >> processes' cannot sensibly use fork/exec to execute external
    >> commands, despite reserving enough swap space to eventually store
    >> all of the contents of the inherited address space is completely
    >> unnecessary for this particular case.

    >
    > But the kernel doesn't know. There are more cases fork() can be used than
    > calling exec(). Backing store (disks) is cheap nowadays.


    Actually, the kernel can make either of two assumptions:

    a) optimistic assumption: Backing store will be available when
    (and if) it is actually needed.

    b) pessimistic assumption: Backing store for the complete
    address space will be needed and won't be available unless
    reserved beforehand.

    Both assumptions can cause something to fail which would otherwise
    have succeeded. The qualitative difference can be expressed by the
    question: Should the system rather try and eventually fail or rather
    refuse to avoid failing?

    And this is a policy question with no 'right' answer at this
    level. A sensible default would (IMHO) be to usually provide 'best
    effort' service and to allow for a policy override in situation where
    failure cannot be tolerated, assuming a general-purpose timesharing/
    internetworking system.

    >>> And what makes this different from system()?

    >>
    >> Why would it be different from system?

    >
    > That's what I'm asking you.


    But since the description was intended to describe system, the
    question appears to be somehwat pointless.

  20. Re: will system() waste memory?

    Rainer Weikusat wrote:
    >> But the kernel doesn't know. There are more cases fork() can be used than
    >> calling exec(). Backing store (disks) is cheap nowadays.

    >
    > a) optimistic assumption: Backing store will be available when
    > (and if) it is actually needed.
    >
    > b) pessimistic assumption: Backing store for the complete
    > address space will be needed and won't be available unless
    > reserved beforehand.
    >
    > Both assumptions can cause something to fail which would otherwise
    > have succeeded. The qualitative difference can be expressed by the
    > question: Should the system rather try and eventually fail or rather
    > refuse to avoid failing?


    No the system doesn't fail in the pessimistic case. A return value of
    NULL to malloc() is not a failure to the system. I prefer the deterministic
    approach. Configuring large swap doesn't cost much nowadays.


    > And this is a policy question with no 'right' answer at this
    > level. A sensible default would (IMHO) be to usually provide 'best
    > effort' service and to allow for a policy override in situation where
    > failure cannot be tolerated, assuming a general-purpose timesharing/
    > internetworking system.


    Don't let the computer try to be clever. He will be wrong most of the time.

    --
    Daniel

+ Reply to Thread
Page 1 of 2 1 2 LastLast