How to make C++ code fork-safe? - Unix

This is a discussion on How to make C++ code fork-safe? - Unix ; Hi all. I've been programming in C++ for quite some time, but I am new in the unix world. One thing that keeps surprizing me is fork()... The software I am working on right now seems to use fork() quite ...

+ Reply to Thread
Page 1 of 2 1 2 LastLast
Results 1 to 20 of 22

Thread: How to make C++ code fork-safe?

  1. How to make C++ code fork-safe?

    Hi all.

    I've been programming in C++ for quite some time, but I am new in the
    unix world. One thing that keeps surprizing me is fork()...

    The software I am working on right now seems to use fork() quite
    extensively. Besides the fact that it runs under Apache (which forks,
    I think, in rather controlled manner), forks are used left and right
    in the module itself. It seems that, when something needs to be done
    asynchronously, a fork() is issued, the work is done in the child
    process, and then the child is terminated.

    First issue I ran into was a thread created in the singleton
    constructor not replicated in the forked process. This was quite easy
    to workaround once I figured what's happenning, but then it became
    worse... The condition variable inited in parent process hanged in
    pthread_cond_destroy() at the child process termination. Then we
    found pthread_atexit...

    Now I think I ran into another related issue. I will not describe it
    here, but rather try to provide some generic description:

    Supposedly you have a global object (singleton, etc.). To make it
    thread-safe you lock some of its operations. In the middle of such
    operation the object may be in an inconsistent state. Let's say a
    fork is issued from another thread during such an operation (most
    likely the thread calling fork() does not lock your global object).
    It copies the object into the child process while it is in the middle
    of something... Later on it crashes in the destructor when the child
    is terminated.

    Even better, recall the problem with the double-check pattern
    (described in http://www.aristeia.com/Papers/DDJ_J...04_revised.pdf).
    Looks like this problem directly applies to fork() if it is issued
    from another thread after the pointer is assigned but before the
    object is initialized.

    Now I am wondering if there exist any rules that can be applied to
    make your C++ code fork-safe? Or does the usage of fork() itself
    needs to be limited?

    Thanks for any advice.

    Arkadiy


  2. Re: How to make C++ code fork-safe?

    On Nov 30, 7:26 pm, Arkadiy wrote:
    > Now I am wondering if there exist any rules that can be applied to
    > make your C++ code fork-safe? Or does the usage of fork() itself
    > needs to be limited?


    A good rule of thumb is don't mix multi threading with multi process
    since it all gets rather messy especially given that if you fork a
    process the child process is single threaded so if you're relying on
    various threads to complete operations in the child you're out of
    luck. There is a function called pthread_atfork() that you may find
    useful however.

    B2003


  3. Re: How to make C++ code fork-safe?

    Arkadiy writes:
    > I've been programming in C++ for quite some time, but I am new in the
    > unix world. One thing that keeps surprizing me is fork()...
    >
    > The software I am working on right now seems to use fork() quite
    > extensively. Besides the fact that it runs under Apache (which forks,
    > I think, in rather controlled manner), forks are used left and right
    > in the module itself. It seems that, when something needs to be done
    > asynchronously, a fork() is issued, the work is done in the child
    > process, and then the child is terminated.


    That would be a usual way to handle asychronicity (term?) in a world
    which has neither async I/O nor multiple threads, ie a fairly
    traditional UNIX(*) environment.

    > First issue I ran into was a thread created in the singleton
    > constructor not replicated in the forked process. This was quite easy
    > to workaround once I figured what's happenning, but then it became
    > worse... The condition variable inited in parent process hanged in
    > pthread_cond_destroy() at the child process termination. Then we
    > found pthread_atexit...


    Well, you cannot just tack multithreaded C++ code written under
    certain assumptions which happen to be false for UNIX(*) onto an
    existing codebase written for UNIX(*).

    > Now I think I ran into another related issue. I will not describe it
    > here, but rather try to provide some generic description:
    >
    > Supposedly you have a global object (singleton, etc.). To make it
    > thread-safe you lock some of its operations. In the middle of such
    > operation the object may be in an inconsistent state. Let's say a
    > fork is issued from another thread during such an operation (most
    > likely the thread calling fork() does not lock your global object).
    > It copies the object into the child process while it is in the middle
    > of something... Later on it crashes in the destructor when the child
    > is terminated.


    Presumably, instead of crashing, the code would (at some point in
    time) deadlock while waiting for release of a mutex which is
    unreleasable in the forked process because it was locked by a certain
    thread in the parent process which does not exist in the child. This
    could be dealt with by setting up atfork-handlers which acquire and
    release all possibly held locks in a controlled manner but it would
    probably be simpler to just accept that certain programming paradigms
    just cannot be mixed this way.

    [...]

    > Now I am wondering if there exist any rules that can be applied to
    > make your C++ code fork-safe? Or does the usage of fork() itself
    > needs to be limited?


    There is the simple rule that multithreaded process cannot use easily
    use fork the same way single-threaded processes can, and usually, one
    would structure the code to be either one or the other and stick to
    it.

    In your case, this appears to imply that you should either get rid of
    the forks or the multi-threading, whatever is more convenient.

  4. Re: How to make C++ code fork-safe?

    On 30 Nov, 19:26, Arkadiy wrote:

    >
    > Now I am wondering if there exist any rules that can be applied to
    > make your C++ code fork-safe? Or does the usage of fork() itself
    > needs to be limited?


    Personal opinion here, but fork() is not the problem. The
    problem is pthread_create(). You should not mix threading
    with forking.

    Threads are a convenient way to get the functionality provided
    by proper IPC, but are pretty much a cop out when you're working
    on a system that provides adequate IPC. AFAIK, the only things
    you gain by using threads are: a more complicated API which
    some people argue is simpler, and the ability to simulate
    proper IPC on systems which do not support it well.

    Always happy to be educated, though, so if anyone knows
    any use of threading that cannot be accomplished via
    fork() (on unix, that is. This is comp.unix.programmer,
    after all), I'm willing to read about it.

  5. Re: How to make C++ code fork-safe?

    William Pursell writes:
    > On 30 Nov, 19:26, Arkadiy wrote:


    [...]

    > Threads are a convenient way to get the functionality provided
    > by proper IPC, but are pretty much a cop out when you're working
    > on a system that provides adequate IPC. AFAIK, the only things
    > you gain by using threads are: a more complicated API which
    > some people argue is simpler, and the ability to simulate
    > proper IPC on systems which do not support it well.


    One of the usual gains would be being able to avoid IPC altogether.
    This can, of course, be achieved using shared memory, too, but this
    would then need full-blown locking as well and would be more
    complicated to program because sharing pointer values would not
    (easily, portably) be possible.

    > Always happy to be educated, though, so if anyone knows
    > any use of threading that cannot be accomplished via
    > fork() (on unix, that is. This is comp.unix.programmer,
    > after all), I'm willing to read about it.


    This is somewhat of a trick question. Because UNIX(*) supports shared
    memory, everything which can be accomplished by using multiple threads
    can be accomplished by using multiple process explicitly sharing the
    necessary memory areas, too.

  6. Re: How to make C++ code fork-safe?

    On Dec 3, 11:48 am, William Pursell wrote:
    > On 30 Nov, 19:26, Arkadiy wrote:
    >
    >
    >
    > > Now I am wondering if there exist any rules that can be applied to
    > > make your C++ code fork-safe? Or does the usage of fork() itself
    > > needs to be limited?

    >
    > Personal opinion here, but fork() is not the problem. The
    > problem is pthread_create(). You should not mix threading
    > with forking.
    >
    > Threads are a convenient way to get the functionality provided
    > by proper IPC, but are pretty much a cop out when you're working
    > on a system that provides adequate IPC. AFAIK, the only things
    > you gain by using threads are: a more complicated API which
    > some people argue is simpler, and the ability to simulate
    > proper IPC on systems which do not support it well.
    >
    > Always happy to be educated, though, so if anyone knows
    > any use of threading that cannot be accomplished via
    > fork() (on unix, that is. This is comp.unix.programmer,
    > after all), I'm willing to read about it.


    Threads are invented to provide finer grained parallelism then
    processes do.
    I think it would be kind of inconvenient to implement Event-Dispater
    (for GUI) as a separate process.


  7. Re: How to make C++ code fork-safe?

    Andrey wrote:
    > Threads are invented to provide finer grained parallelism then
    > processes do.
    > I think it would be kind of inconvenient to implement Event-Dispater
    > (for GUI) as a separate process.


    Exactly. Inconvenient for the black hats, even.


  8. Re: How to make C++ code fork-safe?

    Andrey writes:
    > On Dec 3, 11:48 am, William Pursell wrote:


    [...]

    >> Threads are a convenient way to get the functionality provided
    >> by proper IPC, but are pretty much a cop out when you're working
    >> on a system that provides adequate IPC. AFAIK, the only things
    >> you gain by using threads are: a more complicated API which
    >> some people argue is simpler, and the ability to simulate
    >> proper IPC on systems which do not support it well.
    >>
    >> Always happy to be educated, though, so if anyone knows
    >> any use of threading that cannot be accomplished via
    >> fork() (on unix, that is. This is comp.unix.programmer,
    >> after all), I'm willing to read about it.

    >
    > Threads are invented to provide finer grained parallelism then
    > processes do.


    It is not exactly clear what 'finer grained parallelism' is supposed
    to mean. A process always has an address space of its own. Multiple
    threads being part of the same process share an
    address space.

  9. Re: How to make C++ code fork-safe?

    William Ahern writes:
    > Andrey wrote:
    >> Threads are invented to provide finer grained parallelism then
    >> processes do.
    >> I think it would be kind of inconvenient to implement Event-Dispater
    >> (for GUI) as a separate process.

    >
    > Exactly. Inconvenient for the black hats, even.


    A German proverb goes 'Was der Bauer nicht kennt, dass frisst er
    nicht' ("a farmer won't eat what he isn't used to").

    The interface-part of a GUI-application is usually structured around an
    event dispatching loop. Events represent either user input (mouse
    movements, keypresses) or are synthetically generated in reaction to
    'environmental changes' (eg a redraw-event sent to some application
    because one of its windows has been uncovered because another window
    was moved). The interface part of such an application has soft
    realtime requirements because it needs to react whenever the user does
    something. Therefore, it is imperative that no 'lenghty' calculations
    or anything else consuming 'a lot of time' (such as blocking for I/O)
    take place in the event processing code. The usual way to solve this
    would be to do any such lengthy operation from a separate thread while
    the event handling thread continues to process ecents.

    Instead of waving that dreaded rubber-chicken, an explanation why
    precisely this would be bad idea would be more useful.

  10. Re: How to make C++ code fork-safe?


    > Always happy to be educated, though, so if anyone knows
    > any use of threading that cannot be accomplished via
    > fork() (on unix, that is. This is comp.unix.programmer,
    > after all), I'm willing to read about it.


    Creating a thread is faster than creating a process and if all you
    want to do is swap off a thread that either sets or doesn't set some
    global used by the main thread (eg a seperate DNS name resolver
    thread) then it can make the code a lot simpler than using multi
    process since for that you'd have to put extra code into the main
    process to read the incoming data from the child.

    B2003


  11. Re: How to make C++ code fork-safe?

    On Dec 3, 2:44 pm, Rainer Weikusat wrote:
    > William Pursell writes:
    > > On 30 Nov, 19:26, Arkadiy wrote:

    >
    > [...]
    >
    > > Threads are a convenient way to get the functionality provided
    > > by proper IPC, but are pretty much a cop out when you're working
    > > on a system that provides adequate IPC. AFAIK, the only things
    > > you gain by using threads are: a more complicated API which
    > > some people argue is simpler, and the ability to simulate
    > > proper IPC on systems which do not support it well.

    >
    > One of the usual gains would be being able to avoid IPC altogether.
    > This can, of course, be achieved using shared memory, too, but this
    > would then need full-blown locking as well and would be more
    > complicated to program because sharing pointer values would not
    > (easily, portably) be possible.
    >
    > > Always happy to be educated, though, so if anyone knows
    > > any use of threading that cannot be accomplished via
    > > fork() (on unix, that is. This is comp.unix.programmer,
    > > after all), I'm willing to read about it.

    >
    > This is somewhat of a trick question. Because UNIX(*) supports shared
    > memory, everything which can be accomplished by using multiple threads
    > can be accomplished by using multiple process explicitly sharing the
    > necessary memory areas, too.


    Multithreaded programs are more portable since, in contrast to the
    fork(), threads can be used on both unix and windows.

  12. Re: How to make C++ code fork-safe?

    Andrey wrote:
    > On Dec 3, 2:44 pm, Rainer Weikusat wrote:


    > > This is somewhat of a trick question. Because UNIX(*) supports shared
    > > memory, everything which can be accomplished by using multiple threads
    > > can be accomplished by using multiple process explicitly sharing the
    > > necessary memory areas, too.


    > Multithreaded programs are more portable since, in contrast to the
    > fork(), threads can be used on both unix and windows.


    That doesn't make sense. Does Windows have pthread_create()?

    Threads might be easier to use because of third party libraries which hide
    the details of the plaforms threading implementation.

    But, the same could be said for fork() and inter-process shared memory. On
    Windows I use multiple processes which use socket IPC. The only downside is
    Windows has categorically inferior support for process management in terms
    of child/parent process interaction. But, on my project that's somebody
    else's problem If I had wanted I could have also used shared memory, but
    instead I chose the route which required the least amount of coding or
    library dependency (BSD sockets API exists almost everywhere).

    However, note that Windows Vista _does_ have fork().

    I'm not arguing that threads may be the more prudent choice for portable
    programming (though not 5 years ago threads may have been the least portable
    choice, by far). But the reasons have less to do w/ the fundamental
    capabilities of the platforms (which, as between Unix and Windows are almost
    identical in rough measure) as compared to other externalities.

  13. Re: How to make C++ code fork-safe?

    Rainer Weikusat wrote:
    > William Ahern writes:
    > > Andrey wrote:
    > >> Threads are invented to provide finer grained parallelism then
    > >> processes do.
    > >> I think it would be kind of inconvenient to implement Event-Dispater
    > >> (for GUI) as a separate process.

    > >
    > > Exactly. Inconvenient for the black hats, even.


    > A German proverb goes 'Was der Bauer nicht kennt, dass frisst er
    > nicht' ("a farmer won't eat what he isn't used to").


    > The interface-part of a GUI-application is usually structured around an
    > event dispatching loop. Events represent either user input (mouse
    > movements, keypresses) or are synthetically generated in reaction to
    > 'environmental changes' (eg a redraw-event sent to some application
    > because one of its windows has been uncovered because another window
    > was moved). The interface part of such an application has soft
    > realtime requirements because it needs to react whenever the user does
    > something. Therefore, it is imperative that no 'lenghty' calculations
    > or anything else consuming 'a lot of time' (such as blocking for I/O)
    > take place in the event processing code. The usual way to solve this
    > would be to do any such lengthy operation from a separate thread while
    > the event handling thread continues to process ecents.
    >


    Indeed that is the usual way. I'm unsure why its not more common to employ
    multiple processes, using a more structured and constrained dispatching
    mechanism between processes, and isolating the code doing prolonged
    calculations (often on opaque input data, often untrusted opaque input
    data). I can think of web browers, e-mail clients, image viewing or
    manipulation applications, audio/video applications... I could go on forever
    where this is a prima facie preferred choice absent factors like
    convenience.

    I suppose it might be more convenient, given the dearth of libraries which
    might implement this mechanism and present it with a simple interface. But
    let's break it down. You often have a GUI event which kick starts the
    processing of some datam. AFAIK--though I'm not a GUI app developer--typical
    SOP is to drop the datum and some meta data into a work queue, where a
    worker thread picks it up. In the grander scheme of things, it seems only
    nominally more convenient to use only threads for this, as opposed to
    introducing, say, just a single separate process which itself might use a
    thread worker queue. But that process could be sandboxed, and its
    input/output tightly constrained, at least as compared to a single process
    (I imagine its hard to sandbox the process receiving GUI events).

    Anyhow, my larger point is that convenience is hardly the strongest
    argument. Certainly, convenience isn't generally one of the more important
    motives for writing for an application. And when choosing how one architects
    an application, I would think convenience is likewise not at the top of the
    list of factors. Its a substantial factor, to be sure, but by no means the
    determinative one.


  14. Re: How to make C++ code fork-safe?

    On Dec 3, 3:44 am, Rainer Weikusat wrote:

    > > Always happy to be educated, though, so if anyone knows
    > > any use of threading that cannot be accomplished via
    > > fork() (on unix, that is. This is comp.unix.programmer,
    > > after all), I'm willing to read about it.


    > This is somewhat of a trick question. Because UNIX(*) supports shared
    > memory, everything which can be accomplished by using multiple threads
    > can be accomplished by using multiple process explicitly sharing the
    > necessary memory areas, too.


    Well, almost anything you can do with a computer can be done with any
    particular set of tools. You can write a 3D CAD/CAM application
    entirely in Z80 assembler if you want to. That doesn't mean they're
    all equally good.

    I don't think I'm disagreeing with either of you.

    For some problems, 'fork' is a the perfect solution. Some are really
    nicely handled with threads. Some are best solved with shared memory
    and some without. It's nice to have the variety of tools. You can bash
    a screw in with a hammer, but there is also a better way.

    DS

  15. Re: How to make C++ code fork-safe?

    > > Multithreaded programs are more portable since, in contrast to the
    > > fork(), threads can be used on both unix and windows.

    >
    > That doesn't make sense. Does Windows have pthread_create()?
    >
    > Threads might be easier to use because of third party libraries which hide
    > the details of the plaforms threading implementation.
    >


    I didn't profess that Windows has pthreads (however it has as well as
    boost/thread and QT).
    I meant that precise semantic of fork() can't be implemented on
    Windows, can it?


  16. Support for process management in terms of child/parent processinteraction

    WA> On Windows I use multiple processes which use socket IPC. The
    WA> only downside is Windows has categorically inferior support for
    WA> process management in terms of child/parent process interaction.

    Untrue. (This is not least in part because Windows NT natively has
    all of the requisite mechanism for providing the POSIX API functions
    for process handling. It does, after all, have a POSIX subsystem.)
    Indeed, native facilities aside, Win32 is actually superior in several
    areas. For example: In this very newsgroup a mere three days ago
    someone was asking how a child process could receive notification that
    its parent process had terminated. That's tricky to do in Unix, and
    the best solution if one really does want this is usually to change
    the application architecture, either reversing the roles of parent and
    child or waiting for IPC mechanisms to be closed rather than for
    processes to terminate. On Win32, in contrast, the mechanism for
    waiting for process termination is not arbitrarily constrained by
    parent-child relationships, and is more coherent and uniform than on
    Unix. To wait for one's parent process, or indeed for any other
    process that one didn't create onesself and so doesn't already have an
    open handle for, to terminate, one obtains the target process ID in
    some fashion (such as via the ToolHelp API) and then simply calls
    OpenProcess() specifying the SYNCHRONIZE access right (which will of
    course fail if the access control list on the process does not grant
    one that right) and the universal wait functions WaitForSingleObject()/
    WaitForMultipleObjects() (and then closes the handle, of course).

  17. Re: Support for process management in terms of child/parent processinteraction

    J de Boyne Pollard wrote:
    > WA> On Windows I use multiple processes which use socket IPC. The
    > WA> only downside is Windows has categorically inferior support for
    > WA> process management in terms of child/parent process interaction.
    >
    > Untrue.


    [counter-example snipped]

    Though your example makes sense, I don't think it's directly on point.
    In fact I think it goes some way to illustrate the previous poster's
    point. With specific regard to parent/child interaction, Windows is
    weaker precisely because it deliberately chooses to suppress the
    distinction.

    The way I like to explain it is this: Unix is *mammalian* in that
    parents gestate a child (by "forking"!) which looks much like them and
    the relationship lasts throughout their life. Granted, the metaphor
    breaks down somewhat when the parent waits for the child to die. By
    contrast, Windows is *reptilian* because processes spawn other processes
    but never look back and there is no special filial relationship once the
    new process is born.

    Your point that any process can wait for any other (or do anything with
    any other, permissions allowing, once it digs up the process handle) is
    illustrative of this. Yes, Windows has useful capabilities that Unix
    doesn't. But in the specific area of parent/child structure, Unix has it
    and Windows, by design, does not. None of which is to say that Unix
    couldn't stand to add a few features, possibly even (gasp) learned from
    Windows.

    All this can be summarized and symbolized by getppid(), which is the way
    to say "who's my daddy?" Unix has it, Windows doesn't[*]. Not because it
    couldn't but because the process model doesn't care.

    HT
    [*] And yes, there are ways of mocking up a getppid for Windows. I speak
    as one who has done so. But the lack is still a clue to the Windows
    architecture.

  18. Support for process management in terms of child/parent processinteraction

    WA> On Windows I use multiple processes which use socket IPC. The
    WA> only downside is Windows has categorically inferior support for
    WA> process management in terms of child/parent process interaction.

    JdeBP> Untrue.

    HT> [counter-example snipped]

    You should have read the part that you excised.

    HT> Though your example makes sense, I don't think it's
    HT> directly on point. In fact I think it goes some way to
    HT> illustrate the previous poster's point. With specific
    HT> regard to parent/child interaction, Windows is weaker
    HT> precisely because it deliberately chooses to suppress
    HT> the distinction. [...] in the specific area of parent/child
    HT> structure, Unix has it and Windows, by design, does
    HT> not.

    .... except that it does no such thing, so your whole argument, based
    upon that premise, falls apart. There's a parent process ID field in
    the EPROCESS structure of a Windows NT process (at offset 0x1c8) just
    as there are a parent process ID fields in the process structures of
    processes on Unices. It's not suppressed at all. Windows NT has
    exactly the structure that you are talking about. As I wrote in the
    text that you should have read, Windows NT has all of the requisite
    mechanism for providing the POSIX API functions for process handling.
    It has to, in order to support its POSIX subsystem. This includes
    maintaining parent process IDs for every process.

    This whole notion that "Windows NT doesn't have POSIX parent-child
    relationships for processes." is a folk myth; highly prevalent on
    Usenet and on WWW discussion fora, but quite untrue. Beware of basing
    arguments upon it.

    HT> None of which is to say that Unix couldn't stand to add
    HT> a few features, possibly even (gasp) learned from Windows.

    The ability to inform the kernel about the intended sequential/random
    access use of a file description (i.e. FILE_FLAG_RANDOM_ACCESS and
    FILE_FLAG_SEQUENTIAL_SCAN) comes immediately to mind.

  19. Re: Support for process management in terms of child/parent processinteraction

    J de Boyne Pollard wrote:
    > HT> [counter-example snipped]
    >
    > You should have read the part that you excised.


    I did.

    > ... except that it does no such thing, so your whole argument, based
    > upon that premise, falls apart. There's a parent process ID field in
    > the EPROCESS structure of a Windows NT process (at offset 0x1c8)


    Does it have a native API for getting the parent process id, aka
    getppid? If it's not part of the documented API it doesn't count.

    If I am a parent process, what are the differences in my relationships
    with two other processes, only one of which is my child? Am I informed
    when my child dies, for instance?

    HT

  20. Re: Support for process management in terms of child/parent process interaction

    Henry Townsend writes:
    >J de Boyne Pollard wrote:
    >> WA> On Windows I use multiple processes which use socket IPC. The
    >> WA> only downside is Windows has categorically inferior support for
    >> WA> process management in terms of child/parent process interaction.
    >>
    >> Untrue.

    >
    >[counter-example snipped]
    >
    > None of which is to say that Unix
    >couldn't stand to add a few features, possibly even (gasp) learned from
    >Windows.


    Well, considering that particular windows feature derives from VMS
    (and likely has precendents in Multics, MVS, MCP et. al.), it's unclear
    what new OS capabilities one can learn from windows.

    scott

+ Reply to Thread
Page 1 of 2 1 2 LastLast