Apache2/MP2/Parllel::ForkManager and Segmentation Faults - modperl

This is a discussion on Apache2/MP2/Parllel::ForkManager and Segmentation Faults - modperl ; ...lots of them. First off, let me thank Torsten for the help understanding some of the environment issues with forking. I have created a class for running external programs that will be a little bit smart and a little bit ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

  1. Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    ...lots of them.

    First off, let me thank Torsten for the help understanding some of the
    environment issues with forking.

    I have created a class for running external programs that will be a
    little bit smart and a little bit configurable to allow processes to be
    run by system() calls directly, IPC::Run3, which uses system, or by
    $r->spawn_proc_prog() under mod_perl and also it provides for setting
    environment variables via Env::C or by using $r->subprocess_env->set(
    $var => $vars->{$var} ) if running under Apache/mod_perl.

    Ok, now I've got my programs forking with the proper environment, but
    I'm experiencing numerous segfaults. Same request issued multiple times
    and a good number of them result in segmentation faults. That would be
    on the order of once every other request, where each request could fork
    about 7 or 8 times.

    We're using DBI here as well, and I understand that there were related
    issues with pre 1.3 versions of DBI.

    I instantiate the ForkManager, and get into the loop, print out a
    message right before I do my '$pm->start and next;', and then I get
    this:

    [Tue Aug 26 13:56:37 2008] [notice] child pid 19991 exit signal
    Segmentation fault (11)

    From what I've been able to determine so far, this happens when forking
    the first process after creating the ForkManager object.

    I'm running:

    RHE Linux
    Apache/2.2.9 (Unix) DAV/2 mod_apreq2-20051231/2.6.0
    mod_perl/2.0.4
    Perl/v5.10.0
    DBI 1.605

    I've tried changing the numbers the prefork mpm to allow for more
    Clients, more MaxSpareServers. My config isn't tuned at all as you can
    see:

    StartServers 5
    MinSpareServers 5
    MaxSpareServers 15
    MaxClients 150
    MaxRequestsPerChild 128

    Any thoughts or questions?

    Eric D. Berg
    Lehman Brothers
    > Fixed Income Research
    > 745 7th Avenue, 15th floor
    > New York, NY 10019
    > Phone +1 212 526 8118
    >
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  2. Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    On Tue, Aug 26, 2008 at 2:04 PM, Berg, Eric wrote:
    > We're using DBI here as well


    That's going to make forking a lot more complicated. You'll need to
    close your DBI handles before you fork, or set InactiveDestroy on
    them. You need to open new ones in the forked process to do any DBI
    work.

    If you strip out the DBI stuff from the forked processes, does it
    still segfault?

    - Perrin


  3. RE: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    I'm using Apache:BI.

    I stripped out calls to code that use DBI and it still segfaults.

    I've gone into the code and made sure that any method that does a DBI
    call also uses $dbh->disconnect;

    Eric

    > -----Original Message-----
    > From: pharkins@gmail.com [mailtoharkins@gmail.com] On
    > Behalf Of Perrin Harkins
    > Sent: Tuesday, August 26, 2008 2:12 PM
    > To: Berg, Eric
    > Cc: modperl@perl.apache.org; dlux@kapu.hu
    > Subject: Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
    >
    > On Tue, Aug 26, 2008 at 2:04 PM, Berg, Eric
    > wrote:
    > > We're using DBI here as well

    >
    > That's going to make forking a lot more complicated. You'll need to
    > close your DBI handles before you fork, or set InactiveDestroy on
    > them. You need to open new ones in the forked process to do any DBI
    > work.
    >
    > If you strip out the DBI stuff from the forked processes, does it
    > still segfault?
    >
    > - Perrin
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  4. Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    On Tue, Aug 26, 2008 at 2:28 PM, Berg, Eric wrote:
    > I'm using Apache:BI.
    >
    > I stripped out calls to code that use DBI and it still segfaults.
    >
    > I've gone into the code and made sure that any method that does a DBI
    > call also uses $dbh->disconnect;


    With Apache:BI, you actually need to call $dbh->SUPER::disconnect.
    However, you would only need to do this right before forking, not on
    every call.

    - Perrin


  5. RE: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    Hey, Perrin.

    My DBI calls are within the code that is executed in the loop. Not
    before. Even so, I tried using $dbh->super::disconnect(), and that had
    no apparent affect. Still segfaulting.

    I'm feeling like it's not the DBI stuff. Where else can I look?

    Eric

    > -----Original Message-----
    > From: pharkins@gmail.com [mailtoharkins@gmail.com] On
    > Behalf Of Perrin Harkins
    > Sent: Tuesday, August 26, 2008 2:35 PM
    > To: Berg, Eric
    > Cc: modperl@perl.apache.org; dlux@kapu.hu
    > Subject: Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
    >
    > On Tue, Aug 26, 2008 at 2:28 PM, Berg, Eric
    > wrote:
    > > I'm using Apache:BI.
    > >
    > > I stripped out calls to code that use DBI and it still segfaults.
    > >
    > > I've gone into the code and made sure that any method that

    > does a DBI
    > > call also uses $dbh->disconnect;

    >
    > With Apache:BI, you actually need to call $dbh->SUPER::disconnect.
    > However, you would only need to do this right before forking, not on
    > every call.
    >
    > - Perrin
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  6. Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    On Tue, Aug 26, 2008 at 3:25 PM, Berg, Eric wrote:
    > I'm feeling like it's not the DBI stuff. Where else can I look?


    Your next step is probably to get a backtrace of the segfault and
    figure out where it crashed.

    - Perrin


  7. RE: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    Ok. I appreciate your help here, Perrin.

    mod_backtrace gave me this:

    [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace backtrace for sig 11
    (thread "pid" 24456)
    [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace main() is at 8062fb8
    /home/pointsrv/apache/config/dev/modules/mod_backtrace.so[0xa3fa43]
    /home/erberg/public_html/work/point-www/aut(ap_run_fatal_exception+0x2e)
    [0x8079ce2]
    /home/erberg/public_html/work/point-www/aut[0x807b0a0]
    /lib/tls/libpthread.so.0[0x5b9a98]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_PerlIO_flush+0
    x68)[0x116e648]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_pp_fork+0x45)[
    0x1149b55]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_runops_standar
    d+0x18)[0x1102fe8]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_call_sv+0x49a)
    [0x110106a]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback+0x
    180)[0x10764f0]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_ru
    n_handlers+0x35a)[0x1076b3a]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_pe
    r_dir+0x53)[0x1076e83]
    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_response_ha
    ndler_cgi+0x11b)[0x107354b]
    /home/erberg/public_html/work/point-www/aut(ap_run_handler+0x32)[0x80735
    6e]
    /home/erberg/public_html/work/point-www/aut(ap_invoke_handler+0xad)[0x80
    73939]
    /home/erberg/public_html/work/point-www/aut(ap_process_request+0x18d)[0x
    807e669]
    /home/erberg/public_html/work/point-www/aut[0x807c145]
    /home/erberg/public_html/work/point-www/aut(ap_run_process_connection+0x
    32)[0x8078e12]
    /home/erberg/public_html/work/point-www/aut[0x80822c9]
    /home/erberg/public_html/work/point-www/aut[0x808249e]
    /home/erberg/public_html/work/point-www/aut[0x808252c]
    [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace end of backtrace
    [Wed Aug 27 18:29:30 2008] [notice] child pid 24456 exit signal
    Segmentation fault (11), possible coredump in /tmp

    BTW, no core file was created in /tmp.

    What do you think?

    Eric

    > -----Original Message-----
    > From: pharkins@gmail.com [mailtoharkins@gmail.com] On
    > Behalf Of Perrin Harkins
    > Sent: Wednesday, August 27, 2008 11:18 AM
    > To: Berg, Eric
    > Cc: modperl@perl.apache.org; dlux@kapu.hu
    > Subject: Re: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
    >
    > On Tue, Aug 26, 2008 at 3:25 PM, Berg, Eric
    > wrote:
    > > I'm feeling like it's not the DBI stuff. Where else can I look?

    >
    > Your next step is probably to get a backtrace of the segfault and
    > figure out where it crashed.
    >
    > - Perrin
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  8. RE: Apache2/MP2/Parllel::ForkManager and Segmentation Faults

    Does anyone have time to take a look at this? I sure would appreciate
    being able to move this forward and find a solution.

    Thanks much in advance.

    Eric

    > -----Original Message-----
    > From: Berg, Eric
    > Sent: Wednesday, August 27, 2008 6:33 PM
    > To: 'Perrin Harkins'
    > Cc: modperl@perl.apache.org
    > Subject: RE: Apache2/MP2/Parllel::ForkManager and Segmentation Faults
    >
    > Ok. I appreciate your help here, Perrin.
    >
    > mod_backtrace gave me this:
    >
    > [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace backtrace
    > for sig 11 (thread "pid" 24456)
    > [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace main() is
    > at 8062fb8
    > /home/pointsrv/apache/config/dev/modules/mod_backtrace.so[0xa3fa43]
    >

    /home/erberg/public_html/work/point-www/aut(ap_run_fatal_exception+0x2e)
    [0x8079ce2]
    > /home/erberg/public_html/work/point-www/aut[0x807b0a0]
    > /lib/tls/libpthread.so.0[0x5b9a98]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_PerlIO_flush+0
    x68)[0x116e648]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_pp_fork+0x45)[
    0x1149b55]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_runops_standar
    d+0x18)[0x1102fe8]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(Perl_call_sv+0x49a)
    [0x110106a]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback+0x
    180)[0x10764f0]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_ru
    n_handlers+0x35a)[0x1076b3a]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_callback_pe
    r_dir+0x53)[0x1076e83]
    >

    /home/pointsrv/apache/config/dev/modules/mod_perl.so(modperl_response_ha
    ndler_cgi+0x11b)[0x107354b]
    >

    /home/erberg/public_html/work/point-www/aut(ap_run_handler+0x32)[0x80735
    6e]
    >

    /home/erberg/public_html/work/point-www/aut(ap_invoke_handler+0xad)[0x80
    73939]
    >

    /home/erberg/public_html/work/point-www/aut(ap_process_request+0x18d)[0x
    807e669]
    > /home/erberg/public_html/work/point-www/aut[0x807c145]
    >

    /home/erberg/public_html/work/point-www/aut(ap_run_process_connection+0x
    32)[0x8078e12]
    > /home/erberg/public_html/work/point-www/aut[0x80822c9]
    > /home/erberg/public_html/work/point-www/aut[0x808249e]
    > /home/erberg/public_html/work/point-www/aut[0x808252c]
    > [Wed Aug 27 18:29:29 2008] pid 24456 mod_backtrace end of backtrace
    > [Wed Aug 27 18:29:30 2008] [notice] child pid 24456 exit
    > signal Segmentation fault (11), possible coredump in /tmp
    >
    > BTW, no core file was created in /tmp.
    >
    > What do you think?
    >
    > Eric
    >
    > > -----Original Message-----
    > > From: pharkins@gmail.com [mailtoharkins@gmail.com] On
    > > Behalf Of Perrin Harkins
    > > Sent: Wednesday, August 27, 2008 11:18 AM
    > > To: Berg, Eric
    > > Cc: modperl@perl.apache.org; dlux@kapu.hu
    > > Subject: Re: Apache2/MP2/Parllel::ForkManager and

    > Segmentation Faults
    > >
    > > On Tue, Aug 26, 2008 at 3:25 PM, Berg, Eric
    > > wrote:
    > > > I'm feeling like it's not the DBI stuff. Where else can I look?

    > >
    > > Your next step is probably to get a backtrace of the segfault and
    > > figure out where it crashed.
    > >
    > > - Perrin
    > >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


+ Reply to Thread