How to debug bizarre memory corruption in mod_perl - modperl

This is a discussion on How to debug bizarre memory corruption in mod_perl - modperl ; I will preface this message with this: this is not a mod_perl bug or problem, but it involves a mod_perl application, so this is probably a good place to get advice from experienced mod_perl users/developers. I have an application that ...

+ Reply to Thread
Results 1 to 3 of 3

Thread: How to debug bizarre memory corruption in mod_perl

  1. How to debug bizarre memory corruption in mod_perl

    I will preface this message with this: this is not a mod_perl bug or
    problem, but it involves a mod_perl application, so this is probably a good
    place to get advice from experienced mod_perl users/developers.

    I have an application that ran on mod_perl 1.3 and perl 5.8 for a long time,
    that I recently ported over to a Fedora 9 box, thus moving it to mod_perl
    2.0 and perl 5.10.

    Under this new environment, I am getting some of the most bizarre and
    insidious perl core errors I've ever seen in my 15 years of using perl.
    Here's a sampling:

    Attempt to free unreferenced scalar: SV 0xbd266be4, Perl interpreter:
    0xba01c410 at /usr/lib/perl5/vendor_perl/5.10.0/HTML/Mason/Component.pm line
    147.
    panic: attempt to copy value 20 to a freed scalar bd49dffc
    Use of freed value in iteration
    panic: free from wrong pool

    Not to mention countless application errors involving dereferencing, where
    things that are supposed to be arrays and hashrefs have magically turned
    into strings and key values, and vice versa.

    Now I realize this is not an issue with mod_perl itself. Obviously
    something is doing a massive fandango on core and corrupting the Perl
    stack/arena over time. Note that running Apache with MaxRequestsPerChild =
    1 seems to mitigate the issue, so it's a problem that occurs at runtime.

    My problem is that I don't even know where or how to start tracing or
    debugging such an issue to determine precisely what is scribbling on perl's
    memory. If this was a standalone app, it would be simple enough to run it
    through gdb and trace everything out, but in a mod_perl environment....

    Any advice on how to proceed is greatly appreciated.

    --
    Stephen Clouse


  2. Re: How to debug bizarre memory corruption in mod_perl

    On Tue, Jul 8, 2008 at 1:23 PM, Stephen Clouse wrote:
    > I have an application that ran on mod_perl 1.3 and perl 5.8 for a long time,
    > that I recently ported over to a Fedora 9 box, thus moving it to mod_perl
    > 2.0 and perl 5.10.


    Are you running prefork, or threaded? Did you change anything else?
    The kind of problems you're seeing usually come from bad XS code. You
    might try reverting to Perl 5.8 or mod_perl 1.3 to determine what is
    causing the problem.

    - Perrin


  3. Re: How to debug bizarre memory corruption in mod_perl


    "Stephen Clouse" wrote:
    > Under [mod_perl 2 and perl 5.10 on Fedora 9] I am getting some of the
    > most bizarre and insidious perl core errors I've ever seen in my 15
    > years of using perl.
    >
    > Attempt to free unreferenced scalar: SV 0xbd266be4, Perl interpreter:
    > 0xba01c410 at /usr/lib/perl5/vendor_perl/5.10.0/HTML/Mason/Component.pm
    > line
    > 147.
    > panic: attempt to copy value 20 to a freed scalar bd49dffc
    > Use of freed value in iteration
    > panic: free from wrong pool
    >
    > Not to mention countless application errors involving dereferencing.


    Dereferencing demons, eh? Your error rang a bell with me. But it wasn't
    until I saw List/Util/Util.so in your follow-up post that I remembered,
    List::Util did that same scary crashing thing to me a while back while I
    was debugging krang (which is mod_perl 1.3x-based).

    It boiled down to the (IMO) senseless and unnecessary use of weak
    referrences somewhere deep in the dark bowels of the Scalar-List-Utils
    package. I'd read somewhere that weak references were not all that stable
    in certain recent versions of perl, so when I saw all the weakening being
    done in there (and all the XS bugs the module has had to fix, in its change
    log:

    http://search.cpan.org/src/GBARR/Sca...s-1.19/Changes

    At the time, upgrading the module to the latest release didn't stop my big
    scary memory reference errors. So I read through the code that used the
    module and was astonished at why the module was being used: the *one* line
    of code in the entire module that I was troubleshooting that *depended* on
    List::Util was a single, simple call to its exported method: first()

    What does first() do? It invokes kilobytes of fast, compiled XS code to
    ....huh? Return to me the first element in a list I pass to it that returns
    true for a code block I pass to it? Yup. Like this:

    # from the POD doco at
    #
    http://search.cpan.org/~gbarr/Scalar...pm#DESCRIPTION

    $foo = first { defined($_) } @list; # first defined value in @list


    Who needs to install a CPAN module to do that? I personally would have
    written it as:

    $foo = (grep defined, @list)[0]; # first defined value in @list


    But thats kind of non-obvious to the programmer who finds it later (if they
    dont read the comment, that is). You could also simply replace "first"
    with "grep" (and a pair of parens) and get the same effect:

    ($foo) = grep { defined($_) } @list; # first defined value in @list

    (at the cost of one extra byte and some wasted cpu cycles if the compiler
    doesn't notice that the lvalue is a list of length one, and lets grep
    process the whole list anyway, even after it finds and assigns a defined
    element in there).

    IMO, even this would be better than loading List::Utils and calling
    first():

    my $foo;
    for (@list) {
    next unless defined($_);
    $foo = $_;
    last;
    }

    So I fixed my bug by writing a line or two of perl to in order to remove
    one call to the List::Util module's first() function.


    > My problem is that I don't even know where or how to start tracing or
    > debugging such an issue to determine precisely what is scribbling on
    > perl's
    > memory.


    Starting from your error message (line 147 of HTML::Mason::Component) it
    appears HTML::Mason is calling a "dynamic method" method (one named in a
    variable). No help there. Okay, what else is HTML::Mason::Component
    using? What libraries has it loaded? Is List::Utils being used?

    This looked suspicious to me:

    use HTML::Mason::Tools qw(absolute_comp_path can_weaken);

    Just because you "can_weaken" a reference, does that mean you should? Sure
    enough HTML::Mason::Tools loads Scalar::Util apparently to do some
    reference weakening.

    I'd check my Mason component for use of the functions that the
    Scalar-List-Utils modules provide:

    List::Util

    first max maxstr min minstr reduce shuffle sum

    Scalar::Util

    blessed dualvar isweak readonly refaddr reftype
    tainted weaken isvstring looks_like_number set_prototype

    And see if rewriting these in pure perl helps. Also, if it does, but would
    require changing code that you'd rather not monkey with, it looks like
    there is a pure-perl version of the module that can be installed by passing
    a -pm option to Makefile.PL:

    perl Makefile.PL -pm
    make
    make test
    make install

    Anyway, I hope somehow, some of this helps!

    -dave


+ Reply to Thread