some flawed benchmarks - modperl

This is a discussion on some flawed benchmarks - modperl ; A couple of months ago i was going through slides from gozers "From CGI to mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun and registry to each other. At which point i realized that i've never really ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: some flawed benchmarks

  1. some flawed benchmarks

    A couple of months ago i was going through slides from gozers "From
    CGI to mod_perl 2.0, Fast!" talk, which has some benchmarks comparing
    CGI, perlrun and registry to each other. At which point i realized
    that i've never really known how much faster using straight handlers
    is than using one of the CGI emulation layers. I also didn't have any
    idea how much faster SetHandler modperl was vs SetHandler perl-script.

    So i decided to see what i could figure out. I took gozers CGI from
    the slides (slightly modified) and ran it through the paces on my
    laptop, then converted the script to run as a straight handler.

    here's the CGI version:

    #!/usr/bin/perl


    print qq[Content-Type: text/html\r\n\r\n];

    print(qq[

    Hello Worlds



    GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
    MOD_PERL: $ENV{MOD_PERL}



    ]);


    Here's the Handler version

    package Kabob::HelloWorld;

    use strict;
    use warnings;

    use Apache2::RequestRec ();

    use Apache2::Const -compile =>qw(:common);

    sub handler {
    my $r = shift;

    $r->content_type('text/html');
    $r->print(qq[

    Hello Worlds



    GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
    MOD_PERL: $ENV{MOD_PERL}




    ]);

    return Apache2::Const::OK;
    }

    1;

    and here's the conf (these tests were all running through a light
    mod_proxy front end too)




    ProxyPass http://localhost:8080/cgi/
    ProxyPassReverse http://localhost:8080/cgi/



    ScriptAlias /cgi/ /www/p/



    Alias /perlrun/ /www/p/


    ProxyPass http://localhost:8080/perlrun/
    ProxyPassReverse http://localhost:8080/perlrun/


    SetHandler perl-script
    PerlHandler ModPerl::PerlRun
    Options +ExecCGI
    PerlSendHeader On



    Alias /registry/ /www/p/


    ProxyPass http://localhost:8080/registry/
    ProxyPassReverse http://localhost:8080/registry/


    SetHandler perl-script
    PerlHandler ModPerl::Registry
    Options +ExecCGI
    PerlSendHeader On







    ProxyPass http://localhost:8080/perlscript/
    ProxyPassReverse http://localhost:8080/perlscript/


    SetHandler perl-script
    PerlResponseHandler Kabob::HelloWorld





    ProxyPass http://localhost:8080/modperl/
    ProxyPassReverse http://localhost:8080/modperl/


    SetHandler modperl
    PerlResponseHandler Kabob::HelloWorld




    and here's the results (which are no doubt flawed for a number of reasons)

    running: ab -n 10000 [url]

    CGI
    Requests per second: 217.80 [#/sec] (mean)
    Time per request: 4.591 [ms] (mean)
    Transfer rate: 53.17 [Kbytes/sec] received

    PerlRun
    Requests per second: 482.49 [#/sec] (mean)
    Time per request: 2.073 [ms] (mean)
    Transfer rate: 114.49 [Kbytes/sec] received

    Registry
    Requests per second: 693.33 [#/sec] (mean)
    Time per request: 1.442 [ms] (mean)
    Transfer rate: 164.53 [Kbytes/sec] received

    SetHandler perl-script
    Requests per second: 772.12 [#/sec] (mean)
    Time per request: 1.295 [ms] (mean)
    Transfer rate: 189.94 [Kbytes/sec] received

    SetHandler modperl
    Requests per second: 1048.66 [#/sec] (mean)
    Time per request: 0.954 [ms] (mean)
    Transfer rate: 250.84 [Kbytes/sec] received

    I'm not sure how well you can really compare the CGI emulation numbers
    to the PerlHandler numbers, but personally i think the 30%ish
    improvement from perl-script to modperl is pretty amazing. I wouldn't
    have imagined it would have been that high.

    Adam


  2. Re: some flawed benchmarks

    I appreciate this, as I'd been wondering.

    But it also prompts me to.. I gotta ask...

    No offence but...
    Don't you know what a here_doc is?

    --
    Dodger

    2008/7/9 :
    > A couple of months ago i was going through slides from gozers "From CGI to
    > mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun
    > and registry to each other. At which point i realized that i've never
    > really known how much faster using straight handlers is than using one of
    > the CGI emulation layers. I also didn't have any idea how much faster
    > SetHandler modperl was vs SetHandler perl-script.
    >
    > So i decided to see what i could figure out. I took gozers CGI from the
    > slides (slightly modified) and ran it through the paces on my laptop, then
    > converted the script to run as a straight handler.
    >
    > here's the CGI version:
    >
    > #!/usr/bin/perl
    >
    >
    > print qq[Content-Type: text/html\r\n\r\n];
    >
    > print(qq[
    >
    >

    Hello Worlds


    >

    > GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
    > MOD_PERL: $ENV{MOD_PERL}
    >

    >
    >
    > ]);
    >
    >
    > Here's the Handler version
    >
    > package Kabob::HelloWorld;
    >
    > use strict;
    > use warnings;
    >
    > use Apache2::RequestRec ();
    >
    > use Apache2::Const -compile =>qw(:common);
    >
    > sub handler {
    > my $r = shift;
    >
    > $r->content_type('text/html');
    > $r->print(qq[
    >
    >

    Hello Worlds


    >

    > GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
    > MOD_PERL: $ENV{MOD_PERL}
    >

    >
    >
    >
    > ]);
    >
    > return Apache2::Const::OK;
    > }
    >
    > 1;
    >
    > and here's the conf (these tests were all running through a light mod_proxy
    > front end too)
    >
    >
    >
    >
    > ProxyPass http://localhost:8080/cgi/
    > ProxyPassReverse http://localhost:8080/cgi/
    >

    >

    >
    > ScriptAlias /cgi/ /www/p/
    >

    >
    >
    > Alias /perlrun/ /www/p/
    >
    >
    > ProxyPass http://localhost:8080/perlrun/
    > ProxyPassReverse http://localhost:8080/perlrun/
    >

    >
    > SetHandler perl-script
    > PerlHandler ModPerl::PerlRun
    > Options +ExecCGI
    > PerlSendHeader On
    >

    >

    >
    > Alias /registry/ /www/p/
    >
    >
    > ProxyPass http://localhost:8080/registry/
    > ProxyPassReverse http://localhost:8080/registry/
    >

    >
    > SetHandler perl-script
    > PerlHandler ModPerl::Registry
    > Options +ExecCGI
    > PerlSendHeader On
    >

    >

    >
    >
    >
    >
    >
    > ProxyPass http://localhost:8080/perlscript/
    > ProxyPassReverse http://localhost:8080/perlscript/
    >

    >
    > SetHandler perl-script
    > PerlResponseHandler Kabob::HelloWorld
    >

    >

    >
    >
    >
    > ProxyPass http://localhost:8080/modperl/
    > ProxyPassReverse http://localhost:8080/modperl/
    >

    >
    > SetHandler modperl
    > PerlResponseHandler Kabob::HelloWorld
    >

    >

    >
    >
    > and here's the results (which are no doubt flawed for a number of reasons)
    >
    > running: ab -n 10000 [url]
    >
    > CGI
    > Requests per second: 217.80 [#/sec] (mean)
    > Time per request: 4.591 [ms] (mean)
    > Transfer rate: 53.17 [Kbytes/sec] received
    >
    > PerlRun
    > Requests per second: 482.49 [#/sec] (mean)
    > Time per request: 2.073 [ms] (mean)
    > Transfer rate: 114.49 [Kbytes/sec] received
    >
    > Registry
    > Requests per second: 693.33 [#/sec] (mean)
    > Time per request: 1.442 [ms] (mean)
    > Transfer rate: 164.53 [Kbytes/sec] received
    >
    > SetHandler perl-script
    > Requests per second: 772.12 [#/sec] (mean)
    > Time per request: 1.295 [ms] (mean)
    > Transfer rate: 189.94 [Kbytes/sec] received
    >
    > SetHandler modperl
    > Requests per second: 1048.66 [#/sec] (mean)
    > Time per request: 0.954 [ms] (mean)
    > Transfer rate: 250.84 [Kbytes/sec] received
    >
    > I'm not sure how well you can really compare the CGI emulation numbers to
    > the PerlHandler numbers, but personally i think the 30%ish improvement from
    > perl-script to modperl is pretty amazing. I wouldn't have imagined it would
    > have been that high.
    >
    > Adam
    >
    >
    >
    >
    >
    >
    >
    >
    >




    --
    Dodger


  3. Re: some flawed benchmarks

    Oh. I would also recommend three variants, based on what people often
    do, what people sometimes do, and what people probably should do when
    using CGI.pm, which can make a difference (just for thoroughness):

    Usually done:
    #!/usr/bin/perl
    use CGI;
    print header;

    print <<"EOF";


    Environment dump:



    @{[map "
    $_
    \n
    $ENV{$_}
    \n", sort keys %ENV]}



    EOF

    Sometimes do:
    #!/usr/bin/perl
    use strict;
    use CGI;
    my $cgi = new CGI;
    print $cgi->header;

    print <<"EOF";


    Environment dump:



    @{[map "
    $_
    \n
    $ENV{$_}
    \n", sort keys %ENV]}



    EOF

    Might do occassionally, and probably should do all the time if using CGI:

    #!/usr/bin/perl
    use strict;
    use CGI(); # note the difference -- using CGI in OO mode, don't import
    *anything*
    my $cgi = new CGI;
    print $cgi->header;

    print <<"EOF";


    Environment dump:



    @{[map "
    $_
    \n
    $ENV{$_}
    \n", sort keys %ENV]}



    EOF

    2008/7/9 :
    > A couple of months ago i was going through slides from gozers "From CGI to
    > mod_perl 2.0, Fast!" talk, which has some benchmarks comparing CGI, perlrun
    > and registry to each other. At which point i realized that i've never
    > really known how much faster using straight handlers is than using one of
    > the CGI emulation layers. I also didn't have any idea how much faster
    > SetHandler modperl was vs SetHandler perl-script.
    >
    > So i decided to see what i could figure out. I took gozers CGI from the
    > slides (slightly modified) and ran it through the paces on my laptop, then
    > converted the script to run as a straight handler.
    >
    > here's the CGI version:
    >
    > #!/usr/bin/perl
    >
    >
    > print qq[Content-Type: text/html\r\n\r\n];
    >
    > print(qq[
    >
    >

    Hello Worlds


    >

    > GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
    > MOD_PERL: $ENV{MOD_PERL}
    >

    >
    >
    > ]);
    >
    >
    > Here's the Handler version
    >
    > package Kabob::HelloWorld;
    >
    > use strict;
    > use warnings;
    >
    > use Apache2::RequestRec ();
    >
    > use Apache2::Const -compile =>qw(:common);
    >
    > sub handler {
    > my $r = shift;
    >
    > $r->content_type('text/html');
    > $r->print(qq[
    >
    >

    Hello Worlds


    >

    > GATEWAY_INTERFACE: $ENV{GATEWAY_INTERFACE}
    > MOD_PERL: $ENV{MOD_PERL}
    >

    >
    >
    >
    > ]);
    >
    > return Apache2::Const::OK;
    > }
    >
    > 1;
    >
    > and here's the conf (these tests were all running through a light mod_proxy
    > front end too)
    >
    >
    >
    >
    > ProxyPass http://localhost:8080/cgi/
    > ProxyPassReverse http://localhost:8080/cgi/
    >

    >

    >
    > ScriptAlias /cgi/ /www/p/
    >

    >
    >
    > Alias /perlrun/ /www/p/
    >
    >
    > ProxyPass http://localhost:8080/perlrun/
    > ProxyPassReverse http://localhost:8080/perlrun/
    >

    >
    > SetHandler perl-script
    > PerlHandler ModPerl::PerlRun
    > Options +ExecCGI
    > PerlSendHeader On
    >

    >

    >
    > Alias /registry/ /www/p/
    >
    >
    > ProxyPass http://localhost:8080/registry/
    > ProxyPassReverse http://localhost:8080/registry/
    >

    >
    > SetHandler perl-script
    > PerlHandler ModPerl::Registry
    > Options +ExecCGI
    > PerlSendHeader On
    >

    >

    >
    >
    >
    >
    >
    > ProxyPass http://localhost:8080/perlscript/
    > ProxyPassReverse http://localhost:8080/perlscript/
    >

    >
    > SetHandler perl-script
    > PerlResponseHandler Kabob::HelloWorld
    >

    >

    >
    >
    >
    > ProxyPass http://localhost:8080/modperl/
    > ProxyPassReverse http://localhost:8080/modperl/
    >

    >
    > SetHandler modperl
    > PerlResponseHandler Kabob::HelloWorld
    >

    >

    >
    >
    > and here's the results (which are no doubt flawed for a number of reasons)
    >
    > running: ab -n 10000 [url]
    >
    > CGI
    > Requests per second: 217.80 [#/sec] (mean)
    > Time per request: 4.591 [ms] (mean)
    > Transfer rate: 53.17 [Kbytes/sec] received
    >
    > PerlRun
    > Requests per second: 482.49 [#/sec] (mean)
    > Time per request: 2.073 [ms] (mean)
    > Transfer rate: 114.49 [Kbytes/sec] received
    >
    > Registry
    > Requests per second: 693.33 [#/sec] (mean)
    > Time per request: 1.442 [ms] (mean)
    > Transfer rate: 164.53 [Kbytes/sec] received
    >
    > SetHandler perl-script
    > Requests per second: 772.12 [#/sec] (mean)
    > Time per request: 1.295 [ms] (mean)
    > Transfer rate: 189.94 [Kbytes/sec] received
    >
    > SetHandler modperl
    > Requests per second: 1048.66 [#/sec] (mean)
    > Time per request: 0.954 [ms] (mean)
    > Transfer rate: 250.84 [Kbytes/sec] received
    >
    > I'm not sure how well you can really compare the CGI emulation numbers to
    > the PerlHandler numbers, but personally i think the 30%ish improvement from
    > perl-script to modperl is pretty amazing. I wouldn't have imagined it would
    > have been that high.
    >
    > Adam
    >
    >
    >
    >
    >
    >
    >
    >
    >




    --
    Dodger


  4. Re: some flawed benchmarks

    On 10 Jul 2008, at 06:28, adam.prime@utoronto.ca wrote:
    > I'm not sure how well you can really compare the CGI emulation
    > numbers to the PerlHandler numbers, but personally i think the
    > 30%ish improvement from perl-script to modperl is pretty amazing. I
    > wouldn't have imagined it would have been that high.



    It would be interesting to see how FCGI compares to those numbers.

    --
    Andy Armstrong, Hexten


  5. Re: some flawed benchmarks

    I deliberately removed CGI from the script because i personally would
    never use CGI in something written to be run as straight handlers, and
    it obviously wouldn't make any sense to use CGI in the CGI emulations,
    and then not use it in the Handler version.

    Not using heredoc's shouldn't really have any effect on the (already
    flawed) results, because they weren't used in any of the examples. But
    yeah, it was late and i should have been sleeping but i was dorking
    around with this instead.

    I'm personally not really interested in how using CGI affects the
    numbers, but if other people would like to see them i can do this later
    tonight.

    Adam


    Dodger wrote:
    > Oh. I would also recommend three variants, based on what people often
    > do, what people sometimes do, and what people probably should do when
    > using CGI.pm, which can make a difference (just for thoroughness):
    >
    > Usually done:
    > #!/usr/bin/perl
    > use CGI;
    > print header;
    >
    > print <<"EOF";
    >
    >
    >

    Environment dump:


    >

    > @{[map "
    $_
    \n
    $ENV{$_}
    \n", sort keys %ENV]}
    >

    >
    >
    > EOF
    >
    > Sometimes do:
    > #!/usr/bin/perl
    > use strict;
    > use CGI;
    > my $cgi = new CGI;
    > print $cgi->header;
    >
    > print <<"EOF";
    >
    >
    >

    Environment dump:


    >

    > @{[map "
    $_
    \n
    $ENV{$_}
    \n", sort keys %ENV]}
    >

    >
    >
    > EOF
    >
    > Might do occassionally, and probably should do all the time if using CGI:
    >
    > #!/usr/bin/perl
    > use strict;
    > use CGI(); # note the difference -- using CGI in OO mode, don't import
    > *anything*
    > my $cgi = new CGI;
    > print $cgi->header;
    >
    > print <<"EOF";
    >
    >
    >

    Environment dump:


    >

    > @{[map "
    $_
    \n
    $ENV{$_}
    \n", sort keys %ENV]}
    >

    >
    >
    > EOF
    >



  6. Re: some flawed benchmarks

    Andy Armstrong wrote:
    > It would be interesting to see how FCGI compares to those numbers.


    I don't know anything about fastcgi, but i suppose i could look at that
    also this evening.

    Adam


  7. Re: some flawed benchmarks

    On Thu, Jul 10, 2008 at 1:28 AM, wrote:
    > and here's the conf (these tests were all running through a light mod_proxy
    > front end too)


    Note that CGI and FastCGI don't need the proxy frontend.

    > I'm not sure how well you can really compare the CGI emulation numbers to
    > the PerlHandler numbers, but personally i think the 30%ish improvement from
    > perl-script to modperl is pretty amazing. I wouldn't have imagined it would
    > have been that high.


    It is pretty cool that it works so well. I feel like I should point
    out though, for the benefit of those using Registry, that the
    difference is only this big because the code isn't doing anything. In
    a real-world scenario you'd see an improvement, but nothing close to
    30%.

    - Perrin


  8. Re: some flawed benchmarks

    Quoting Perrin Harkins :

    > Note that CGI and FastCGI don't need the proxy frontend.


    The only reason I did it that way was because that's how apache was
    already set up on my laptop, and i didn't feel like dorking around
    with it too much. I could certainly change it around so that CGI and
    FGCGI (if added) were handled by the light apache. I wasn't sure how
    apples to oranges that would be, but i guess in reality, people that
    are running plain CGI or fast CGI probably aren't doing it on a
    mod_perl enabled server.

    > It is pretty cool that it works so well. I feel like I should point
    > out though, for the benefit of those using Registry, that the
    > difference is only this big because the code isn't doing anything. In
    > a real-world scenario you'd see an improvement, but nothing close to
    > 30%.


    It's probably also worth noting that under sethandler modperl the
    GATEWAY_INTERFACE env variable has no value, which results in the page
    being 3 bytes shorter (iirc), which is actually a pretty big deal
    since the whole page is pretty short. The examples should likely be
    modified to actually return pages that are the same length.

    anything else stupid i'm missing?

    adam


+ Reply to Thread