Skip or minimise Apache log phase - modperl

This is a discussion on Skip or minimise Apache log phase - modperl ; Hi. Following a message posted on the Apache users list, I am just curious if via mod_perl there could be a solution to the following issue : A busy Apache server (with several VirtualHosts, why not ?) is being accessed ...

+ Reply to Thread
Results 1 to 8 of 8

Thread: Skip or minimise Apache log phase

  1. Skip or minimise Apache log phase

    Hi.

    Following a message posted on the Apache users list, I am just curious
    if via mod_perl there could be a solution to the following issue :

    A busy Apache server (with several VirtualHosts, why not ?) is being
    accessed from internal network clients (IP address 192.168.*) as well as
    by external clients (any other IPs).
    Among the internal clients are some GoogleBots, which generate thousands
    of accesses, all logged in the access logs of the hosts. These accesses
    come to generate more than 90% of the total, which really bothers the
    sysadmins when they have to scan any logfile for something else.

    Would there be any way, using mod_perl, to detect such accesses early,
    and to either cancel the log phase for them, or else redirect the
    logging to some sink file, or else at least set some parameter so that
    the verbosity of the log for these accesses would be drastically reduced ?
    (Of course the requests themselves should just go through and still be
    handled properly)

    Looking at the description of the PerlLogHandler at
    http://perl.apache.org/docs/2.0/user...PerlLogHandler
    I find thhe following paragraph, which tends to indicate that Apache log
    handlers will run anyway, but maybe there is still a devious solution ?

    quote
    First the handler tries to figure out what username the request is
    issued for, if it fails to match the URI, it simply returns
    Apache2::Const:ECLINED, letting other log handlers to do the logging.
    Though it could return Apache2::Const::OK since all other log handlers
    will be run anyway.
    unquote

    Thanks


  2. Re: Skip or minimise Apache log phase

    On Wed, 08 Oct 2008 14:32:26 +0200 André Warnier wrote:

    > Hi.
    >
    > Following a message posted on the Apache users list, I am just curious
    > if via mod_perl there could be a solution to the following issue :
    >
    > A busy Apache server (with several VirtualHosts, why not ?) is being
    > accessed from internal network clients (IP address 192.168.*) as well
    > as by external clients (any other IPs).
    > Among the internal clients are some GoogleBots, which generate
    > thousands of accesses, all logged in the access logs of the hosts.
    > These accesses come to generate more than 90% of the total, which
    > really bothers the sysadmins when they have to scan any logfile for
    > something else.
    >
    > Would there be any way, using mod_perl, to detect such accesses early,
    > and to either cancel the log phase for them, or else redirect the
    > logging to some sink file, or else at least set some parameter so that
    > the verbosity of the log for these accesses would be drastically
    > reduced ?


    You can do this without involving mod_perl. Simply use one of the
    directives SetEnvIf or BrowserMatch to detect these bots and set a
    variable like 'is_googlebot', and then use a conditional log-statement:

    CustomLog .... env=!is_googlebot

    See also:
    http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html
    http://httpd.apache.org/docs/2.2/mod...html#customlog

    - Vegard V -


  3. Re: Skip or minimise Apache log phase

    Vegard Vesterheim wrote:
    >
    > CustomLog .... env=!is_googlebot
    >
    > See also:
    > http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html
    > http://httpd.apache.org/docs/2.2/mod...html#customlog
    >
    > - Vegard V -
    >


    That looks just too easy to be true..
    But I believe you of course, and Thanks.

    It's just that I have seen this same kind of question pop up regularly
    on forums, and I don't remember ever seeing the simple suggestion above.
    ....
    As a matter of fact, the original poster on the other forum just got the
    same answer from someone else.

    I do remain interested in a possible mod_perl solution though, because I
    already have some add-on Apache/perl handler modules where such a
    functionality would come in handy sometimes.

    I have a suspicion that I might have to look at the ServerRec or
    something like that, and make the change in the configuration on-the-fly...


  4. Re: Skip or minimise Apache log phase

    On Wednesday 08 October 2008, André Warnier wrote:
    > Vegard Vesterheim wrote:
    > > CustomLog .... env=!is_googlebot
    > >
    > > See also:
    > > http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html
    > > http://httpd.apache.org/docs/2.2/mod...html#customlog
    > >
    > > - Vegard V -

    >
    > That looks just too easy to be true..
    > But I believe you of course, and Thanks.


    Is that easy... I've used that type of configuration many times, setting up
    ENV vars for various things and then logging them to separate files.

    Most common thing I've used it for is to ignore logging of requests for
    images.

    --
    Graham TerMarsch
    Howling Frog Internet Development, Inc.


  5. Re: Skip or minimise Apache log phase

    André Warnier wrote:
    > Vegard Vesterheim wrote:
    >>
    >> CustomLog .... env=!is_googlebot
    >>
    >> See also: http://httpd.apache.org/docs/2.2/mod/mod_setenvif.html
    >> http://httpd.apache.org/docs/2.2/mod...html#customlog
    >>
    >> - Vegard V -
    >>

    >
    > That looks just too easy to be true..
    > But I believe you of course, and Thanks.
    >
    > It's just that I have seen this same kind of question pop up regularly
    > on forums, and I don't remember ever seeing the simple suggestion above.
    > ...
    > As a matter of fact, the original poster on the other forum just got the
    > same answer from someone else.
    >
    > I do remain interested in a possible mod_perl solution though, because I
    > already have some add-on Apache/perl handler modules where such a
    > functionality would come in handy sometimes.


    You could use a transhandler to set your own loghandler based on the
    request url.

    $r->set_handlers( PerlResponseHandler => [ 'My::PerlLogHandler' ] )
    if $should_skip_apache_loghandler;

    And then write your own PerlLogHandler which returns Apache2::Const::OK
    (which should tell Apache to skip the log handler phase).

    package My::PerlLogHandler;

    use strict;
    use warnings;

    use Apache2::Const -compile => qw( OK );

    sub handler;
    my $r = shift;
    return Apache2::Const::OK;
    }

    >
    > I have a suspicion that I might have to look at the ServerRec or
    > something like that, and make the change in the configuration on-the-fly...



  6. Re: Skip or minimise Apache log phase

    André Warnier wrote:
    > Fred Moyer wrote:
    >> André Warnier wrote:

    >
    > Thanks for the answer and the above code.
    > But what about the following ?
    >
    > http://perl.apache.org/docs/2.0/user...PerlLogHandler
    > quote
    > [...]
    > it simply returns Apache2::Const:ECLINED, letting other log handlers
    > to do the logging. Though it could return Apache2::Const::OK since all
    > other log handlers will be run anyway.


    Ah right, PerlLogHandler is RUN_ALL - my bad. I remember wanting to do
    something similar a while ago and ended up just marking those log
    entries so that I could 'grep -v MY_IGNORE' the log and not have to deal
    with them.

    > unquote
    >
    > as compared to your
    >
    > And then write your own PerlLogHandler which returns Apache2::Const::OK
    > (which should tell Apache to skip the log handler phase).
    >
    > ?
    >
    > Of course I can try it, but if you already know..
    >



  7. Re: Skip or minimise Apache log phase

    Hi Andr,
    If you're looking into the mod_perl approach, you might want to have a peek
    at Apache::LogIgnore. I developed the module for Apache v1 and haven't
    tested it on v2 but it might be a good place to start.

    http://search.cpan.org/~beatnik/Apache-LogIgnore-0.03/

    HTH

    Hendrik

    2008/10/8 Andr Warnier

    > Hi.
    >
    > Following a message posted on the Apache users list, I am just curious if
    > via mod_perl there could be a solution to the following issue :
    >
    > A busy Apache server (with several VirtualHosts, why not ?) is being
    > accessed from internal network clients (IP address 192.168.*) as well as by
    > external clients (any other IPs).
    > Among the internal clients are some GoogleBots, which generate thousands of
    > accesses, all logged in the access logs of the hosts. These accesses come
    > to generate more than 90% of the total, which really bothers the sysadmins
    > when they have to scan any logfile for something else.
    >
    > Would there be any way, using mod_perl, to detect such accesses early, and
    > to either cancel the log phase for them, or else redirect the logging to
    > some sink file, or else at least set some parameter so that the verbosityof
    > the log for these accesses would be drastically reduced ?
    > (Of course the requests themselves should just go through and still be
    > handled properly)
    >
    > Looking at the description of the PerlLogHandler at
    > http://perl.apache.org/docs/2.0/user...PerlLogHandler
    > I find thhe following paragraph, which tends to indicate that Apache log
    > handlers will run anyway, but maybe there is still a devious solution ?
    >
    > quote
    > First the handler tries to figure out what username the request is issued
    > for, if it fails to match the URI, it simply returns
    > Apache2::Const:ECLINED, letting other log handlers to do the logging.
    > Though it could return Apache2::Const::OK since all other log handlers will
    > be run anyway.
    > unquote
    >
    > Thanks
    >




    --
    Hendrik Van Belleghem
    Spine - The backbone for your website - http://spine.sf.net


  8. Re: Skip or minimise Apache log phase

    Hendrik Van Belleghem wrote:
    > Hi Andr,
    > If you're looking into the mod_perl approach, you might want to have a peek
    > at Apache::LogIgnore. I developed the module for Apache v1 and haven't
    > tested it on v2 but it might be a good place to start.
    >
    > http://search.cpan.org/~beatnik/Apache-LogIgnore-0.03/
    >

    Many thanks.
    I just had a look at the module.
    From what I understand, it returns DONE whenever a condition matches,
    the idea being probably then to skip other logging steps.
    But according to the mod_perl 2.x docs quoted in previous messages, that
    would still let the other log handlers run.
    But, maybe the mp2 docs are wrong ?
    I guess I'll have to test that under Apache2/mp2.


+ Reply to Thread