mp2, IPC::Run3 && Environment Variables giving "variable not set" error - modperl

This is a discussion on mp2, IPC::Run3 && Environment Variables giving "variable not set" error - modperl ; Hey, folks, As the migration to the Apache2/mp2/perl-5.10 server progresses, I just ran into something that I'm not quite understanding. I've got a module that gets information from a program using IPC::Run3. This script relies on a number of environment ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

  1. mp2, IPC::Run3 && Environment Variables giving "variable not set" error

    Hey, folks,

    As the migration to the Apache2/mp2/perl-5.10 server progresses, I just
    ran into something that I'm not quite understanding.

    I've got a module that gets information from a program using IPC::Run3.
    This script relies on a number of environment variables, which I've been
    setting successfully ust using plain old " $ENV{MY_VAR} = 'avalue';"
    under mp1.3.

    Now, though I'm setting the variable explicitly within the EVAL block
    within which I'm running run3(), I'm getting errors from the external
    program saying that the variable is not set.

    These variables are set based on parameters passed into the script, so I
    can't use PerlSetVar, and I don't think that PerlPassVar is propriate
    here.

    Am I missing something? Anybody have any ideas?

    Thanks


    Eric D. Berg

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  2. Re: mp2, IPC::Run3 && Environment Variables giving "variable notset" error

    Berg, Eric wrote:
    [...]
    I don't know the exact answer to your question, but assuming you are in
    a Unix environment, you could try the following :

    In a Unix-like environment, if you launch a command using the following
    form :

    VAR1=value1 VAR2=value2 command

    then VAR1 and VAR2 are automatically "exported" as environment variables
    for the program "command".

    Thus, if in your script you have the possibility to start your external
    command in that way, it may be a usable work-around to your problem.

    André


  3. RE: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

    Thanks again, Andre.

    This issue of the environment variables' not being passed to forked processes is turning into a fairly substantial problem for me. We have a number of places in our code -- both the modules that are tightly controlled and fairly easy to manage, as well as a bunch of scripts and CGI's that use them -- that use system or backticks (sometimes via IPC::Run3) to get info from external tools.

    Does anybody know anything about this issue or about the code involved? How difficult would it be to address this? I took a look at the code in modperl_env.c as suggested in the docs, but I'm a Perl guy, not a C guy.

    This is a deal killer for us.

    I'd appreciate any comments on this. One option seems to be to pass the environment variables if we're not in a threaded (worker) server. Maybe that would work for us.

    Anybody?

    Thanks a lot.

    Eric.

    > -----Original Message-----
    > From: André Warnier [mailto:aw@ice-sa.com]
    > Sent: Tuesday, August 12, 2008 10:59 AM
    > To: modperl@perl.apache.org
    > Cc: Berg, Eric
    > Subject: Re: mp2, IPC::Run3 && Environment Variables giving
    > "variable not set" error
    >
    > Berg, Eric wrote:
    > [...]
    > I don't know the exact answer to your question, but assuming
    > you are in
    > a Unix environment, you could try the following :
    >
    > In a Unix-like environment, if you launch a command using the
    > following
    > form :
    >
    > VAR1=value1 VAR2=value2 command
    >
    > then VAR1 and VAR2 are automatically "exported" as
    > environment variables
    > for the program "command".
    >
    > Thus, if in your script you have the possibility to start
    > your external
    > command in that way, it may be a usable work-around to your problem.
    >
    > André
    >
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  4. Re: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

    On Fri 15 Aug 2008, Berg, Eric wrote:
    > This issue of the environment variables' not being passed to forked
    > processes is turning into a fairly substantial problem for me. * We
    > have a number of places in our code -- both the modules that are
    > tightly controlled and fairly easy to manage, as well as a bunch of
    > scripts and CGI's that use them -- that use system or backticks
    > (sometimes via IPC::Run3) to get info from external tools.
    >
    > Does anybody know anything about this issue or about the code
    > involved? *How difficult would it be to address this? *I took a look
    > at the code in modperl_env.c as suggested in the docs, but I'm a Perl
    > guy, not a C guy. *
    >
    > This is a deal killer for us. *


    Let's have a look at the reason for this behavior first. It's always
    better to know what you are doing. Apache 2.x can use various MPMs.
    Some of them use threads instead of processes to achieve parallelism.
    Now, the environment is a process global resource. So, if those values
    are changed all threads are affected. This is surely no what you want.

    Therefore there is a request specific data structure called
    subprocess_env where environment variables for CGI scripts are kept. If
    you use the Apache interface to spawn subprocesses
    (Apache2::SubProcess) you'll notice that variables set in
    subprocess_env appear as environment variables of the spawned process.
    Under mod_perl %ENV is tied to $r->subprocess_env.

    If you are using the prefork MPM (as you probably do) you can still
    manipulate the environment without affecting other requests. But you
    have to use the C-level functions. There is a module called Env::C on
    CPAN that provides a perl interface to them.

    Torsten

    --
    Need professional mod_perl support?
    Just hire me: torsten.foertsch@gmx.net


  5. RE: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

    Mr. Foertsch, that was a wonderful reply. Just what the doctor ordered. Thanks.

    > If you are using the prefork MPM (as you probably do) you can still
    > manipulate the environment without affecting other requests. But you
    > have to use the C-level functions. There is a module called Env::C on
    > CPAN that provides a Perl interface to them.


    Yes, I am using the prefork MPM.

    I have put together a test of this, and Env::C works as advertised. For those variables that need to be passed to the subprocess, lines like this suffice:

    foreach my $var_name ( qw/DB_SVR TEST1 TEST2/ ) {
    Env::C::setenv( $var_name, $ENV{ $varname}, 1);
    }

    Very simple. Of course there's scant documentation on the required environment variables in most of the scripts that I'll be calling, but that's a different matter....

    > Let's have a look at the reason for this behavior first. It's always
    > better to know what you are doing. Apache 2.x can use various MPMs.
    > Some of them use threads instead of processes to achieve parallelism.
    > Now, the environment is a process global resource. So, if
    > those values are changed all threads are affected. This is surely no
    > what you want.


    When you say that the environment is a global process, global to what? Each forked process has its own environment, doesn't it? It seems to me from the brief piece in the User Guide about this and what you've said, that threaded environments are the real problem here. In the case of threaded environments, the environment is shared among all threads, whereas forked processes can and do have distinct environments per process. Is this correct? That means (as suggested in the relevant source comments) that a solution for the forked MPM would be much easier than for those that are threaded. Right?

    > Therefore there is a request specific data structure called
    > subprocess_env where environment variables for CGI scripts
    > are kept. If you use the Apache interface to spawn subprocesses
    > (Apache2::SubProcess) you'll notice that variables set in
    > subprocess_env appear as environment variables of the spawned
    > process. Under mod_perl %ENV is tied to $r->subprocess_env.


    Do you need to explicitly set environment variables in subprocess_env to have them show up in the environment of the forked process, or do they get passed automatically?

    Just because I wanted to think this thing through... if you fork off a Perl script, for example, and that Perl script forks off another process, that is done using standard forking so the environment is passed to the forked process. That was what my experimentation showed up anyway.

    Eric


    > -----Original Message-----
    > From: Torsten Foertsch [mailto:torsten.foertsch@gmx.net]
    > Sent: Friday, August 15, 2008 3:15 PM
    > To: modperl@perl.apache.org
    > Cc: Berg, Eric
    > Subject: Re: mp2, IPC::Run3 && Environment Variables giving
    > "variable not set" error
    >
    > On Fri 15 Aug 2008, Berg, Eric wrote:
    > > This issue of the environment variables' not being passed to forked
    > > processes is turning into a fairly substantial problem for me. * We
    > > have a number of places in our code -- both the modules that are
    > > tightly controlled and fairly easy to manage, as well as a bunch of
    > > scripts and CGI's that use them -- that use system or backticks
    > > (sometimes via IPC::Run3) to get info from external tools.
    > >
    > > Does anybody know anything about this issue or about the code
    > > involved? *How difficult would it be to address this? *I took a look
    > > at the code in modperl_env.c as suggested in the docs, but

    > I'm a Perl
    > > guy, not a C guy. *
    > >
    > > This is a deal killer for us. *

    >
    >
    > Therefore there is a request specific data structure called
    > subprocess_env where environment variables for CGI scripts
    > are kept. If you use the Apache interface to spawn subprocesses
    > (Apache2::SubProcess) you'll notice that variables set in
    > subprocess_env appear as environment variables of the spawned
    > process. Under mod_perl %ENV is tied to $r->subprocess_env.
    >
    >
    > Torsten
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


  6. Re: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

    On Sat 16 Aug 2008, Berg, Eric wrote:
    > > Now, the environment is a process global resource. So, if
    > > those values are changed all threads are affected. This is surely
    > > no what you want.

    >
    > When you say that the environment is a global process, global to
    > what? *Each forked process has its own environment, doesn't it? *It
    > seems to me from the brief piece in the User Guide about this and
    > what you've said, that threaded environments are the real problem
    > here. *In the case of threaded environments, the environment is
    > shared among all threads, whereas forked processes can and do have
    > distinct environments per process. *Is this correct? *That means (as
    > suggested in the relevant source comments) that a solution for the
    > forked MPM would be much easier than for those that are threaded.
    > *Right?


    Correct. The words "a process global resource" above mean a resource
    that is global to the process.

    Btw, another way to manipulate the environment is using the modperl
    handler instead of perl-script. This way %ENV remains tied to the
    environment and changes will show up after a perl fork.

    See also
    http://perl.apache.org/docs/2.0/user...t_by_Perl_Code

    Torsten

    --
    Need professional mod_perl support?
    Just hire me: torsten.foertsch@gmx.net


  7. RE: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

    I've taken a few minutes to regurgitate this into something that I can submit to the documentation project. Mind taking a look and giving me some feedback (anyone) before I do?

    >>>>>>>>>>>>>>

    =head1 Issues with Environment Variables

    There are several thread-safety issues with setting environment
    variables.

    =head2 Environment Variables and C Code

    Environment variables set during request time won't be seen by C
    code. See the L issue|docs::2.0::user::troubleshooting::troublesho oting/C_Libraries_Don_t_See_C__ENV__Entries_Set_by_Perl_ Code> for possible workarounds.

    =head2 Environment Variables and Forked Processes

    Forked processes (accomplished using backticks, system, or exec )
    won't see CGI emulation environment variables. That means that the
    environment variables as set in the calling context will not be set in
    the forked process. This is different from the behavior in Apache
    1.3. (META: This will hopefully be resolved in the future, it's
    documented in modperl_env.c:modperl_env_magic_set_all.)

    Apache, at the time of this writing, does not distinguish between
    threaded and forked MPM types, and the processes as which they run.
    So all MPMs have a standard behavior which is that the environment is
    not passed to the forked process.

    The environment is a process-global resource. In Apache terms, that
    means that when using a threaded MPM, all of the children are threads
    of the parent apache server process, whereas in the preforked MPM, the
    children are each their own separate process.

    The environment for CGI scripts is kept in a request-specific data
    structure called C. Remembering that in a threaded
    MPM, a forked process is handled by another thread of the parent
    process, we know that if the environment in C is
    changed, it will also change the environment for all other threads of
    it's parent process. But not so for a non-threaded MPM (i.e.,
    preforked), because each subprocess -- such as the Apache children or
    processes forked off of them -- has it's own process, so changing that
    doesn't affect the environment of any other process.

    That said, if you're running the prefork MPM, then there are several
    work arounds that will allow you to pass environment variables to
    forked processes from within mod_perl code.

    =head3 The Env::C Solution

    You can manipulate the environment using the C-level functions. There
    is a module called Env::C on CPAN that provides a perl interface to
    them.

    # To set a single variable for the subprocess
    Env::C::setenv( 'ENVTEST', $ENV{ENVTEST}, 1);

    # To set all environment variables for the subprocess unless
    # they're already set. map { Env::C::setenv( '$_', $ENV{$_} ) } keys %ENV;

    =head3 The Apache2::SubProcess Solution

    Again, for non-threaded MPMs only: If you use the Apache interface
    directly to spawn subprocesses (Apache2::SubProcess) you'll notice
    that variables set in subprocess_env do appear as environment
    variables of the spawned process. Under mod_perl %ENV is tied to
    $r->subprocess_env.

    =head3 The "Passing-The-Environment-Variable-On-The-Command-Line" Solution

    Again, for non-threaded MPMs only: This is probably just a UNIX thing,
    but you can set variables for a process by passing them on the command
    line before calling the command. For example, you may have seen this
    in the README file of any number of software packages:

    C

    Again, for non-threaded MPMs only: This tells the shell to set the
    environment variable C to the value "gcc" in the process
    in which you run C.

    You can do that with your own environment varibles too:

    my $cmd = 'MYVAR1="My value" MYVAR2="your value" do_that_thing.cgi';
    system( $cmd );

    Then MYVAR1 and MYVAR2 will be set to "My value" and "your value",
    respectively in the environment of the process in which
    do_that_thing.cgi runs.

    Links:

    http://perl.apache.org/docs/2.0/user...t_by_Perl_Code

    <<<<<<<<<<<<<<<<<<<<<<<<
    Thanks.

    Eric

    > -----Original Message-----
    > From: Torsten Foertsch [mailto:torsten.foertsch@gmx.net]
    > Sent: Friday, August 15, 2008 3:15 PM
    > To: modperl@perl.apache.org
    > Cc: Berg, Eric
    > Subject: Re: mp2, IPC::Run3 && Environment Variables giving
    > "variable not set" error
    >
    > On Fri 15 Aug 2008, Berg, Eric wrote:
    > > This issue of the environment variables' not being passed to forked
    > > processes is turning into a fairly substantial problem for me. * We
    > > have a number of places in our code -- both the modules that are
    > > tightly controlled and fairly easy to manage, as well as a bunch of
    > > scripts and CGI's that use them -- that use system or backticks
    > > (sometimes via IPC::Run3) to get info from external tools.
    > >
    > > Does anybody know anything about this issue or about the code
    > > involved? *How difficult would it be to address this? *I took a look
    > > at the code in modperl_env.c as suggested in the docs, but

    > I'm a Perl
    > > guy, not a C guy. *
    > >
    > > This is a deal killer for us. *

    >
    > Let's have a look at the reason for this behavior first. It's always
    > better to know what you are doing. Apache 2.x can use various MPMs.
    > Some of them use threads instead of processes to achieve parallelism.
    > Now, the environment is a process global resource. So, if
    > those values
    > are changed all threads are affected. This is surely no what you want.
    >
    > Therefore there is a request specific data structure called
    > subprocess_env where environment variables for CGI scripts
    > are kept. If
    > you use the Apache interface to spawn subprocesses
    > (Apache2::SubProcess) you'll notice that variables set in
    > subprocess_env appear as environment variables of the spawned
    > process.
    > Under mod_perl %ENV is tied to $r->subprocess_env.
    >
    > If you are using the prefork MPM (as you probably do) you can still
    > manipulate the environment without affecting other requests. But you
    > have to use the C-level functions. There is a module called Env::C on
    > CPAN that provides a perl interface to them.
    >
    > Torsten
    >
    > --
    > Need professional mod_perl support?
    > Just hire me: torsten.foertsch@gmx.net
    >

    - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

    This message is intended only for the personal and confidential use of the designated recipient(s) named above. If you are not the intended recipient of this message you are hereby notified that any review, dissemination, distribution or copying of this message is strictly prohibited. This communication is for information purposes only and should not be regarded as an offer to sell or as a solicitation of an offer to buy any financial product, an official confirmation of any transaction, or as an official statement of Lehman Brothers. Email transmission cannot be guaranteed to be secure or error-free. Therefore, we do not represent that this information is complete or accurate and it should not be relied upon as such. All information is subject to change without notice.

    --------
    IRS Circular 230 Disclosure:
    Please be advised that any discussion of U.S. tax matters contained within this communication (including any attachments) is not intended or written to be used and cannot be used for the purpose of (i) avoiding U.S. tax related penalties or (ii) promoting, marketing or recommending to another party any transaction or matter addressed herein.


+ Reply to Thread