mp2, IPC::Run3 && Environment Variables giving "variable not set" error

Hey, folks,

As the migration to the Apache2/mp2/perl-5.10 server progresses, I just
ran into something that I'm not quite understanding.

I've got a module that gets information from a program using IPC::Run3.
This script relies on a number of environment variables, which I've been
setting successfully ust using plain old =22 =24ENV=7BMY_VAR=7D =3D =
'avalue';=22
under mp1.3.

Now, though I'm setting the variable explicitly within the EVAL block
within which I'm running run3(), I'm getting errors from the external
program saying that the variable is not set.

These variables are set based on parameters passed into the script, so I
can't use PerlSetVar, and I don't think that PerlPassVar is propriate
here.

Am I missing something? Anybody have any ideas?

Thanks


Eric D. Berg

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - =
- - - - - - -

This message is intended only for the personal and confidential use of the =
designated recipient(s) named above. If you are not the intended =
recipient of this message you are hereby notified that any review, =
dissemination, distribution or copying of this message is strictly =
prohibited. This communication is for information purposes only and =
should not be regarded as an offer to sell or as a solicitation of an =
offer to buy any financial product, an official confirmation of any =
transaction, or as an official statement of Lehman Brothers. Email =
transmission cannot be guaranteed to be secure or error-free. Therefore, =
we do not represent that this information is complete or accurate and it =
should not be relied upon as such. All information is subject to change =
without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within =
this communication (including any attachments) is not intended or written =
to be used and cannot be used for the purpose of (i) avoiding U.S. tax =
related penalties or (ii) promoting, marketing or recommending to another =
party any transaction or matter addressed herein.

Re: mp2, IPC::Run3 && Environment Variables giving "variable notset" error

Berg, Eric wrote:
[...]
I don't know the exact answer to your question, but assuming you are in
a Unix environment, you could try the following :

In a Unix-like environment, if you launch a command using the following
form :

VAR1=value1 VAR2=value2 command

then VAR1 and VAR2 are automatically "exported" as environment variables
for the program "command".

Thus, if in your script you have the possibility to start your external
command in that way, it may be a usable work-around to your problem.

André

RE: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

Thanks again, Andre.=20

This issue of the environment variables' not being passed to forked =
processes is turning into a fairly substantial problem for me. We have a =
number of places in our code -- both the modules that are tightly =
controlled and fairly easy to manage, as well as a bunch of scripts and =
CGI's that use them -- that use system or backticks (sometimes via =
IPC::Run3) to get info from external tools.

Does anybody know anything about this issue or about the code involved? =
How difficult would it be to address this? I took a look at the code in =
modperl_env.c as suggested in the docs, but I'm a Perl guy, not a C guy. =20

This is a deal killer for us. =20

I'd appreciate any comments on this. One option seems to be to pass the =
environment variables if we're not in a threaded (worker) server. Maybe =
that would work for us.

Anybody?

Thanks a lot.

Eric.

> -----Original Message-----
> From: Andr=E9 Warnier =5Bmailto:aw=40ice-sa.com]
> Sent: Tuesday, August 12, 2008 10:59 AM
> To: modperl=40perl.apache.org
> Cc: Berg, Eric
> Subject: Re: mp2, IPC::Run3 && Environment Variables giving=20
> =22variable not set=22 error
>=20
> Berg, Eric wrote:
> =5B...=5D
> I don't know the exact answer to your question, but assuming=20
> you are in=20
> a Unix environment, you could try the following :
>=20
> In a Unix-like environment, if you launch a command using the=20
> following=20
> form :
>=20
> VAR1=3Dvalue1 VAR2=3Dvalue2 command
>=20
> then VAR1 and VAR2 are automatically =22exported=22 as=20
> environment variables=20
> for the program =22command=22.
>=20
> Thus, if in your script you have the possibility to start=20
> your external=20
> command in that way, it may be a usable work-around to your problem.
>=20
> Andr=E9
>=20
>=20
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - =
- - - - - - -

This message is intended only for the personal and confidential use of the =
designated recipient(s) named above. If you are not the intended =
recipient of this message you are hereby notified that any review, =
dissemination, distribution or copying of this message is strictly =
prohibited. This communication is for information purposes only and =
should not be regarded as an offer to sell or as a solicitation of an =
offer to buy any financial product, an official confirmation of any =
transaction, or as an official statement of Lehman Brothers. Email =
transmission cannot be guaranteed to be secure or error-free. Therefore, =
we do not represent that this information is complete or accurate and it =
should not be relied upon as such. All information is subject to change =
without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within =
this communication (including any attachments) is not intended or written =
to be used and cannot be used for the purpose of (i) avoiding U.S. tax =
related penalties or (ii) promoting, marketing or recommending to another =
party any transaction or matter addressed herein.

Re: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

On Fri 15 Aug 2008, Berg, Eric wrote:
> This issue of the environment variables' not being passed to forked
> processes is turning into a fairly substantial problem for me. =A0 We
> have a number of places in our code -- both the modules that are
> tightly controlled and fairly easy to manage, as well as a bunch of
> scripts and CGI's that use them -- that use system or backticks
> (sometimes via IPC::Run3) to get info from external tools.
>
> Does anybody know anything about this issue or about the code
> involved? =A0How difficult would it be to address this? =A0I took a look
> at the code in modperl_env.c as suggested in the docs, but I'm a Perl
> guy, not a C guy. =A0
>
> This is a deal killer for us. =A0

Let's have a look at the reason for this behavior first. It's always=20
better to know what you are doing. Apache 2.x can use various MPMs.=20
Some of them use threads instead of processes to achieve parallelism.=20
Now, the environment is a process global resource. So, if those values=20
are changed all threads are affected. This is surely no what you want.

Therefore there is a request specific data structure called=20
subprocess_env where environment variables for CGI scripts are kept. If=20
you use the Apache interface to spawn subprocesses=20
(Apache2::SubProcess) you'll notice that variables set in=20
subprocess_env appear as environment variables of the spawned process.=20
Under mod_perl %ENV is tied to $r->subprocess_env.

If you are using the prefork MPM (as you probably do) you can still=20
manipulate the environment without affecting other requests. But you=20
have to use the C-level functions. There is a module called Env::C on=20
CPAN that provides a perl interface to them.

Torsten

=2D-
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net

RE: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

Mr. Foertsch, that was a wonderful reply. Just what the doctor ordered. =
Thanks.

> If you are using the prefork MPM (as you probably do) you can still=20
> manipulate the environment without affecting other requests. But you=20
> have to use the C-level functions. There is a module called Env::C on=20
> CPAN that provides a Perl interface to them.

Yes, I am using the prefork MPM.

I have put together a test of this, and Env::C works as advertised. For =
those variables that need to be passed to the subprocess, lines like this =
suffice:

foreach my =24var_name ( qw/DB_SVR TEST1 TEST2/ ) =7B
=09Env::C::setenv( =24var_name, =24ENV=7B =24varname=7D, 1);
=7D

Very simple. Of course there's scant documentation on the required =
environment variables in most of the scripts that I'll be calling, but =
that's a different matter....

> Let's have a look at the reason for this behavior first. It's always=20
> better to know what you are doing. Apache 2.x can use various MPMs.=20
> Some of them use threads instead of processes to achieve parallelism.=20
> Now, the environment is a process global resource. So, if=20
> those values are changed all threads are affected. This is surely no=20
> what you want.

When you say that the environment is a global process, global to what? =
Each forked process has its own environment, doesn't it? It seems to me =
from the brief piece in the User Guide about this and what you've said, =
that threaded environments are the real problem here. In the case of =
threaded environments, the environment is shared among all threads, =
whereas forked processes can and do have distinct environments per =
process. Is this correct? That means (as suggested in the relevant =
source comments) that a solution for the forked MPM would be much easier =
than for those that are threaded. Right?

> Therefore there is a request specific data structure called=20
> subprocess_env where environment variables for CGI scripts=20
> are kept. If you use the Apache interface to spawn subprocesses=20
> (Apache2::SubProcess) you'll notice that variables set in=20
> subprocess_env appear as environment variables of the spawned=20
> process. Under mod_perl %ENV is tied to =24r->subprocess_env.

Do you need to explicitly set environment variables in subprocess_env to =
have them show up in the environment of the forked process, or do they get =
passed automatically?

Just because I wanted to think this thing through... if you fork off a =
Perl script, for example, and that Perl script forks off another process, =
that is done using standard forking so the environment is passed to the =
forked process. That was what my experimentation showed up anyway.

Eric


> -----Original Message-----
> From: Torsten Foertsch =5Bmailto:torsten.foertsch=40gmx.net]
> Sent: Friday, August 15, 2008 3:15 PM
> To: modperl=40perl.apache.org
> Cc: Berg, Eric
> Subject: Re: mp2, IPC::Run3 && Environment Variables giving=20
> =22variable not set=22 error
>=20
> On Fri 15 Aug 2008, Berg, Eric wrote:
> > This issue of the environment variables' not being passed to forked
> > processes is turning into a fairly substantial problem for me. =A0 We
> > have a number of places in our code -- both the modules that are
> > tightly controlled and fairly easy to manage, as well as a bunch of
> > scripts and CGI's that use them -- that use system or backticks
> > (sometimes via IPC::Run3) to get info from external tools.
> >
> > Does anybody know anything about this issue or about the code
> > involved? =A0How difficult would it be to address this? =A0I took a look
> > at the code in modperl_env.c as suggested in the docs, but=20
> I'm a Perl
> > guy, not a C guy. =A0
> >
> > This is a deal killer for us. =A0
>=20
>=20
> Therefore there is a request specific data structure called=20
> subprocess_env where environment variables for CGI scripts=20
> are kept. If you use the Apache interface to spawn subprocesses=20
> (Apache2::SubProcess) you'll notice that variables set in=20
> subprocess_env appear as environment variables of the spawned=20
> process. Under mod_perl %ENV is tied to =24r->subprocess_env.
>=20
>=20
> Torsten
>=20
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - =
- - - - - - -

This message is intended only for the personal and confidential use of the =
designated recipient(s) named above. If you are not the intended =
recipient of this message you are hereby notified that any review, =
dissemination, distribution or copying of this message is strictly =
prohibited. This communication is for information purposes only and =
should not be regarded as an offer to sell or as a solicitation of an =
offer to buy any financial product, an official confirmation of any =
transaction, or as an official statement of Lehman Brothers. Email =
transmission cannot be guaranteed to be secure or error-free. Therefore, =
we do not represent that this information is complete or accurate and it =
should not be relied upon as such. All information is subject to change =
without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within =
this communication (including any attachments) is not intended or written =
to be used and cannot be used for the purpose of (i) avoiding U.S. tax =
related penalties or (ii) promoting, marketing or recommending to another =
party any transaction or matter addressed herein.

Re: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

On Sat 16 Aug 2008, Berg, Eric wrote:
> > Now, the environment is a process global resource. So, if
> > those values are changed all threads are affected. This is surely
> > no what you want.
>
> When you say that the environment is a global process, global to
> what? =A0Each forked process has its own environment, doesn't it? =A0It
> seems to me from the brief piece in the User Guide about this and
> what you've said, that threaded environments are the real problem
> here. =A0In the case of threaded environments, the environment is
> shared among all threads, whereas forked processes can and do have
> distinct environments per process. =A0Is this correct? =A0That means (as
> suggested in the relevant source comments) that a solution for the
> forked MPM would be much easier than for those that are threaded.
> =A0Right?

Correct. The words "a process global resource" above mean a resource=20
that is global to the process.

Btw, another way to manipulate the environment is using the modperl=20
handler instead of perl-script. This way %ENV remains tied to the=20
environment and changes will show up after a perl fork.

See also=20
http://perl.apache.org/docs/2.0/user/troubleshooting/trouble shooting.html#C=
_Libraries_Don_t_See_C__ENV__Entries_Set_by_Perl_Code

Torsten

=2D-
Need professional mod_perl support?
Just hire me: torsten.foertsch@gmx.net

RE: mp2, IPC::Run3 && Environment Variables giving "variable not set" error

I've taken a few minutes to regurgitate this into something that I can =
submit to the documentation project. Mind taking a look and giving me =
some feedback (anyone) before I do?

>>>>>>>>>>>>>>
=3Dhead1 Issues with Environment Variables

There are several thread-safety issues with setting environment
variables.

=3Dhead2 Environment Variables and C Code
=20
Environment variables set during request time won't be seen by C
code. See the L issue=7Cdocs::2.0::user::troubleshooting::troubleshooting/C_ Libraries_Don_t=
_See_C__ENV__Entries_Set_by_Perl_Code> for possible workarounds.

=3Dhead2 Environment Variables and Forked Processes

Forked processes (accomplished using backticks, system, or exec )
won't see CGI emulation environment variables. That means that the
environment variables as set in the calling context will not be set in
the forked process. This is different from the behavior in Apache
1.3. (META: This will hopefully be resolved in the future, it's
documented in modperl_env.c:modperl_env_magic_set_all.)

Apache, at the time of this writing, does not distinguish between
threaded and forked MPM types, and the processes as which they run.
So all MPMs have a standard behavior which is that the environment is
not passed to the forked process.

The environment is a process-global resource. In Apache terms, that
means that when using a threaded MPM, all of the children are threads
of the parent apache server process, whereas in the preforked MPM, the
children are each their own separate process.

The environment for CGI scripts is kept in a request-specific data
structure called C. Remembering that in a threaded
MPM, a forked process is handled by another thread of the parent
process, we know that if the environment in C is
changed, it will also change the environment for all other threads of
it's parent process. But not so for a non-threaded MPM (i.e.,
preforked), because each subprocess -- such as the Apache children or
processes forked off of them -- has it's own process, so changing that
doesn't affect the environment of any other process.

That said, if you're running the prefork MPM, then there are several
work arounds that will allow you to pass environment variables to
forked processes from within mod_perl code.

=3Dhead3 The Env::C Solution

You can manipulate the environment using the C-level functions. There
is a module called Env::C on CPAN that provides a perl interface to
them.

=23 To set a single variable for the subprocess
Env::C::setenv( 'ENVTEST', =24ENV=7BENVTEST=7D, 1);

=23 To set all environment variables for the subprocess unless
=23 they're already set. map =7B Env::C::setenv( '=24_', =
=24ENV{$_=7D ) =7D keys %ENV;

=3Dhead3 The Apache2::SubProcess Solution

Again, for non-threaded MPMs only: If you use the Apache interface
directly to spawn subprocesses (Apache2::SubProcess) you'll notice
that variables set in subprocess_env do appear as environment
variables of the spawned process. Under mod_perl %ENV is tied to
=24r->subprocess_env.

=3Dhead3 The =
"Passing-The-Environment-Variable-On-The-Command-Line" Solution

Again, for non-threaded MPMs only: This is probably just a UNIX thing,
but you can set variables for a process by passing them on the command
line before calling the command. For example, you may have seen this
in the README file of any number of software packages:

C

Again, for non-threaded MPMs only: This tells the shell to set the
environment variable C to the value "gcc" in the process
in which you run C.

You can do that with your own environment varibles too:

my =24cmd =3D 'MYVAR1="My value=22 MYVAR2="your value=22 =
do_that_thing.cgi';
system( =24cmd );

Then MYVAR1 and MYVAR2 will be set to =22My value=22 and =22your value=22,
respectively in the environment of the process in which
do_that_thing.cgi runs.

Links:

=
http://perl.apache.org/docs/2.0/user/troubleshooting/trouble shooting.html=
=23C_Libraries_Don_t_See_C__ENV__Entries_Set_by_Perl_Code

<<<<<<<<<<<<<<<<<<<<<<<<
Thanks.

Eric=20

> -----Original Message-----
> From: Torsten Foertsch =5Bmailto:torsten.foertsch=40gmx.net]
> Sent: Friday, August 15, 2008 3:15 PM
> To: modperl=40perl.apache.org
> Cc: Berg, Eric
> Subject: Re: mp2, IPC::Run3 && Environment Variables giving=20
> =22variable not set=22 error
>=20
> On Fri 15 Aug 2008, Berg, Eric wrote:
> > This issue of the environment variables' not being passed to forked
> > processes is turning into a fairly substantial problem for me. =A0 We
> > have a number of places in our code -- both the modules that are
> > tightly controlled and fairly easy to manage, as well as a bunch of
> > scripts and CGI's that use them -- that use system or backticks
> > (sometimes via IPC::Run3) to get info from external tools.
> >
> > Does anybody know anything about this issue or about the code
> > involved? =A0How difficult would it be to address this? =A0I took a look
> > at the code in modperl_env.c as suggested in the docs, but=20
> I'm a Perl
> > guy, not a C guy. =A0
> >
> > This is a deal killer for us. =A0
>=20
> Let's have a look at the reason for this behavior first. It's always=20
> better to know what you are doing. Apache 2.x can use various MPMs.=20
> Some of them use threads instead of processes to achieve parallelism.=20
> Now, the environment is a process global resource. So, if=20
> those values=20
> are changed all threads are affected. This is surely no what you want.
>=20
> Therefore there is a request specific data structure called=20
> subprocess_env where environment variables for CGI scripts=20
> are kept. If=20
> you use the Apache interface to spawn subprocesses=20
> (Apache2::SubProcess) you'll notice that variables set in=20
> subprocess_env appear as environment variables of the spawned=20
> process.=20
> Under mod_perl %ENV is tied to =24r->subprocess_env.
>=20
> If you are using the prefork MPM (as you probably do) you can still=20
> manipulate the environment without affecting other requests. But you=20
> have to use the C-level functions. There is a module called Env::C on=20
> CPAN that provides a perl interface to them.
>=20
> Torsten
>=20
> --
> Need professional mod_perl support?
> Just hire me: torsten.foertsch=40gmx.net
>=20
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - =
- - - - - - -

This message is intended only for the personal and confidential use of the =
designated recipient(s) named above. If you are not the intended =
recipient of this message you are hereby notified that any review, =
dissemination, distribution or copying of this message is strictly =
prohibited. This communication is for information purposes only and =
should not be regarded as an offer to sell or as a solicitation of an =
offer to buy any financial product, an official confirmation of any =
transaction, or as an official statement of Lehman Brothers. Email =
transmission cannot be guaranteed to be secure or error-free. Therefore, =
we do not represent that this information is complete or accurate and it =
should not be relied upon as such. All information is subject to change =
without notice.

--------
IRS Circular 230 Disclosure:
Please be advised that any discussion of U.S. tax matters contained within =
this communication (including any attachments) is not intended or written =
to be used and cannot be used for the purpose of (i) avoiding U.S. tax =
related penalties or (ii) promoting, marketing or recommending to another =
party any transaction or matter addressed herein.