Creating a "load simulator" by calling Perl Programs - or Forking?

Greetings,

I would like to create a 'driver' (simulation) program that calls
existing perl programs, waits for them to complete and provides some
feedback. Ideally I would like to have the driver echo to the screen
that the simulation is still running and perhaps provide some
information. That may be too ambitious, but that is the basic idea.

I have a handful of perl programs that do things - read a database
(kudos to the DBI modules) and output a report, from a few lines to a
few thousand.

Question #1: What is the proper way to execute the perl programs?

I am familiar with exec() and system() and have done this:

system("sallmr1 -n50000 >/dev/null");
system("srngmr1 -sr -es -n10000 >/dev/null");
system("srngmr2 -sr -es -n10000 >/dev/null");

Which executes these three programs sequentially - or by adding '&' at
the end I can execute them at the same time. Of course the 'calling'
program cannot wait for the background tasks because of the shell
return.

Which leads me down the path of fork(). If I were to code the
following:

for(1..2)
{
if($pid=fork)
{
# parent
} elsif (defined $pid)
{
# child
exec("sallmr1 -n50000 >/dev/null");
} # end child (elsif)
} # end for loop
do { $x=wait; print "$x died\n";} until $x==-1;

Then I would get a pair of runs, and wait until each child dies.

Question #2: Is there a way to echo to the screen that the wait() is
happening and provide some indication of progress?

To clarify this question, I had originally tried:
do {$x=wait; print ".";} until $x==-1;

Thinking I would get a period printed during the loop. (silly me -
wait only comes back when a child process dies...).

Of course if this is much to complicated a way to so a simulated load,
then so be it - just a plain old fork and wait works fine... And I
could always remove the /dev/null and actually see the called programs
output, which will tell me that it's still running... But I was
curious if there is a more elegant or at least interesting way of
doing this?

thanks,
phil

Re: Creating a "load simulator" by calling Perl Programs - or Forking?

pgodfrin writes:

> Greetings,
>
> I would like to create a 'driver' (simulation) program that calls
> existing perl programs, waits for them to complete and provides some
> feedback. Ideally I would like to have the driver echo to the screen
> that the simulation is still running and perhaps provide some
> information. That may be too ambitious, but that is the basic idea.

It may take a bit of getting used to, but POE::Wheel::Run can do this
very nicely, including monitoring of the fork'd procesesses' STDOUT and
STDERR, catching signals etc. And knowing POE will be useful in many
event-based processes.

See:
http://search.cpan.org/~rcaputo/POE-1.0000/lib/POE/Wheel/Run .pm

and the section "Process Management" in the POE cookbook:
http://poe.perl.org/?POE_Cookbook


--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 3, 3:30 pm, Joost Diepenmaat wrote:
> pgodfrin writes:
> > Greetings,
>
> > I would like to create a 'driver' (simulation) program that calls
> > existing perl programs, waits for them to complete and provides some
> > feedback. Ideally I would like to have the driver echo to the screen
> > that the simulation is still running and perhaps provide some
> > information. That may be too ambitious, but that is the basic idea.
>
> It may take a bit of getting used to, but POE::Wheel::Run can do this
> very nicely, including monitoring of the fork'd procesesses' STDOUT and
> STDERR, catching signals etc. And knowing POE will be useful in many
> event-based processes.
>
> See:http://search.cpan.org/~rcaputo/POE-1.0000/lib/POE/Wheel /Run.pm
>
> and the section "Process Management" in the POE cookbook:http://poe.perl.org/?POE_Cookbook
>
> --
> Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/

I was looking at IPC::Run as well. What do you think of it?
pg

Re: Creating a "load simulator" by calling Perl Programs - or Forking?

pgodfrin writes:

> I was looking at IPC::Run as well. What do you think of it?

Haven't used it all. And to be honest, from skimming the docs I can't
really see if it will even do what you want it to do if you have more
than one child process running, which I can guarantee you POE will.

But as I implied, POE has a bit of a learning curve, and it requires you
to (re)structure your code in a event handling way. YMMV, probably :-)

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 3, 3:49 pm, Joost Diepenmaat wrote:
> pgodfrin writes:
> > I was looking at IPC::Run as well. What do you think of it?
>
> Haven't used it all. And to be honest, from skimming the docs I can't
> really see if it will even do what you want it to do if you have more
> than one child process running, which I can guarantee you POE will.
>
> But as I implied, POE has a bit of a learning curve, and it requires you
> to (re)structure your code in a event handling way. YMMV, probably :-)
>
> --
> Joost Diepenmaat | blog:http://joost.zeekat.nl/| work:http://zeekat.nl/

Yeah - I agree. As I've mentioned in the past, I'm just a lowly DBA
(out of work btw) who's trying to not only learn Perl real good, but
use it simply to drive my databases so I can do DBA stuff - you know
like look at CPU util, I/O, funky SQL, etc.

So - I'm certain the POE will do the trick, on your say-so, but I'm
not sure if it is worth the learning curve at this juncture. But I
have bookmarked it and I will at least peruse it to see what it's all
about.

You never know...

smiles,
pg

Re: Creating a "load simulator" by calling Perl Programs - or Forking?

pgodfrin wrote:

....
> Question #1: What is the proper way to execute the perl programs?

You gave two examples, sequentially and in parallel. Which one is
appropriate depends on what kind of load you want to simulate, which in
turn depends on why are you are doing the simulation.


> Question #2: Is there a way to echo to the screen that the wait() is
> happening and provide some indication of progress?
>
> To clarify this question, I had originally tried:
> do {$x=wait; print ".";} until $x==-1;
>
> Thinking I would get a period printed during the loop. (silly me -
> wait only comes back when a child process dies...).

So you do get a period printed, once per child-exit. This is what I would
want to do, anyway, but I seem to be in a philosophical minority in this
regard (I hate progress indicators that merely indicate that the system
clock is ticking, and nothing about the actual progress of the actual
program.) I'm not sure what you originally thought it might do instead.
Periods to be printed as fast as Perl possible can print them?



> Of course if this is much to complicated a way to so a simulated load,
> then so be it - just a plain old fork and wait works fine... And I
> could always remove the /dev/null and actually see the called programs
> output, which will tell me that it's still running... But I was
> curious if there is a more elegant or at least interesting way of
> doing this?

If the waiting program is still running, and is still waiting, then the
program it is waiting on must still be running. What other possibilities
are there?

If you really want the "time has not come to a halt" thing, then
something like this:

perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
$SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
do {$x=wait; print "$x";} until $x==-1; \
alarm 0'
..
..
..
20287
..
..
..
..
20288
..
..
..
..
20289
-1


Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 3, 4:22 pm, xhos...@gmail.com wrote:
> pgodfrin wrote:
>
> ...
>
> > Question #1: What is the proper way to execute the perl programs?
>
> You gave two examples, sequentially and in parallel. Which one is
> appropriate depends on what kind of load you want to simulate, which in
> turn depends on why are you are doing the simulation.
>
> > Question #2: Is there a way to echo to the screen that the wait() is
> > happening and provide some indication of progress?
>
> > To clarify this question, I had originally tried:
> > do {$x=wait; print ".";} until $x==-1;
>
> > Thinking I would get a period printed during the loop. (silly me -
> > wait only comes back when a child process dies...).
>
> So you do get a period printed, once per child-exit. This is what I would
> want to do, anyway, but I seem to be in a philosophical minority in this
> regard (I hate progress indicators that merely indicate that the system
> clock is ticking, and nothing about the actual progress of the actual
> program.) I'm not sure what you originally thought it might do instead.
> Periods to be printed as fast as Perl possible can print them?
>
> > Of course if this is much to complicated a way to so a simulated load,
> > then so be it - just a plain old fork and wait works fine... And I
> > could always remove the /dev/null and actually see the called programs
> > output, which will tell me that it's still running... But I was
> > curious if there is a more elegant or at least interesting way of
> > doing this?
>
> If the waiting program is still running, and is still waiting, then the
> program it is waiting on must still be running. What other possibilities
> are there?
>
> If you really want the "time has not come to a halt" thing, then
> something like this:
>
> perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
> $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
> do {$x=wait; print "$x";} until $x==-1; \
> alarm 0'
> .
> .
> .
> 20287
> .
> .
> .
> .
> 20288
> .
> .
> .
> .
> 20289
> -1
>
> Xho
>
> --
> --------------------http://NewsReader.Com/------------------ --
> The costs of publication of this article were defrayed in part by the
> payment of page charges. This article must therefore be hereby marked
> advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> this fact.

Hi Xho,
As always - good points. I am looking to load up the system, so a
parallel simulation seems to be the best approach. And frankly, the
period printing was just a way to see that indeed something was
happening, but clearly and wholly unnecessary - although I did get a
chuckle over the time halting comment...

And anyway 'watch ps u' in anotehr session celarly shows me the
processes still running.

But - the question still remains - what's the best way to call other
perl programs? Exec, system or something else?

pg

Re: Creating a "load simulator" by calling Perl Programs - or Forking?

pgodfrin wrote:
>
> But - the question still remains - what's the best way to call other
> perl programs? Exec, system or something else?

I'd do fork and exec. If you want to do something in the child after
the other thing was done, then I'd do fork and system.

Xho

--
-------------------- http://NewsReader.Com/ --------------------
The costs of publication of this article were defrayed in part by the
payment of page charges. This article must therefore be hereby marked
advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
this fact.

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 4, 6:12 am, pgodfrin wrote:
> On Apr 3, 4:22 pm, xhos...@gmail.com wrote:
>
>
>
> > pgodfrin wrote:
>
> > ...
>
> > > Question #1: What is the proper way to execute the perl programs?
>
> > You gave two examples, sequentially and in parallel. Which one is
> > appropriate depends on what kind of load you want to simulate, which in
> > turn depends on why are you are doing the simulation.
>
> > > Question #2: Is there a way to echo to the screen that the wait() is
> > > happening and provide some indication of progress?
>
> > > To clarify this question, I had originally tried:
> > > do {$x=wait; print ".";} until $x==-1;
>
> > > Thinking I would get a period printed during the loop. (silly me -
> > > wait only comes back when a child process dies...).
>
> > So you do get a period printed, once per child-exit. This is what I would
> > want to do, anyway, but I seem to be in a philosophical minority in this
> > regard (I hate progress indicators that merely indicate that the system
> > clock is ticking, and nothing about the actual progress of the actual
> > program.) I'm not sure what you originally thought it might do instead.
> > Periods to be printed as fast as Perl possible can print them?
>
> > > Of course if this is much to complicated a way to so a simulated load,
> > > then so be it - just a plain old fork and wait works fine... And I
> > > could always remove the /dev/null and actually see the called programs
> > > output, which will tell me that it's still running... But I was
> > > curious if there is a more elegant or at least interesting way of
> > > doing this?
>
> > If the waiting program is still running, and is still waiting, then the
> > program it is waiting on must still be running. What other possibilities
> > are there?
>
> > If you really want the "time has not come to a halt" thing, then
> > something like this:
>
> > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
> > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
> > do {$x=wait; print "$x";} until $x==-1; \
> > alarm 0'
> > .
> > .
> > .
> > 20287
> > .
> > .
> > .
> > .
> > 20288
> > .
> > .
> > .
> > .
> > 20289
> > -1
>
> > Xho
>
> > --
> > --------------------http://NewsReader.Com/------------------ --
> > The costs of publication of this article were defrayed in part by the
> > payment of page charges. This article must therefore be hereby marked
> > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> > this fact.
>
> Hi Xho,
> As always - good points. I am looking to load up the system, so a
> parallel simulation seems to be the best approach. And frankly, the
> period printing was just a way to see that indeed something was
> happening, but clearly and wholly unnecessary - although I did get a
> chuckle over the time halting comment...
>
> And anyway 'watch ps u' in anotehr session celarly shows me the
> processes still running.
>
> But - the question still remains - what's the best way to call other
> perl programs? Exec, system or something else?
>
> pg

Hi pg,

a module that I've written couple of months ago was for that very
purpose.
Basically your script will have to instantiate an PreforkAgent, tell
it how many children to beget (degree of parallelization) and provide
it with a couple of callbacks as means of communication.
These callback routines may provide any task, execute it as you like
and perform evaluation of results and logging.
The module's initial purpose was it to generate as much load on a
remote system as the local server would allow. You can however, setup
a small number of children and even delay the task assignments without
interfering with the former tasks.
Children will live and be re-used until the job-queue is depleted.

You might download the module from http://www.atablis.com/temp/PreforkAgent.pm
and give it a try.

Cheers,
Steffen



It is rather easy to integrate

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 4, 1:44 am, sheinr...@my-deja.com wrote:
> On Apr 4, 6:12 am, pgodfrin wrote:
>
>
>
> > On Apr 3, 4:22 pm, xhos...@gmail.com wrote:
>
> > > pgodfrin wrote:
>
> > > ...
>
> > > > Question #1: What is the proper way to execute the perl programs?
>
> > > You gave two examples, sequentially and in parallel. Which one is
> > > appropriate depends on what kind of load you want to simulate, which in
> > > turn depends on why are you are doing the simulation.
>
> > > > Question #2: Is there a way to echo to the screen that the wait() is
> > > > happening and provide some indication of progress?
>
> > > > To clarify this question, I had originally tried:
> > > > do {$x=wait; print ".";} until $x==-1;
>
> > > > Thinking I would get a period printed during the loop. (silly me -
> > > > wait only comes back when a child process dies...).
>
> > > So you do get a period printed, once per child-exit. This is what I would
> > > want to do, anyway, but I seem to be in a philosophical minority in this
> > > regard (I hate progress indicators that merely indicate that the system
> > > clock is ticking, and nothing about the actual progress of the actual
> > > program.) I'm not sure what you originally thought it might do instead.
> > > Periods to be printed as fast as Perl possible can print them?
>
> > > > Of course if this is much to complicated a way to so a simulated load,
> > > > then so be it - just a plain old fork and wait works fine... And I
> > > > could always remove the /dev/null and actually see the called programs
> > > > output, which will tell me that it's still running... But I was
> > > > curious if there is a more elegant or at least interesting way of
> > > > doing this?
>
> > > If the waiting program is still running, and is still waiting, then the
> > > program it is waiting on must still be running. What other possibilities
> > > are there?
>
> > > If you really want the "time has not come to a halt" thing, then
> > > something like this:
>
> > > perl -le '$|=1; fork or do {sleep 4*$_; exit} foreach (1..3); \
> > > $SIG{ALRM}=sub{print "."; alarm 1}; alarm 1; \
> > > do {$x=wait; print "$x";} until $x==-1; \
> > > alarm 0'
> > > .
> > > .
> > > .
> > > 20287
> > > .
> > > .
> > > .
> > > .
> > > 20288
> > > .
> > > .
> > > .
> > > .
> > > 20289
> > > -1
>
> > > Xho
>
> > > --
> > > --------------------http://NewsReader.Com/------------------ --
> > > The costs of publication of this article were defrayed in part by the
> > > payment of page charges. This article must therefore be hereby marked
> > > advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate
> > > this fact.
>
> > Hi Xho,
> > As always - good points. I am looking to load up the system, so a
> > parallel simulation seems to be the best approach. And frankly, the
> > period printing was just a way to see that indeed something was
> > happening, but clearly and wholly unnecessary - although I did get a
> > chuckle over the time halting comment...
>
> > And anyway 'watch ps u' in anotehr session celarly shows me the
> > processes still running.
>
> > But - the question still remains - what's the best way to call other
> > perl programs? Exec, system or something else?
>
> > pg
>
> Hi pg,
>
> a module that I've written couple of months ago was for that very
> purpose.
> Basically your script will have to instantiate an PreforkAgent, tell
> it how many children to beget (degree of parallelization) and provide
> it with a couple of callbacks as means of communication.
> These callback routines may provide any task, execute it as you like
> and perform evaluation of results and logging.
> The module's initial purpose was it to generate as much load on a
> remote system as the local server would allow. You can however, setup
> a small number of children and even delay the task assignments without
> interfering with the former tasks.
> Children will live and be re-used until the job-queue is depleted.
>
> You might download the module fromhttp://www.atablis.com/temp/PreforkAgent.pm
> and give it a try.
>
> Cheers,
> Steffen
>
> It is rather easy to integrate

cool! I'll take a look...
phil

Re: Creating a "load simulator" by calling Perl Programs - or Forking?

sheinrich@my-deja.com wrote:
[...]
> a module that I've written couple of months ago was for that very
> purpose.
> Basically your script will have to instantiate an PreforkAgent, tell
> it how many children to beget (degree of parallelization) and provide
> it with a couple of callbacks as means of communication.

Hu.. sounds pretty much like Parallel::ForkManager.

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 4, 6:19 pm, "J. Gleixner"
wrote:
> sheinr...@my-deja.com wrote:
>
> [...]
>
> Hu.. sounds pretty much like Parallel::ForkManager.

That might well be.
PreforkAgent came into being after I'd 'inherited' a customized
loadtesting script to which more and more functionality was to add. So
I was looking for a means of abstraction. Only after that was
finished, did I peak into the Parallel namespace.
Yet I didn't bother to explore how much of my work, which I had
enjoyed, has been gratuitous.
:-)

steffen

Re: Creating a "load simulator" by calling Perl Programs - or

On Apr 4, 6:19 pm, "J. Gleixner"
wrote:
> sheinr...@my-deja.com wrote:
>
> [...]
>
> > a module that I've written couple of months ago was for that very
> > purpose.
> > Basically your script will have to instantiate anPreforkAgent, tell
> > it how many children to beget (degree of parallelization) and provide
> > it with a couple of callbacks as means of communication.
>
> Hu.. sounds pretty much like Parallel::ForkManager.

Now I got the chance to look at Parallel::ForkManager and get back on
this.

Well, the obvious difference is that PreforkAgent preforks while
Parallel::ForkManager does very much the same as fork(), only limiting
the child number and abstracting waitpid.
And as I wrote above
> > Children will live and be re-used until the job-queue is depleted.

While the Parallel::ForkManager is creating a new process on every job
(with all the performance overhead that involves), PreforkAgent does
no forking once the work 's been taken up and its children do exit
only after the last job was assigned.

The basic incentive for PreforkAgent.pm was making the most efficient
use of the host's ressources. (E. g., to wreak havoc with real big
application servers for the sake of load simulation.)

steffen