DTrace in Perl: What probes should we have?

As of patch 32953 dtrace support is in bleadperl (5.11.0). The probes
are based on Alan Burlinson's original blog post on the subject:

http://blogs.sun.com/alanbur/date/20050909

By guarding the probes with PERL_SUB_*_ENABLED the performance hit is
unmeasurable.

All the necessary bits already existed in the wild. I just assembled
them and made the necessary changes to Perl's Configure script. The
patched Perl works with the examples in DTraceToolkit.

Currently we have probes on subroutine entry and return.

Next I'd like to solicit input about what other probes would be
useful. I have Sven Dowideit's patch that adds probes on new__sv,
del__sv (creation and deletion of values) and, main__enter, main__exit
(interpreter lifecycle) and load__module (require, use etc).

Where else might we usefully probe? What would people find useful?

-- notes --

I'm crossposting this to what I hope are a few interested lists. If
anyone reading this needs a primer on dtrace this video is highly
recommended:

http://video.google.com/videoplay?docid=-8002801113289007228

It's 90 minutes of your life but it'll be time well spent :)

If anyone would like to try bleadperl it is available here:

rsync -avz --exclude .svn/ --delete \
rsync://ftp.linux.activestate.com/perl-current/ bleadperl

You can configure it like this:

../Configure -de -Dusedevel -Dinc_version_list=none \
-Dprefix=$install -Dldflags=-Dman3ext=3pm \
-Duseithreads -Duseshrplib -Uversiononly \
-Dusedtrace

where $install is your preferred install base.

Right now ./Configure displays a harmless error about '!' being an
unknown command just after the questions. There's a patch in the
pipeline to fix that.

--
Andy Armstrong, Hexten

Re: DTrace in Perl: What probes should we have?

excellent stuff Andy!

To start with the obvious :)

Due to the lack of ustack helper, I am hoping to add a stack probe - I
really could do with being able to write a dtrace that shows me a perl
stack trace on file system access - yesterday I spent a borish day using
inotifywait and some perl tracing on linux :/


Another other thing I'm missing is the arguments to ops. Oh, and can I
trace exception throws and catches?


Sven

Andy Armstrong wrote:
> As of patch 32953 dtrace support is in bleadperl (5.11.0). The probes
> are based on Alan Burlinson's original blog post on the subject:
>
> http://blogs.sun.com/alanbur/date/20050909
>
> By guarding the probes with PERL_SUB_*_ENABLED the performance hit is
> unmeasurable.
>
> All the necessary bits already existed in the wild. I just assembled
> them and made the necessary changes to Perl's Configure script. The
> patched Perl works with the examples in DTraceToolkit.
>
> Currently we have probes on subroutine entry and return.
>
> Next I'd like to solicit input about what other probes would be
> useful. I have Sven Dowideit's patch that adds probes on new__sv,
> del__sv (creation and deletion of values) and, main__enter, main__exit
> (interpreter lifecycle) and load__module (require, use etc).
>
> Where else might we usefully probe? What would people find useful?
>
> -- notes --
>
> I'm crossposting this to what I hope are a few interested lists. If
> anyone reading this needs a primer on dtrace this video is highly
> recommended:
>
> http://video.google.com/videoplay?docid=-8002801113289007228
>
> It's 90 minutes of your life but it'll be time well spent :)
>
> If anyone would like to try bleadperl it is available here:
>
> rsync -avz --exclude .svn/ --delete \
> rsync://ftp.linux.activestate.com/perl-current/ bleadperl
>
> You can configure it like this:
>
> ./Configure -de -Dusedevel -Dinc_version_list=none \
> -Dprefix=$install -Dldflags=-Dman3ext=3pm \
> -Duseithreads -Duseshrplib -Uversiononly \
> -Dusedtrace
>
> where $install is your preferred install base.
>
> Right now ./Configure displays a harmless error about '!' being an
> unknown command just after the questions. There's a patch in the
> pipeline to fix that.
>
>


--
Professional Wiki Innovation and Support
Sven Dowideit - http://DistributedINFORMATION.com
A WikiRing Partner http://wikiring.com

Re: DTrace in Perl: What probes should we have?

On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
> excellent stuff Andy!
>
> To start with the obvious :)
>
> Due to the lack of ustack helper, I am hoping to add a stack probe

Two things: first of all DTrace could really benefit from a
general purpose mechanism of hooking up custom stack helpers
with the rest of the system. Frankly, the way jstack is done
doesn't strike me as too generic a mechanism.

Second, I would like to encourage those who do DTrace integration
into Perl to take a look at what JavaScript is doing. They don't
have a custom stack helper either but the set of probes they offer
makes stack synthesis a trivial (albeit costly) matter.

Thanks,
Roman.

Re: DTrace in Perl: What probes should we have?

Roman Shaposhnik wrote:
> On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
>
>> excellent stuff Andy!
>>
>> To start with the obvious :)
>>
>> Due to the lack of ustack helper, I am hoping to add a stack probe
>>
>
> Two things: first of all DTrace could really benefit from a
> general purpose mechanism of hooking up custom stack helpers
> with the rest of the system. Frankly, the way jstack is done
> doesn't strike me as too generic a mechanism.
>
> Second, I would like to encourage those who do DTrace integration
> into Perl to take a look at what JavaScript is doing. They don't
> have a custom stack helper either but the set of probes they offer
> makes stack synthesis a trivial (albeit costly) matter.
>
> Thanks,
> Roman.
>
Yeah, I'm pondering (naively) the idea of activating the perl debugger
using isenabled.

--
Professional Wiki Innovation and Support
Sven Dowideit - http://DistributedINFORMATION.com
A WikiRing Partner http://wikiring.com

Re: DTrace in Perl: What probes should we have?

On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:

> > Due to the lack of ustack helper, I am hoping to add a stack probe
>
> Two things: first of all DTrace could really benefit from a
> general purpose mechanism of hooking up custom stack helpers
> with the rest of the system. Frankly, the way jstack is done
> doesn't strike me as too generic a mechanism.

Do you have something in mind?

regards
john

Re: DTrace in Perl: What probes should we have?

On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:
> On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
> > excellent stuff Andy!
> >
> > To start with the obvious :)
> >
> > Due to the lack of ustack helper, I am hoping to add a stack probe
>
> Two things: first of all DTrace could really benefit from a
> general purpose mechanism of hooking up custom stack helpers
> with the rest of the system. Frankly, the way jstack is done
> doesn't strike me as too generic a mechanism.

Do you actually understand how it's implemented? It's actually quite
general -- witness the presence of ustack helpers for both Python and
PHP. (Indeed, the underlying mechanism is _so_ general that for some
time we weren't sure if we had fallen prey to false generality.) The
problem -- such as it is -- is not the lack of generality, it's that they
are brutally difficult to write, in large part because the problem that
ustack helpers are trying to solve is a brutally difficult problem.
(Most VMs are not prepared to perform stack introspection in an arbitrary
context.)

> Second, I would like to encourage those who do DTrace integration
> into Perl to take a look at what JavaScript is doing. They don't
> have a custom stack helper either but the set of probes they offer
> makes stack synthesis a trivial (albeit costly) matter.

While the JavaScript approach is better than nothing, it is _not_
preferable to the ustack approach -- but a JavaScript ustack helper is
prohibitively difficult at the moment because the information required
simply doesn't exist when it's needed.

So, Perl folks: if you can do it, a ustack helper is the way to go. It's
brutal, but the payoff is substantial, as it will be much easier to connect
misbehaving Perl to the symptoms of that misbehavior elsewhere in the
system. If you're interested in seeing a demo of this, check out John
Levon's blog on the Python ustack helper that he implemented:

http://blogs.sun.com/levon/entry/python_and_dtrace_in_build

And my description (and video) of demonstrating John's outstanding work
at Google:

http://blogs.sun.com/bmc/entry/dtrace_at_google

- Bryan

------------------------------------------------------------ --------------
Bryan Cantrill, Sun Microsystems FishWorks. http://blogs.sun.com/bmc

Re: [dtrace-discuss] DTrace in Perl: What probes should we have?

On 12 Jan 2008, at 01:32, Bryan Cantrill wrote:
> So, Perl folks: if you can do it, a ustack helper is the way to
> go. It's
> brutal, but the payoff is substantial, as it will be much easier to
> connect
> misbehaving Perl to the symptoms of that misbehavior elsewhere in the
> system. If you're interested in seeing a demo of this, check out John
> Levon's blog on the Python ustack helper that he implemented:
>
> http://blogs.sun.com/levon/entry/python_and_dtrace_in_build


I read that Bryan. I got the impression - maybe incorrectly - that it
relies on the Python interpreter recursing when it enters a Python
function - so that a walk up the C stack visits all entries in the
Python stack. Did I get that wrong?

Perl doesn't recurse in C when it calls a Perl subroutine so walking
up the stack won't yield a Perl call chain.

--
Andy Armstrong, Hexten

Re: DTrace in Perl: What probes should we have?

On Sat, 2008-01-12 at 01:30 +0000, John Levon wrote:
> On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:
>
> > > Due to the lack of ustack helper, I am hoping to add a stack probe
> >
> > Two things: first of all DTrace could really benefit from a
> > general purpose mechanism of hooking up custom stack helpers
> > with the rest of the system. Frankly, the way jstack is done
> > doesn't strike me as too generic a mechanism.
>
> Do you have something in mind?

Discalimer: I can't really claim to be an expert in DTrace, my
main familiarity with it comes from working on a tool that sometimes
pushes DTrace to its limits and that way I'm forced to explore how
these limits could be overcome. If what I have to say below is a
wrong perception of the technology I'd love to be educated by the
subject experts.

My main beef with stack helpers is that they have to be executed
inside the kernel and are subject to the usual DTrace restrictions.
I would like to see a possibility of offloading more work to user
space in a generic fashion. For example, we have a libcollector.so
facility that is capable of reconstructing stacks under the most
challenging of situations (inside the function preamble, etc.) it is
way more accurate in what it reports back than ustack() it is also
capable of more stack unwinding than jstack(). Of course, it runs in
user space.

We do have some preliminary ideas on how such functionality
can be hooked to the rest of DTrace machinery. If there's a
significant portion of the DTrace community that is interested in
extending DTrace functionality in a way that would make it more
reliant on userspace -- perhaps it is a good thing to have on the
DTrace conference agenda.

Thanks,
Roman.

Re: DTrace in Perl: What probes should we have?

On Fri, 2008-01-11 at 17:32 -0800, Bryan Cantrill wrote:
> On Fri, Jan 11, 2008 at 04:39:25PM -0800, Roman Shaposhnik wrote:
> > On Sat, 2008-01-12 at 11:21 +1100, Sven Dowideit wrote:
> > > excellent stuff Andy!
> > >
> > > To start with the obvious :)
> > >
> > > Due to the lack of ustack helper, I am hoping to add a stack probe
> >
> > Two things: first of all DTrace could really benefit from a
> > general purpose mechanism of hooking up custom stack helpers
> > with the rest of the system. Frankly, the way jstack is done
> > doesn't strike me as too generic a mechanism.
>
> Do you actually understand how it's implemented?

Not fully, but I've spent a great deal of time staring at
jhelper.d ;-)

Seriously, though -- my experience comes mostly from reading the
source. And the source code for DTrace can get pretty tricky. If you
have any suggestion for me as to what would be a good intro into
stack helpers -- I'd appreciate that.

> It's actually quite
> general -- witness the presence of ustack helpers for both Python and
> PHP. (Indeed, the underlying mechanism is _so_ general that for some
> time we weren't sure if we had fallen prey to false generality.)

I guess we are talking about two different kinds of generality here.
What I have in mind is the generality that would allow DTrace to be
able to offload some of the heavylifting to the userspace. I DO
understand the ramifications of that and I know full well that pushing
DTrace into userspace might be perceived as the deviation from the
original design. To some extent it might even reduce the generality
that you seem to be referring to.

> > Second, I would like to encourage those who do DTrace integration
> > into Perl to take a look at what JavaScript is doing. They don't
> > have a custom stack helper either but the set of probes they offer
> > makes stack synthesis a trivial (albeit costly) matter.
>
> While the JavaScript approach is better than nothing, it is _not_
> preferable to the ustack approach

Agreed. Still it is WAY better than nothing ;-)

Thanks,
Roman.

Re: DTrace in Perl: What probes should we have?

On Fri, Jan 11, 2008 at 06:23:14PM -0800, Roman Shaposhnik wrote:
>
> I guess we are talking about two different kinds of generality here.
> What I have in mind is the generality that would allow DTrace to be
> able to offload some of the heavylifting to the userspace. I DO
> understand the ramifications of that and I know full well that pushing
> DTrace into userspace might be perceived as the deviation from the
> original design. To some extent it might even reduce the generality
> that you seem to be referring to.
>

The issue is not one of design, but of the constraints on the problem.
DTrace actions must be executed in probe context. Heavy lifting can be
done via post-processing in userland (symbol translation, system()
actions, etc), but the data gathering must be done in arbitrary context.
How would you call arbitrary userspace code from, say, an interrupt
handler?

- Eric

--
Eric Schrock, FishWorks http://blogs.sun.com/eschrock

Re: DTrace in Perl: What probes should we have?

On Sat, Jan 12, 2008 at 02:03:42AM +0000, Andy Armstrong wrote:

> > http://blogs.sun.com/levon/entry/python_and_dtrace_in_build
>
> I read that Bryan. I got the impression - maybe incorrectly - that it
> relies on the Python interpreter recursing when it enters a Python
> function - so that a walk up the C stack visits all entries in the
> Python stack. Did I get that wrong?

Nope, that's right.

> Perl doesn't recurse in C when it calls a Perl subroutine so walking
> up the stack won't yield a Perl call chain.

I always got the impression the difficulty was in walking the stack for
the relevant Perl thread at all, not just the C vs. Perl stack issue.

Whilst the C stack issue is a problem, it's not necessarily a major one.
Presuming (and I really don't know) that it's easy to get to the top of
the Perl stack from the available initial state, and it's feasible to
traverse the relevant structures, in theory we could extend the ustack()
implementation with a yield()-style approach: that is, the helper is
called multiple times and each time yields a relevant string until
"done".

I'm sure Bryan will jump and tell me I'm crazy.

BTW this has been discussed before:

http://mail.opensolaris.org/pipermail/dtrace-discuss/2007-Se ptember/004572.html

regards
john

Re: DTrace in Perl: What probes should we have?

--===============2025466273==
Content-type: multipart/alternative;
boundary="Boundary_(ID_ZKZfbxcA6WJGPrZ1WBqXVw)"


--Boundary_(ID_ZKZfbxcA6WJGPrZ1WBqXVw)
Content-type: text/plain; format=flowed; delsp=yes; charset=US-ASCII
Content-transfer-encoding: 7BIT

On Jan 11, 2008, at 6:26 PM, Eric Schrock wrote:

> On Fri, Jan 11, 2008 at 06:23:14PM -0800, Roman Shaposhnik wrote:
>>
>> I guess we are talking about two different kinds of generality
>> here.
>> What I have in mind is the generality that would allow DTrace to be
>> able to offload some of the heavylifting to the userspace. I DO
>> understand the ramifications of that and I know full well that
>> pushing
>> DTrace into userspace might be perceived as the deviation from the
>> original design. To some extent it might even reduce the generality
>> that you seem to be referring to.
>>
>
> The issue is not one of design, but of the constraints on the problem.

If we are talking about *u*stack, than I disagree with the way you're
setting up constraints. The *u*stack is by definition a userspace
related
operation. Not only that, but calling ustack() is declared void for a
reason.
Calling it changes the output of the DTrace but has no way of affecting
the rest of the DTrace script's internal logic.

> DTrace actions must be executed in probe context.

Please don't redefine the problem. We are not talking about arbitrary
actions here.

> Heavy lifting can be
> done via post-processing in userland (symbol translation, system()
> actions, etc), but the data gathering must be done in arbitrary
> context.

What data gathering? Could you, please, be more specific? What data
there is that can't be leveraged from the userspace? And again, I'm
talking specifically about ustack() here.

> How would you call arbitrary userspace code from, say, an interrupt
> handler?

I'm not suggesting that at all. And in fact for Project D-Light we
devised
a mechanism for getting much more accurate stacktraces from userspace
compared to what DTrace is capable of. E.g.:

self uint32_t stkIdx;

collector$1:::ustack
{
printf("0 %d %d %u\n", tid, self->stkIdx++, arg0<<32|
(arg1&0xffffffff));
}

syscall::read:return
/pid == $1/
{
fname = (self->fd == 0 ) ? "" : fnames[self->fd];
printf("1 %d %d %d %d \"%s\" %d %d\n",
cpu,
tid,
timestamp,
0,
fname,
arg1,
self->stkIdx);
raise(29);
}

The awkwardness here comes from the fact that we have to:
* make libcollector available in the address space of the process
* use a raise(29) in order to poke userspace instead of
something more
DTrace specific.

Otherwise it works very nice. We get all the benefits of userspace
code:
we can match dlopen/dlclose events and such and our Java stack are
better than jstack().

Thanks,
Roman.

--Boundary_(ID_ZKZfbxcA6WJGPrZ1WBqXVw)
Content-type: text/html; charset=ISO-8859-1
Content-transfer-encoding: QUOTED-PRINTABLE

-webkit-line-break: after-white-space; ">

style=3D"margin-top: 0px; margin-right: 0px; margin-bottom: 0px; marg=
in-left: 0px; ">DTrace actions must be executed in probe context. n class=3D"Apple-converted-space">=A0