I've read that the best way to learn DTrace is to study existing D scripts. However, upon doing so I quickly find myself asking "where is that documented?" And it turns out, with tens of thousands of probes, as is often the case, it's not. So I'd love to hear what others with more DTrace experience do, but these are the steps I seem to have stumbled upon.

As an example, let's take the script /opt/DTT/Proc/rwbytype.d, which shows the bytes read and written by process by vnode type. Here's an example of the output:
bleonard@opensolaris:/opt/DTT/Proc$ rwbytype.d Tracing... Hit Ctrl-C to end. ^C PID CMD VTYPE DIR BYTES 21124 gnome-terminal chr W 1 10523 gnome-power-mana sock W 4 10565 gnome-netstatus- fifo R 7 10547 gnome-netstatus- fifo W 8 4817 conky fifo R 9 10634 firefox-bin fifo R 9 10634 firefox-bin fifo W 9 15379 whoami fifo W 9 10547 gnome-netstatus- fifo R 20 10565 gnome-netstatus- fifo W 20 15377 dtrace chr W 31 10412 Xorg chr R 32 10469 gnome-settings-d sock R 32 10523 gnome-power-mana sock R 32 21124 gnome-terminal chr R 33 10472 metacity sock W 56 10472 metacity sock R 200 21124 gnome-terminal sock R 220 15379 conky reg R 529 15379 ksh93 reg R 529 4817 conky sock R 2720 10412 Xorg sock W 3204 4817 conky reg R 6696 21124 gnome-terminal sock W 6852 4817 conky sock W 37952 10412 Xorg sock R 44864
And here's the D script that produces that output:

typedef struct vtype2str { string code; }; translator struct vtype2str < int T > { /* the order has been picked for performance reasons */ code = T == 1 ? "reg" : T == 9 ? "sock" : T == 4 ? "chr" : T == 6 ? "fifo" : T == 8 ? "proc" : T == 2 ? "dir" : T == 3 ? "blk" : T == 5 ? "lnk" : T == 7 ? "door" : T == 10 ? "port" : T == 11 ? "bad" : "non"; }; dtrace:::BEGIN { printf("Tracing... Hit Ctrl-C to end.\n"); } fbt::fop_read:entry, fbt::fop_write:entry { self->type = xlate (args[0]->v_type)->code; self->size = args[1]->uio_resid; self->uiop = args[1]; } fbt::fop_read:return /self->uiop/ { this->resid = self->uiop->uio_resid; @bytes[pid, execname, self->type, "R"] = sum(self->size - this->resid); self->type = 0; self->size = 0; self->uiop = 0; } /* this is delibrately redundant code for performance reasons */ fbt::fop_write:return /self->uiop/ { this->resid = self->uiop->uio_resid; @bytes[pid, execname, self->type, "W"] = sum(self->size - this->resid); self->type = 0; self->size = 0; self->uiop = 0; } dtrace:::END { printf("%-6s %-16s %6s %4s %9s\n", "PID", "CMD", "VTYPE", "DIR", "BYTES"); printa("%-6d %-16s %6s %4s %@9d\n", @bytes); }
First of all, this D script is only enabling 6 probes:
dtrace:::BEGIN dtrace:::END dtrace::fop_read:entry dtrace::fob_read:return dtrace::fob_write:entry dtrace::fob_write:return
The BEGIN and END clauses are self-explanatory. Where I get tripped up is looking at the 2nd clause:

fbt::fop_read:entry, fbt::fop_write:entry { self->type = xlate (args[0]->v_type)->code; self->size = args[1]->uio_resid; self->uiop = args[1]; }
First, I see that we're setting 3 thread-local variables, type, size and uiop. All of these variables are set using probe arguments, and this is where things start to get confusing - where are these arguments defined? I can run the following dtrace command to get verbose information on a probe:

bleonard@opensolaris:/opt/DTT/Proc$ dtrace -lv -n fbt::fop_read:entry ID PROVIDER MODULE FUNCTION NAME 17725 fbt genunix fop_read entry Probe Description Attributes Identifier Names: Private Data Semantics: Private Dependency Class: Unknown Argument Attributes Identifier Names: Private Data Semantics: Private Dependency Class: ISA Argument Types args[0]: vnode_t * args[1]: uio_t * args[2]: int args[3]: cred_t * args[4]: caller_context_t *
So I learn that the data semantics of the argument attributes are private, which means this probe is accessing private implementation details of the kernel, but that's the case with most DTrace kernel probes. Of more interest is the argument list:

args[0]: vnode_t * args[1]: uio_t * args[2]: int args[3]: cred_t * args[4]: caller_context_t
So I can see there are 5 total arguments available for this probe and I know my clause is accessing args[0] and args[1]. But what now? What exactly is vnode_t and uio_t? As far as I know, there's no further dtrace command to drill down into these data types. At this point I turn to the OpenSolaris Source Browser.


I begin by searching on the probe function fop_read, which returns 2 results, the vnode.h header file and the vnode.c source file:

Searched refs:fop_read (Results 1 -2 of 2) sorted by null /onnv/onnv-gate/usr/src/uts/common/sys/ vnode.h [all...] /onnv/onnv-gate/usr/src/uts/common/fs/ vnode.c 3177 fop_read( function
[all...]
The fop_read function is defined on line 3177 of vnode.c and I can click on that link to be taken directly to the source. Sure enough, the arguments to the fop_read function in vnode.c match what I was told by dtrace:

3176 int 3177 fop_read( 3178 vnode_t *vp, 3179 uio_t *uiop, 3180 int ioflag, 3181 cred_t *cr, 3182 caller_context_t *ct)
Now I can click on vnode_t to see its definition and this returns 3 results - the logical one being the vnode.h header file returned by our first search:
/onnv/onnv-gate/usr/src/uts/common/sys/ vnode.h 257 } vnode_t; typedef in typeref:struct:vnode
800 int (*vop_open)(vnode_t **, int, cred_t *, \
802 int (*vop_close)(vnode_t *, int, int, offset_t, cred_t *, \
804 int (*vop_read)(vnode_t *, uio_t *, int, cred_t *, \
806 int (*vop_write)(vnode_t *, uio_t *, int, cred_t *, \
808 int (*vop_ioctl)(vnode_t *, int, intptr_t, int, cred_t *, \
810 int (*vop_setfl)(vnode_t *, int, int, cred_t *, \
812 int (*vop_getattr)(vnode_t *, vattr_t *, int, cred_t *, \
814 int (*vop_setattr)(vnode_t *, vattr_t *, int, cred_t *, \
816 int (*vop_access)(vnode_t *, int, int, cred_t *,
[all...]
Here we see that vnode_t is defined on line 257, which you can click to navigate directly to that section of the source. Our D script above is referencing args[0]->v_type, and we can see v_type is an enum:

225 typedef struct vnode { 226 kmutex_t v_lock; /* protects vnode fields */ 227 uint_t v_flag; /* vnode flags (see below) */ 228 uint_t v_count; /* reference count */ 229 void *v_data; /* private data for fs */ 230 struct vfs *v_vfsp; /* ptr to containing VFS */ 231 struct stdata *v_stream; /* associated stream */ 232 enum vtype v_type; /* vnode type */ 233 dev_t v_rdev; /* device (VCHR, VBLK) */
Which you can also click on to see the type definition:

151 /* 152 * vnode types. VNON means no type. These values are unrelated to 153 * values in on-disk inodes. 154 */ 155 typedef enum vtype { 156 VNON = 0, 157 VREG = 1, 158 VDIR = 2, 159 VBLK = 3, 160 VCHR = 4, 161 VLNK = 5, 162 VFIFO = 6, 163 VDOOR = 7, 164 VPROC = 8, 165 VSOCK = 9, 166 VPORT = 10, 167 VBAD = 11 168 } vtype_t;
Which matches up perfectly to the struct at the beginning of our D script. So, now I've got a better handle of exactly what's going on with this D script.


Now, as a test, see if you can follow the same procedure to find the documented comment in the OpenSolaris source code for arg[1]->uio_resid :-).



More...