Last week, an Oracle DBA wanted me to prove that directio was active on a Solaris 10 server. Oracle data resided on UFS filesystems and mounted with "forcedirectio". When set like this at the filesystem level, in theory, all Oracle accesses to the filesystems are done via direct I/O independently of what is set at the Oracle instance level (filesystemio_options). In practise, there is no real evidence that directio is truly in effect.

Let's consider such a server. To reiterate, the server is using UFS filesystems mounted with forcedirectio. Oracle "filesystemio_options" is set to "asynch" (don't ask me why it is not set to a more recommended value). I want to prove that direct I/O is in effect.

1) I can run "vmstat -p" and check that there is no I/O going through the cache filesystem. But it is more a hint than a proof.

2) I could have used "directiostat" from http://www.solarisinternals.com/si/t...stat/index.php. But I was not authorized to transfer and run it on the server under study.

3) Trussing an Oracle process - here, an Oracle Checkpoint process - yields the following:

root@server1 # pgrep -lf ckpt
22806 ora_ckpt_SID1
root@server1 # truss -f -t !lwp_park,semtimedop -p 22806
22806/2: kaio(6, 0x00000000, 0xFFFFFFFF7CE480C0, 0x00000010, 0xFFFFFFFF7CE49D58, 0xFFFFFFFF7F700A00) (sleeping...)
22806/1: kill(22884, SIG#0) = 0
22806/1: open("/proc/22884/psinfo", O_RDONLY) = 16
22806/1: read(16, "02\0\0\0\0\0\0\v\0\0 Y d".., 416) = 416
22806/1: close(16) = 0
22806/1: kill(22886, SIG#0) = 0
22806/1: open("/proc/22886/psinfo", O_RDONLY) = 16
22806/1: read(16, "02\0\0\0\0\0\0\v\0\0 Y f".., 416) = 416
22806/1: close(16) = 0
22806/1: kaio(AIOWAIT, 0xFFFFFFFF7FFFD950) = 1
22806/16: pwrite(257, "15C2\0\0\0\0\003\0\0\0\0".., 16384, 49152) = 16384
22806/17: pwrite(258, "15C2\0\0\0\0\003\0\0\0\0".., 16384, 49152) = 16384
22806/16: kaio(AIONOTIFY, 0) = 0
22806/17: kaio(AIONOTIFY, 0) = 0
22806/10: pwrite(256, "15C2\0\0\0\0\003\0\0\0\0".., 16384, 49152) = 16384
22806/1: kaio(AIOWAIT, 0xFFFFFFFF7FFFD950) = 1
22806/10: kaio(AIONOTIFY, 0) = 0


So, the process is doing asynchronous I/O writes using kaio. It continues to run without waiting for its writes to be acknowledged. The I/Os are handled asynchronously by the Solaris kernel. But the process truss looks the same whether or not directio is enabled.

4) The only way I have found to ensure directio is enabled is with DTrace. The first DTrace command below comes fromhttp://www.forsythesunsolutions.com/node/34. It checks how I/O is done to a particular file (here an Oracle Control file). The stack shows that the file is accessed through directio.

root@server1 # pfiles 22806 | grep ctl
/oracle/d1/SID1/ctl/control_01.ctl
/oracle/d2/SID1/ctl/control_02.ctl
/oracle/d3/SID1/ctl/control_03.ctl
root@server1 # dtrace -n io:::start'/(args[0]->b_flags & B_READ) == 0 && args[2]->fi_pathname == "/oracle/d3/SID1/ctl/control_03.ctl"/{@[stack()] = count()}'
dtrace: description 'io:::start' matched 6 probes
^C

genunix`bdev_strategy+0x44
ufs`directio_start+0xfc
ufs`ufs_directio_write+0x8c4
ufs`ufs_write+0x174
genunix`fop_write+0x20
genunix`pwrite+0x22c
unix`syscall_trap+0xac
8

genunix`bdev_strategy+0x44
md_stripe`md_stripe_strategy+0x2fc
md_mirror`mirror_write_strategy+0x7dc
md`mdstrategy+0xd4
ufs`directio_start+0xfc
ufs`ufs_directio_write+0x8c4
ufs`ufs_write+0x174
genunix`fop_write+0x20
genunix`pwrite+0x22c
unix`syscall_trap+0xac
8

An even better proof is provided by the DTrace script from http://wikis.sun.com/display/SAPonSu...ying+Oracle+IO. I copied the script verbatim to /var/tmp/sun/dio2.d. Running the script on the same Oracle Checkpoint process trussed above clearly shows what the process is performing directio.
root@server1 # cat /var/tmp/sun/dio2.d
#!/usr/sbin/dtrace -s
#pragma D option flowindent
fbt:ufs::entry
/pid == $1/
{
self->ts[probefunc] = timestamp;
}

fbt:ufs::return
/pid == $1 && self->ts[probefunc] != 0/
{
self->duration[probefunc] = (timestamp - self->ts[probefunc])/1000;
}

fbt:ufs::
/pid == $1/
{
trace(self->duration[probefunc]);
self->duration[probefunc] = 0;

}


root@server1 # /var/tmp/sun/dio2.d 22806
dtrace: script '/var/tmp/sun/dio2.d' matched 1826 probes
CPU FUNCTION
68 -> ufs_rwlock 0
68 ufs_write 0
68 -> ufs_check_rewrite 0
68 -> bmap_has_holes 0
68 ufs_lockfs_is_under_rawlockfs 0
68 bmap_read 0
68 -> findextent 0
68 directio_wait_one 0
68 ufs_check_rewrite 0
0 -> bmap_has_holes 0
0 ufs_lockfs_is_under_rawlockfs 0
0 ufs_imark 0
0 bmap_write 0
0 ufs_putpage 0
0 ufs_putpages 0
0 -> ufs_putapage 0
0 -> bmap_read 0
0 -> findextent 0
0 deltamap_remove 0
0 findextent 0
0 deltamap_remove 0
0