How to calculate number of files on a disk

Hello all,

I'm working on gathering disk stats. I've used sys$getdviw to gather
freeblocks, maxblocks, maxfiles, errorcount, mountcount...

But I'd like to also get the number of files on a disk as well. In
case I'm getting close to maxfiles, or incase the number of files
increase/decrease over time.


A sample of C code would be appreciated...


Thanks,

Lyndon

On May 6, 11:29*pm, lyndonbart...@yahoo.com wrote:[color=blue]
> Hello all,
>
> I'm working on gathering disk stats. I've used sys$getdviw to gather
> freeblocks, maxblocks, maxfiles, errorcount, mountcount...
>
> But I'd like to also get the number of files on a disk as well. In
> case I'm getting close to maxfiles, or incase the number of files
> increase/decrease over time.
>
> A sample of C code would be appreciated...
>
> Thanks,
>
> Lyndon[/color]

Use DFU. [url]http://www.digiater.nl/dfu.html[/url]
Spawn that if need be.

Of course you _could_ open [000000]INDEXF.SYS and read <maxfiles+4095>/
4096 blocks at VBN <bucket_size>*4+1
Then proceed to count the bits.

hth,
Hein.

[email]lyndonbartels@yahoo.com[/email] wrote:[color=blue]
> Hello all,
>
> I'm working on gathering disk stats. I've used sys$getdviw to gather
> freeblocks, maxblocks, maxfiles, errorcount, mountcount...
>
> But I'd like to also get the number of files on a disk as well. In
> case I'm getting close to maxfiles, or incase the number of files
> increase/decrease over time.
>
>
> A sample of C code would be appreciated...
>[/color]

Extract the following code to count_files.c.
Compile and link as follows:

$ cc count_files
$ link count_files

An example run would be:

$ write sys$output f$getdvi ("dsa0:", "devnam")
_DSA0:
$ set process/priv=readall
$ mcr []count_files dsa0:
Number of files on dsa0: is 41376


-------------------begin code---------------------------
#define __NEW_STARLET 1

#include <stdio.h>
#include <stdlib.h>
#include <ssdef.h>
#include <stsdef.h>
#include <rms.h>
#include <descrip.h>
#include <lib$routines.h>

#define errchk_sig(arg) if (!$VMS_STATUS_SUCCESS(arg)) (void)lib$signal
(arg);


/******************************************************************************/
static void usage (void) {

(void)printf ("Usage: $ mcr []count_files device_name\n"
"\twhere device name is the name of a disk, including"
"\t the colon. For example, DSA0:\n");
}


/******************************************************************************/
int main (int argc, char *argv[]) {

static int r0_status;
static unsigned int file_count = 0;
static unsigned int context = 0;
static int finished = FALSE;

static char wild[] = "[000000...]*.*;*";
static char file[NAML$C_MAXRSS+1];
static char spec[64+sizeof(wild)];

static struct dsc$descriptor_s spec_d = { 0,
DSC$K_DTYPE_T,
DSC$K_CLASS_S,
spec };
static struct dsc$descriptor_s file_d = { 0,
DSC$K_DTYPE_T,
DSC$K_CLASS_S,
file };

if (argc < 2) {
usage ();
exit (EXIT_FAILURE);
}

spec_d.dsc$w_length = sprintf (spec, "%s%s", argv[1], wild);

while (!finished) {
file_d.dsc$w_length = NAML$C_MAXRSS;
r0_status = lib$find_file (&spec_d,
&file_d,
&context,
0,
0,
0,
0);
if (r0_status == RMS$_NMF) {
finished = TRUE;
continue;
} else {
errchk_sig (r0_status);
}
file_count++;
}

(void)printf ("Number of files on %s is %u\n",
argv[1],
file_count);
return (EXIT_SUCCESS);
}
-----------end code----------------------------------

Additional examples of calls to system services and LIB$ routines can be
found at <http://www.eight-cubed.com/examples.shtml>

You might like to look at the LIB$SET_SYMBOL call to set a DCL symbol
rather than printing out the result.

Cheers,
Jim.
--
[url]www.eight-cubed.com[/url]

Jim Duff wrote:[color=blue]
> [email]lyndonbartels@yahoo.com[/email] wrote:
>[color=green]
>> Hello all,
>>
>> I'm working on gathering disk stats. I've used sys$getdviw to gather
>> freeblocks, maxblocks, maxfiles, errorcount, mountcount...
>>
>> But I'd like to also get the number of files on a disk as well. In
>> case I'm getting close to maxfiles, or incase the number of files
>> increase/decrease over time.
>>
>>
>> A sample of C code would be appreciated...
>>[/color]
>
> Extract the following code to count_files.c.
> Compile and link as follows:
>
> $ cc count_files
> $ link count_files
>
> An example run would be:
>
> $ write sys$output f$getdvi ("dsa0:", "devnam")
> _DSA0:
> $ set process/priv=readall
> $ mcr []count_files dsa0:
> Number of files on dsa0: is 41376
>
>
> -------------------begin code---------------------------
> #define __NEW_STARLET 1
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <ssdef.h>
> #include <stsdef.h>
> #include <rms.h>
> #include <descrip.h>
> #include <lib$routines.h>
>
> #define errchk_sig(arg) if (!$VMS_STATUS_SUCCESS(arg)) (void)lib$signal
> (arg);
>
>
> /******************************************************************************/
>
> static void usage (void) {
>
> (void)printf ("Usage: $ mcr []count_files device_name\n"
> "\twhere device name is the name of a disk, including"
> "\t the colon. For example, DSA0:\n");
> }
>
>
> /******************************************************************************/
>
> int main (int argc, char *argv[]) {
>
> static int r0_status;
> static unsigned int file_count = 0;
> static unsigned int context = 0;
> static int finished = FALSE;
>
> static char wild[] = "[000000...]*.*;*";
> static char file[NAML$C_MAXRSS+1];
> static char spec[64+sizeof(wild)];
>
> static struct dsc$descriptor_s spec_d = { 0,
> DSC$K_DTYPE_T,
> DSC$K_CLASS_S,
> spec };
> static struct dsc$descriptor_s file_d = { 0,
> DSC$K_DTYPE_T,
> DSC$K_CLASS_S,
> file };
>
> if (argc < 2) {
> usage ();
> exit (EXIT_FAILURE);
> }
>
> spec_d.dsc$w_length = sprintf (spec, "%s%s", argv[1], wild);
>
> while (!finished) {
> file_d.dsc$w_length = NAML$C_MAXRSS;
> r0_status = lib$find_file (&spec_d,
> &file_d,
> &context,
> 0,
> 0,
> 0,
> 0);
> if (r0_status == RMS$_NMF) {
> finished = TRUE;
> continue;
> } else {
> errchk_sig (r0_status);
> }
> file_count++;
> }
>
> (void)printf ("Number of files on %s is %u\n",
> argv[1],
> file_count);
> return (EXIT_SUCCESS);
> }
> -----------end code----------------------------------
>
> Additional examples of calls to system services and LIB$ routines can be
> found at <http://www.eight-cubed.com/examples.shtml>
>
> You might like to look at the LIB$SET_SYMBOL call to set a DCL symbol
> rather than printing out the result.
>
> Cheers,
> Jim.[/color]

If I understand this correctly, this is doing a wildcard lookup
on <disk:>[000000...]*.*;* and counting the hits, i.e. traversing
the directory tree. It won't find files that aren't in a directory
(lost files), and it will count multiple times any file that has
alias entries (produced by set file/enter...)

What the OP didn't ask for, but what he actually needs is the
number of file headers available. The only way to get this is
to count the clear bits in the bitmap (as Hein proposed.) This
could still be off, since the system may not be able to actually
allocate that many headers for new files. If the free space is
badly fragmented, the new files will also be badly fragmented and
if severe enough, many of them may require multiple headers to
map all the fragments. Also, if you use lots of ACLs, this can
increase the number of headers required for an average file.
A third limit is if INDEXF.SYS has not yet been fully extended,
when it comes time to allocate more files, the disk may already
be full or so fragmented that INDEXF.SYS requires another header,
which I think is prohibited. Every file requires at least one
header, but it is hard to predict how many will be required for
future files!




--
John Santos
Evans Griffiths & Hart, Inc.
781-861-0670 ext 539

I was wondering if, due to caching, could the bitmap be out of date?

On 7 May, 10:01, IanMiller <g...@uk2.net> wrote:[color=blue]
> I was wondering if, due to caching, could the bitmap be out of date?[/color]

Not necessarily a bad assumption to make, but wouldn't this be risky
for the OS? If it doesn't know at some level or other what headers
are available and which aren't it's potentially going to try and use
one that's already being used...

<lyndonbartels@yahoo.com> wrote in message
news:f0cb80eb-76d0-4de5-90b0-a7bdeafa0c55@f36g2000hsa.googlegroups.com...[color=blue]
> Hello all,
>
> I'm working on gathering disk stats. I've used sys$getdviw to gather
> freeblocks, maxblocks, maxfiles, errorcount, mountcount...
>
> But I'd like to also get the number of files on a disk as well. In
> case I'm getting close to maxfiles, or incase the number of files
> increase/decrease over time.
>
> A sample of C code would be appreciated...[/color]

Not C code, but wouldn't DIR/GRAND [000000...] be a starting point?
--
David Biddulph

On May 7, 7:47*am, "David Biddulph" <groups [at] biddulph.org.uk>
wrote:[color=blue]
> <lyndonbart...@yahoo.com> wrote in message
>
> news:f0cb80eb-76d0-4de5-90b0-a7bdeafa0c55@f36g2000hsa.googlegroups.com...
>[color=green]
> > Hello all,[/color]
>[color=green]
> > I'm working on gathering disk stats. I've used sys$getdviw to gather
> > freeblocks, maxblocks, maxfiles, errorcount, mountcount...[/color][/color]
:[color=blue]
> Not C code, but wouldn't DIR/GRAND [000000...] be a starting point?
> David Biddulph[/color]

That's what our friend Jim Duff already proposed.

It is not entirely correct for the reasons outlined by John Santos

John S> What the OP didn't ask for, but what he actually needs is the
number of file headers available. The only way to get this is
to count the clear bits in the bitmap.

Exactly.

Ian Miller asked: "I was wondering if, due to caching, could the
bitmap be out of date?"

It would be, if you opened INDEXF read-only, but it should be 'close
enough' for the intended usage purpose.

If a tools, such as briefly outlined, were to open INDEXF with WRITE
access, then the XQP will flush the caches rigth there.
See... Black Bible 8.6.7 User Invalidation of Cached Buffers

Hein.

On May 6, 10:29*pm, lyndonbart...@yahoo.com wrote:[color=blue]
> Hello all,
>
> I'm working on gathering disk stats. I've used sys$getdviw to gather
> freeblocks, maxblocks, maxfiles, errorcount, mountcount...
>
> But I'd like to also get the number of files on a disk as well. In
> case I'm getting close to maxfiles, or incase the number of files
> increase/decrease over time.
>
> A sample of C code would be appreciated...
>
> Thanks,
>
> Lyndon[/color]

First, I thank all of you for your responses. Especially Jim Duff, I
found your sight, and your examples have been invaluable...!

Second, I should further define my needs/limitations.

I'm writing an extension onto the T4 monitoring swuite. So.
1. It must be very fast.
2. Must use very little I/O.
3. It must be very fast.

As others have stated, I don't need an absolutely perfectly accurate
number. Inaccuracy due to caching, etc. is acceptable. I simply want a
real close number I can track.

Oh, did I mention it has to be fast???

T4, by default, samples every second, so I need to be able loop
through all the disks on a system, and get this information (If it's
mounted) and write this info to an output csv file every sampling.

This is the last piece of the performance data I want to gather, it's
a "It'd be nice to have." If I can't gather this data quickly and
efficiently, I can leave it out of the final product.


Thanks in advance,

Lyndon

In article <8bc540b0-5e7c-4853-9ada-6e255430a87a@e53g2000hsa.googlegroups.com>, IanMiller <gxys@uk2.net> writes:[color=blue]
> I was wondering if, due to caching, could the bitmap be out of date?
>[/color]

The bitmap on the platters is always out of date when the disk is mounted
unless /nowrite was used in the mount (or hardware write lock is used).

The bitmap will show allocated blocks that are actually preallocated
(in the cache for future use). The bitmap is corrected on proper
dismount, mount without /norebuild, or an explicit set volume/rebuild.

So you don't have to worry about blocks in use that are marked free,
but if you need a rebuild and haven't done it yet there are blocks
marked used that aren't in use.

I generally mount/norebuild and do the explicit rebuild once a week
prior to full backups (overnight) at work, and let the rebuild happen
during mount at home.

It is possible (I've seen it about twice in the last 30 years) for
hardware failures to corrupt the bitmap, leading to blocks marked
free that actually have data in them. No file system can prevent
that kind of hardware failure. That's one of the reasons I do
backups.

On May 7, 9:38*am, lyndonbart...@yahoo.com wrote:[color=blue]
> On May 6, 10:29*pm, lyndonbart...@yahoo.com wrote:
>[color=green]
> > Hello all,[/color]
>[color=green]
> > I'm working on gathering disk stats. I've used sys$getdviw to gather
> > freeblocks, maxblocks, maxfiles, errorcount, mountcount...[/color][/color]

Along with the cluster size, that's all you need to know to read the
header-bitmap
in a single QIO, or IO_PERFORM, from INDEXF.SYS.

Open the file by FID (1,1) or better still, keep it open.

[color=blue]
> First, I thank all of you for your responses. Especially Jim Duff, I
> found your sight, and your examples have been invaluable...![/color]

"website" or "insight" would work.
[color=blue]
> Oh, did I mention it has to be fast???
> T4, by default, samples every second, nor every second.[/color]

Isn't it every minute?

Anyway, either rule out any [*...] walk.

IMHO it also rules out opening INDEXF.SYS for write,
because I would not want (risk) flush those caches every second.

I would just open for read and accept any inaccuracies.
They can (rightfully!) be explained as 'active files'.
[color=blue]
> This is the last piece of the performance data I want to gather, it's
> a "It'd be nice to have." If I can't gather this data quickly and
> efficiently, I can leave it out of the final product.[/color]

You could perhaps fake it (cache it!) somewhat.

Get the easy (GETDVI) stuff every interval.
Read the bitmaps every 30 or 60 intervals and just repeat the same
info 29 (59) times more.

Bob K> The bitmap will show allocated blocks that are actually
preallocated

Bob, it seems to me you are referring to the STORAGE bitmap from
BITMAP.SYS where
every bit represents and allocation cluster.
We are talking about the header bitmap, which lives inside INDEXF.SYS
and where
each set bit represents a free block in that file and each cleared bit
represents a block
used for a header or saved in the cache.

Hein.

>...[color=blue]
> As others have stated, I don't need an absolutely perfectly
> accurate
> number. Inaccuracy due to caching, etc. is acceptable. I simply
> want a
> real close number I can track.
>
> Oh, did I mention it has to be fast???
>
> T4, by default, samples every second, so I need to be able loop
> through all the disks on a system, and get this information (If
> it's
> mounted) and write this info to an output csv file every
> sampling.
>...[/color]

Just because T4 records some information fast (I have not checked
but I do not remember any T4 data being sampled every second, but
it has been a couple of years since I last did anything serious
with T4) doesn't mean that every piece of information needs to be
collected that fast. I sample the disk space information once a
day using T4 and that is good enough for my application, other
sites may want the information recorded less often or more often.


Peter Weaver
[url]www.WeaverConsulting.ca[/url] [url]www.OpenVMSvirtualization.com[/url]
[url]www.VAXvirtualization.com[/url] [url]www.AlphaVirtualization.com[/url]

[color=blue][color=green]
> > Oh, did I mention it has to be fast???
> > T4, by default, samples every second, nor every second.[/color]
>
> Isn't it every minute?
>[/color]


You're right.. My typo. I meant every minute.


But my earlier desires for high speed and low I/O still apply. I do
NOT want the process of collecting data become a performance burden. I
don't want T4 collecting to become a statistic on the performance
graphs.

Hein RMS van den Heuvel wrote:[color=blue]
> On May 7, 9:38 am, lyndonbart...@yahoo.com wrote:[color=green]
>> On May 6, 10:29 pm, lyndonbart...@yahoo.com wrote:
>>[color=darkred]
>>> Hello all,
>>> I'm working on gathering disk stats. I've used sys$getdviw to gather
>>> freeblocks, maxblocks, maxfiles, errorcount, mountcount...[/color][/color]
>
> Along with the cluster size, that's all you need to know to read the
> header-bitmap
> in a single QIO, or IO_PERFORM, from INDEXF.SYS.
>
> Open the file by FID (1,1) or better still, keep it open.
>
>[color=green]
>> First, I thank all of you for your responses. Especially Jim Duff, I
>> found your sight, and your examples have been invaluable...![/color]
>
> "website" or "insight" would work.
>[color=green]
>> Oh, did I mention it has to be fast???
>> T4, by default, samples every second, nor every second.[/color]
>
> Isn't it every minute?
>
> Anyway, either rule out any [*...] walk.
>
> IMHO it also rules out opening INDEXF.SYS for write,
> because I would not want (risk) flush those caches every second.
>
> I would just open for read and accept any inaccuracies.
> They can (rightfully!) be explained as 'active files'.
>[color=green]
>> This is the last piece of the performance data I want to gather, it's
>> a "It'd be nice to have." If I can't gather this data quickly and
>> efficiently, I can leave it out of the final product.[/color]
>
> You could perhaps fake it (cache it!) somewhat.
>
> Get the easy (GETDVI) stuff every interval.
> Read the bitmaps every 30 or 60 intervals and just repeat the same
> info 29 (59) times more.
>
> Bob K> The bitmap will show allocated blocks that are actually
> preallocated
>
> Bob, it seems to me you are referring to the STORAGE bitmap from
> BITMAP.SYS where
> every bit represents and allocation cluster.
> We are talking about the header bitmap, which lives inside INDEXF.SYS
> and where
> each set bit represents a free block in that file and each cleared bit
> represents a block
> used for a header or saved in the cache.
>[/color]

Of course, knowing the speed requirements for this now, doing
lib$find_file or alternatively, sys$search (so we can eliminate aliased
files) is out of the question.

However, reading the bitmap is only going to give us an approximation as
well, as the bitmap does not distinguish between primary and extension
headers.

To get an accurate number, you would have to take out the volume
serialization lock, read the bitmap, and read each in-use header to see
if it's a primary header, then release the volume serialization lock.

Cheers,
Jim.
--
[url]www.eight-cubed.com[/url]

On May 7, 5:55*pm, Jim Duff <spam.t...@127.0.0.1> wrote:
:[color=blue]
> Cheers,
> Jim.
> --www.eight-cubed.com- Hide quoted text -[/color]

Jim, I tried to Email you but got a 'Mailbox full' message?
Fix that or send me an Email from an other account to reply to?
Nothing special, cheers, Hein.

Hein RMS van den Heuvel wrote:[color=blue]
> On May 7, 5:55 pm, Jim Duff <spam.t...@127.0.0.1> wrote:
> :[color=green]
>> Cheers,
>> Jim.
>> --www.eight-cubed.com- Hide quoted text -[/color]
>
> Jim, I tried to Email you but got a 'Mailbox full' message?
> Fix that or send me an Email from an other account to reply to?
> Nothing special, cheers, Hein.
>[/color]

Fixed. Thanks for mentioning this. I hate spam. I especially hate
spam when I get 4000+ of them in less than an hour :-(

Cheers,
Jim.
--
[url]www.eight-cubed.com[/url]

On May 7, 5:55 pm, Jim Duff <spam.t...@127.0.0.1> wrote:[color=blue]
> Hein RMS van den Heuvel wrote:
>
>
>[color=green]
> > On May 7, 9:38 am, lyndonbart...@yahoo.com wrote:[color=darkred]
> >> On May 6, 10:29 pm, lyndonbart...@yahoo.com wrote:[/color][/color]
>[color=green][color=darkred]
> >>> Hello all,
> >>> I'm working on gathering disk stats. I've used sys$getdviw to gather
> >>> freeblocks, maxblocks, maxfiles, errorcount, mountcount...[/color][/color]
>[color=green]
> > Along with the cluster size, that's all you need to know to read the
> > header-bitmap
> > in a single QIO, or IO_PERFORM, from INDEXF.SYS.[/color]
>[color=green]
> > Open the file by FID (1,1) or better still, keep it open.[/color]
>[color=green][color=darkred]
> >> First, I thank all of you for your responses. Especially Jim Duff, I
> >> found your sight, and your examples have been invaluable...![/color][/color]
>[color=green]
> > "website" or "insight" would work.[/color]
>[color=green][color=darkred]
> >> Oh, did I mention it has to be fast???
> >> T4, by default, samples every second, nor every second.[/color][/color]
>[color=green]
> > Isn't it every minute?[/color]
>[color=green]
> > Anyway, either rule out any [*...] walk.[/color]
>[color=green]
> > IMHO it also rules out opening INDEXF.SYS for write,
> > because I would not want (risk) flush those caches every second.[/color]
>[color=green]
> > I would just open for read and accept any inaccuracies.
> > They can (rightfully!) be explained as 'active files'.[/color]
>[color=green][color=darkred]
> >> This is the last piece of the performance data I want to gather, it's
> >> a "It'd be nice to have." If I can't gather this data quickly and
> >> efficiently, I can leave it out of the final product.[/color][/color]
>[color=green]
> > You could perhaps fake it (cache it!) somewhat.[/color]
>[color=green]
> > Get the easy (GETDVI) stuff every interval.
> > Read the bitmaps every 30 or 60 intervals and just repeat the same
> > info 29 (59) times more.[/color]
>[color=green]
> > Bob K> The bitmap will show allocated blocks that are actually
> > preallocated[/color]
>[color=green]
> > Bob, it seems to me you are referring to the STORAGE bitmap from
> > BITMAP.SYS where
> > every bit represents and allocation cluster.
> > We are talking about the header bitmap, which lives inside INDEXF.SYS
> > and where
> > each set bit represents a free block in that file and each cleared bit
> > represents a block
> > used for a header or saved in the cache.[/color]
>
> Of course, knowing the speed requirements for this now, doing
> lib$find_file or alternatively, sys$search (so we can eliminate aliased
> files) is out of the question.
>
> However, reading the bitmap is only going to give us an approximation as
> well, as the bitmap does not distinguish between primary and extension
> headers.
>
> To get an accurate number, you would have to take out the volume
> serialization lock, read the bitmap, and read each in-use header to see
> if it's a primary header, then release the volume serialization lock.
>
> Cheers,
> Jim.
> --www.eight-cubed.com[/color]

But isn't the number of headers what he really needs? Isn't that what
he's worried of running out of? My question is: How does he get so
many headers in the first place?

AEF

"Bob Koehler" <koehler@eisner.nospam.encompasserve.org> wrote in message
news:rAPCDs2gP637@eisner.encompasserve.org...[color=blue]
> In article[/color]
<8bc540b0-5e7c-4853-9ada-6e255430a87a@e53g2000hsa.googlegroups.com>,
IanMiller <gxys@uk2.net> writes:[color=blue][color=green]
> > I was wondering if, due to caching, could the bitmap be out of date?
> >[/color]
>
> The bitmap on the platters is always out of date when the disk is[/color]
mounted[color=blue]
> unless /nowrite was used in the mount (or hardware write lock is used).
>
> The bitmap will show allocated blocks that are actually preallocated
> (in the cache for future use). The bitmap is corrected on proper
> dismount, mount without /norebuild, or an explicit set volume/rebuild.
>
> So you don't have to worry about blocks in use that are marked free,
> but if you need a rebuild and haven't done it yet there are blocks
> marked used that aren't in use.[/color]

It's been a while since I read the fiche on this, but back when I was
writing utilities to read & modify the Files-11 disk metadata files, opening
them for write caused them to flush their contents to disk. Don't know if
this is true past V4.4...it's been a while. Naturally, if you open them for
write, you have to Be Careful.
--
Lee K. Gleason N5ZMR
Control-G Consultants
[email]lee.gleason@comcast.net[/email]