Re: Too many files in one directory (again) - VMS

This is a discussion on Re: Too many files in one directory (again) - VMS ; From: Hein RMS van den Heuvel > > =A0 =A0So, it couldn't be improved to handle this case better? > > Yes. Maybe, Just maybe, Andy will get to release a b-tree based > implementation. I'm ready. > > =A0 ...

+ Reply to Thread
Results 1 to 2 of 2

Thread: Re: Too many files in one directory (again)

  1. Re: Too many files in one directory (again)

    From: Hein RMS van den Heuvel

    > > =A0 =A0So, it couldn't be improved to handle this case better?

    >
    > Yes. Maybe, Just maybe, Andy will get to release a b-tree based
    > implementation.


    I'm ready.

    > > =A0 =A0I didn't. =A0I didn't realize how doomed I was until I stepped well=

    > into this tar-pit.
    >
    > Oh yeah. Classic. Can't kill it now.. it must almost be done. :-)
    > Just wait till you get to delete them!


    I started out running it on my main XP1000 system. When that got
    into trouble, I fired up the zx2000, which soon got ahead of the Alpha,
    so I killed the job there. Only about 50k files at that time. DFU
    DELETE got rid of them pretty quickly. (I hadn't installed DFU on the
    zx2000 yet, but it's there now.)

    > > > > [...] =A0Even growing the allocation by, say, 25%
    > > > > instead of one block might pay off without a big penalty

    >
    > Mark Hopkins whisperred to me
    > "1) A directory extends by the maximum of 100 blocks or 1/2 the
    > current
    > file size (I just looked in the code). I'm pretty sure that this is
    > rounded up to the next cluster factor. None of the default extends
    > are honored here.


    That sounds good, but as I showed before, I wasn't seeing anything
    bigger than that 26384 - 26272 = 112, which is the smallest N* C_F >=
    100. No sigh of any "1/2 the current file size" here.

    > Steven>> it should also (now) be using 127
    > for the multi-block count ("fil_rab.rab$b_mbc"), and using two
    > buffers
    > ("fil_rab.rab$b_mbf"), and setting write-behind ("fil_rab.rab$l_rop |
    > =3D
    > RAB$M_WBH;") on
    >
    > Some argue to make it a multiple of 4 (124) to make live easier on
    > certain storage types (EVA Raid-5).
    > [...]


    The RMS parameter code here (which I use everywhere) will use any
    non-default SET RMS parameters set by the user instead of its own
    internal defaults. (And it's all open-source.)

    For the record:

    IT $ pipe t ; vmstar xf zip.tar ; t
    20-MAR-2008 13:43:57
    IT::_RTA3: 21:27:39 VMSTAR CPU=01:54:16.82 PF=1182 IO=44381972 MEM=357
    IT::_RTA3: 23:33:39 VMSTAR CPU=02:36:08.14 PF=1182 IO=57161427 MEM=357
    IT::_RTA3: 09:19:04 VMSTAR CPU=05:55:49.96 PF=1182 IO=120353399 MEM=357
    IT::_RTA3: 15:33:39 VMSTAR CPU=08:08:25.16 PF=1182 IO=163580113 MEM=357
    IT::_RTA3: 21:10:42 VMSTAR CPU=10:07:42.02 PF=1182 IO=202912695 MEM=357
    IT::_RTA3: 23:47:42 VMSTAR CPU=11:02:55.68 PF=1182 IO=221024209 MEM=357
    22-MAR-2008 00:35:48
    IT $
    IT::_RTA3: 08:25:11 (DCL) CPU=11:18:33.07 PF=1344 IO=225889468 MEM=247

    So, it took only about 35 hours on a (newer, faster) VMS system to do
    what took about 35 minutes on the HP-UX system.

    Grand total of 1 directory, 189833 files, 1076441 blocks.


    This test case is easy to get, if anyone wants to share the fun.

    ------------------------------------------------------------------------

    Steven M. Schweda sms@antinode-org
    382 South Warwick Street (+1) 651-699-9818
    Saint Paul MN 55105-2547

  2. Re: Too many files in one directory (again)

    On Mar 22, 9:56*am, s...@antinode.org (Steven M. Schweda) wrote:
    > From: Hein RMS van den Heuvel


    > > Mark Hopkins whisperred to me
    > > "1) A directory extends by the maximum of 100 blocks or 1/2 the
    > > current * file size (I just looked in the code). *I'm pretty sure that this is
    > > * rounded up to the next cluster factor


    > * *That sounds good, but as I showed before, I wasn't seeing anything
    > bigger than that 26384 - 26272 = 112, which is the smallest N* C_F >=
    > 100. *No sigh of any "1/2 the current file size" here.


    I thought the description a little odd, so that's why I quoted it.
    It must be the minimum or 100 and 1/2 current, rounded up to cluster
    size.

    > > Steven>> it should also (now) be using 127
    > > for the multi-block count ("fil_rab.rab$b_mbc"), and using two

    :
    > * *The RMS parameter code here (which I use everywhere) will use any
    > non-default SET RMS parameters set by the user instead of its own
    > internal defaults. *(And it's all open-source.)


    Super!
    btw... beware of the plumbing: pipes/paren(thesi)s. Compare
    $SET RMS/IND/BUF=123
    $PIPE SHOW RMS
    ... Process 0 | 123
    $ PIPE SHOW RMS | SEARCH SYS$PIPE Pro,"|"/MATCH=AND
    Process 0 | 0
    $PIPE ( SHOW RMS )
    ... Process 0 | 0

    > * *For the record:
    > IT $ pipe t ; vmstar xf zip.tar ; t

    :
    > IT::_RTA3: 08:25:11 * (DCL) * CPU=11:18:33.07 PF=1344 IO=225,889,468 MEM=247


    That's a big number. 1189 IOs per file. 75 IOs per second.
    If we assume that IOs are synchroneous and that the job does not
    trigger parallel IO,
    then we should discount for 11 hours CPU time used and come to 100 IO/
    second
    That's not great, but not unreasonable either for a single disk
    randomly accessed with short IOs.

    The directory expansion shuffle does NOT explain that number.
    Best I can tell that should acount for 'only' 170 * 594 = 100,000 IO.
    - number of shuffles = 19000/112 = 170 (right?)
    - average IO per shuffle = 2 * (19000/2) / 32 = 594 (1 read, 1 write,
    many times. right ?)
    So pre-allocating would not have helped much/any, other then taking
    away rampant speculation.

    The only hint we have to explain the high IO rats is the Dir Data
    Attempt Rate @ 6700.
    Apply those 70% hits and that leaves a potential for 2000 IO/sec.

    Was that a quick worst-case picture, or over minutes ?
    Unfortunately the MONITOR output sorely lacks in the space.
    Plenty of room on that screen! T4 would help there.

    The displayed XFC stats do not help explain that 2000 number back to
    75 IO/sec

    > So, it took only about 35 hours on a (newer, faster) VMS system to do
    > what took about 35 minutes on the HP-UX system.


    And that's what hurts and needs to be pointed out.
    Thank you for the topic!

    Hein.

+ Reply to Thread