Re: Too many files in one directory (again) - VMS

This is a discussion on Re: Too many files in one directory (again) - VMS ; From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?= > > I just pass this along as a reminder that there's still some room > > for improvement in dealing with cases like this which,... > > And some of them are under your control, such as ...

+ Reply to Thread
Results 1 to 4 of 4

Thread: Re: Too many files in one directory (again)

  1. Re: Too many files in one directory (again)

    From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?=

    > > I just pass this along as a reminder that there's still some room
    > > for improvement in dealing with cases like this which,...

    >
    > And some of them are under your control, such as using a large
    > enough /ALLOCATION=n on the CRE/DIR command. I guess that it
    > would also help to INIT the device with a large enought
    > /HEADERS=n to begin with.


    Probably would. Drawbacks of this (and several other potentially
    helpful suggestions) is that _I_ didn't create the directory, VMSTAR
    did. The archive entries look like
    "zip/020989f4d6c2f32768d0535c1815344d.zip". Now I could, in principle,
    analyze the whole "tar" archive, puzzle out the optimal directory
    characteristics, and pre-create the things myself rather than trusting
    VMSTAR to know how to create directories. How many people believe that
    a user should need to go through that ordeal to avoid this problem?

    > Even if 190.000 files "works" on VMS, it might not be what
    > VMS was primarily designed for...


    So, it couldn't be improved to handle this case better?

    > How large/small is each individual file ?


    Pretty small, I believe.

    > Would it be possible (with 4 Gb available) to
    > create a DECram disk and run the un-tar against
    > that ?


    May be, but it'd probably be tight. And this is all on a system
    which I don't keep powered on constantly.


    From: "Main, Kerry"

    > Just a WAG, but since you are doing mostly write activities, can we assume
    > That you have removed disk highwater marking and set OpenVMS to the file
    > system default That UNIX uses (write back) vs OpenVMS's file system default
    > (write through)?


    I didn't. I didn't realize how doomed I was until I stepped well
    into this tar-pit.

    Volume Status: ODS-5, subject to mount verification, protected subsystems
    enabled, file high-water marking, write-through caching enabled, hard
    links enabled.

    When I get a chance, I'll try to look into VMSTAR to see if it is (or
    could be) using SQO to help avoid highwater marking trouble. In the
    past, it only bothered me when allocating space for a big file, and I
    believe that the files themselves are all pretty small. Should I still
    worry?


    From: Hein RMS van den Heuvel

    > > [...] Even growing the allocation by, say, 25%
    > > instead of one block might pay off without a big penalty

    >
    > It will be growing by at least a disk cluster at a time.
    > I don't think it'll honor the SET RMS/EXT


    It's probably the default for 143374741 blocks (SEAGATE ST373405LW)
    ODS5:
    Cluster size 16
    but directory size growth seems to be better than that:

    IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir
    21-MAR-2008 15:23:00
    IT$DKA100:[cert]zip.DIR;1
    26272/26272
    IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir
    21-MAR-2008 15:23:04
    IT$DKA100:[cert]zip.DIR;1
    26273/26384

    Noticable (ten second?) pause waiting for that "26273/26384".

    > MONI FILE would be interesting


    FILE SYSTEM CACHING STATISTICS
    on node IT
    21-MAR-2008 15:18:54.22

    CUR AVE MIN MAX

    Dir FCB (Hit %) 100.00 100.00 100.00 100.00
    (Attempt Rate) 10.66 10.22 10.00 10.66
    Dir Data (Hit %) 70.00 70.66 70.00 72.00
    (Attempt Rate) 6701.00 6738.11 6330.33 7183.00
    File Hdr (Hit %) 89.00 88.66 88.00 89.00
    (Attempt Rate) 9.66 9.33 9.00 9.66
    File ID (Hit %) 100.00 100.00 100.00 100.00
    (Attempt Rate) 1.00 1.00 1.00 1.00

    Extent (Hit %) 100.00 100.00 100.00 100.00
    (Attempt Rate) 1.00 1.00 1.00 1.00
    Quota (Hit %) 0.00 0.00 0.00 0.00
    (Attempt Rate) 0.00 0.00 0.00 0.00
    Bitmap (Hit %) 0.00 0.00 0.00 0.00
    (Attempt Rate) 0.00 0.00 0.00 0.00


    IT $ sho mem /cac
    System Memory Resources on 21-MAR-2008 15:21:07.91

    Extended File Cache (Time of last reset: 20-MAR-2008 05:59:48.57)
    Allocated (GBytes) 1.48 Maximum size (GBytes) 1.99
    Free (GBytes) 0.00 Minimum size (GBytes) 0.00
    In use (GBytes) 1.48 Percentage Read I/Os 22%
    Read hit rate 78% Write hit rate 0%
    Read I/O count 75035 Write I/O count 258987
    Read hit count 58781 Write hit count 0
    Reads bypassing cache 81 Writes bypassing cache 1521
    Files cached open 548 Files cached closed 18757
    Vols in Full XFC mode 0 Vols in VIOC Compatible mode 4
    Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

    (Must be that lack of "Full XFC mode". Tee-hee-hee.)

    If I could find some dirt-cheap 2GB memory modules, I'd double it again
    to 8GB, but 4GB is all the budget would allow.


    > btw, do a semi random:
    > $ pipe dump/dir/blo=3D(star=3D10000,count=3D10) bad.dir | searc sys$pipe
    > "End of records"


    IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir
    21-MAR-2008 15:33:24
    IT$DKA100:[cert]zip.DIR;1
    26376/26384

    IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir | sear sys$
    pipe "End of records"
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)

    IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir
    21-MAR-2008 15:48:19
    IT$DKA100:[cert]zip.DIR;1
    26523/26608

    IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir | sear sys$
    pipe "End of records"
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)
    012C End of records (-1)

    Seems pretty stable at "012C".


    From: koehler@eisner.nospam.encompasserve.org (Bob Koehler)

    > Can you put a listing in a file using tar -t and then break down
    > the output into multiple tar passes into multiple directories?


    And how should I specify the division of labor to VMSTAR?

    > I sure think I'd try that if I were in your situation. A little
    > time editing to create a script might save a great many hours.


    Complaining can be more fun than working, and (in my dreams) could be
    more likely to lead to some actual improvement than quietly finding some
    annoying work-around.


    From: JF Mezei

    > TAR unpacks in mydir1.dir
    > You create mydir2 through mydir200
    >
    > You have a separate process which runs say every minute, and does a
    > RENAME [.mydir1]*.* [.mydirX] where X increases with every run and goes
    > back to 2 after 200.


    Or I could repackage the stuff in smaller pieces on the HP-UX system,
    or I could try breaking NFS by getting the files served from the HP-UX
    system, ...

    Many things are possible. Most of them, I claim, shouldn't be
    needed.


    From: "Richard B. Gilbert"

    > A sane VMS user, rather than create 190,000 files, might consider an
    > indexed sequential file to hold the information. [...]


    I didn't design this exercise. Someone else created a
    dump-truck-full of Zip (and other compress/archive) test files, mostly
    diddled in creative ways (they say) in an attempt to find problems in
    various expand/extract software. The packaging was probably done on a
    Linux system, where, I'd assume, no one saw any particular problems with
    the scheme. I just thought that it might be productive to run the stuff
    through the next version of UnZip to see if we had any problems on VMS.

    ------------------------------------------------------------------------

    Steven M. Schweda sms@antinode-org
    382 South Warwick Street (+1) 651-699-9818
    Saint Paul MN 55105-2547

  2. RE: Too many files in one directory (again)


    > -----Original Message-----
    > From: Steven M. Schweda [mailto:sms@antinode.org]
    > Sent: March 21, 2008 3:46 PM
    > To: Info-VAX@Mvb.Saic.Com
    > Subject: Re: Too many files in one directory (again)
    >


    [snip ..]

    >
    > From: "Main, Kerry"
    >
    > > Just a WAG, but since you are doing mostly write activities, can we

    > assume
    > > That you have removed disk highwater marking and set OpenVMS to the

    > file
    > > system default That UNIX uses (write back) vs OpenVMS's file system

    > default
    > > (write through)?

    >
    > I didn't. I didn't realize how doomed I was until I stepped well
    > into this tar-pit.
    >
    > Volume Status: ODS-5, subject to mount verification, protected
    > subsystems
    > enabled, file high-water marking, write-through caching enabled,
    > hard
    > links enabled.
    >
    > When I get a chance, I'll try to look into VMSTAR to see if it is (or
    > could be) using SQO to help avoid highwater marking trouble. In the
    > past, it only bothered me when allocating space for a big file, and I
    > believe that the files themselves are all pretty small. Should I still
    > worry?
    >
    >


    Might not be applicable, but perhaps worth a try to see if the process
    appears to speed up some:

    Disk high water marking can be changed dynamically without dismounting the
    drive, so while you might be afraid to stop the job while it is running,
    you should be able to change it without impacting the job by entering:
    $ set volume $1$duax /nohighwater

    The same is true for modifying the sysgen parameter RMS_SEQFILE_WBH

    Reference:
    sysgen> help sys_p RMS_SEQFILE_WBH
    (Alpha and I64) RMS_SEQFILE_WBH can enable the RMS write behind feature
    as a system default for any unshared sequential disk file if the file
    is opened for image I/O with write access specified.

    [snip ..]



    Regards

    Kerry Main
    Senior Consultant
    HP Services Canada
    Voice: 613-254-8911
    Fax: 613-591-4477
    kerryDOTmainAThpDOTcom
    (remove the DOT's and AT)

    OpenVMS - the secure, multi-site OS that just works.





  3. Re: Too many files in one directory (again)

    In article <08032114454569_2020CE0A@antinode.org>, sms@antinode.org (Steven M. Schweda) writes:
    > From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?=
    >
    >> > I just pass this along as a reminder that there's still some room
    >> > for improvement in dealing with cases like this which,...

    >>
    >> And some of them are under your control, such as using a large
    >> enough /ALLOCATION=n on the CRE/DIR command. I guess that it
    >> would also help to INIT the device with a large enought
    >> /HEADERS=n to begin with.

    >
    > Probably would. Drawbacks of this (and several other potentially
    > helpful suggestions) is that _I_ didn't create the directory, VMSTAR
    > did. The archive entries look like
    > "zip/020989f4d6c2f32768d0535c1815344d.zip". Now I could, in principle,
    > analyze the whole "tar" archive, puzzle out the optimal directory
    > characteristics, and pre-create the things myself rather than trusting
    > VMSTAR to know how to create directories.


    The nice part of that approach is that from the general description
    of the problem it would seem you would not have to create many
    directories :-)

  4. Re: Too many files in one directory (again)

    On Mar 21, 3:45*pm, s...@antinode.org (Steven M. Schweda) wrote:
    > From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?=
    >
    > > > * *I just pass this along as a reminder that there's still some room
    > > > for improvement in dealing with cases like this which,...


    > > Even if 190.000 files "works" on VMS, it might not be what
    > > VMS was primarily designed for...

    >
    > * *So, it couldn't be improved to handle this case better?


    Yes. Maybe, Just maybe, Andy will get to release a b-tree based
    implementation.

    > > create a DECram disk and run the un-tar against that ?

    >
    > * *May be, but it'd probably be tight. *And this is all on a system
    > which I don't keep powered on constantly


    Create an LD container first. Copy the container to a the real disk.

    > * *I didn't. *I didn't realize how doomed I was until I stepped wellinto this tar-pit.


    Oh yeah. Classic. Can't kill it now.. it must almost be done. :-)
    Just wait till you get to delete them!

    > > > [...] *Even growing the allocation by, say, 25%
    > > > instead of one block might pay off without a big penalty


    Mark Hopkins whisperred to me
    "1) A directory extends by the maximum of 100 blocks or 1/2 the
    current
    file size (I just looked in the code). I'm pretty sure that this is
    rounded up to the next cluster factor. None of the default extends
    are honored here.

    2) If someone hasn't already suggested it, increase acp_maxread from
    32 to 64.

    3) Turn on RMS_SEQFILE_WBH if possible."


    > Noticable (ten second?) pause waiting for that "26273/26384".


    That suggests that there was no free space found next to the current
    allocation
    and the directory was copied ACP_MAXREAD blocks at a time
    I made the DUMP/HEAD | grep LBN suggestion to check just that.
    Pre-allocation would have prevented that delay.
    Suspend the job (if not done by now), use DFU to expand, resume.

    > > MONI FILE would be interesting

    > * * * * * * * * * * * * *FILE SYSTEM CACHING STATISTICS
    > * * * * * * * * * * * * * * * * * * * *CUR * * * *AVE * * * *MIN * * * *MAX
    > * * Dir Data * (Hit %) * * * * * * * 70.00 * * *70.66 * * *70.00 * * *72.00
    > * * * * * * * *(Attempt Rate) * * *6701.00 * *6738.11 * *6330.33 * *7183.00

    Yikes! Is is doing some readdir before doing the create files?


    > IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir| sear sys$
    > pipe "End of records"
    > 012C *End of records (-1)

    :
    > 012C *End of records (-1)
    > Seems pretty stable at "012C".


    So all blocks are nicely filled, suggesting files are nicely added in
    sequence.
    In that case actuall adding of files 100..200 should be no slower than
    100100 .. 100200.
    But the directory growth / reallocation will be getting progressively
    worse as you noticed.


    > * *Complaining can be more fun than working, and (in my dreams) could be
    > more likely to lead to some actual improvement than quietly finding some
    > annoying work-around.


    Agreed! Too many poor souls out there do not even have the first clue
    to work around this.


    > From: JF Mezei
    >
    > > TAR unpacks in mydir1.dir
    > > You create mydir2 through mydir200

    >
    > > You have a separate process which runs say every minute, and does a
    > > RENAME [.mydir1]*.* [.mydirX] *where X increases with every run and goes
    > > back to 2 after 200.


    clueless

    > * *Many things are possible. *Most of them, I claim, shouldn't be needed.


    Agreed.

    > From: "Richard B. Gilbert"
    >
    > > A sane VMS user, rather than create 190,000 files, might consider an
    > > indexed sequential file to hold the information. *[...]


    Or an indexed file to hold name/fid tupples for directory less
    files. :-)

    JF>> Out of curiosity, why would creating 190,000 files in a VMS
    directory be
    significantly slower than on Unix ?

    Because it uses a writeback model, not write through. That works most
    of the time.
    If 10 entries are made to a directory block, VMS will do 10 actual
    write IO.
    Your typical Unix will just do 1 IO, or none. Depends on the update
    deamon.
    Unix'es also have a variety of filesystems, many log based, making it
    a little safer.

    JF> did you CREATE/DIRECTORY/ALLOCATION=xxxxxxx ?
    JF> Would this have made a huge difference in the file creation
    speed ?

    Yes, with the evidence currently in play.

    JF> Would it make sense to SET FILE mydir.dir;1/global=5000 (or
    whatever) ?

    Absolutely not. For starters, directories do use the normal RMS buffer
    caches.
    But if they did... Global Buffers cache buffers at certain VBN
    location.
    Well, after a few not-at-the-end inserts all blocks are shuffled up.
    So the contents of VBN 10 moves to VBN 11 and so. Not good for a VBN
    tag cache.

    JF> So when I do an "ls" in Unix, it does an in memory sort of the
    directory
    before listing it ?

    Unix has these 'inodes' which are like a mini file header, but they
    live in the directory.
    They hold the 'stat' info: byte-size, dates, owner,...
    So for unix is it trivial to list sizes or present sorted by date.
    OpenVMS has to do 1 IO per file for attributes other than name or file-
    id.
    This IOs may be cached, but still.

    Steven>> it should also (now) be using 127
    for the multi-block count ("fil_rab.rab$b_mbc"), and using two
    buffers
    ("fil_rab.rab$b_mbf"), and setting write-behind ("fil_rab.rab$l_rop |
    =
    RAB$M_WBH;") on

    Some argue to make it a multiple of 4 (124) to make live easier on
    certain storage types (EVA Raid-5).
    Others recommend a multiple of 16 (112) to make life easier for the
    XFC (16 block cache lines)
    RMS WHB is a simple to use way to get concurrent IO going and more
    often than not well worth the CPU price. If CPU usage becomes a
    concern, then self-managed $WRITE, with $RAB64 may become desirable.
    For even less CPU time, go down to QIO and for the least possible CPU
    overhead, $IO_PERFORM.

    2 buffers should be just fine. 4 often helps a little more. Beyond
    that I find rapidly diminishing returns.

    Cheers,
    Hein.

+ Reply to Thread