On Oct 10, 5:50*am, hel...@astro.multiCLOTHESvax.de (Phillip Helbig---
remove CLOTHES to reply) wrote:
> VMS MAIL, of course:
:
> Behind the scenes, is the effect the same (except for the copy of the
> uncompressed file one gets with COMPRESS)?

Compress (straight convert) will be cleaner, tighter.
Purge (convert/reclaim) will only recycle completely empty buckets.
The effect of re-cycle will also cause randomish spread of records
over the VBN range.

> Is there the locked-file problem causing messages to bounce with
> COMPRESS like there is with PURGE/RECLAIM? *If so, is this time always
> shorter than is the case with PURGE/RECLAIM?

I believe the Compress to behave nicer, but have no time to try/ check
back in my old notes. Sorry.
Both reclaim and convert will read each databucket from the file.
Reclaim will then re-write those index and data buckets foudn to be
empty, one at a time.

Convert (8.2+) will build large (127 block) buffers of buckets with
valid records.
Which one is faster clearly depends on the deleted vs retained ratio.
I find a full convert often faster than a significant reclaim and
prefer the result, so a full convert is a no-brainer for me. But I
suppose if you faithfull purge regularly, then it that woudl be
quicker each time ( but slower in aggregation).

> I just moved 220 MB of stuff from one mail file from another. *What
> should I do to "clean up" the file with the deleted stuff?

VMSmail files a basic indexed file and can and could/should be tuned
as such.
While ideally you'd tune each VMSmail file with EDIT/FDL/NOINT based
on the number of records it is expected to have, a more coarse
approach is fine.
I used to use a little script to choose between SMALL, MEDIUM and BIG.
I'll attach it below.

Now for expected to be stable files, you may want to tweak the
allocations towards exact fit. Possibly just one area with an
ALLOCATION of 80% of anticipated need and an extent of 4 or 5%.

*(I'm sure Hein will respond.)

:-).

Regards,
Hein van den Heuvel

$! MAIL_CONVERT.COM Cleans up and optimizes VMSmail indexed files.
$! Hein van den Heuvel, March 1996
$! Oct-2008, Make 'big' FDL bigger.
$
$ BIG = 5000
$ SMALL = 500
$
$! This command file will convert the MAIL.MAI file in the current
$! directory selecting a reasonable FDL file based on the current
size.
$! It will NOT compress as agressivly as standard MAIL COMPRESS would
$! do as it is believed that current mail file size is an indication
$! of the future size.
$! It makes an attempt to protect against incoming mail during the
convert.
$! Such new mail would create a fresh MAIL.MAI file which is merged
in.
$! Hein van den Heuvel
$!
$if f$search("mail.mai").eqs."" then goto help
$open/read/write/erro=help x mail.mai
$close x
$size = f$file ("mail.mai","alq")
$gbc = f$file ("mail.mai","gbc")
$set prot mail.mai;*
$rena/log mail.mai;* mail.old;*
$!
$! Select the convert with appropriate FDL
$!
$if size.gt.big then goto big_fdl
$if size.lt.small then goto small_fdl
$goto medium_fdl
$!
$! Come back here after convert and make sure no new mail has arrived
$!
$newmail:
$set noon
$if f$search("mail.mai").eqs."" then goto nonews
$write sys$output "New MAIL.MAI file appeared. Merging in with .NEW
file."
$convert/merge/stat MAIL.MAI MAIL.NEW
$set prot MAIL.MAI
$delete/conf MAIL.MAI.
$goto newmail
$
$nonews:
$rena/log mail.new mail.mai
$set prot=(ow:rw,sy:rw,gr,wo) mail.mai
$if gbc.gt.0 then set file/global='gbc' mail.mai
$dele/conf mail.old.*
$exit
$
$big_fdl:
$conv/fast/stat/nosort/fdl=sys$input mail.old mail.new
FILE; ORG indexed;
RECORD; FORMAT variable; SIZE 2048
AREA 0; BEST_TRY_CONTIGUOUS yes; BUCKET 16; ALLOCATION 16;
EXTENSION 160;
AREA 1; BEST_TRY_CONTIGUOUS yes; BUCKET 16; ALLOCATION 3200; EXTENSION
6400;
AREA 2; BEST_TRY_CONTIGUOUS yes; BUCKET 16; ALLOCATION 320;
EXTENSION 640;
KEY 0; DATA_KEY_COMPR no; DATA_RECORD_COMPR yes; INDEX_COMPRES no
DATA_AREA 1; DATA_FILL 100; DUPLICATES no; PROLOG 3
INDEX_AREA 0; INDEX_FILL 100; LEVEL1_INDEX_AREA 0
SEG0_LENGTH 8; SEG0_POSITION 0; TYPE bin8
KEY 1; DATA_KEY_COMPR yes; INDEX_COMPR yes; NULL_KEY yes; NULL_VALUE 0
DATA_AREA 2; DATA_FILL 70; CHANGES yes; DUPLICATES yes
INDEX_AREA 2; INDEX_FILL 90; LEVEL1_INDEX_AREA 2
SEG0_LENGTH 39; SEG0_POSITION 9; TYPE string
$
$goto newmail
$
$medium_fdl:
$conv/fast/stat/nosort/fdl=sys$input mail.old mail.new
FILE; ORG indexed;
RECORD; FORMAT variable; SIZE 2048
AREA 0; BEST_TRY_CONTIGUOUS yes; BUCKET 8; ALLOCATION 8; EXTENSION
64;
AREA 1; BEST_TRY_CONTIGUOUS yes; BUCKET 8; ALLOCATION 480; EXTENSION
320;
KEY 0; DATA_KEY_COMPR no; DATA_RECORD_COMPR yes; INDEX_COMPRES no
DATA_AREA 1; DATA_FILL 100; DUPLICATES no; PROLOG 3
INDEX_AREA 0; INDEX_FILL 100; LEVEL1_INDEX_AREA 0
SEG0_LENGTH 8; SEG0_POSITION 0; TYPE bin8
KEY 1; DATA_KEY_COMPR yes; INDEX_COMPR yes; NULL_KEY yes; NULL_VALUE 0
DATA_AREA 1; DATA_FILL 70; CHANGES yes; DUPLICATES yes
INDEX_AREA 1; INDEX_FILL 90; LEVEL1_INDEX_AREA 1
SEG0_LENGTH 39; SEG0_POSITION 9; TYPE string
$
$goto newmail
$
$small_fdl:
$conv/fast/stat/nosort/fdl=sys$input mail.old mail.new
FILE; ORG indexed;
RECORD; FORMAT variable; SIZE 2048
AREA 0; BEST_TRY_CONTIGUOUS yes; BUCKET 6; ALLOCATION 60;
EXTENSION 120;
KEY 0; DATA_KEY_COMPR no; DATA_RECORD_COMPR yes; INDEX_COMPRES no
DATA_FILL 100; INDEX_FILL 100; DUPLICATES no; PROLOG 3
SEG0_LENGTH 8; SEG0_POSITION 0; TYPE bin8
KEY 1; DATA_KEY_COMPR yes; INDEX_COMPR yes; NULL_KEY yes; NULL_VALUE 0
DATA_FILL 70; INDEX_FILL 90; CHANGES yes; DUPLICATES yes
SEG0_LENGTH 39; SEG0_POSITION 9; TYPE string
$
$goto newmail
$
$help:
$type sys$input

MAIL.MAI file note found, or could not be opened exclusivly
(in use).
Please execute this command file while default set to mail
directory.
Exit from any and all active VMSmail/DECwindowsmail processed.

$exit

In article
<708916ee-526b-470d-a0cc-8f2a476aec4b@n1g2000prb.googlegroups.com>, Hein
RMS van den Heuvel writes:

> Compress (straight convert) will be cleaner, tighter.

OK, doing COMPRESS now. Will probably take another 15 hours.

> > Is there the locked-file problem causing messages to bounce with
> > COMPRESS like there is with PURGE/RECLAIM? If so, is this time always
> > shorter than is the case with PURGE/RECLAIM?
>
> I believe the Compress to behave nicer, but have no time to try/ check
> back in my old notes. Sorry.

The PURGE/RECLAIM will lock it for the whole time during the reclaim.
COMPRESS makes a copy then compresses the copy before renaming it to
MAIL.MAI. So, the time the file is blocked is MUCH less than with
PURGE/RECLAIM (at least for a large number of deleted messages).

> I used to use a little script to choose between SMALL, MEDIUM and BIG.
> I'll attach it below.

Thanks. I'm doing a big reorg now, so the file will shrink
significantly. After the reorg, the size should stay roughly constant,
so in future I will maintain it with your procedure.

> OK, doing COMPRESS now. *Will probably take another 15 hours.

How can that be ????

Use ^T to see where it is
and again 1 minute later.,
Verify with the PROCIO SDA extention if you feel like seeing details.

And then stop it.
Surely it is nickle and diming the output file. (MONI FILE,FCP,MODE)
Pre-allocate and be happy!

The bucket size is probably just 5 blocks (the mimimum).
So 220MB = 440K blocks = 90,000 read IOs at the most.
Give it 100 IO/sec --> 900 seconds = 15 minutes to read ?
Another 5 minutes to write, 1 minute for the secondary key.

Shouldn't take more then 1/2 hour for an 220MB VMSmail file...
with proper pre-allocation and CONVERT/FAST.
I would really expect less than 10 minutes on any storage from this
century.

fwiw,
Hein.

Phillip Helbig---remove CLOTHES to reply wrote:

> PURGE
>
> /RECLAIM
>
> Releases deleted message space back to OpenVMS RMS for reuse.


HELP CONVERT/RECLAIM

If you have 5 records in a bucket and delete all of them, then the
bucket remains marked as in use. New records can be placed in that
bucket only if they fit the key range of the index entry pointing to
that bucket.


CONVERT/RECLAIM spot all those buckets that are in the index but contain
only deleted records and mark those buckets as free (and I assume will
update the index to give the previous bucket a new key range to include
the range of the former bucket, is that correct ?)

This work is done in-situ with the file opened for read/write with
exclusive status. Nobody can insert new records in it while you
re-arrange the deck chairs in the index structure.

This is the moral equivalent of emptying the trash can on a desktop.




> COMPRESS
>
> Makes an indexed mail file smaller by recovering space that is
> left by deleted mail messages. When you compress a file, the
> following steps occur:

This is a CONVERT/FDL operation. It renames your MAIL.MAI file to
MAIL.OLD, then it does a CONVERT MAIL.OLD to MAIL.MAI. This creates a
new empty file and copies record by record from the old to the new file.

and then tries to delete MAIL.OLD (which usually fails because those
files are created by default so that the user can't delete them)


This is the moral equivalent of using BACKUP/IMAGE to defragment your
disk. This takes a lot more time than the convert/reclaim which is
generally very fast.

Reception of new mail during this time is problematic. While CONVERT
write the new MAIL.MAI, incoming new mail is not possible. If you get
mail betwene the time the old file has been renamed and the new file
created, than MAIL will create a new MAIL.MAI, and the CONVERT may
create a MAIL.MAI;2

In article
<1125b6f0-28fb-4183-a734-ef1e06b39261@40g2000prx.googlegroups.com>, Hein
RMS van den Heuvel writes:

> > OK, doing COMPRESS now. Will probably take another 15 hours.
>
> How can that be ????

Big file, slow system. :-) I did delete aboutu 220 MB of stuff from
the file.

> Use ^T to see where it is
> and again 1 minute later.,
> Verify with the PROCIO SDA extention if you feel like seeing details.

This is the COMPRESS from within MAIL on 7.3-2. Control-T just has the
standard output. (Control-T on PURGE/RECLAIM does show more stuff, at
least sometimes (just checked and it didn't).)

> I would really expect less than 10 minutes on any storage from this
> century.

The disk is a shadow set (10 MB/s SCSI) mounted on two VAXes in the
cluster. The command is running on one of them, a VAXstation 4000/90.
So all writes are going to both disks over the LAN.

In article <48ef875a$0$9644$c3e8da3@news.astraweb.com>, JF Mezei
writes:

> This is a CONVERT/FDL operation. It renames your MAIL.MAI file to
> MAIL.OLD, then it does a CONVERT MAIL.OLD to MAIL.MAI. This creates a
> new empty file and copies record by record from the old to the new file.
>
> and then tries to delete MAIL.OLD (which usually fails because those
> files are created by default so that the user can't delete them)

Are you sure about the delete stuff? I just tried it from an account
with all privs enabled, and it DOESN'T delete the file. There is also
no indication that it tried.

> Reception of new mail during this time is problematic. While CONVERT
> write the new MAIL.MAI, incoming new mail is not possible. If you get
> mail betwene the time the old file has been renamed and the new file
> created, than MAIL will create a new MAIL.MAI, and the CONVERT may
> create a MAIL.MAI;2

Hein's procedure handles this much better. His idea should be
integrated into VMS MAIL. When I get rich and buy all the old DEC
stufuf from HP (or whoever owns it when I get rich), I'll hire him to do
it!

In article , helbig@astro.multiCLOTHESvax.de
(Phillip Helbig---remove CLOTHES to reply) writes:

> > I would really expect less than 10 minutes on any storage from this
> > century.
>
> The disk is a shadow set (10 MB/s SCSI) mounted on two VAXes in the
> cluster. The command is running on one of them, a VAXstation 4000/90.
> So all writes are going to both disks over the LAN.

It is definitely from the LAST century:

device type SEAGATE SX910800N

which is, I guess, from sometime in the 1990s. When old hardware stops
working, I do replace it, usually with new stuff, but my Seagate disks
and VAX stuff is so well built that some of it has been quite literally
running practically continuously for 20 years!

In article , helbig@astro.multiCLOTHESvax.de
(Phillip Helbig---remove CLOTHES to reply) writes:

> In article , helbig@astro.multiCLOTHESvax.de
> (Phillip Helbig---remove CLOTHES to reply) writes:
>
> > > I would really expect less than 10 minutes on any storage from this
> > > century.
> >
> > The disk is a shadow set (10 MB/s SCSI) mounted on two VAXes in the
> > cluster. The command is running on one of them, a VAXstation 4000/90.
> > So all writes are going to both disks over the LAN.
>
> It is definitely from the LAST century:
>
> device type SEAGATE SX910800N
>
> which is, I guess, from sometime in the 1990s. When old hardware stops
> working, I do replace it, usually with new stuff, but my Seagate disks
> and VAX stuff is so well built that some of it has been quite literally
> running practically continuously for 20 years!

My mistake. It's the PURGE which will take about 15 hours! After that
I'll do the COMPRESS!

On Oct 10, 12:45*pm, hel...@astro.multiCLOTHESvax.de (Phillip Helbig---
remove CLOTHES to reply) wrote:
> In article
> <1125b6f0-28fb-4183-a734-ef1e06b39...@40g2000prx.googlegroups.com>, Hein
> RMS van den Heuvel writes:
>
> > > OK, doing COMPRESS now. *Will probably take another 15 hours.
>
> > How can that be ????
>
> Big file, slow system. *:-) *I did delete aboutu 220 MB of stuff from
> the file.

Ay yes, I read too quickly. 220 MB deleted.
No telling (for me) how much is left.


> This is the COMPRESS from within MAIL on 7.3-2. *

Ah... 7.3-2 no ^T, no large output buffers on convert
The size of the output may be your best progress indicator.
ANAL/SYS... SHOW PROC/RMS=bdbsum may also give clues.
And PROCIO or the WCB can give READ & WRITE stats.

> The disk is a shadow set (10 MB/s SCSI) mounted on two VAXes in the
> cluster. *The command is running on one of them, a VAXstation 4000/90.

Ah, just a vax. nice vax, but just a vax.

> So all writes are going to both disks over the LAN.

Which is hopefully 100mb/FD, but maybe not.

So stop it already!
Check my procedure,
Get out of vaxmail,
CONV/FDL... with that 'large' FDL further tweaked for your appoximate
output size.
convert input from a local disk.
convert output to a non-shadowed local disk.
When done, copy result to target location
Optionally use convert again to copy new to target for optimal size.
That convert will not take muck longer than a copy, as (supposedly)
you picked a large bucket size.

JFM> convert/reclaim which is generally very fast.

Both have to read all data buckets.
Reclaim has to read index buckets and alternate indexes as well.
Basic convert uses SYS$GET (RMS RECORD IO) for read sych..
Reclaim uses SYS$READ... also synchroneously.
Convert can use an optimized bucket size for output, reclaim does not.
Convert double-buffers output, reclaim does not.

JFM> Reception of new mail during this time is problematic

Yes and no.
A new file will be created, which can be merged into the comverted
file... if you notice the extra file.
But I believe Philip is not converting his main mail file (or at least
he should not be imho ;-) so then it becomes a moot point.
Hmm, I guess he is converting the main file, as he was worried about
the file being locked.

Hein.

Phillip Helbig---remove CLOTHES to reply wrote:

> My mistake. It's the PURGE which will take about 15 hours! After that
> I'll do the COMPRESS!

CONVERT/RECLAIM (purge) is done implicitely in a rebuild of the file
(COMPRESS). So it is a complete waste of time.

The main key of the MAIL.MAI file, if I am not mistaken, (can someone
confirm) is a time stamp. And this should be an important consideration
for managing the file.

If you keep only one out of 10 received messages, then you will end up
with buckets that contain 1 record and 9 deleted records. the "PURGE"
will not reclaim any of those buckets. Say your bucket contains 5
blocks. This means that over the years, you will be retaining old mail
records that will occupy 5 blocks each.

Doing a rebuild (compress aka convert/fdl) will move all those old
records to be next to each other and fill those 5 blocks to capacity,
and will thus be using far less paces in the record psace, but also far
fewer index entries.

The CONVERT/RECLAIM works well in a "queue" situation where all old
records are deleted as new records are inserted at the end since the old
buckets become empty and made available for new records to be used at
the end.

Hein RMS van den Heuvel wrote:

> JFM> convert/reclaim which is generally very fast.
>
> Both have to read all data buckets.
> Reclaim has to read index buckets and alternate indexes as well.


While your arguments are compelling to show that /reclaim should be
slower, in my experience, a /reclaim seems to be much faster. Perhaps
because it has to do far fewer writes ?

> My mistake. *It's the PURGE which will take about 15 hours! *

> After that I'll do the COMPRESS!- Hide quoted text -

Ah... you still have to empty out those 220MB from the 'wastebasket'
huh?!

I suggest you write a small program, or even just SEARCH/EXACT/
MATCH=NOR "WASTEBASKET" to grab all non-wastebasket messages (or
whatever you called your wastebasket).
AWK, or PERL might be better.. loking just at the rigth place.

If you write a program (I might... after I finsih some customer
stuff), then you want to MEMCMP match 40 bytes at offset 8 (0 based)
with the first byte being the length (11 for "wastebasket") followed
by the name for the wastebasked folder, zero filled.
Exclude on full match or first all-zero longword
Include on first mis-matching longword.

Convert the output sequential back to a fresh indexed file.
If using a simple search for this, then later verfiy for false
positives.

fwiw,
Hein.

[ Seem to have lost a reply .. quick retry... ]


On Oct 10, 12:59*pm, hel...@astro.multiCLOTHESvax.de (Phillip Helbig---
remove CLOTHES to reply) wrote:
:
> My mistake. *It's the PURGE which will take about 15 hours! *

Ah... don't do that!

$RENAME MAIL.MAI MAIL.OLD
$SEARCH/STAT/MAT=NOR/EXACT/OUT=MAIL.SEQ mail.mai "WASTEBASKET"
$CONVERT/FAST/STAT/NOSORT/FDL.... MAIL.SEQ MAIL.NEW
$RENA/LOG MAIL.NEW MAIL.MAI

Later, check with SEARH MAIL.OLD WASTEBASKET for false positives.

Actually you should use a program (or AWK or PERL) to compare 40 bytes
at offset 8 (0 based) and input if that differs from a a 40 buffer
with byte-0 being the length of the WASTEBASKET folder name, then the
wastebasket folder name itself, zero-filled.

Pre-allocate generously OUT-ALQ=input-ALQ/5? OUT-DEQ=input-ALQ/20
FAB$V_TEF=1, WBH=1, MBF=4, MBC=127

I may write that some day, just not today.

Regards,

Hein.

Hein RMS van den Heuvel wrote:
>
> $RENAME MAIL.MAI MAIL.OLD
> $SEARCH/STAT/MAT=NOR/EXACT/OUT=MAIL.SEQ mail.mai "WASTEBASKET"
> $CONVERT/FAST/STAT/NOSORT/FDL.... MAIL.SEQ MAIL.NEW
> $RENA/LOG MAIL.NEW MAIL.MAI

Wouldn't someone normally empty the wastebasket at regular intervals ?
Besides, a MAIL> PURGE/RECLAIM won't empty the wastebasket, will it ?

In article <48ef910c$0$12358$c3e8da3@news.astraweb.com>, JF Mezei
writes:

> Phillip Helbig---remove CLOTHES to reply wrote:
>
> > My mistake. It's the PURGE which will take about 15 hours! After that
> > I'll do the COMPRESS!
>
> CONVERT/RECLAIM (purge) is done implicitely in a rebuild of the file
> (COMPRESS). So it is a complete waste of time.

First I have to do a PURGE (/NORECLAIM) to actually empty the
WASTEBASKET folder; then the convert comes.

In article <48efb395$0$12383$c3e8da3@news.astraweb.com>, JF Mezei
writes:

> Hein RMS van den Heuvel wrote:
> >
> > $RENAME MAIL.MAI MAIL.OLD
> > $SEARCH/STAT/MAT=NOR/EXACT/OUT=MAIL.SEQ mail.mai "WASTEBASKET"
> > $CONVERT/FAST/STAT/NOSORT/FDL.... MAIL.SEQ MAIL.NEW
> > $RENA/LOG MAIL.NEW MAIL.MAI
>
> Wouldn't someone normally empty the wastebasket at regular intervals ?

I normally do. In fact, I normally do so regularly in a batch job.
This is a special case of one-time maintenance.

> Besides, a MAIL> PURGE/RECLAIM won't empty the wastebasket, will it ?

PURGE, by definition, empties the wastebasket. /RECLAIM is an option,
not enabled by default, to actually free up the used space.

I gather from this thread that many of the defaults for VMS MAIL are not
optimal for large files. On the other hand, I started using VMS MAIL in
1992 and in principle still have the main mail file. (My user disk has
been migrated several times. As someone said, hardware is ephemeral,
software is inertial.) I have a huge number of mail messages in various
folders and files (not all in the same directory, but on the same disk)
and am happy that VMS MAIL satisfies almost all of my needs. It works
in an environment far beyond what it could have been tested with 16
years ago. (Yes, there have been improvements since then, but probably
not for several years now and there were some problems introduced by the
Bliss-->C rewrite.)

The only bug I've seen is that when a message is marked in a folder with
more than 9999 messages, one gets five stars in the left columns. (No,
I normally don't use folders this big, just for archival purposes in
special cases.)