junkfiles-bays_toks.expire\d{4-5} - SpamAssassin

This is a discussion on junkfiles-bays_toks.expire\d{4-5} - SpamAssassin ; In my .spamassassin dir, I see lots of files that look like: bayes_toks.expire1098 bayes_toks.expire1243 bayes_toks.expire13494 bayes_toks.expire15029 bayes_toks.expire15761 bayes_toks.expire16349 bayes_toks.expire17370 bayes_toks.expire17385 bayes_toks.expire1754 bayes_toks.expire18183 bayes_toks.expire18584 bayes_toks.expire18813 bayes_toks.expire19274 bayes_toks.expire19481 bayes_toks.expire20721 bayes_toks.expire2264 bayes_toks.expire2265 bayes_toks.expire2266 bayes_toks.expire2267 bayes_toks.expire22670 bayes_toks.expire2268 bayes_toks.expire2324 bayes_toks.expire2327 bayes_toks.expire2355 bayes_toks.expire2356 bayes_toks.expire2385 bayes_toks.expire23960 bayes_toks.expire2443 ...

+ Reply to Thread
Results 1 to 7 of 7

Thread: junkfiles-bays_toks.expire\d{4-5}

  1. junkfiles-bays_toks.expire\d{4-5}

    In my .spamassassin dir, I see lots of files that look like:

    bayes_toks.expire1098 bayes_toks.expire1243 bayes_toks.expire13494
    bayes_toks.expire15029 bayes_toks.expire15761 bayes_toks.expire16349
    bayes_toks.expire17370 bayes_toks.expire17385 bayes_toks.expire1754
    bayes_toks.expire18183 bayes_toks.expire18584 bayes_toks.expire18813
    bayes_toks.expire19274 bayes_toks.expire19481 bayes_toks.expire20721
    bayes_toks.expire2264 bayes_toks.expire2265 bayes_toks.expire2266
    bayes_toks.expire2267 bayes_toks.expire22670 bayes_toks.expire2268
    bayes_toks.expire2324 bayes_toks.expire2327 bayes_toks.expire2355
    bayes_toks.expire2356 bayes_toks.expire2385 bayes_toks.expire23960
    bayes_toks.expire2443 bayes_toks.expire2447 bayes_toks.expire25435
    bayes_toks.expire26900 bayes_toks.expire29828 bayes_toks.expire31304
    bayes_toks.expire3343 bayes_toks.expire3442 bayes_toks.expire3444
    bayes_toks.expire4002 bayes_toks.expire4334 bayes_toks.expire4877
    bayes_toks.expire5636 bayes_toks.expire5683 bayes_toks.expire5779
    bayes_toks.expire6464 bayes_toks.expire9281 bayes_toks.expire9300

    They are all a few hundred K or more long (just deleted the bunch).

    I've also noticed spamd going off and cranking for more than an hour -- seems to
    produce one of these files...

    Any idea what they are for? or why SA would keep leaving them in my .sa (user) dir?


  2. Re: junkfiles-bays_toks.expire\d{4-5}

    Linda Walsh wrote:
    > In my .spamassassin dir, I see lots of files that look like:
    >
    > bayes_toks.expire1098 bayes_toks.expire1243 bayes_toks.expire13494
    >


    >
    > They are all a few hundred K or more long (just deleted the bunch).
    >
    > I've also noticed spamd going off and cranking for more than an hour
    > -- seems to
    > produce one of these files...
    >
    > Any idea what they are for?

    They are a tempfile SA creates when it's trying to expire old tokens out
    of your database. IIRC, it essentially creates a new DB with just the
    tokens it wants, nukes the old one, and renames the tempfile.
    > or why SA would keep leaving them in my .sa (user) dir?

    The fact that they keep laying around is a problem. This suggests SA
    keeps getting killed before the expire can complete. Do you have any
    kind of limits set such as CPU time or memory that SA might be running
    against and dying?

    You can try kicking off an expire manually using sa-learn
    --force-expire. (add -D if you want some debug output)..
    note: this could run for a long time, particularly if bayes_toks is
    really large.


  3. Re: junkfiles-bays_toks.expire\d{4-5}



    Matt Kettler wrote:
    > The fact that they keep laying around is a problem. This suggests SA
    > keeps getting killed before the expire can complete. Do you have any
    > kind of limits set such as CPU time or memory that SA might be running
    > against and dying?
    >
    > You can try kicking off an expire manually using sa-learn
    > --force-expire. (add -D if you want some debug output)..
    > note: this could run for a long time, particularly if bayes_toks is
    > really large.

    ----


    Another one of the fileles appeared -- 17M long.
    while bayes_toks is 8.8M. auto-whitelist is 78M -- that seems a bit excessive...

    Don't know what really large means -- bayes_toks isn't that large
    compared to some of the other files. No limits that I know of...
    Ahh...seeing some oddness in the log though:
    (Interrupted? timeouts?...weird...)...

    Jul 25 15:23:59 Ishtar spamd[2447]: bayes: cannot open bayes databases
    /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
    Jul 25 15:24:48 Ishtar spamd[2443]: bayes: cannot open bayes databases
    /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
    Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child processing
    timeout at /usr/bin/spamd line 1085, line 22.
    Jul 25 15:36:55 Ishtar spamd[2447]: bayes: cannot open bayes databases
    /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
    Jul 25 15:41:38 Ishtar spamd[2443]: bayes: expire_old_tokens: child processing
    timeout at /usr/bin/spamd line 1085.
    Jul 25 16:14:14 Ishtar spamd[2355]: bayes: cannot open bayes databases
    /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
    Jul 25 16:19:00 Ishtar spamd[2385]: bayes: expire_old_tokens: child processing
    timeout at /usr/bin/spamd line 1085.
    Jul 25 16:29:05 Ishtar spamd[2356]: bayes: expire_old_tokens: child processing
    timeout at /usr/bin/spamd line 1085.
    Jul 25 17:06:02 Ishtar spamd[2385]: bayes: cannot open bayes databases
    /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call


  4. Re: junkfiles-bays_toks.expire\d{4-5}

    Linda Walsh wrote:
    >
    > Matt Kettler wrote:
    >> The fact that they keep laying around is a problem. This suggests SA
    >> keeps getting killed before the expire can complete. Do you have any
    >> kind of limits set such as CPU time or memory that SA might be
    >> running against and dying?
    >>
    >> You can try kicking off an expire manually using sa-learn
    >> --force-expire. (add -D if you want some debug output)..
    >> note: this could run for a long time, particularly if bayes_toks is
    >> really large.

    > ----
    >
    >
    > Another one of the fileles appeared -- 17M long.
    > while bayes_toks is 8.8M.

    That's quite reasonable. Should be able to crank through that reasonably
    quick. It strikes me as odd that the .expire got larger than the toks
    file, but I might be missing something about how the process works.

    > auto-whitelist is 78M -- that seems a bit excessive...

    Yeah, the AWL has no expire process, other than manually running the
    check-whitelist script on it. (found at:
    http://svn.apache.org/repos/asf/spam...hes/3.2/tools/ )

    >
    > Don't know what really large means -- bayes_toks isn't that large
    > compared to some of the other files. No limits that I know of...
    > Ahh...seeing some oddness in the log though:
    > (Interrupted? timeouts?...weird...)...


    > Jul 25 15:24:48 Ishtar spamd[2443]: bayes: cannot open bayes databases
    > /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system
    > call
    > Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child
    > processing timeout at /usr/bin/spamd line 1085, line 22.

    The R/W lock fails are not surprising at all. If an expire process is
    running, other scanners will fail to get a write-lock on the bayes DB.
    That's not really that big a deal, all it means is that autolearning
    can't occur during the expire run. Rather than logjam your mail queue,
    SA just moves on and skips the autolearning, treating it as better to
    process mail in a timely fashion than to wait to perform every automatic
    learning possible. Same thing happens when two message try to autolearn
    at the same time, only one gets the R/W lock, and the other moves on..

    Now if it can't get a R/O lock (read only), start to worry. But R/W lock
    failures happen fairly often.

    The child processing timeouts are a bigger deal, that's what's causing
    all the expire files to be left around. I can only guess this is caused
    by the fact that --timeout-child defaults to 300 seconds. However, I
    wouldn't expect that to apply to expire runs... Of course I'm no expert
    here, as I don't use spamd (I use MailScanner, which loads the API
    directly).

    What version are you running? reading around the child processing
    timeout seems to have been a common problem in the 3.1.x series, but
    I've not seen it reported in the 3.2.x series.

    https://issues.apache.org/SpamAssass...ug.cgi?id=4650


  5. Re: junkfiles-bays_toks.expire\d{4-5}



    Matt Kettler wrote:
    > What version are you running? reading around the child processing
    > timeout seems to have been a common problem in the 3.1.x series, but
    > I've not seen it reported in the 3.2.x series.

    ---
    Erp. I'll try upgrading and see what happens...still have a
    3.1.7 installed.


  6. Re: junkfiles-bays_toks.expire\d{4-5}


    On Fri, 2008-07-25 at 18:35 -0700, Linda Walsh wrote:

    > Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child processing
    > timeout at /usr/bin/spamd line 1085, line 22.


    Your autoexpire is taking longer than SA is willing to wait.

    This is a fairly common question, there's lots of discussion in the list
    archives.

    Consensus: disable autoexpire and run a dedicated expiry from cron,
    weekly or daily based on your token volume.

    --
    John Hardin KA7OHZ http://www.impsec.org/~jhardin/
    jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
    key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
    -----------------------------------------------------------------------
    There is no doubt in my mind that millions of lives could have been
    saved if the people were not "brainwashed" about gun ownership and
    had been well armed. ... Gun haters always want to forget the Warsaw
    Ghetto uprising, which is a perfect example of how a ragtag,
    half-starved group of Jews took 10 handguns and made asses out of
    the Nazis. -- Theodore Haas, Dachau Survivor
    -----------------------------------------------------------------------
    9 days until the 273rd anniversary of John Peter Zenger's acquittal


  7. Re: junkfiles-bays_toks.expire\d{4-5}

    A manual expire run took less than 2 minutes -- closer to 1 minute....How impatient
    is SA ??



    John Hardin wrote:
    > On Fri, 2008-07-25 at 18:35 -0700, Linda Walsh wrote:
    >
    >> Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child processing
    >> timeout at /usr/bin/spamd line 1085, line 22.

    >
    > Your autoexpire is taking longer than SA is willing to wait.
    >
    > This is a fairly common question, there's lots of discussion in the list
    > archives.
    >
    > Consensus: disable autoexpire and run a dedicated expiry from cron,
    > weekly or daily based on your token volume.
    >



+ Reply to Thread