Linda Walsh wrote:
> In my .spamassassin dir, I see lots of files that look like:
>
> bayes_toks.expire1098 bayes_toks.expire1243 bayes_toks.expire13494
>

>
> They are all a few hundred K or more long (just deleted the bunch).
>
> I've also noticed spamd going off and cranking for more than an hour
> -- seems to
> produce one of these files...
>
> Any idea what they are for?
They are a tempfile SA creates when it's trying to expire old tokens out
of your database. IIRC, it essentially creates a new DB with just the
tokens it wants, nukes the old one, and renames the tempfile.
> or why SA would keep leaving them in my .sa (user) dir?
The fact that they keep laying around is a problem. This suggests SA
keeps getting killed before the expire can complete. Do you have any
kind of limits set such as CPU time or memory that SA might be running
against and dying?

You can try kicking off an expire manually using sa-learn
--force-expire. (add -D if you want some debug output)..
note: this could run for a long time, particularly if bayes_toks is
really large.

Matt Kettler wrote:
> The fact that they keep laying around is a problem. This suggests SA
> keeps getting killed before the expire can complete. Do you have any
> kind of limits set such as CPU time or memory that SA might be running
> against and dying?
>
> You can try kicking off an expire manually using sa-learn
> --force-expire. (add -D if you want some debug output)..
> note: this could run for a long time, particularly if bayes_toks is
> really large.
----


Another one of the fileles appeared -- 17M long.
while bayes_toks is 8.8M. auto-whitelist is 78M -- that seems a bit excessive...

Don't know what really large means -- bayes_toks isn't that large
compared to some of the other files. No limits that I know of...
Ahh...seeing some oddness in the log though:
(Interrupted? timeouts?...weird...)...

Jul 25 15:23:59 Ishtar spamd[2447]: bayes: cannot open bayes databases
/home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
Jul 25 15:24:48 Ishtar spamd[2443]: bayes: cannot open bayes databases
/home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child processing
timeout at /usr/bin/spamd line 1085, line 22.
Jul 25 15:36:55 Ishtar spamd[2447]: bayes: cannot open bayes databases
/home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
Jul 25 15:41:38 Ishtar spamd[2443]: bayes: expire_old_tokens: child processing
timeout at /usr/bin/spamd line 1085.
Jul 25 16:14:14 Ishtar spamd[2355]: bayes: cannot open bayes databases
/home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call
Jul 25 16:19:00 Ishtar spamd[2385]: bayes: expire_old_tokens: child processing
timeout at /usr/bin/spamd line 1085.
Jul 25 16:29:05 Ishtar spamd[2356]: bayes: expire_old_tokens: child processing
timeout at /usr/bin/spamd line 1085.
Jul 25 17:06:02 Ishtar spamd[2385]: bayes: cannot open bayes databases
/home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system call

Linda Walsh wrote:
>
> Matt Kettler wrote:
>> The fact that they keep laying around is a problem. This suggests SA
>> keeps getting killed before the expire can complete. Do you have any
>> kind of limits set such as CPU time or memory that SA might be
>> running against and dying?
>>
>> You can try kicking off an expire manually using sa-learn
>> --force-expire. (add -D if you want some debug output)..
>> note: this could run for a long time, particularly if bayes_toks is
>> really large.
> ----
>
>
> Another one of the fileles appeared -- 17M long.
> while bayes_toks is 8.8M.
That's quite reasonable. Should be able to crank through that reasonably
quick. It strikes me as odd that the .expire got larger than the toks
file, but I might be missing something about how the process works.

> auto-whitelist is 78M -- that seems a bit excessive...
Yeah, the AWL has no expire process, other than manually running the
check-whitelist script on it. (found at:
http://svn.apache.org/repos/asf/spam...hes/3.2/tools/ )

>
> Don't know what really large means -- bayes_toks isn't that large
> compared to some of the other files. No limits that I know of...
> Ahh...seeing some oddness in the log though:
> (Interrupted? timeouts?...weird...)...

> Jul 25 15:24:48 Ishtar spamd[2443]: bayes: cannot open bayes databases
> /home/user/.spamassassin/bayes_* R/W: lock failed: Interrupted system
> call
> Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child
> processing timeout at /usr/bin/spamd line 1085, line 22.
The R/W lock fails are not surprising at all. If an expire process is
running, other scanners will fail to get a write-lock on the bayes DB.
That's not really that big a deal, all it means is that autolearning
can't occur during the expire run. Rather than logjam your mail queue,
SA just moves on and skips the autolearning, treating it as better to
process mail in a timely fashion than to wait to perform every automatic
learning possible. Same thing happens when two message try to autolearn
at the same time, only one gets the R/W lock, and the other moves on..

Now if it can't get a R/O lock (read only), start to worry. But R/W lock
failures happen fairly often.

The child processing timeouts are a bigger deal, that's what's causing
all the expire files to be left around. I can only guess this is caused
by the fact that --timeout-child defaults to 300 seconds. However, I
wouldn't expect that to apply to expire runs... Of course I'm no expert
here, as I don't use spamd (I use MailScanner, which loads the API
directly).

What version are you running? reading around the child processing
timeout seems to have been a common problem in the 3.1.x series, but
I've not seen it reported in the 3.2.x series.

https://issues.apache.org/SpamAssass...ug.cgi?id=4650

Matt Kettler wrote:
> What version are you running? reading around the child processing
> timeout seems to have been a common problem in the 3.1.x series, but
> I've not seen it reported in the 3.2.x series.
---
Erp. I'll try upgrading and see what happens...still have a
3.1.7 installed.

On Fri, 2008-07-25 at 18:35 -0700, Linda Walsh wrote:

> Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child processing
> timeout at /usr/bin/spamd line 1085, line 22.

Your autoexpire is taking longer than SA is willing to wait.

This is a fairly common question, there's lots of discussion in the list
archives.

Consensus: disable autoexpire and run a dedicated expiry from cron,
weekly or daily based on your token volume.

--
John Hardin KA7OHZ http://www.impsec.org/~jhardin/
jhardin@impsec.org FALaholic #11174 pgpk -a jhardin@impsec.org
key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C AF76 D822 E6E6 B873 2E79
-----------------------------------------------------------------------
There is no doubt in my mind that millions of lives could have been
saved if the people were not "brainwashed" about gun ownership and
had been well armed. ... Gun haters always want to forget the Warsaw
Ghetto uprising, which is a perfect example of how a ragtag,
half-starved group of Jews took 10 handguns and made asses out of
the Nazis. -- Theodore Haas, Dachau Survivor
-----------------------------------------------------------------------
9 days until the 273rd anniversary of John Peter Zenger's acquittal

A manual expire run took less than 2 minutes -- closer to 1 minute....How impatient
is SA ??



John Hardin wrote:
> On Fri, 2008-07-25 at 18:35 -0700, Linda Walsh wrote:
>
>> Jul 25 15:28:21 Ishtar spamd[2355]: bayes: expire_old_tokens: child processing
>> timeout at /usr/bin/spamd line 1085, line 22.
>
> Your autoexpire is taking longer than SA is willing to wait.
>
> This is a fairly common question, there's lots of discussion in the list
> archives.
>
> Consensus: disable autoexpire and run a dedicated expiry from cron,
> weekly or daily based on your token volume.
>