RE: Memory Leak? - SpamAssassin

This is a discussion on RE: Memory Leak? - SpamAssassin ; Ron Smith wrote: > > On Jul 25, 2008, at 10:54 AM, Bowie Bailey wrote: > > > I am running spamd like this: > > > > /usr/bin/spamd -d -m 4 --max-conn-per-child=50 -timeout-child=180 > > -u mailuser > > ...

+ Reply to Thread
Results 1 to 9 of 9

Thread: RE: Memory Leak?

  1. RE: Memory Leak?

    Ron Smith wrote:
    >
    > On Jul 25, 2008, at 10:54 AM, Bowie Bailey wrote:
    >
    > > I am running spamd like this:
    > >
    > > /usr/bin/spamd -d -m 4 --max-conn-per-child=50 -timeout-child=180
    > > -u mailuser
    > >
    > > That's a maximum of 4 spamd child processes and they restart after
    > > 50 connections each. The processes are using 60-65M each.
    > >
    > > Drop your maximum number of children until the system runs without
    > > using swap. You could also drop any unnecessary 3rd party rulesets
    > > to lower the memory consumption of each process. If you can't keep
    > > it out of swap, you should consider adding some memory. My system
    > > has 1GB of ram, but the only things running are the mailserver,
    > > spamd, clamd, and named. I have some swap space available, but it
    > > is never used.

    >
    > That's a great suggestion, Bowie. I'm going to give that a shot.
    >
    > Now that you bring up connections per child... how many usually occur
    > or can occur if I don't specify? I seem to have missed that in the man
    > pages.
    >
    > I'm running 4gb on my system all this week (up from 1gb originally)
    > and have a total of 8gb in my hot little hands to install if need be.
    >
    > And I too am only using my mailserver, spamd, clamd, named, but I also
    > run freshclamd which updates the viral database four times a day.


    According to the spamd man page, the default number of children is 5.

    If you have 4gb on your server, I'm not sure why you are having memory
    problems. Take a look at 'top' and see which processes are using the
    memory. How big are your spamd processes?

    I am using freshclamd as well. I didn't mention it since it doesn't
    use a significant amount of memory.

    --
    Bowie


  2. Re: Memory Leak?


    ----- Original Message -----
    From: "Ron Smith"
    Sent: Friday, 2008, July 25 18:02


    > Processes: 61 total, 3 running, 58 sleeping... 342
    > threads 20:19:27
    > Load Avg: 0.45, 0.99, 1.21 CPU usage: 2.80% user, 3.74% sys,
    > 93.46% idle
    > SharedLibs: num = 2, resident = 34M code, 0 data, 3104K
    > linkedit.
    > MemRegions: num = 8452, resident = 253M + 14M private, 81M shared.
    > PhysMem: 199M wired, 489M active, 3365M inactive, 4054M used, 42M
    > free.
    > VM: 3418M + 131M 53452(0) pageins, 6452(0) pageouts
    >
    > PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD
    > RSIZE VSIZE
    > 74354 top 5.9% 0:02.43 1 19 33 460K 212K
    > 1064K 19M

    *** > 74288 mdworker 0.0% 0:00.40 3 52 37 744K 2504K
    > 2492K 31M
    > 74059 sh 0.0% 0:00.03 1 14 18 224K 764K
    > 976K 18M
    > 74054 sshd 0.0% 0:00.36 1 17 49 356K 200K
    > 1440K 21M

    NOTE* > 73445 perl 0.0% 0:13.79 1 11 155 17M 29M
    > 26M 50M
    > 71709 perl 0.0% 0:48.19 1 11 155 18M 29M
    > 27M 50M

    *** > 314 mdworker 0.0% 0:32.72 4 55 36 1292K 2140K
    > 2892K 31M
    > 313 pmTool 0.0% 9:25.72 1 24 26 628K 1048K
    > 1468K 27M
    > 312 Activity M 0.5% 11:10.62 5 110 229 4340K 15M
    > 11M 197M
    > 309 usbmuxd 0.0% 0:00.01 2 21 32 208K 224K
    > 728K 19M
    > 306 iTunesHelp 0.0% 0:00.10 2 53 50 512K 588K
    > 2280K 89M
    > 188 Finder 0.0% 19:26.91 9 162 203 3728K 12M
    > 16M 185M
    > 187 coreaudiod 0.0% 0:00.08 2 96 29 260K 232K
    > 928K 18M
    > 186 ATSServer 0.0% 0:00.45 2 80 82 700K 1164K
    > 2360K 62M
    > 185 AppleVNCSe 0.9% 6:18:17 6 101 91 13M 8228K
    > 17M 143M
    > 184 SystemUISe 0.0% 0:02.40 6 211 186 2164K 4732K
    > 6292K 158M
    >
    > Here is a breakdown from the Activity Monitor showing the parent spamd
    > (perl) and two (perl) children called 'nobody.' The sh process is the
    > scanspam script from which I excerpted the two lines which call spamc
    > and then change the .tmp file to .sub once spamd is through processing
    > it. I'v thrown in freshclam and the CG PRO process also:
    >
    > PID proc user CPU Thrds Real Mem Vir Mem Kind(Platform)

    COMPARE* > 73445 perl nobody 0.0 1 25.79 MB 618.09 MB PowerPC
    > 156 perl root 0.0 1 24.55 MB 608.68 MB PowerPC
    > 71709 perl nobody 0.0 1 26.82 MB 618.35 MB PowerPC
    > 75 freshclam root 0.0 1 456.00 KB 586.43 MB PowerPC
    > 74059 sh root 0.0 1 976.00 KB 586.16 MB PowerPC
    > 103 sh root 0.0 1 772.00 KB 586.16 MB PowerPC
    > 102 CGServer root 4.8 141 13.45 MB 617.47 MB PowerPC


    WTH is Activity Monitor claiming 618.09 MB for this perl function when
    top reports a more sane 50M? Can ActivityMonitor be believed?

    Note the mdworker threads. I Googled it. First answer was this thread:
    http://forum.insanelymac.com/index.php?showtopic=24789

    Here is another discussion thread to consider:
    "http://forums.macnn.com/90/mac-os-x/361819/great-crusade-against-10-5-mdworker/"

    In some circles mdworker is not considered to be a very friendly tool.
    I'd nice it to the point it barely runs in the background if I didn't
    kill it altogether or simply run a FreeBSD or Linux based solution. For
    servers Apples and non-server versions of Windows are dreadful. The raw
    'ix solutions do far better.

    Take it up with Apple. You have activity on your machine so mdworker
    dutifully tries to search out changes to the system it may have caused
    thus eating your machine alive.

    {^_^} Joanne


  3. Re: Memory Leak?

    Ron Smith wrote on Fri, 25 Jul 2008 21:02:02 -0400:

    > Processes: 61 total, 3 running, 58 sleeping... 342
    > threads 20:19:27
    > Load Avg: 0.45, 0.99, 1.21 CPU usage: 2.80% user, 3.74% sys,
    > 93.46% idle
    > SharedLibs: num = 2, resident = 34M code, 0 data, 3104K
    > linkedit.
    > MemRegions: num = 8452, resident = 253M + 14M private, 81M shared.
    > PhysMem: 199M wired, 489M active, 3365M inactive, 4054M used, 42M
    > free.


    Finally: figures :-)
    These figures do not indicate any problems. That "inactive memory" is -
    according to the explanation link you gave - what we call "buffers" in
    Linux. However, it's not the same "buffers" that free shows. Growing
    buffers tells there is enough RAM available to let it grow. There's no
    indication that there is a memory problem.
    Few "free memory" doesn't mean anything on Unix-like systems. That's what
    they are programmed to do, keep as much as possible in the buffers. Only
    when your used swap (VM) increases to more than 10% of the available swap
    there *might* be a problem. Please run "free" (I hope that is available on
    Mac, too), that will give much better figures about RAM usage than top.
    The figures from Activity Viewer you quoted are *very* suspicious, I
    wouldn't trust them at all. It's interesting that they all fluctuate
    around 600 MB. Very suspicious.
    As Joanne already pointed out that mdworker process takes a lot of memory.
    That was probably the reason why you had to up the RAM in this machine.
    According to the given links it's a realtime content indexer. Hell! That
    means it indexes every mail and every file that gets created/changed on
    your machine! The more mail you get the more mdworker will do! If we are
    not very mistaken *that* is the source of your problem. What's relevant
    for your problem is not the memory, it's the load! I expect that load to
    grow heavily when you get mail because mdworker grabs a good portion of
    your cpu indexing it.
    You can see the load in top in the first few lines. There are two main
    figures. % idle shows how much your system is idle, if it decreases your
    CPU is getting used. If it stays constantly low that means that your CPU
    is in heavy usage. Look at it while the system has problems! The load
    average shows how much the system "has to work". It's calculated from I
    don't know what. It's a relative figure and needs interpretation. Usually,
    if it goes over 2 that means that the system is loaded, but still working
    at a reasonable level. Normally, it shouldn't stay at this level for
    hours, though, and it should not go much higher, unless for short peaks.
    You can "calibrate" these figures by yourself by looking at the load at
    various times and correlating that to the current tasks and how the
    machine behaves. F.i. if you tar cz a 1GB directory the load will go up
    very soon, maybe to 2, 3, 4, 5 and other task may becomne a bit sluggish,
    depending on your horse power.
    The first figure is the current load, the other two are 5 and 15 (or ten?)
    minutes ago. The figures from this "light" system load are already too
    high for my taste, I suspect that this "base load" comes from the fact
    that this machine is not only a server but also runs GUI (X) and is used
    as workstation, too, at least temporarily (?).
    If you want to run this system as a server you have to shut down as much
    stuff as you can that is irrelevant for that. And the no. 1 candidate is
    that mdworker process.
    I think if you want to use a Mac as a server you hit the limit where it
    suffices to use and know all the nice click-a-dee-click stuff that Mac and
    Windows have in common. You need to read up on the BSD-ish nature of Mac
    OS and how it works under the hood and how to use the unixish tools it
    offers.



    Kai

    --
    Kai Schätzl, Berlin, Germany
    Get your web at Conactive Internet Services: http://www.conactive.com


  4. Re: Memory Leak?

    Thanks, Kai, for you great comments.

    Ron Smith
    postmaster@pmbx.net

    "Having an email problem is painful, but character-building."

    On Jul 26, 2008, at 8:31 AM, Kai Schaetzl wrote:

    > Finally: figures :-)
    > These figures do not indicate any problems. That "inactive memory"
    > is -
    > according to the explanation link you gave - what we call "buffers" in
    > Linux. However, it's not the same "buffers" that free shows. Growing
    > buffers tells there is enough RAM available to let it grow. There's no
    > indication that there is a memory problem.


    That's been my perception when I look at the figures, though the
    actual performance hits during peak mail throughput had given me pause.

    > Few "free memory" doesn't mean anything on Unix-like systems. That's
    > what
    > they are programmed to do, keep as much as possible in the buffers.
    > Only
    > when your used swap (VM) increases to more than 10% of the available
    > swap
    > there *might* be a problem. Please run "free" (I hope that is
    > available on
    > Mac, too), that will give much better figures about RAM usage than
    > top.
    > The figures from Activity Viewer you quoted are *very* suspicious, I
    > wouldn't trust them at all. It's interesting that they all fluctuate
    > around 600 MB. Very suspicious.


    Hmmm. That's interesting and notable. I must bear that in mind as we
    continue to test. However that is a common number to see among all the
    Mac servers that I run.

    > As Joanne already pointed out that mdworker process takes a lot of
    > memory.
    > That was probably the reason why you had to up the RAM in this
    > machine.
    > According to the given links it's a realtime content indexer. Hell!
    > That
    > means it indexes every mail and every file that gets created/changed
    > on
    > your machine! The more mail you get the more mdworker will do! If we
    > are
    > not very mistaken *that* is the source of your problem. What's
    > relevant
    > for your problem is not the memory, it's the load! I expect that
    > load to
    > grow heavily when you get mail because mdworker grabs a good portion
    > of
    > your cpu indexing it.


    Spotlight (mdworker) doesn't index any of the incoming mail because it
    is located in the /var/CommuniGate/ directory. When I view it in real
    time, it fluctuates very minimally and rarely can I recall ever seeing
    it above 0.3% CPU time. Usually it is 0. The indexing is done
    primarily on home directories and since this is a server, nothing is
    added to my directory save any rare file that I put there that
    pertains to the server software itself.

    > You can see the load in top in the first few lines. There are two main
    > figures. % idle shows how much your system is idle, if it decreases
    > your
    > CPU is getting used. If it stays constantly low that means that your
    > CPU
    > is in heavy usage. Look at it while the system has problems! The load
    > average shows how much the system "has to work". It's calculated
    > from I
    > don't know what. It's a relative figure and needs interpretation.
    > Usually,
    > if it goes over 2 that means that the system is loaded, but still
    > working
    > at a reasonable level. Normally, it shouldn't stay at this level for
    > hours, though, and it should not go much higher, unless for short
    > peaks.
    > You can "calibrate" these figures by yourself by looking at the load
    > at
    > various times and correlating that to the current tasks and how the
    > machine behaves. F.i. if you tar cz a 1GB directory the load will go
    > up
    > very soon, maybe to 2, 3, 4, 5 and other task may becomne a bit
    > sluggish,
    > depending on your horse power.


    Yes, yes. That's what I've been doing and why its so confusing. The
    CPU usage does not rise under load. The cgserver process percentage
    remains steady! That's been so confusing because the Queue files
    continue to increase, but mail is not being delivered... unless I
    stopped the spamd process.

    > The first figure is the current load, the other two are 5 and 15 (or
    > ten?)
    > minutes ago. The figures from this "light" system load are already too
    > high for my taste, I suspect that this "base load" comes from the fact
    > that this machine is not only a server but also runs GUI (X) and is
    > used
    > as workstation, too, at least temporarily (?).


    No, it is not a workstation. I run no other GUI apps on it. Most of my
    work is done with terminal, except to remote in to view the activity
    monitor itself. The Activity Monitor has no effect on system
    performance, CPU times or anything else that I have ever seen in all
    the years I've run my small server herd.

    > If you want to run this system as a server you have to shut down as
    > much
    > stuff as you can that is irrelevant for that. And the no. 1
    > candidate is
    > that mdworker process.


    I have that process set not to run on the next reboot, should the
    current changes in the DNS of the machines network settings and the
    changes in the spamd call prove not to resolve this issue. I would be
    so glad NOT to find that that this is a memory leak issue. If this is
    something so simple and stupid as me miss-setting the DNS server
    preferences, then I deserve a swift kick!

    > I think if you want to use a Mac as a server you hit the limit where
    > it
    > suffices to use and know all the nice click-a-dee-click stuff that
    > Mac and
    > Windows have in common. You need to read up on the BSD-ish nature of
    > Mac
    > OS and how it works under the hood and how to use the unixish tools it
    > offers.


    I don't think the issue is that the Mac as a platform. This same
    server performed under a DDoS load just about a month ago that lasted
    a week and a half and was STILL getting mail delivered. It handled
    1000 smtp channels and only began to falter at 2000 to 2500. That's
    another story to tell if you need help fighting DDoS off.

    No as a server, I've always been VERY pleased over the years with the
    performance and as I said this issue is very recent.

    Thanks again so much for your kind comments, Kai.

    > Kai



  5. Re: Memory Leak?

    How recent is the "mdworker" daemon to the Apple OS architecture?

    It is clearly taking a sizable chunk of memory. And if it manages to
    drive spamd into swapping you're toast for throughput. If the indexing
    is not absolutely needed kill it and tell the system not to start it,
    ever. Then see how the system performs.

    (I am also concerned about the apparently huge, order of magnitude or
    more, difference in memory reports for top and "ActivityMonitor." I'd
    not trust that latter "thing" off a lead foil wrapped heavily scratched
    CDROM buried in a cemetery by the light of a full Moon with a stake
    through its heart.)

    {^_^}
    ----- Original Message -----
    From: "Ron Smith"
    Sent: Saturday, 2008, July 26 07:09


    > Thanks, Kai, for you great comments.
    >
    > Ron Smith
    > postmaster@pmbx.net
    >
    > "Having an email problem is painful, but character-building."
    >
    > On Jul 26, 2008, at 8:31 AM, Kai Schaetzl wrote:
    >
    >> Finally: figures :-)
    >> These figures do not indicate any problems. That "inactive memory"
    >> is -
    >> according to the explanation link you gave - what we call "buffers" in
    >> Linux. However, it's not the same "buffers" that free shows. Growing
    >> buffers tells there is enough RAM available to let it grow. There's no
    >> indication that there is a memory problem.

    >
    > That's been my perception when I look at the figures, though the
    > actual performance hits during peak mail throughput had given me pause.
    >
    >> Few "free memory" doesn't mean anything on Unix-like systems. That's
    >> what
    >> they are programmed to do, keep as much as possible in the buffers.
    >> Only
    >> when your used swap (VM) increases to more than 10% of the available
    >> swap
    >> there *might* be a problem. Please run "free" (I hope that is
    >> available on
    >> Mac, too), that will give much better figures about RAM usage than
    >> top.
    >> The figures from Activity Viewer you quoted are *very* suspicious, I
    >> wouldn't trust them at all. It's interesting that they all fluctuate
    >> around 600 MB. Very suspicious.

    >
    > Hmmm. That's interesting and notable. I must bear that in mind as we
    > continue to test. However that is a common number to see among all the
    > Mac servers that I run.
    >
    >> As Joanne already pointed out that mdworker process takes a lot of
    >> memory.
    >> That was probably the reason why you had to up the RAM in this
    >> machine.
    >> According to the given links it's a realtime content indexer. Hell!
    >> That
    >> means it indexes every mail and every file that gets created/changed
    >> on
    >> your machine! The more mail you get the more mdworker will do! If we
    >> are
    >> not very mistaken *that* is the source of your problem. What's
    >> relevant
    >> for your problem is not the memory, it's the load! I expect that
    >> load to
    >> grow heavily when you get mail because mdworker grabs a good portion
    >> of
    >> your cpu indexing it.

    >
    > Spotlight (mdworker) doesn't index any of the incoming mail because it
    > is located in the /var/CommuniGate/ directory. When I view it in real
    > time, it fluctuates very minimally and rarely can I recall ever seeing
    > it above 0.3% CPU time. Usually it is 0. The indexing is done
    > primarily on home directories and since this is a server, nothing is
    > added to my directory save any rare file that I put there that
    > pertains to the server software itself.
    >
    >> You can see the load in top in the first few lines. There are two main
    >> figures. % idle shows how much your system is idle, if it decreases
    >> your
    >> CPU is getting used. If it stays constantly low that means that your
    >> CPU
    >> is in heavy usage. Look at it while the system has problems! The load
    >> average shows how much the system "has to work". It's calculated
    >> from I
    >> don't know what. It's a relative figure and needs interpretation.
    >> Usually,
    >> if it goes over 2 that means that the system is loaded, but still
    >> working
    >> at a reasonable level. Normally, it shouldn't stay at this level for
    >> hours, though, and it should not go much higher, unless for short
    >> peaks.
    >> You can "calibrate" these figures by yourself by looking at the load
    >> at
    >> various times and correlating that to the current tasks and how the
    >> machine behaves. F.i. if you tar cz a 1GB directory the load will go
    >> up
    >> very soon, maybe to 2, 3, 4, 5 and other task may becomne a bit
    >> sluggish,
    >> depending on your horse power.

    >
    > Yes, yes. That's what I've been doing and why its so confusing. The
    > CPU usage does not rise under load. The cgserver process percentage
    > remains steady! That's been so confusing because the Queue files
    > continue to increase, but mail is not being delivered... unless I
    > stopped the spamd process.
    >
    >> The first figure is the current load, the other two are 5 and 15 (or
    >> ten?)
    >> minutes ago. The figures from this "light" system load are already too
    >> high for my taste, I suspect that this "base load" comes from the fact
    >> that this machine is not only a server but also runs GUI (X) and is
    >> used
    >> as workstation, too, at least temporarily (?).

    >
    > No, it is not a workstation. I run no other GUI apps on it. Most of my
    > work is done with terminal, except to remote in to view the activity
    > monitor itself. The Activity Monitor has no effect on system
    > performance, CPU times or anything else that I have ever seen in all
    > the years I've run my small server herd.
    >
    >> If you want to run this system as a server you have to shut down as
    >> much
    >> stuff as you can that is irrelevant for that. And the no. 1
    >> candidate is
    >> that mdworker process.

    >
    > I have that process set not to run on the next reboot, should the
    > current changes in the DNS of the machines network settings and the
    > changes in the spamd call prove not to resolve this issue. I would be
    > so glad NOT to find that that this is a memory leak issue. If this is
    > something so simple and stupid as me miss-setting the DNS server
    > preferences, then I deserve a swift kick!
    >
    >> I think if you want to use a Mac as a server you hit the limit where
    >> it
    >> suffices to use and know all the nice click-a-dee-click stuff that
    >> Mac and
    >> Windows have in common. You need to read up on the BSD-ish nature of
    >> Mac
    >> OS and how it works under the hood and how to use the unixish tools it
    >> offers.

    >
    > I don't think the issue is that the Mac as a platform. This same
    > server performed under a DDoS load just about a month ago that lasted
    > a week and a half and was STILL getting mail delivered. It handled
    > 1000 smtp channels and only began to falter at 2000 to 2500. That's
    > another story to tell if you need help fighting DDoS off.
    >
    > No as a server, I've always been VERY pleased over the years with the
    > performance and as I said this issue is very recent.
    >
    > Thanks again so much for your kind comments, Kai.
    >
    >> Kai



  6. Re: Memory Leak?

    Hello, all.

    Here's the current situation with what I was concerned was a memory
    leak. Since the Network Settings DNS was changed to the named process
    on the same machine as the CG Pro mail server some 36 hours ago, and
    the spamd settings were changed to

    spamd -d -x -q -u nobody -x -m 4 --max-conn-per-child=50 --timeout-
    child=180 -r /var/Communigate/spamdprocessid

    There has been NO further apparent memory issues and no backlog of
    email (and Spotlight, i.e. mdworker, was still running, Joanne.)

    One other thing that is of note is that we had also institute
    blocklists which had previously been disabled. In order to get a
    better picture of how much mail was then coming into the server prior
    to this change, I've gone back to the logs for some tallies. Here's
    the way those numbers stack up:

    Email that was not culled by the blocklists that required spamd
    scanning: ~13,000 pieces in 24 hours.
    SMTP connections that were refused outright (no spamd scanning
    required): ~100,000 pieces in 24 hours.

    Total then for the last 24 hours of spam was 113,000, if you
    considered all 13,000 pieces that got through the blocklist as spam.
    (Now I suspect that about 60% to 70% of what got through was really
    determined to be spam.) That comes to 1.307 spam pieces per second.
    Now Joanne suggest I do a time spamc test which I did:

    My results for time spamc < (test email):

    real 0m0.354s
    user 0m0.001s
    sys 0m0.005s

    My results for time spamassassin < (test email):

    real 0m5.310s
    user 0m1.969s
    sys 0m0.521s

    I clearly see slowdowns in spam on the weekend so these numbers
    represent likely the lower end of the scale for incoming, so a valid
    estimate would certainly be that there are 150,000 to 200,000 spam
    connections under a weekday load.

    Now my question is this. Could the apparent memory issue simply under
    the crush of the above load actually have been a combination of not
    having the blocklists on, having the DNS for the mail machine set to
    reference another DNS host server on my LAN, and perhaps a less than
    optimal spamd process?

    I'm not sure how the statistics here shake out here. How many pieces
    of email would a spamd child process be able to process per second
    given the numbers above? If I have 3 child processes, would they be
    overwhelmed by this load so that I have just misinterpreted a memory
    leak issue?

    Ron Smith
    postmaster@pmbx.net

    "Having an email problem is painful, but character-building."


  7. Re: Memory Leak?

    From: "Ron Smith"
    Sent: Sunday, 2008, July 27 06:16


    > My results for time spamc < (test email):
    >
    > real 0m0.354s
    > user 0m0.001s
    > sys 0m0.005s
    >
    > My results for time spamassassin < (test email):
    >
    > real 0m5.310s
    > user 0m1.969s
    > sys 0m0.521s
    >
    > I clearly see slowdowns in spam on the weekend so these numbers
    > represent likely the lower end of the scale for incoming, so a valid
    > estimate would certainly be that there are 150,000 to 200,000 spam
    > connections under a weekday load.


    These are important results. They show that you have a plenty fast
    enough machine for 100,000 emails per day, although given the fact
    that daytime is pretty bad compared to night time you'd probably see
    significant slowdowns in throughput during the day as the machine
    overloads. You'd be in deadly trouble with 200,000 messages filtered.

    The first result is pure throughput for spamd. Figure most of that
    ..354 seconds is CPU time for spamd. That gives you an upper limit on
    what the machine is likely to be able to handle, 86400/.354 messages
    per day. Of course you have other things running as well. So the
    nearly 250,000 message capacity isn't really there. Figure it's maybe
    100,000 max if the MTA and other utilities are running.

    The second results show you what'd happen if you ran spamassassin
    rather than spamc/spamd. (It also makes me wonder where the extra
    calendar time went compared to the CPU time measures. But that's not
    particularly important, I suspect.

    One thing I know is that doing this sort of analysis is worthwhile and
    often quite discouraging. (Pushing multiple sets of uncompressed frames
    of 1080i across a PCI-e bus can do strange things to a machine's
    throughput. Heck, composing four 1080i frames to provide an
    output frame can do strange things to taskmanager's eight processor
    display. It's disheartening when they all show 50% to 80% processor
    utilization. At least I know WHY some of what I want
    to do won't work. {o.o})

    Earlier results you produced showed that spamd is using a fairly
    nominal amount of memory for your installation, 50 megs a pop. For
    children is 200 megs. The mdworker threads, at least two of which
    were shown, each took 50 megs or so if my wetware memory is still
    valid. So that's 300 megs. The Comunigate server seemed to be another
    in that "price range". So you'd have been easily pushing half the
    machine's memory with four spamd children running. Unless you had
    tried to up the number of children or have enough other "little things"
    to add up you'd not push a gigabyte. A half gigabyte would have been
    swamped, most likely.

    > Now my question is this. Could the apparent memory issue simply under
    > the crush of the above load actually have been a combination of not
    > having the blocklists on, having the DNS for the mail machine set to
    > reference another DNS host server on my LAN, and perhaps a less than
    > optimal spamd process?


    See above. YES - sort of.

    You NOW have sufficient memory to run a solid dns cache plus, perhaps,
    your own BL copies. (Somebody will correct me if I am wrong, please; but,
    I believe you're at a high enough query rate that many of the BL people
    will allow you to periodically download their database so that you can
    run your own mirror of their servers. It won't be as immediately up to
    date. But it'd reduce communications problems.)

    And it comes to me that "communications problems" are a whole new measure
    in this picture. Suppose at peak traffic you're running maybe 3/4 of the
    day's traffic in half a day or maybe a rate of 12,500 an hour, or almost
    4 a second. Let's say 10k of various transfers/message that's about half
    a 768k pipe. If that pipe is shared with WWW service and other such things
    make sure it is sized big enough it does not become a bottleneck.

    > I'm not sure how the statistics here shake out here. How many pieces
    > of email would a spamd child process be able to process per second
    > given the numbers above? If I have 3 child processes, would they be
    > overwhelmed by this load so that I have just misinterpreted a memory
    > leak issue?


    I still say your readings on the Activewhazzit that Apple provides are
    outrageously out of line. Something has it displaying numbers an order
    of magnitude bigger than seem realistic compared to 'top' readings. I run
    tons of rules and don't seem to ever get over about 70megs in use on a
    spamc run. Yet it says 600 meg numbers. So I suspect it needs to get
    recalibrated as to what it's numbers really mean.

    I suspect it mislead you into thinking SpamAssassin used an utterly
    outrageous amount of memory. (side note - as someone who used to think
    64k was hog heaven 64 megabytes is utterly outrageous in its own
    account. ) The real problem was probably simply in overloading the
    machine with requests when you were not using the prefiltering.

    Note that if a "time spamc" run starts taking materially more real time
    than CPU time (sum of system and user times) it may be time to use more
    children since DNS lookups have slowed for one reason or another. "Use
    those cycles"; but, don't over do it. Leave time for the other stuff to
    run. {^_-}

    In the advanced techniques section there are people, I understand, who
    choose to stress the network a bit to get more time for spamd to run.
    They move spamd to another machine on the network and run it there with
    100% of a machine to work with. You are at a high enough traffic level
    that it would be a good idea to get someone more experienced than I am
    to help you run through a system design for the data flow you have to
    deal with. I touched on only a few of the things I can think of in my
    relative lack of experience. (I chiefly write software. I "perforce"
    administer the system on which Loren and I work because somebody has
    to do it; and, I had some fun learning Linux during a time work was real
    slow. So I work it in bursts - make it work smoothly then enter a period
    of benign neglect punctuated by frequent moments of "yum update" and
    bitching at the slow updates for critical items from the Fedora Core
    people. (They are at least a rev behind on clamav, for example.)

    {^_^} Joanne - a crazed technophile.


  8. Re: Memory Leak?

    On 7/27/08 at 8:28 AM -0700 jdow wrote:
    >These are important results. They show that you have a plenty fast
    >enough machine for 100,000 emails per day, although given the fact
    >that daytime is pretty bad compared to night time you'd probably see
    >significant slowdowns in throughput during the day as the machine
    >overloads. You'd be in deadly trouble with 200,000 messages filtered.


    Ron Smith's machine could easily handle 200,000 messages filtered per day.
    I know, I've got one just like it.

    >The first result is pure throughput for spamd. Figure most of that
    >.354 seconds is CPU time for spamd. That gives you an upper limit on
    >what the machine is likely to be able to handle, 86400/.354 messages
    >per day. Of course you have other things running as well. So the
    >nearly 250,000 message capacity isn't really there. Figure it's maybe
    >100,000 max if the MTA and other utilities are running.


    Of course that is if the machine is only running ONE spamd process. He is
    running 4 at the moment.

    >Earlier results you produced showed that spamd is using a fairly
    >nominal amount of memory for your installation, 50 megs a pop.


    Actually it is closer to 25M a pop. Check his top results in his email of
    7/25/08 at 9:02 PM -0400.
    PID proc user CPU Thrds Real Mem Vir Mem
    73445 perl nobody 0.0 1 25.79 MB 618.09 MB
    156 perl root 0.0 1 24.55 MB 608.68 MB
    71709 perl nobody 0.0 1 26.82 MB 618.35 MB


    >I still say your readings on the Activewhazzit that Apple provides are
    >outrageously out of line. Something has it displaying numbers an order
    >of magnitude bigger than seem realistic compared to 'top' readings.


    I use Active Monitor constantly and the data displayed is identical to what
    is displayed in top. It's a very useful application. I think the part of
    the problem was not understanding what the numbers represent. Ron has
    plenty of available memory, as in more than 80% of it.

    Nedry

  9. Re: Memory Leak?

    From: "Larry Nedry"
    Sent: Sunday, 2008, July 27 12:02


    > On 7/27/08 at 8:28 AM -0700 jdow wrote:
    >>These are important results. They show that you have a plenty fast
    >>enough machine for 100,000 emails per day, although given the fact
    >>that daytime is pretty bad compared to night time you'd probably see
    >>significant slowdowns in throughput during the day as the machine
    >>overloads. You'd be in deadly trouble with 200,000 messages filtered.

    >
    > Ron Smith's machine could easily handle 200,000 messages filtered per day.
    > I know, I've got one just like it.
    >
    >>The first result is pure throughput for spamd. Figure most of that
    >>.354 seconds is CPU time for spamd. That gives you an upper limit on
    >>what the machine is likely to be able to handle, 86400/.354 messages
    >>per day. Of course you have other things running as well. So the
    >>nearly 250,000 message capacity isn't really there. Figure it's maybe
    >>100,000 max if the MTA and other utilities are running.

    >
    > Of course that is if the machine is only running ONE spamd process. He is
    > running 4 at the moment.
    >
    >>Earlier results you produced showed that spamd is using a fairly
    >>nominal amount of memory for your installation, 50 megs a pop.

    >
    > Actually it is closer to 25M a pop. Check his top results in his email of
    > 7/25/08 at 9:02 PM -0400.
    > PID proc user CPU Thrds Real Mem Vir Mem
    > 73445 perl nobody 0.0 1 25.79 MB 618.09 MB
    > 156 perl root 0.0 1 24.55 MB 608.68 MB
    > 71709 perl nobody 0.0 1 26.82 MB 618.35 MB


    Um, OK - with column lineups I'd missed the declaration of 618 megabytes
    per "perl" session. Under Linux running a large collection of rules I do
    not go over 70 megabytes virtual memory and 60 megs "resident" memory. So
    the 620 megs sounds "absurd" to me. And surely if one perl session gobbles
    that much memory from the system I'd expect 4 children plus the momma would
    even be touchy on a 4 gigabyte machine. Where does all that memory go on
    the Apple machines? Or is it mega-bits they read out?

    {^_^}


+ Reply to Thread