OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Re: "Out of memory!" errors running sa-learn on 3.6-STABLE AMD64

From: Kevin (kkadowgmail.com)
Date: Mon Jan 31 2005 - 23:21:12 CST

On Mon, 31 Jan 2005 22:46:07 -0600 (CST), C. Bensend
<bennybennyvision.com> wrote:
> I just cut over my personal web/mailserver to a 3.6-STABLE system,
> running an AMD64 3000+ with 2GB of DDR400 RAM. For the most part,
> things are running smoothly, but I'm seeing some strange memory
> issues every now and again.
. . .
> The one I'm fighting with now is training my SpamAssassin Bayes
> database using 'sa-learn'. I'm attempting to train a directory of
> 5000 corpus spam emails (something I did often on the old PIII-800
> with 512MB of RAM).

The sa-learn process can consume a significant amount of RAM,
and depending on the OS version you were running on the P3/800,
memory consumption of perl (sa-learn is a perl script) may have
increased significantly with the move to ELF, etc.

> What I'm getting is this (wrapped for your viewing comfort):
>
> [bennyfusion ~]$ time sa-learn --showdots --spam --dir
> /home/benny/Maildir/.SPAM.corpus.2004.archive10/cur/
> ..................................................................
> ..................................................................
> ..................................................................
> ..................................................................
> ..............................................Out of memory!
>
> Out of memory? I still have over a gig of RAM free, no swap is
> being used, and my user is in the 'staff' login group. OK, so I'm
> obviously hitting some sort of limit - I changed my user to the
> 'daemon' login group (and yes, logged out and back in). Same thing.
> Hrm.

OpenBSD's default limits are set in /etc/login.conf based on the 'class',
you will want to adjust datasize-cur. If you use the 'ksh' shell, the
built-in command 'ulimit -a' will show the current limits.

The whole 'class' thing took me a while to get used to, being completely
distinct from the account's primary group and all.

> This machine is running SpamAssassin 3.02 built from source, as I
> opted for a newer version than is in the ports tree.

I've encountered the exact same issue with sparc64, perhaps the issue
is in part related to 64-bit mode? Or more likely, sa-learn is simply a
memory hog :)

Kevin