OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Content Filtering performance
From: Brad Knowles (blkskynet.be)
Date: Tue Jun 06 2000 - 01:41:22 CDT

At 6:43 PM -0400 2000/6/5, Bennett Todd wrote:

> Summary: I wrote a script that dumped 1000 messages in on a single
> smtp connection, each a little over 1KB, as fast as possible.

        Is there a reason you didn't use Wietse's own "smtp-source"
program to do this?

> I set
> up postfix to deliver to /dev/null, and found that the filtering
> setup with my tailbiter daemon seems to be delivering at a bit under
> 1/4 the rate without filtering.

        Can you compare this performance to the dual postfix
configuration you've previously mentioned?

> Neither disabling the "chattr +S" on the postfix spool, nor placing
> it on Reiserfs, seems to have made a significant difference.

        How about putting it on a RAM disk? You wouldn't want to run
this way normally, but I'd be interested to know what the absolute
maximum top-end performance is like on your system for both unaltered
postfix versus your "tail-biter".

> disable_dns_lookups didn't seem to make a difference, I got similar
> figures without it (local dnscache). The unfiltered case never
> developed a backlog; when smtp-stress finished, the deliveries were
> done; smtp-stress ran as quickly with filtering, but accumulated
> a backlog of c. 850 messages; most of the time was waiting for it
> to drain.

        Hmm. Fascinating. This implies that there is a major bottleneck
getting into the filtering, perhaps due to overhead of synchronous
meta-data updates, fork/exec, or maybe something else.

        It would be interesting to profile both the postfix and
"tail-biter" code during a run of this sort, so that we can try to
get an idea what the bottlenecks are.

> msg/sec was computed by dividing 1000 by the number of
> seconds from the first log entry for a run to the last log entry for
> a run; rotated logs between runs so each run started with a fresh
> log.

        This is a good point. In doing benchmarking of MTAs, this is
something we should suggest that everyone does, so that we can both
retain consistency of testing, but also so that we can be sure that
we don't get confused or miss something.

> In most of the "chattr -S" tests (but not all --- not on unfiltered,
> nprocs=5, or nprocs=10), some 38-40 messages got held for "(site
> destination queue overflow)" until I manually released 'em with a
> "sendmail -q". I released them in the middle of the queue drain; I
> settled on an invocation "smtp-stress;sleep 60;sendmail -q" which
> seemed to reliably do the job.

        Hmm. That's quite strange. Have you considered turning up the
logging, so that we can try to figure out why qmgr semi-permanently
decided to defer those messages?

> Here's the code I used:

        I'd encourage you to run the same test, but this time using
Wietse's own smtp-source program, with the standard test parameters
that everyone else has been reporting on so far, for example:

                /usr/bin/time ./smtp-source -m1000 -s20 -l15360 -c
-tnulllocalhost localhost 

--
   These are my opinions -- not to be taken as official Skynet policy
======================================================================
Brad Knowles, <blkskynet.be>                || Belgacom Skynet SA/NV
Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124
Phone/Fax: +32-2-706.13.11/12.49             || B-1140 Brussels
http://www.skynet.be                         || Belgium