OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Content Filtering performance
From: Wietse Venema (wietseporcupine.org)
Date: Tue Jun 06 2000 - 07:57:14 CDT

Running the tailbiter on RAMdisk should be feasible, provided that
the software is modified to report success if and only if it passed
the mail back into Postfix successfully.

I'm curious why the performance drops by 4. I would expect 2 for
disk overhead for content filtering without temp file, 3 for a
content filter that creates/removes a tempfile. Is this because
you're using a lot of CPU power? It would make sense to compare
iostat and vmstat measurements with/without content filtering.

        Wietse

Brad Knowles:
> At 6:43 PM -0400 2000/6/5, Bennett Todd wrote:
>
> > Summary: I wrote a script that dumped 1000 messages in on a single
> > smtp connection, each a little over 1KB, as fast as possible.
>
> Is there a reason you didn't use Wietse's own "smtp-source"
> program to do this?
>
> > I set
> > up postfix to deliver to /dev/null, and found that the filtering
> > setup with my tailbiter daemon seems to be delivering at a bit under
> > 1/4 the rate without filtering.
>
> Can you compare this performance to the dual postfix
> configuration you've previously mentioned?
>
> > Neither disabling the "chattr +S" on the postfix spool, nor placing
> > it on Reiserfs, seems to have made a significant difference.
>
> How about putting it on a RAM disk? You wouldn't want to run
> this way normally, but I'd be interested to know what the absolute
> maximum top-end performance is like on your system for both unaltered
> postfix versus your "tail-biter".
>
> > disable_dns_lookups didn't seem to make a difference, I got similar
> > figures without it (local dnscache). The unfiltered case never
> > developed a backlog; when smtp-stress finished, the deliveries were
> > done; smtp-stress ran as quickly with filtering, but accumulated
> > a backlog of c. 850 messages; most of the time was waiting for it
> > to drain.
>
> Hmm. Fascinating. This implies that there is a major bottleneck
> getting into the filtering, perhaps due to overhead of synchronous
> meta-data updates, fork/exec, or maybe something else.
>
> It would be interesting to profile both the postfix and
> "tail-biter" code during a run of this sort, so that we can try to
> get an idea what the bottlenecks are.
>
> > msg/sec was computed by dividing 1000 by the number of
> > seconds from the first log entry for a run to the last log entry for
> > a run; rotated logs between runs so each run started with a fresh
> > log.
>
> This is a good point. In doing benchmarking of MTAs, this is
> something we should suggest that everyone does, so that we can both
> retain consistency of testing, but also so that we can be sure that
> we don't get confused or miss something.
>
> > In most of the "chattr -S" tests (but not all --- not on unfiltered,
> > nprocs=5, or nprocs=10), some 38-40 messages got held for "(site
> > destination queue overflow)" until I manually released 'em with a
> > "sendmail -q". I released them in the middle of the queue drain; I
> > settled on an invocation "smtp-stress;sleep 60;sendmail -q" which
> > seemed to reliably do the job.
>
> Hmm. That's quite strange. Have you considered turning up the
> logging, so that we can try to figure out why qmgr semi-permanently
> decided to defer those messages?
>
> > Here's the code I used:
>
> I'd encourage you to run the same test, but this time using
> Wietse's own smtp-source program, with the standard test parameters
> that everyone else has been reporting on so far, for example:
>
> /usr/bin/time ./smtp-source -m1000 -s20 -l15360 -c
> -tnulllocalhost localhost
>
> --
> These are my opinions -- not to be taken as official Skynet policy
> ======================================================================
> Brad Knowles, <blkskynet.be> || Belgacom Skynet SA/NV
> Systems Architect, Mail/News/FTP/Proxy Admin || Rue Colonel Bourg, 124
> Phone/Fax: +32-2-706.13.11/12.49 || B-1140 Brussels
> http://www.skynet.be || Belgium
>
>
>
>