OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Content filtering 101
From: Liviu Daia (Liviu.Daiaimar.ro)
Date: Wed Jun 07 2000 - 08:16:00 CDT

    Ok, let me clear the backlog first.

(1) Tan Swee Heng's macofida, version 0.3.

    Nice and clean code, again. It now returns '450' for safe error
recovery, it allows easy rejection by being able to return '550' at any
point, and it saves the message to a temporary file without loading it
in memory first. It can still be easily modified to keep the message in
memory if needed. I have only a few cosmetic suggestions:

- since the filter must be able to write to the spool it should be run
    under a dedicated UID/GID; user nobody shouldn't be allowed to own
    any files;

- macofida.pid should probably live in /var/run instead of /tmp;

- the filter should never return a '5xx' if it doesn't specifically want
    to reject the message; in particular "550 Broken connection" should
    be changed to "450 Broken connection" (although it's not clear to me
    whether it still makes sense to try to return a code in that case;
    anyway, if the incoming connection is broken you probably have much
    bigger problems);

- the filter should allow a ".\n" sequence to terminate the DATA instead
    of insisting for a ".\r\n"; not an issue when getting the data from
    Postfix, but one might want to play with the filter by telnet-ing
    directly to port 10025...

    A patch addressing these (minor) issues is included after the
signature.

(2) Bennett Todd's tailbiter, version 1.1 (with the latest patch).

    It has the same major drawbacks as the previous versions of
macofida:

- it can't send back an error code to the client process, so if
    something goes wrong the best it can do is to save the message to
    /var/tmp/tailbiter.<PID> (provided it managed to read it first ---
    otherwise the message is simply lost;

- the messages is loaded completely in memory; that happens because of
    the way Net::SMTP::Server::Client works, so saving it to a temporary
    file won't change that;

- it uses Net::SMTP::Server for spawning children, which looks much less
    robust than Net::Daemon, at least AFAICT.

    It also has a few other minor annoyances, like calling
"gethostbyaddr" for no real reason (fun when not running a DNS), and not
being able to cope with my (admittedly not the latest-and-greatest) Perl
5.004_04 at home, because:

: Bareword "qr" not allowed while "strict subs" in use at ./tailbiter line 67.
: Unquoted string "qr" may clash with future reserved word at ./tailbiter line 67.
: Bareword "im" not allowed while "strict subs" in use at ./tailbiter line 67.
: Unquoted string "im" may clash with future reserved word at ./tailbiter line 67.
: syntax error at ./tailbiter line 106, near ") for "
: Global symbol "smtp" requires explicit package name at ./tailbiter line 107.
: Global symbol "msg" requires explicit package name at ./tailbiter line 110.
: syntax error at ./tailbiter line 111, near "} else"
: Execution of ./tailbiter aborted due to compilation errors.

(3) Performance tests.

    Here's what I did (please read on, I promise I do raise a few points
below).

- Setup:

* Linux 2.0.38, libc5, ext2 (yeah, not the latest-and-greatest, again)

* main.cf:

    alias_database = hash:/etc/postfix/aliases
    alias_maps = hash:/etc/postfix/aliases
    biff = no
    content_filter = smtp:localhost:10025
    default_destination_concurrency_limit = 20
    default_privs = nobody
    disable_dns_lookups = yes
    mail_owner = postfix
    mydestination = $myhostname, localhost.$mydomain
    mydomain = imar.ro
    myhostname = euler.imar.ro
    program_directory = /usr/postfix
    queue_directory = /var/spool/postfix

* master.cf:

    smtp inet n - y - 1000 smtpd
    localhost:10026 inet n - n - 0 smtpd
                -o content_filter= -o myhostname=localhost.imar.ro

* macofida version 0.3

* async logging, that is

    mail.info -/var/log/maillog

* tinydns, configured with localhost = euler.imar.ro = 127.0.0.1

- Testbed:

    /usr/bin/time /usr/postfix/smtp-source -m100 -s20 -l15360 -c \
        -tnulleuler.imar.ro euler.imar.ro

("null" being aliased to |/dev/null).

- Findings:

(a) Basically, the above setup is useless for testing filter
    performance. What seems to happen here is that Postfix will
    happily pump up the messages to the filter at full speed (because
    of the 1000 limit above I get essentially the same rate as in the
    unfiltered case), a huge queue is created by the second smtpd, and
    smtp-source returns without waiting for it to drain. stat-ing the
    spool would probably interfere with the results, so how do we test
    this? Comments / corrections / suggestions welcome.

(b) Running tinydns as above seems to be important. Even with
    "disable_dns_lookups = yes", Postfix tries to resolve localhost.
    Without actually looking into it, I'd say $disable_dns_lookups
    only affects smtp, while smtpd still tries to resolve client's IP.
    Wietse?

(c) I didn't try running tailbiter instead of macofida, but I suspect
    the same thing happens with it: the "backlog" observed earlier by
    Bennett is actually the queue created by the second smtpd, and
    the big speed difference is actually due to "gethostbyaddr" and /
    or other DNS lookup failures. Again, comments / corrections are
    welcome.

    Regards,

    Liviu Daia

--- macofida.old Tue Jun 6 00:32:50 2000
+++ macofida Wed Jun 7 14:34:58 2000
-44,8 +44,8
 use vars qw ( %Params );
 %Params = (
     # parameters you most likely want to change.
- user => 'nobody', # run as this user
- group => 'nobody', # run under this group
+ user => 'filter', # run as this user
+ group => 'filter', # run under this group
     spooldir => '/var/spool/macofida', # spool directory
 
     # parameters that you might want to change.
-58,7 +58,7
     facility => 'mail', # use this syslog facility
     logname => 'macofida', # use this name in syslog
     version => '0.3', # version number of this program
- pidfile => '/tmp/macofida.pid', # name of pid file or 'none'
+ pidfile => '/var/run/macofida.pid', # name of pid file or 'none'
 );
 1;
 # end of Macofida::Config ==================================================
-97,14 +97,14
     my $sock = $self->{ SOCK };
     my $done = undef;
     while (<$sock>) {
- if (/^\.\r\n$/) { $done = 1; last };
+ if (/^\.\r?\n$/) { $done = 1; last };
         s/^\.\./\./o;
         s/\r\n$/\n/o;
         print SPOOL_FILE;
     }
 
     # check that data end with <CR><LF>.<CR><LF>
- $done or $self->_put("550 Broken connection") && return 1;
+ $done or $self->_put("450 Broken connection") && return 1;
 
     # close spool file
     unless (close SPOOL_FILE) { 

-- 
Dr. Liviu Daia               e-mail:   Liviu.Daiaimar.ro
Institute of Mathematics     web page: http://www.imar.ro/~daia
of the Romanian Academy      PGP key:  http://www.imar.ro/~daia/daia.asc