OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Some questions on postfix, SUMMARY
From: Wietse Venema (wietseporcupine.org)
Date: Thu Mar 02 2000 - 16:52:03 CST

Karimov, Rashid:
> Thank you all for timely responses.
>
> Here's the summary of what I've gathered from the replies:
>
>
> A. The fastest/easiest way to generate the messages (see the original
> post) is to use SMTP to inject messages into postfix and
> let it handle the rest.
> Here is the logic (I am a bit rusty here):
>
> for all recepients {
> generate message
> fork or new thread
> SMTP into postfix (enable it to relay)
> end
> }

I see no control on the number of parallel sessions. Not a problem
as long as you configure postfix to run only a limited number SMTP
processes at a time. The default process limit of 50 is probably
too high for a continuous burst of email. This is controlled in
the master.cf file.

> 2. If the SMTP server (postfix(es) you're realying to) are down, you
> wouldn't be able to generate the messages (there is no way
> to spool them, unless you code for it).
>
>
> B. Another way is to link your app against the same routines that
> sendmail uses to generate a message in the maildrop (enqueue() in
> sendmail.c).
>
> for all recepients {
> generate message
> enqueue(.......)
> }
>
>
> This one I am getting a mixed feeling about.
>
> I am being told (by a whole buncha people, including Wietse) that
> SMTP would be faster here, and w/o having test data I can
> not disagree with that :)

However, it might still be fast enough for your purposes.

> Possible problems with that:
>
> 1. Inner working of postfix might change, and some1 has to maintain
> the custom code.
>
> 2. (??) It is no faster (slower ?) than the SMTP version

It's twice as slow, actually. Postfix wasn;t optimized for local
submission.

> C. Is modification of B, when instead of directly using the same
> routines as sendmail uses, one just fork()s and exec*() sendmail
> program to drop the message into the maildrop.
>
> Possible problems with that:
>
> 1. On a slow system, fork()/exec() pair will be slow :)

Have you tried this? Unless you run a dog of an operating system
you should be able to fork/exec a hundred or more times a second.
That's more than enough, given that it takes much more time to
write mail to disk.

There is the other variant, and that is to take a few Postfix
routines and write a sendmail-like program that injects directly
into the queue. However, that program would have to run set-uid
postfix, which is OK only in a tightly-controlled environment.

If that is an option to consider, then I could write such a beast.
But it would be an optional thing, not recommended for general use.

> D. Using NFS to let message generators to populate the maildrop is
> not a good idea due to usual problems with NFS. How about NFS V3
> on a fast NFS server, say Net App toaster ?

This seems to be the most overlooked fact of email.

    1) The mail system is disk I/O limited.

    2) Adding processors isn't going to help.

> E. Can some1 provide some data as to how many messages are generated
> locally in what time (we're looking to 50K - 1Mil in a
> HURRY !), size is under 4K.

With submission via SMTP, my laptop can accept 1000 4k messages in
47 seconds using 20 parallel SMTP sessions, that's over 20 a second.
I'd expect the same timings with direct into the queue submission,
since all the time is spent waiting for the disk.

You can expect to get much higher rates as your file system is
striped over multiple disks, and as the file system can use write
buffering in non-volatile buffers (maybe the Auspex has an edge
here).

That should get you into the hundreds of messages a second.
So, queueing up a million would take an hour or so.

Is that within your limits of feasibility? If not, you could throw
more iron at the problem and divide the load over even more disks.

> How long would it take to fan them out to Internet, assuming regular
> client mix (AOL/Hotmail/etc) ? What do folx use in terms
> setting on postfix systems ?

Here, you'll find that performance is utterly painful. In fact, if
you send lots of mail into AOL they are likely to drop it silently
on the floor until to register as a bona fide mail provider.

You will want to run up to a thousand SMTP client processes, with
per-destination concurrency limits turned off.

        Wietse