OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Some questions on postfix, SUMMARY
From: Brad Knowles (blkskynet.be)
Date: Fri Mar 03 2000 - 11:32:12 CST

At 10:44 AM -0500 2000/3/3, Wietse Venema wrote:

> I prefer an open(), write() and close() routine instead. That allows
> one to submit messages that are larger than is convenient to keep
> in memory. Maybe it is because I am an old fart who grew up with
> machines with less than a megabyte of main memory.

        This also allows the for streaming the connection (a.k.a.,
"pipelining" in SMTP jargon), so that you can get maximum possible
throughput. Pipelining is good. Pipelining is your friend.
Especially when you have millions of recipients, and a large
percentage of them are likely to be at highly congested sites like
AOL, etc....

> Queueing up a million files per hour is 300 files a second. With
> the present queue organization of one message per file, that requires
> a lot of disk spindles to share the load, and a persistent write
> cache that allows the system to sort disk writes for optimal speed.

        You need a decent set of relatively fast disks in RAID-0 or
RAID0+1, with a fast filesystem (e.g., softupdates) in order to do
this. I've been doing some benchmarking, and I can give you some
idea of what kind of hardware it would take to get these kinds of
numbers.

> Since this spends most of the time in file creation overhead, it
> could be done quicker with a different queue organization where
> messages are appended to a log, say a thousand messages per log,
> and where each log is processed sequentially.

        Remember my suggestion from SANE'98, where I said that you should
have a set of queue files that are always kept open, so that you just
append to them when you want to write and then truncate them when
you're done? This sounds pretty familiar.... ;-)

        Of course, this is going one step further, by implementing a
timecaf-like method of storing the queue files, although I'm not
entirely sure that it would be all that much of an improvement over
having a set of constantly open queue files. 

-- 
  These are my opinions and should not be taken as official Skynet policy
=========================================================================
Brad Knowles, <blkskynet.be>       Sys. Arch., Mail/News/FTP/Proxy Admin

Note: No Microsoft programs were used in the creation or distribution of
this message. If you are using a Microsoft program to view this message,
be forewarned that I am not responsible for any harm you may encounter as
a result.

See <http://i-want-a-website.com/about-microsoft/twelve-step.html> for
details.