OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: defer pickup notification?
From: Bennett Todd (betrahul.net)
Date: Sun Apr 16 2000 - 12:18:21 CDT

2000-04-16-12:43:02 Viraj Alankar:
> I am trying to do a mass mail to a large customer address list
> (around 500k email addresses).

Whew. That's a pile o' work. At 5/second, which is a pretty good
rate for getting messages out the door, that's over a solid day.

To look at it another way, supposing your note is reasonably short,
say 4KB including headers, c. 30-40/sec will be saturating a T1
with outgoing email --- very, very hard to do --- and at 40/sec
you're still talking quite a few hours.

Picture what happens sending a single message. It does its DNS
stuff, it opens its SMTP connection, it says howdy, here's a
message, catch, it waits for it to be accepted, then hangs up.
That's an appreciable fraction of a second to an unloaded server
on your LAN. That's many seconds on average to a randomly-selected
SMTP server Out There somewhere. You want to get thousands of smtp
clients working on your side. I'd recommend splitting the load
around a small farm of servers, as that's the cheapest, simplest way
to multiply your disk-to-memory bandwidth and get a bunch of disks
all humming at once, along with simply acres of RAM to hold all the
zillions of smtp clients.

I'd also recommended setting up one really really memory-rich server
and running dnscache <URL:http://cr.yp.to/dnscache.html> on it; it
handles heavy traffic very well indeed. Let all your servers share
one common caching nameserver.

And 500k message is about 500 times as many as you want to have
sitting in a single flat directory, ever. Unless you're running
something exotic, like Reiserfs, your directory lookup and modify
times go up linearly with directory size, which means filling and
draining a directory file-at-a-time goes up quadratically --- ouch!
You're over the knee.

I'd recommend setting up a feeder process; the fastest way to feed
messages into Postfix is via SMTP, so just use Net::SMTP from libnet
and do it in perl. That cuts out some disk I/O relative to using
/usr/lib/sendmail, which is helpful here:-).

I don't know what's the best way to do it, but what you _really_
want is to tune your smtpd receiving processes so they refuse to
accept messages faster than some really large number of smtp clients
--- how ever many you can hold in real memory --- can drain them.

I guess if you put the incoming queue on a separate filesystem,
small enough that it couldn't hold more than a couple thousand of
your messages, then set queue_minfree, that'd be one way to do it.
There has to be something less painful, though.

-Bennett

  • application/pgp-signature attachment: stored