OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: ideas on determining socket / OS latency
From: Nagendra Mishr (nboldfish.com)
Date: Tue Mar 14 2000 - 03:21:36 CST

Some evidence...
        
        Over the last few days, we have been collecting data and making one or
two modifications in each run.. The architecture for our sending engine
is very simple...

1. Fetch the content for distrubution and save to disk (heavly disk I/O)
2. When all content has been fetched, we start queueing the messages
into memory.. allowing 2500 to be in memory at a time. A thread always
wakes up every 30 seconds to make sure that the queue is full.
3. A set of dedicated threads pull emails out of the in-memory queue and
distribute emails to 1 postfix server... (3 threads per server).. So if
a server responds faster, it gets messages faster.

In our tests, we deliver 43xx indivual emails.. none of them locally.
each message is about 20-22k. the largest aaround 81k and the smallest
some 400 bytes.. Each Machione has its oen Caching DNS server.

Two tests show the problem I'm trying to address

In this run, we deliver to three postfix servers, and we counted the
number of "sent" lines in /var/log/maillog over a 10 minute period.

, M1, M2, M3
Mar 9 12:00 , 189 , 126 , 143
Mar 9 12:10 , 381 , 348 , 344
Mar 9 12:20 , 256 , 429 , 359
Mar 9 12:30 , 285 , 485 , 270
Mar 9 12:40 , 66 , 555 , 138
Mar 9 12:50 , 31 , 33 , 23
Mar 9 13:00 , 1 , 74 , 1

In this test, we ran against two machines.. leaving the third idle..

, M1, M2, M3
Mar 13 13:50 , 1 , 0 , 0
Mar 13 14:00 , 582 , 615 , 1
Mar 13 14:10 , 897 , 845 , 1
Mar 13 14:20 , 859 , 540 , 1
Mar 13 14:30 , 17 , 16 , 1
Mar 13 14:40 , 5 , 14 , 0
Mar 13 14:50 , 0 , 0 , 0

The interestng point here is that if we turn off M3, then M1, and M2
speed up? points to network congestion to me... but we're told that
this is a 100MBPS switched with 1gB byte backplane.. and going out of
the server, we have 45 MBPS to play with... we're only touching about
400 Kbps.

On the sending side, I see that we are feeding the queue's slowly...
i.e. very few messages are pulled out of the queues.. The first minute
is very quick.. but then, that's just the number of messages we put in
the queue before it was full. So we can read 1760 messages and feed
them to the queue in a minute.

Mon Mar 13 12:03:03 CST 2000 Starting queueing emails for
IEMailMDPortfolio.dat
Mon Mar 13 12:04:08 CST 2000 Processed 1760 files this run for
IEMailMDPortfolio.dat. 2677 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:04:45 CST 2000 Processed 30 files this run for
IEMailMDPortfolio.dat. 2647 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:05:31 CST 2000 Processed 64 files this run for
IEMailMDPortfolio.dat. 2583 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:06:19 CST 2000 Processed 148 files this run for
IEMailMDPortfolio.dat. 2435 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:07:00 CST 2000 Processed 50 files this run for
IEMailMDPortfolio.dat. 2385 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:07:49 CST 2000 Processed 182 files this run for
IEMailMDPortfolio.dat. 2203 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:08:30 CST 2000 Processed 43 files this run for
IEMailMDPortfolio.dat. 2160 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:09:19 CST 2000 Processed 194 files this run for
IEMailMDPortfolio.dat. 1966 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:10:00 CST 2000 Processed 44 files this run for
IEMailMDPortfolio.dat. 1922 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:10:52 CST 2000 Processed 292 files this run for
IEMailMDPortfolio.dat. 1630 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:11:31 CST 2000 Processed 79 files this run for
IEMailMDPortfolio.dat. 1551 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:12:17 CST 2000 Processed 101 files this run for
IEMailMDPortfolio.dat. 1450 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:13:05 CST 2000 Processed 198 files this run for
IEMailMDPortfolio.dat. 1252 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:13:45 CST 2000 Processed 34 files this run for
IEMailMDPortfolio.dat. 1218 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:14:31 CST 2000 Processed 111 files this run for
IEMailMDPortfolio.dat. 1107 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:15:20 CST 2000 Processed 235 files this run for
IEMailMDPortfolio.dat. 872 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:16:04 CST 2000 Processed 187 files this run for
IEMailMDPortfolio.dat. 685 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:16:45 CST 2000 Processed 57 files this run for
IEMailMDPortfolio.dat. 628 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:17:36 CST 2000 Processed 167 files this run for
IEMailMDPortfolio.dat. 461 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:18:16 CST 2000 Processed 39 files this run for
IEMailMDPortfolio.dat. 422 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:19:15 CST 2000 Processed 285 files this run for
IEMailMDPortfolio.dat. 137 files left. ... SMTPq is full. waiting for
30 seconds
Mon Mar 13 12:20:08 CST 2000 Processed 128 files this run for
IEMailMDPortfolio.dat. 9 files left. ... SMTPq is full. waiting for 30
seconds

Here are the items in the main.cf minus the comments..

alias_maps = dbm:/etc/aliases
alias_maps = hash:/etc/aliases
command_directory = /var/postfix/bin
daemon_directory = /var/postfix/lib
debug_peer_list = 198.137.221.211
default_destination_concurrency_limit = 40
hash_queue_names = defer, active
local_destination_concurrency_limit = 4
mail_owner = postfix
maximal_queue_lifetime = 1
mydestination = $myhostname, localhost.$mydomain,
qmail2.noc.stockpoint.com
mydomain = noc.stockpoint.com
myhostname = qmail1.noc.stockpoint.com
mynetworks = 198.137.221.0/24, 127.0.0.0/8, 204.71.102/24
myorigin = $mydomain
myorigin = $myhostname
program_directory = /var/postfix/bin
qmgr_fudge_factor = 10
queue_directory = /var/postfix/queue

I also have a tcp dump for the hour period that all this happened for
M2.

Nagendra