OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: pop-before-smtp (was Re: FreeInet and checking mail?)
From: Greg A. Woods (woodsweird.com)
Date: Tue Jun 06 2000 - 12:52:04 CDT

[ On Tuesday, June 6, 2000 at 16:32:00 (+0200), Brad Knowles wrote: ]
> Subject: Re: pop-before-smtp (was Re: FreeInet and checking mail?)
>
> It may do reading and writing of the message store (mostly
> deleting messages or moving them from folder to folder), but from the
> server side that is totally unrelated to the injection of mail
> messages.

Though I've only run relatively small IMAP servers I do see a pattern of
users moving mail from their clients up to the server, at least within
their allocated storage limits (and when no limits are imposed users
will tend to use their server folders as secondary storage for all kinds
of junk). I.e. IMAP is already almost always used to move messages in
both directions. Conceptually speaking it's just as if you have an
"outbox" folder that some "agent" comes along and "empties" on occasion
for you. In fact this might even be an obvious way to implement things
internally too (which would make it trivial to move any additional
header processing into a separate, and thus more scalable, phase). I've
not looked at existing IMAP extensions which implement message
submission, but perhaps they already use this model.

> I don't give a flying mongolian cluster sexual intercourse about
> client-side "efficiency", since this has absolutely nothing
> whatsoever to do with server-side scalability.

I'm not talking about client-side "efficiency" in any way either (though
it does of course matter to some folks! ;-).

> Users don't keep the connections open all the time. They
> download their e-mail, read it locally, write replies, then upload
> their outgoing e-mail and check their inbox again. Anyone who keeps
> the connection open all the time is wasting major resources on the
> server, the access system (now they have to have a modem for every
> user), and on the transport network itself.

You should study the behaviour of existing IMAP clients a little more
closely before you make such claims. Some are rather "nasty" in the way
they hold even multiple connections open.

As RFC 2683 says: "IMAP's design encourages long-lived (and mostly
inactive) client/server sessions." The apparently upcoming revision of
IMAPv4 will probably specify that an autologout timer "MUST" allow at
least 30 minutes inactivity before closing the connection.

Any IMAP server implementation that wastes major resources just because
users keep the IMAP connection open for long periods is not scalable.

I.e. I don't buy this part of your argument at any level! ;-)

> No, not at all. I use different authentication systems that are
> architected exclusively for the outbound mail system and which are
> dedicated to that purpose. They are totally separate and distinct
> from the authentication systems that are used for users logging in to
> download their e-mail.

Well then that's outside the scope of my concerns. I want strong
accountability from the postmaster's point of view -- i.e. I do not want
an annonymous collection box on the street corner, but rather I want
something more akin to commercial couriers who often require an existing
valid account before they'll accept a package for delivery, or indeed
the outbox on everyone's desk such that the mail clerk can verify who's
doing the sending. In my scenario separate authentication would be out
of the question -- it must be the same anyway. This argues for even
stronger security of the outbound relay server(s) and indeed of the
ability to make outbound SMTP connections too, of course.

> Furthermore, I have my outbound mail relays outside the firewall
> (although they have their own host-level firewalling) because the
> majority of traffic our users have to send is for other addresses off
> the system, while the POP3/IMAP servers are inside the firewall since
> obviously that traffic must be 100% for our users exclusively.

You seem to be solving different problems that I am. Indeed outbound
mail relays still have to be secured, as do the IMAP servers, but that's
only part of the picture that I'm interested in.

> I don't want to unnecessarily tie the inbound to the outbound
> systems. Right now, I have one primary inbound MX, one backup
> inbound MX, and one back-end POP3 server (which we can scale by
> adding more POP3 servers). However, I have multiple outbound mail
> relays.

Once again I can't see where I've made any such assumption.

Just as there's no need to point your inbound MX at your IMAP server,
neither is there any reason to force you to make your IMAP server also
deliver outbound mail to the Internet. With IMAP message submission I
too can have as many IMAP servers as I need, and I can also have as many
separate outbound mail relay servers as I need (running Postfix, or
whatever! ;-).

> I want to be able to add more outbound mail relays if that is
> what I need, without being forced to integrate them into the
> POP3/IMAP/TMSAP/whatever-the-hell-you-want-to-think-of-it-as system,
> and if I need more resources on the front-end MX servers or the
> POP3/IMAP system, I want to be able to add those without affecting
> the outbound mail relay system.

Sure! Go ahead! The more the merrier. I've already said this. You do
not have to integrate them at all. There are an almost infinite number
of ways of forwarding the submitted message to the outbound mail
delivery system(s) -- that's merely an implementation detail.

> When you scale systems to handle millions of users, you want to
> be able to define the roles of machines as tightly as you possibly
> can, so that when it comes time to have dozens or hundreds of them,
> they are all cookie-cutter installs, and you can add resources to
> only those parts of the system that actually need them.

Once again: "No Problem(tm)!"

In fact as I've already said the problem is much more tractable when
you've already scaled the IMAP server to handle the expected user base.
All you do is add a very small factor to the RAM and CPU (and *maybe*
the network) resource calculation. This is trivial and still extremely
scalable. You can then separately, and dynamically, scale your outbound
mail relay servers depending on whether your users exchange a lot of
mail amongst themselves, or whether they send most of their mail
off-site. The only difference is that you need to be aware of the
additional resources necessary to forward mail from the IMAP server to
the mail relay host(s).

Note that if you've already solved the IMAP server scaling and
reliability issues then you'll also have much more direct control over,
and many more options for, the way you handle outbound mail relay load
sharing and reliability if you handle message submission through IMAP.

> See above. You are not comprehending the scalability issues.

On the contrary -- I see them *very* clearly.

> If you choose to tie all your servers into the "one machine does
> all" model, then so be it. However, I should not be required to do
> the same.

I certainly did not say anything of the kind.

I have never liked the idea of using SMTP for message submission. When
I first read about POP over a dozen years ago I was dismayed that it
didn't immediately deal with the issue of message submission, either
within itself or with a companion protocol.

I certainly do not buy the apparent IETF "party line" that "Some other
protocol should be used for message submission."

Hmmm.... I wonder if my "outbox" folder idea could be implemented
successfully without ever having to change the IMAP protocol itself --
just define an "outbox" folder either for each user or for the system as
a whole and then teach the clients to use the most specific one they can
find for submission. A "mail-clerk" process could collect messages from
each outbox and after correcting any sender addresses, adding any
necessary postage (;-), etc., could forward them to the outbound relay
server(s).... Successfully sent messages could even be moved to the
user's "sent-mail" folder and first-level bounces moved to the "bounces"
folder, and of course annotated to describe the reason for the bounce.

> True enough. People need to use message encryption if they want
> to prevent that from occurring. However, assuming you "trust" the
> admins of the sites where the MTAs encrypt communications between
> each other, then link-level encryption does help ensure message
> integrity and freedom from spoofing and man-in-the-middle attacks, if
> not message security from the administrators of those systems.

How can you trust an admin who may have been issued a "wire tap" order
or whatever -- something the end user would explicitly be prevented from
discovering? Even if you're not the subject of the wire tap you may not
want your e-mail to be scanned! ;-) 

-- 
							Greg A. Woods

+1 416 218-0098      VE3TCP      <gwoodsacm.org>      <robohack!woods>
Planix, Inc. <woodsplanix.com>; Secrets of the Weird <woodsweird.com>