OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: hmmmm....
From: Lawrence Greenfield (leg+andrew.cmu.edu)
Date: Tue Sep 19 2000 - 14:13:50 CDT

Courier IMAP implements IMAP over maildirs in a supposed NFS safe
manner. It does so by storing flags, etc., the filename and using
metadata operations as you suggest.

I don't know how well it works.

The Courier author is a little religious at times about how things
should work, even if the standard says something else, and I think
this rubs off on his server, but I don't have any operational
experience.

Finally, there's the issue of performance with maildir. It's likely
that a maildir-based IMAP server does significantly more I/O
operations than other IMAP servers (both free and commercial) out
there, and will incur significant scaling penalties because of it.
(Obviously an opinion.)

I'm not aware of anyone using Courier IMAP for large (greater than a
couple hundred) user populations, though I'm sure the author is more
reliable in this.

Larry

   Date: Tue, 19 Sep 2000 20:23:37 +0200
   From: Thomas Roessler <roesslerdoes-not-exist.org>
   Cc: postfix-userspostfix.org

   On 2000-09-19 19:43:12 +0200, Brad Knowles wrote:

> It is my understanding that maildir with IMAP still requires file
> locking to work (because of the inherent multiple simultaneous
> processes that are accessing the mailbox) , and therefore is
> fundamentally incompatible with NFS.

   I think you can actually work around this, at the cost of metadata
   updates. (Note that this is merely a back-of-the-envelope idea;
   what I'm proposing is not efficient, but it may be a starting point
   for doing things reliably.)

   The most important thing here is that you can do all the
   modifications which don't affect the message file's content in a way
   which doesn't require file locking (the actual locking happens on
   the file system level). IMAP doesn't need to modify files'
   contents.

   The hard part about accessing maildir folders is not concurrent
   access between MUA/imapd and the local delivery agent (that's
   entirely harmless since the LDA doesn't get into the MUA's way), but
   concurrent access of different MUAs which may delete or modify
   messages under each others' feet.

   The simplest way to deal with this is to regularly scan for updates,
   using the unique part of the file name as a unique message-ID, and
   appropriately updating structures. This works iff you can say "ups,
   the message doesn't exist any longer" when the user wants to read it.

   However, if the IMAP details in the back of my head are correct, an
   EXPUNGE is not permitted/desirable as a reply to message-reading
   commands from the client side. Thus, we need to make sure that we
   have a copy of (or rather: a link to) each message available which
   can be accessed when the message no longer exists in the "real"
   folder.

   To achieve this, extend the maildir structure by a working directory
   for each imapd accessing the folder:

           Maildir/
                   cur/
                   new/
                   tmp/

                   work.$HOSTNAME.$TIMESTAMP.$PID/

   Whenever imapd updates it's internal image of the maildir, it
   creates a hard link in the work directory to the actual message file
   in cur/ or new/. Since this kind of updates will happen at
   opportunities where an EXPUNGE is permitted, failure to create the
   link can be safely interpreted as a deleted message, and the
   possibly necessary EXPUNGE can be sent to the client (or the server
   can just forget about that message if the client didn't know about
   it).

   Since the message flags are stored in memory, the unique part of the
   file name should be used for the backup link. Normally, imapd
   accesses the "real" maildir. However, whenever a file to be
   accessed no longer exists (and there is no file with the same unique
   name, but a different flags part - checking for such a file can be
   reduced to a couple of open() or stat() calls if walking the
   directory in user space is undesirable), imapd has to access the
   working copy of the file. At the next opportunity, the client is
   told about the fact that the file was expunged, and the working copy
   is unlinked.

   The problem with this approach is obviously the fact that there are
   lots of metadata updates involved.

   Flames and all kinds of criticism are welcome.

   --
   Thomas Roessler <roesslerdoes-not-exist.org>