OSEC

Neohapsis is currently accepting applications for employment. For more information, please visit our website www.neohapsis.com or email hr@neohapsis.com
 
Subject: Re: Some questions on postfix, SUMMARY
From: Wietse Venema (wietseporcupine.org)
Date: Fri Mar 03 2000 - 14:48:19 CST

Karimov, Rashid:
> > I prefer an open(), write() and close() routine instead. That allows
> > one to submit messages that are larger than is convenient to keep
> > in memory. Maybe it is because I am an old fart who grew up with
> > machines with less than a megabyte of main memory.
>
> Whatever way is easier for you. I was proposing the buffer because in our
> case the messages are usually short - may be 2K top, and having the
> maildrop_it() with the proposed signature would provide a clean separation
> between message generation and dropping it for delivery ?

With an open/write/close interface you can always build an interface
that takes a whole message. Doing it the other way around would
be quite difficult.

> > > that one can use in a custom message generation program. If it is
> > > a cross-platform (WinNT), it would be a real killer .
> >
> > I have no idea. My programming experience is limited to UNIX VMS
> > DOS and a little bit of Windoze.
>
> As long as format of the message is not OS and/or HW
> specific(byte order for example),it wouldn't be a problem ?

Um, you deleted my text on NAMING the file after the INODE number.
If Windoze's version of stat() does not know about the file inode
number on the server, that would be a problem.

Postfix queue files are byte-order and word-length independent.

> Say, if a message file is generated on Windoze NT box, and then copied
> into maildrop box on Unix, will it be delivered ?

If you create files on Windoze then copy to UNIX then you have just
thrown away most of the performance.

Remember, mail systems like Postfix are disk I/O bound. And it
is faster than other mail systems...

> > Queueing up a million files per hour is 300 files a second. With
> > the present queue organization of one message per file, that requires
> > a lot of disk spindles to share the load, and a persistent write
> > cache that allows the system to sort disk writes for optimal speed.
>
> We are using Veritas to stripe/mirror over a bunch of spindles.
> I understand that I/O subsystem must be fast.

We're talking about files that live only for a very short amount
of time. What I have seen from journaling file systems is that it
doesn't necessarily speed up create/delete of lots of small files.

On the other hand, writing 1 million 4k messages as parts of
large files means you're talking about writing 4GBytes in an hour,
and that is peanuts even for a single disk.

Hmm.. perhaps multi-message files would help.

If you have some time, set up an alias

    null: /dev/null

and try a Postfix benchmark:

    time ./smtp-source -t nulllocalhost -s 20 -m 1000 -l 4096 -c host

That pumps 1000 4k messages into the machine, 20 connections in
parallel.

> Using memory-based FS (such as /tmp on Sol 2.6 for example), is
> blazing fast, but has its drawbacks (if the system dies before
> the generated messages are copied into more persistent storage).

However, a couple gigs of solid-state disk with battery backup
might do the trick. So would multi-message queue files it seems.

> > BTW did you notice that your word-wrapping algorithm sucks?
>
> Is there a way in Outlook to make it wrap up 78 symbols ?
> I am trying to keep the lines short ... sorry for
> inconvenience.

Can't help you there, I've been a UNIX shop for 15+ years now.

> > Postfix queue files have a very specific format, and if the file
> > format does not match expectations, then they are rejected. The
> > format is enforced by the Postfix queue file writing routines.
>
> Can some1 share a file (short test message) from a maildrop
> directory ? I am wondering what does it look like ...

The postcat program decyphers the queue file. The format is bizarre.
You don't want to bypass the Postfix code for writing it, as future
generations will hate your guts for doing so.

> I really think that adding this ability to postfix (ability
> to generate lotsa messages in a hurry, with clear interface)
> would make postfix even a better product.

Multi-message files would do the job. But that won't happen overnight.

        Wietse