I'm seeing a lot of these:

Apr 29 08:55:20 huey postfix/qmgr[27099]: warning: mail_scan_any: got EOF; expected: string
Apr 29 08:55:20 huey postfix/qmgr[27099]: warning: private/local: malformed response
Apr 29 08:55:20 huey postfix/qmgr[27099]: fatal: qmgr_active_defer: update active/8/6/86C188A49 time stamps: No such file or directory
Apr 29 08:55:21 huey postfix/master[9658]: warning: process /usr/local/sbin/qmgr pid 27099 exit status 1
Apr 29 08:55:35 huey postfix/qmgr[405]: warning: qmgr_active_done: remove B087A8A77 from active: No such file or directory
Apr 29 08:55:35 huey postfix/qmgr[405]: warning: qmgr_active_done: remove B054D8AAE from active: No such file or directory
Apr 29 08:55:35 huey postfix/local[13028]: fatal: open active B0EC28B8E: No such file or directory
Apr 29 08:55:35 huey postfix/local[12159]: fatal: open active BADD58A9C: No such file or directory
Apr 29 08:55:35 huey postfix/smtp[29196]: fatal: open active D8ABB867B: No such file or directory

        They occur during the larger mailings, and there is no "too many open files" error. After these occur, there are tons of "unknown mail transport error" messages for millions of recipients in the log. It appears that something is slow to update, whether it's the disk (unlikely), the filesystem (vxfs), or solaris to pick up the changes in the fs, or postfix to notice changes.

The vxstats for the volume on which the postfix queue and mail log is located is:

                        OPERATIONS BLOCKS AVG TIME(ms)
TYP NAME READ WRITE READ WRITE READ WRITE
vol opt 35410 52643 2656855 812912 11.7 197.0 <-- emc powerpath software install point
vol rootvol 230907 778715 6810633 12245754 9.0 109.1
vol swapvol 5354378 2181587 89343191 393276928 21.3 56.8
vol var 710619 263235 33304775 6326396 8.2 91.7 <-- other syslog output
vol appsvol 5428246 52821892 132937732 354107054 6.1 1.1 <-- postfix queues and syslog mail log

        The appsvol is a raid 0 stripe on an emc array, which is how we get such fast throughput for disk i/o. Postfix actually sees 50% throughput decrease when the syslog maillog is dumped to /var/log, which are local, veritas-mirrored, 10k rpm scsi disks. The last message from the log snippet "fatal: open active D8ABB867B" shows that 45 different smtp processes attempted to open that file. It then shows master logging the subsequent death of the 45 smtp processes.

        A vxtrace during these errors shows no veritas level errors. The filesystem is 3.03 and the volume mgr is 3.02c. I've done no veritas tuning to this point, as I'm still gathering info from the emc array on the parameters that would make sense to change.

-- 
|  Andy   |     e-mail      |      web       |       gpg public key       |
|         | andylewman.com | www.lewman.com |  www.lewman.com/deimos.sig |
You will be attacked by a beast who has the body of a wolf, the tail of
a lion, and the face of Donald Duck.
-
To unsubscribe, send mail to majordomopostfix.org with content
(not subject): unsubscribe postfix-users

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]