poboxverysmall.org
Date: Sun Apr 01 2007 - 08:10:51 CDT

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Sean Ryan wrote:
>> Could you give me a tip how to debug this process, where are the points
>> to check/monitor, except the maillog and queue (the mail server is
>> Postfix and Cyrus)?
>>
>> The account is IMAP, do not know if this helps.
>>
>> What is 'sure' is that -
>>
>> - the MUA setting (SMTP domain) hasn't been changed
>> - the SMTP domain itself hasn't been changed
>> - the mail server configuration hasn't been changed
>> - the problem seemed to appear and disappear on its own (nothing was
>> done to fix it - one day it did not work, the next it worked)
>> - the MUA (both Thunderbird and Outlook Express) reported 'mail sent'
>> - the lost mail could not be found in any queue
>>
>> (Unless something has been hacked, which is excluded as a possibility
>> for now.)
>
> Did anything happen like rebooting ( MUA machine or SMTP machine )? It
> is possible that this might cause something that was working to stop (
> or vice versa ). A good example is that if another mta ( such as exim
> which logs to a different location ) started up instead of postfix due
> to a configuration error.

To my knowledge nothing has been changed on the server (it's a dedicated
server) on that day (the problem as about half a day or so). Before we
had qmail, which is not un-installed, as far as I know. But I am not
aware of any event (i.e. reboot of something) which might have caused
configuration change or some of the possibilities described by you.

> One thing that I'm not clear on from your description ( forgive me if I
> just missed it ), was the exact situation. You have a workstation, and
> this workstation is NOT running an SMTP server ( ie just a windows
> workstation ) ?

Yes.

There is a FreeBSD dedicated server with Postfix/Cyrus, Squirrelmail and
another php application - the Squirrelmail and the php application did
not have any problems to send mail. And there is a Windows workstation
where I tried Thunderbird (my primary mail client) and Outlook Express
(which I set for the test) - during this period both gave the impression
that mail is sent, but mail did not arrive, neither was logged in
maillog, nor was it in the queue, to my experience (I might have missed
something) I could not find any trace of the mail with 'grep'. Somebody
advised me to try to send mail using telnet from the Workstation, but at
this moment I did not know how and when I found out how on the next day,
the problem did not exist anymore. So I can't tell if telnet would have
worked (resp. I do not know what telnet would have reported, i.e.
somebody mentioned that it might be a DNS problem).

> 1) Make sure 100% that the connection you think
> is being made is. If you have access to the smtp server, then do a
> tcpdump on the server side and watch yourself connect.

Oh, thanks a lot! This is actually my main question - tools to debug and
locations to look. I guess now I can't use it, but I wrote it down in my
debug notes for when it happens next time.

> Something like
> this:
>
> There are many options, and if you are inclined you can actually capture
> the full data as it comes in ( thus being able to verify 100% ). You
> could also install ethereal/wireshark ... which does the exact same
> thing ... but in a nice gui form factor.

Yes! I have wireshark, but it did not occur to me that I can use it to
see what the MUAs (Thunderbird/Outlook Express) actually do. Put it also
in my debug notes for next time, thanks.

> A) On the Client Side ( ie where outlook is ). Also note, if this is a
> window machine, you may not see what you type in because echo might be
> turned off :
> telnet smtp.server.com 25

Yes, as I already learned how to use telnet in order to test the SMTP.
Unfortunately after the problem was gone :/

> B) On the Server Side ( ie where postfix is running NOTE: I use port 80
> here in the tcpdump, this would be replaced with 25 ):
> rootserver:~# tcpdump -s0 -XXX port 80
> tcpdump: verbose output suppressed, use -v or -vv for full protocol
> decode
> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 19:00:03.514386 IP client.ip.addr.45482 > server.www: S
> 268794353:268794353(0) win 5840 <mss 1452,sackOK,timestamp 158639826
> 0,nop,wscale 7>
> 0x0000: 0011 09d1 95a0 0001 e8d5 b309 0800 4500
> ..............E.
> 0x0010: 003c df23 4000 2d06 dcca d863 6188 40fb
> .<.#.-....ca..
> 0x0020: 16e7 b1aa 0050 1005 79f1 0000 0000 a002
> .....P..y.......
> 0x0030: 16d0 b332 0000 0204 05ac 0402 080a 0974
> ...2...........t
> 0x0040: a6d2 0000 0000 0103 0307 ..........
> 19:00:03.516255 IP server.www > client.ip.addr.45482: S
> 1611562685:1611562685(0) ack 268794354 win 5792 <mss
> 1460,sackOK,timestamp 1143357142 158639826,nop,wscale 2>

Really thanks a lot. This is what I wanted to know - tools to debug.

> The data is mostly unimportant, what you are looking for is that the
> connection came in.

Yes, correct.

> *IF* that is the case, then you might need to dig
> deeper to ensure that the full message is transferred. Also make sure
> that you see the same connection happening while using outlook.

Yes! This is a great beginning. I assume this will give a lot of data
(or no data, which is also something) to help solve the problem, or move on.

> 2) Conduct a command line smtp session ... make sure everything looks
> ok ... something like:
> sryanoberon:~$ telnet inbound.evercrack.com.emailmx.com 25
> Trying 64.97.156.1...
> Connected to inbound.evercrack.com.emailmx.com.
> Escape character is '^]'.
> 220 Email Defense System ESMTP EMD
> ehlo deimos.evercrack.com
> 250-emd2-imf4.emaildefenseservice.com
> 250-PIPELINING
> 250-SIZE 25000000
> 250-ETRN
> 250-ENHANCEDSTATUSCODES
> 250 8BITMIME
> mail from: <sryanevercrack.com>
> 250 2.1.0 Ok
> rcpt to: <sryanevercrack.com>
> 250 2.1.5 Ok
> data
> 354 End data with <CR><LF>.<CR><LF>
> Subject: testing
> .
> 250 OK

Yes.

> 3) If this is not good enough ... do something client related where you
> check the outbound smtp connections. It's totally anti-rfc, but perhaps
> outlook is decided to deliver the mail right to the MX, bypassing your
> outbound settings ( this is a long shot, but you can check anyway as a
> last resort ).

OK

> 4) Check the mail server to ensure that the process you think is
> running, in fact is.

Yes, I wanted to do this, but I could not figure out what is the smtp
process.

Thanks for this hint as well.

> A) On the server side, see who is listening on port 25
> #netstat -tanwup | grep LISTEN | grep 25
> tcp 0 0 127.0.0.1:25 0.0.0.0:*
> LISTEN 3123/master
>
> B) Then check what he's got open:
> # lsof -i | grep 3123
> master 3123 root 11u IPv4 8960 TCP localhost:smtp
> (LISTEN)
>
> C) And check the proc info:
> rootoberon:~/code/rotk/lib/OGRE# ls -al /proc/3123
> total 0
> dr-xr-xr-x 5 root root 0 2007-03-30 07:30 .
> dr-xr-xr-x 122 root root 0 2007-03-24 11:39 ..
> dr-xr-xr-x 2 root root 0 2007-03-31 20:19 attr
> -r-------- 1 root root 0 2007-03-31 20:19 auxv
> -r--r--r-- 1 root root 0 2007-03-30 23:32 cmdline
> -r--r--r-- 1 root root 0 2007-03-31 20:19 cpuset
> lrwxrwxrwx 1 root root 0 2007-03-31 20:17 cwd -> /var/spool/postfix
> -r-------- 1 root root 0 2007-03-31 20:19 environ
> lrwxrwxrwx 1 root root 0 2007-03-31 07:30 exe ->
> /usr/lib/postfix/master
> dr-x------ 2 root root 0 2007-03-31 20:05 fd
> -r--r--r-- 1 root root 0 2007-03-31 20:19 maps
> -rw------- 1 root root 0 2007-03-31 20:19 mem
> -r--r--r-- 1 root root 0 2007-03-31 20:19 mounts
> -r-------- 1 root root 0 2007-03-31 20:19 mountstats
> -rw-r--r-- 1 root root 0 2007-03-31 20:19 oom_adj
> -r--r--r-- 1 root root 0 2007-03-31 20:19 oom_score
> lrwxrwxrwx 1 root root 0 2007-03-31 20:17 root -> /
> -r--r--r-- 1 root root 0 2007-03-31 20:19 smaps
> -r--r--r-- 1 root root 0 2007-03-30 23:32 stat
> -r--r--r-- 1 root root 0 2007-03-31 20:19 statm
> -r--r--r-- 1 root root 0 2007-03-30 23:32 status
> dr-xr-xr-x 3 root root 0 2007-03-31 20:19 task
> -r--r--r-- 1 root root 0 2007-03-31 20:19 wchan

These are precious tips, thank you so much.

> If I go deeper than that ... we're getting into advanced
> troubleshooting. In all honesty, try the simple stuff first, this is
> almost 99% chance that this is somehow user error in some way. The
> bottom line is that postfix logs all transactions ... so if the mail was
> "sent", and postfix has no record whatsoever ... then either you're
> checking the wrong postfix, or your client is sending to the wrong
> place.

Yes. I am sure that if I knew all this then, I would have found out enough.

Thank you so much for your time to type all this.

If I catch it, I'll write back.

Iv

PS I replied back to the list... hope it's OK.

• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]