Debug your IMAP server with Telnet

Email by IMAP rocks – there are so many benefits to using it over POP3, particularly in an office environment. However many IMAP clients really suck when it comes to manipulating accounts with large messages. We recently watched a server brought to it’s knees during a email server migration – by an 165MB email containing wedding pictures that someone had decided to forward throughout the network!

At that stage you are better off talking directly to your email server with Telnet.You will of course need, a shell prompt or a Windows command prompt. Also some of these commands will give you lots of output. To make life easier use a terminal program that allows you to capture output to a file, or that gives you LOTS of scrollback memory. For those of you stuck on Windows, Putty is excellent for this purpose.

(The first line of each code window shows you what to tyoe – the following lines are the server response.)

Connecting to the server (non-encrypted connection)

Open a Windows CMD prompt or a Login to your server:
[cc lang=’bash’ line_numbers=’false’]$ telnet 192.168.100.100 143
Trying 192.168.100.100…
Connected to 192.168.100.100.
Escape character is ‘^]’.
* OK imapfront ready. + stunnel
[/cc]
… and you are connected.

Connecting to the server (SSL encryption)

From your Linux command prompt (sorry Windows users):

[cc lang=’bash’ line_numbers=’false’]

$ openssl s_client -connect 192.168.100.100:993
CONNECTED(00000003)
depth=0 /C=–/ST=—-/L=Wherever/O=Example/OU=Main/CN=mail.example.com/emailAddress=admin@example.com
verify error:num=18:self signed certificate
verify return:1
depth=0 /C=–/ST=—-/L=Wherever/O=Example/OU=Main/CN=mail.example.com/emailAddress=admin@example.com
verify return:1
—
Certificate chain
0 s:/C=–/ST=—-/L=Wherever/O=Example/OU=Main/CN=mail.example.com/emailAddress=admin@example.com
i:/C=–/ST=—-/L=Wherever/O=Example/OU=Main/CN=mail.example.com/emailAddress=admin@example.com
—
Server certificate
—–BEGIN CERTIFICATE—–
MIIDkjCCAvugAwIBAgIESfEsLjANBgkqhkiG9w0BAQQFADCBkTELMAkGA1UEBhMC
LS0xDTALBgNVBAgTBC0tLS0xDzANBgNVBAcTBkR1YmxpbjEUMBIGA1UEChMLQ3Jv
c3Nlbi5vcmcxDTALBgNVBAsTBE1haW4xGzAZBgNVBAMTEnNhaWRiaC5jcm9zc2Vu
Lm9yZzEgMB4GCSqGSIb3DQEJARYRYWRtaW5AY3Jvc3Nlbi5vcmcwHhcNMDkwNDI0
MDMwNDE0WhcNMTAwNDI0MDMwNDE0WjCBkTELMAkGA1UEBhMCLS0xDTALBgNVBAgT
BC0tLS0xDzANBgNVBAcTBkR1YmxpbjEUMBIGA1UEChMLQ3Jvc3Nlbi5vcmcxDTAL
BgNVBAsTBE1haW4xGzAZBgNVBAMTEnNhaWRiaC5jcm9zc2VuLm9yZzEgMB4GCSqG
SIb3DQEJARYRYWRtaW5AY3Jvc3Nlbi5vcmcwgZ8wDQYJKoZIhvcNAQEBBQADgY0A
MIGJAoGBAK4JxhFkePIqJGjT42XZ4MFZmVtgpqSsEwxKqaJCZuBOA/X3KTYgdC2X
w6E8HC7zpFX+LSdxMTI7FSeVqY0/XqLYqE3TnxVNuK9MIp/kJ0nUy+qFqyE1jfbr
YEfVNQSNqdmMsyK9dZUCaILyafOBZSNWVpXcBZRL+wpsbF8Pc1srAgMBAAGjgfQw
gfEwHQYDVR0OBBYEFAffxXjuAZHby/LrHU824rgJM0QrMIHBBgNVHSMEgbkwgbaA
FAffxXjuAZHby/LrHU824rgJM0QroYGXpIGUMIGRMQswCQYDVQQGEwItLTENMAsG
A1UECBMELS0tLTEPMA0GA1UEBxMGRHVibGluMRQwEgYDVQQKEwtDcm9zc2VuLm9y
ZzENMAsGA1UECxMETWFpbjEbMBkGA1UEAxMSc2FpZGJoLmNyb3NzZW4ub3JnMSAw
HgYJKoZIhvcNAQkBFhFhZG9jGtejcm9zc2VuLm9yZ4IESfEsLjAMBgNVHRMEBTAD
AQH/MA0GCSqGSIb3DQEBBAUAA4GBAFiGJSuYaDoZak4sbz3xa0vNBX9kNbOSZVV2
7Rwinjtve/GdnesXqI+mt31yDKDW5rdHpgtsE35DFIEW4vYlOlvnn3pC1/syI2xX
5Tylbd/mIeG7xo1ZreVefQ9nzycDHxJcoQ7xyqMCrRSCnGhGyZAIxGYxZPmliuGL
YBCtLyv6
—–END CERTIFICATE—–
subject=/C=–/ST=—-/L=Wherever/O=Example/OU=Main/CN=mail.example.com/emailAddress=admin@example.com
issuer=/C=–/ST=—-/L=Wherever/O=Example/OU=Main/CN=mail.example.com/emailAddress=admin@example.com
—
No client certificate CA names sent
—
SSL handshake has read 1096 bytes and written 330 bytes
—
New, TLSv1/SSLv3, Cipher is DES-CBC3-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : SSLv3
Cipher   : DES-CBC3-SHA
Session-ID: 00000000D7DA47BAAEE91BD4A6A5BCC76A3FB92B43A6107BDB96921669B3AC3C
Session-ID-ctx:
Master-Key: 956DACA93DA7592803F628D5F2189E04C0FFE9DF284B2C4E567FC1691258D23C8D830D3EBAC9082671A7800378B50764
Key-Arg  : None
Start Time: 1264961668
Timeout  : 300 (sec)
Verify return code: 18 (self signed certificate)
—
* OK imapfront ready.
[/cc]

Logon and looking at folders

The first word of an IMAP command is an arbitrary command identification tag read up on RFC3501 if you care. For our demo we will use the letter ‘A’. So
[cc lang=’bash’ line_numbers=’false’]
a login username password
a OK Logged in.
[/cc]
List your folders:
[cc lang=’bash’ line_numbers=’false’]
a list “””*”
* LIST (HasNoChildren) “.” “Junk”
* LIST (HasNoChildren) “.” “Drafts”
* LIST (HasNoChildren) “.” “Templates”
* LIST (HasNoChildren) “.” “Archived”
* LIST (HasNoChildren) “.” “Trash”
* LIST (HasNoChildren) “.” “junkmail”
* LIST (HasNoChildren) “.” “Sent”
* LIST (HasNoChildren) “.” “INBOX”
* LIST (HasNoChildren) “.” “sent-mail”
a OK List completed.

[/cc]
In this case we’ll look at the INBOX:

[cc lang=’bash’ line_numbers=’false’]
a select INBOX
* FLAGS (Answered Flagged Deleted Seen Draft)
* OK [PERMANENTFLAGS (Answered Flagged Deleted Seen Draft *)] Flags permitted.
* 1954 EXISTS
* 0 RECENT
* OK [UIDVALIDITY 1179598577] UIDs valid
* OK [UIDNEXT 4572] Predicted next UID
a OK [READ-WRITE] Select completed.
[/cc]

Looking at messages

IMAP messages are indexed in the order in which they appear in a folder. If you to manipulate messages, use the UID field. The UID is unique and doesn’t change. For example lets look at the UID for the first 10 messages in the folder we selected above:

[cc lang=’bash’ line_numbers=’false’]

a fetch 1:10 (uid)
* 1 FETCH (UID 2558)
* 2 FETCH (UID 2559)
* 3 FETCH (UID 2560)
* 4 FETCH (UID 2561)
* 5 FETCH (UID 2562)
* 6 FETCH (UID 2563)
* 7 FETCH (UID 2564)
* 8 FETCH (UID 2565)
* 9 FETCH (UID 2566)
* 10 FETCH (UID 2567)
a OK Fetch completed.

[/cc]

This is important: you do not want to manipulate messages using the message id – it will change as you operate on mailboxes. You should use the UID whenever you are making changes to the mail store.

So, let’s do a wildcard fetch and get the message size as well (this can generate LOTS of output):

[cc lang=’bash’ line_numbers=’false’]

a fetch 1:* (uid, rfc822.size)

* 1 FETCH (UID 2558 RFC822.SIZE 35058)
* 2 FETCH (UID 2559 RFC822.SIZE 17920)
* 3 FETCH (UID 2560 RFC822.SIZE 6741)
* 4 FETCH (UID 2561 RFC822.SIZE 71412)
* 5 FETCH (UID 2562 RFC822.SIZE 4573)
* 6 FETCH (UID 2563 RFC822.SIZE 6404)
* 7 FETCH (UID 2564 RFC822.SIZE 116816)

======= snip =======

* 1949 FETCH (UID 4566 RFC822.SIZE 1309)
* 1950 FETCH (UID 4567 RFC822.SIZE 1304)
* 1951 FETCH (UID 4568 RFC822.SIZE 1417)
* 1952 FETCH (UID 4569 RFC822.SIZE 1433)
* 1953 FETCH (UID 4570 RFC822.SIZE 1448)
* 1954 FETCH (UID 4571 RFC822.SIZE 1375)
a OK Fetch completed.

[/cc]

Now you can search through your terminal scrollback and look for that big message that was causing you trouble. Say, in our case, it is message 1951 above (although here it is only 1417B in size):

[cc lang=’bash’ line_numbers=’false’]

a fetch 1951 (uid rfc822.size flags body.peek[header.fields (date to cc from subject x-priority content-type)])
* 1951 FETCH (UID 4568 RFC822.SIZE 1417 FLAGS () BODY[HEADER.FIELDS (DATE TO CC FROM SUBJECT X-PRIORITY CONTENT-TYPE)] {252}
From: Brock Berg <kmighty@wsitruss.com>
To: me@example.com
Subject: Stupendous summer ph*ar(macy)cya pr|*ce landslide
Date: Sat, 14 Nov 2009 15:55:17 -0500
Content-Type: text/plain;
charset=”windows-1250″
X-Priority: 3

)
a OK Fetch completed.

[/cc]

Manipulating messages

And to delete (note the use of the UID here):

[cc lang=’bash’ line_numbers=’false’]

* 1931 FETCH (UID 4568 FLAGS (Deleted))
a OK Store completed.
[/cc]

Just to prove it to ourselves:

[cc lang=’bash’ line_numbers=’false’]

a uid fetch 4571 (uid flags)
* 1934 FETCH (UID 4571 FLAGS (Deleted))
a OK Fetch completed.
[/cc]

As you may have guessed IMAP doesn’t delete the message, but instead flags the message as deleted. To actually delete we use the EXPUNGE command:

[cc lang=’bash’ line_numbers=’false’]

a expunge
* 1934 EXPUNGE
a OK Expunge completed.
[/cc]

And that’s that.

Conclusion

Because you are issuing commands directly to the server, there is no performance penalty associated with using Outlook, Thunderbird or any other email client.

In my case I was dealing with multiple copies of that 165MB message and the migration script I was using was running out of memory every time it came across the message.

Further reading

An excellent article by Bob Peers (I wish I had this to hand when I first started speaking IMAP!): http://bobpeers.com/technical/telnet_imap