Windows Server Forum / Exchange Server / Administration / July 2008
Exchange 2003 SP2 Greylisting bug - survey
|
|
Thread rating:  |
Yizhar Hurwitz - 15 Jun 2007 11:34 GMT HI.
As some of you already know, there is a bug with Exchange 2003 SP2 that causes outbound mail to domains that implement greylisting to be jailed in a black hole until the SMTP service is restarted. A more detailed description of the problem can be found here: http://www.microsoft.com/communities/newsgroups/list/en-us/default.aspx?mid=df7b cd16-0306-4787-a358-3888c4e1198f&dg=microsoft.public.exchange.admin And/Or just by searching this newsgroup for the word "greylisting".
Please read the detailed story linked above before you comment to this post, to make sure that we all know what we are talking about.
As many of us have also noticed, there is yet no official response from MS about this bug, which is quite a shame...
This is because of the following reasons (among other): 1. At least for my case that I have openned with them, they were not able to reproduce the problem, and also I wasn't able to reproduce it at my client site (I guess the workarounds did the job). 2. MS PSS didn't get the full picture about the scale of this problem, and didn't make the relations between one case to the other. 3. If the workarounds were solving the problem (as was in my case), it was not escelated to the Exchange Development team for further investigation and a proper fix.
So - what are the next steps, and how can we help MS fix this strange and tricky bug?
Here are some answers:
1. If you are having (or had recently) this same problem, please contact your local PSS , but reference my initial case which is: EMEA case SRZ060302000872 This will help PSS relate the different cases to each other and to the same bug. I will quote a response I got from MS about this in reply to my own post here.
2. Participate in this survey by replying to this message, and provide as much related details as you think. Here are some important details: * Do you currently have the problem, or had it in the past? * Your Exchange version and SP (probably Exchange 2003 SP2). * The OS version and SP (for example WIN2003 SP1). * Was the OS upgraded from WIN2000 or is it a clean install? * Anti virus and other related 3rd party software installed on the server. * Have you openned a case with MS PSS? If so, can you write the case number? Is the case still open? * Can you list some domain names that you had trouble sending email to? for example: technion.ac.il (I had problems sending to them). * Do you have SMTP logs (if not - please enalbe them)? Logging the SMTP Service: http://www.msexchange.org/tutorials/Logging_the_SMTP_Service.html * If you like, you can publish your contact information such as phone number. * Any additional related details that you can provide, and a description of the problem that you had/have.
Summary so far: If you had/have this problem please do: 1. Contact MS PSS and reference EMEA case SRZ060302000872 2. Reply to this post and provide details.
Bye,
 Signature Yizhar Hurwitz http://yizhar.mvps.org
Yizhar Hurwitz - 15 Jun 2007 11:41 GMT HI Again.
Here is a quote of what the Team Manager in PSS (second level) had asked me to write, and I provide it here as is:
******** Begin quote **********
If you are having an issue with greylisting, please review the problem description below. Should your problem match the description below and the workaround works for you, Microsoft has advised me to ask you to raise a support call with them. When you raise your support call, please ask that they link the call to the EMEA case SRZ060302000872. This way, should a number of customers be seeing the problem, it can be tracked within Microsoft.
SYMPTOMS ========= When Exchange tries to send mails to certain domains that implement ‘greylisting’, the mails fail to get delivered, without any intimation to the sender on the first attempt. Thereafter on either restarting the server or SMTP service, the mails get delivered to the destination domains. At times, NDRs for these delayed mails are also generated after rebooting.
PROBLEM ======== The issue is intermittent and occurs only when the destination domains have greylisting implemented and mail is sent to the greylisted domain the very first time.
WORKAROUND ============ 1. Write a script to restart the SMTP service at least once a day. 2. Modify the registry on the sender Exchange server, to change the Glitch Retry key. 3. Clarify that there is no 3rd party AV software in the environment which could be causing the issue.
******** End quote **********
 Signature Yizhar Hurwitz http://yizhar.mvps.org
> HI. > [quoted text clipped - 61 lines] > > Bye, Dominik Rappaport - 22 Jun 2007 21:52 GMT Hi Yizhar,
I’m Dominik from Vienna, Austria and I read you postings about the Exchange Server 2003 Greylisting bug. I experienced this bug on at least two machines but was not able to reproduce it. Two days ago I installed a new Small Business Server 2003 for a customer. I sent a test mail to my private e-Mail account and discovered that the e-Mail did not arrive. My ISP uses greylisting for spam protection.
As the bug was familiar to me I immediately made some investigations:
1. The message was sent only one time and the server responded with a 4xx temporary error code 2. Contrary to what should happen, the message was never sent again 3. Even worse, the message did NOT appear in the Exchange Server Queue viewer! 4. I used Mfcmapi to look into the TempTables and there I found the missing message
As the system was fresh installed without any virus scan program or third party spam filter installed, I opened a call to Microsoft PSS. I made the same experiences as you. The friendly woman on the other side on the phone focused just on the configuration of my machine. She advised me to create the GlitchRetrySeconds key in the registry as described in http://technet.microsoft.com/en-us/library/8b43be56-48e6-400b-8014-54c95f87d1de.aspx. After that I had to restart the SMTP service. Now everything behaved as expected. The message (I used a different sender address for proper testing) was correctly delivered after the intended delay.
Now I did the crosscheck and removed the GlitchRetrySeconds key and restarted the SMTP service. Unfortunately the error did not reoccur. I never believed that the bug was related to the GlitchyRetry interval but now I could not reproduce the bug and had nothing to show for PSS.
I discussed this with the Microsoft support engineer and argued that the fact that a message was in the TempTables but is not shown in the queue viewer must be considered as a bug and if they could make further investigations. She completely refused to do so and said that the message can now be properly delivered and that was the goal of the support call that I have consented.
I replied that the error could reoccur every time and that I’m very dissatisfied that she absolutely denied the possibility of a bug. Then she said “Please wait a minute!”. After she reported back she said there exists a brand new hotfix KB 934709 that she could send to me. It is designed to “check SMTP temp table regularly and try to resend corrupted messages”. For me, that sounds like a bug fix. Even more, the hotfix replaces the aqueue.dll file. One core DLL of the advanced queuing engine of the SMTP service. The hotfix is so brand new that today no KB article exists but will be written in the near future (22th of June, 2007).
I can’t say if this hotfix resolves the problem as it did not reappear.
My support call ID was EMEA SRQ070620601190.
I’m curious what the KB article related to the hotfix will say about it and if it will mention Greylisting.
Kind regards, Dominik -- MCSA Messaging
Steve Howden - 06 Jul 2007 05:38 GMT > Hi Yizhar, > [quoted text clipped - 57 lines] > -- > MCSA Messaging Steve Howden - 06 Jul 2007 05:50 GMT Yes, I have been having the same problem intermittently (exchange 2003 SP2) Very old NDRs popping up after a server or SMTP svc restart.
It wasn't until I noticed a very long SMTP conversation in the SMTP svc logs that i started to get suspicious. Exchange was re-sending a mail on receipt of a 450 after only 1 second! In one case it kept trying every second in the same conversation 10 times before quitting.
I contacted the admin for the recipient mail server and he informed me of their Greylisting policy which expects a retry after 120 secs. He also pointed out that any mail server retrying every second for ten times might make them think it is a DOS attack and blacklist our IP address :-(
I found the GlitchRetrySeconds registry edit and made it 120 secs. That seemed to have fixed the problem......... until today when I discovered a mail server with greylisting that expects a 300 sec delay for replies. I wish they would all settle on a standard time. I now have GlitchRetry up to 300 secs and testing with various fresh emails seems to have solved it.
Smells buggy to me.
regards, Steve
> Hi Yizhar, > [quoted text clipped - 57 lines] > -- > MCSA Messaging jrusso@hdp.com - 10 Jul 2007 20:05 GMT I have been going back and forth with this as well. I finally got this hotfix from MS. The engineer I am working with does not think this is my issue as I am not seeing these messages in the Queue as described in this article. We are still doing some tests, but I have a gut feeling this will fix it.
http://support.microsoft.com/default.aspx?scid=kb;EN-US;934709
Yizhar Hurwitz - 13 Aug 2007 21:26 GMT HI Steve.
> I found the GlitchRetrySeconds registry edit and made it 120 secs. That > seemed to have fixed the problem......... until today when I discovered a > mail server with greylisting that expects a 300 sec delay for replies. I wish > they would all settle on a standard time. I now have GlitchRetry up to 300 > secs and testing with various fresh emails seems to have You have missed the actual problem with the bug. As long as your server does retry the message, this is OK. After 3 attempts using glitchretry, the message should go to the regular queue, and the server should retry again every X minutes (defined in the properties of the default SMTP virtual server). So - even if the recipient server has a timeout of, let's say 10 minutes, the message should go through eventually, after several retries.
However, when the problem happens, Exchange fails to retry even on the first attempt, and fails to requeue the message.
Changing the glitchretry might help as some people posted, bug I guess that the "Advanced" queue update might be the real fix (I haven't tested it, and didn't get any details from MS about it other than what was posted here): On a Windows Server 2003-based SMTP gateway server, some messages may remain in the queue folder until the SMTP service is restarted: http://support.microsoft.com/default.aspx?scid=kb;EN-US;934709
I think that using the workarounds that I described in my initial post is also a good practice:
* Using a smart host for outbound mail traffic (defined in the SMTP connector), if possible and applicable.
* daily restart of SMTP service (scheduled tasks).
And the workarounds/fixes described above, such as:
* Setting glitchretry to something like 60-120 seconds.
* Installing hotfix KB934709 .
 Signature Yizhar Hurwitz http://yizhar.mvps.org
> Yes, I have been having the same problem intermittently (exchange 2003 SP2) > Very old NDRs popping up after a server or SMTP svc restart. [quoted text clipped - 81 lines] > > -- > > MCSA Messaging Susan K - 24 Sep 2007 20:42 GMT Yizhar Hurwitz,
Do you know, or does anyone else know, if this MS hotfix patch no. 934709 has been tested on a Small Business Server 2003? We have been experiencing the same problems, I obtained a copy of the patch, but I'm in a very small environment and am very apprehensive to install the hotfix since I don't have the ability to test it. I'll definitely try your other recommendations, but any feed back about testing on SBS would be greatly appreciated.
Thanks
 Signature Susan
> HI Steve. > [quoted text clipped - 121 lines] > > > -- > > > MCSA Messaging Rich Matheisen [MVP] - 25 Sep 2007 02:26 GMT >Do you know, or does anyone else know, if this MS hotfix patch no. 934709 >has been tested on a Small Business Server 2003? SBS, in this area, is no different to any other Windows/Exchange server.
 Signature Rich Matheisen MCSE+I, Exchange MVP MS Exchange FAQ at http://www.swinc.com/resource/exch_faq.htm Don't send mail to this address mailto:h.pott@getronics.com Or to these, either: mailto:h.pott@pinkroccade.com mailto:melvin.mcphucknuckle@getronics.com mailto:melvin.mcphucknuckle@pinkroccade.com
JGurtz - 19 Sep 2007 14:44 GMT > said “Please wait a minute!”. After she reported back she said there exists a > brand new hotfix KB 934709 that she could send to me. It is designed to [quoted text clipped - 3 lines] > hotfix is so brand new that today no KB article exists but will be written in > the near future (22th of June, 2007). It is funny that the KB article only mentions Windows Server 2003 and NOT Exchange at all, even though it seems to fix the problem for exchange also.
Does anyone know if this 4xx response to RCPT TO: is still a problem in Exchange 2007? How about with the beta of Ex'07 SP1? What about Server 2008?
Yizhar Hurwitz - 19 Sep 2007 19:50 GMT HI.
> It is funny that the KB article only mentions Windows Server 2003 and NOT > Exchange at all, even though it seems to fix the problem for exchange also. Yeah, if it wasn't sad, it could be funny...
> Does anyone know if this 4xx response to RCPT TO: is still a problem in > Exchange 2007? How about with the beta of Ex'07 SP1? What about Server 2008? Exchange 2007 has a totaly different SMTP architecture and is not affected by this specific bug. It is a different product which has new bugs to replace the older ones which are no longer needed.
 Signature Yizhar Hurwitz http://yizhar.mvps.org
Jason Gurtz - 19 Sep 2007 20:10 GMT >> Does anyone know if this 4xx response to RCPT TO: is still a problem in >> Exchange 2007? >> > Exchange 2007 has a totaly different SMTP architecture and is not affected > by this specific bug. Wow, I wonder if it's actually getting to the point were it can be directly accessible on the Internet? Do tell, does it play nice now!?
/me wonders how the MIMEfield situation is now.
Cheers,
~JasonG
--
Raj - 14 Jul 2007 01:28 GMT I agree that the Exchange 2003 SP2 that we have behaves just the way you have explained. This is we can confirm 101% because I put up a test bed on a greylist server and made exchange deliver a mail to it. After the first 4.7.1 that exchange recieves during the first attempt , Exchange does not reattempt the delivery at all. The Queue shows that the mail was retried Once.
CHanging the GLITCHRETYSECONDS to 120 and restarting smtp service did not work , nor for 300 nor for 360. I did not try 60 becuase elsewhere it is documented that the default is 60 seconds.
I wanted to confirm if the key QUEUING should be under smtpsvc or under smtpsvc/parameters ?
Is there a link to download the hotfix ?
TIA
Raj
Rich Matheisen [MVP] - 14 Jul 2007 04:08 GMT >I agree that the Exchange 2003 SP2 that we have behaves just the way you >have explained. This is we can confirm 101% because I put up a test bed on a [quoted text clipped - 4 lines] > >CHanging the GLITCHRETYSECONDS Some registry names are case sensitive. Try "GlitchRetrySeconds".
>to 120 and restarting smtp service did not >work , nor for 300 nor for 360. I did not try 60 becuase elsewhere it is >documented that the default is 60 seconds. > >I wanted to confirm if the key QUEUING should be under smtpsvc Yes.
>or under >smtpsvc/parameters ? No.
>Is there a link to download the hotfix ? Call Microsoft.
 Signature Rich Matheisen MCSE+I, Exchange MVP MS Exchange FAQ at http://www.swinc.com/resource/exch_faq.htm Don't send mail to this address mailto:h.pott@getronics.com Or to these, either: mailto:h.pott@pinkroccade.com mailto:melvin.mcphucknuckle@getronics.com mailto:melvin.mcphucknuckle@pinkroccade.com
Alexander - 21 Jan 2008 20:58 GMT Hi everyone !
Yizhar Hurwitz referred me to this thread since I posted a similar thing in the technet exchange newsgroup.
A lot has been said about this problem already. I won´t repeat everything. However, I do have to add one important difference:
This Problem definitely also appears on an Exchange 2000 (latest patchlevel) server on a Windows 2000 (Latest patchlevel) server so it is not only an Exchange 2003 SP2 problem. We recently had this problem on such a server at one of our customers Exchange server. Messages as old as 1 month suddenly were sent and NDRs were received. The boss of that company was very angry with us, he alone was affected with 40 messages.
I do not have access to all NDRs but to a few but they are in German. (The important things are language independant though):
-------------------------------------------------------------------------------- Ihre Nachricht hat einige oder alle Empfänger nicht erreicht.
Betreff: xxxxxxxxxxxxxxxx Gesendet am: 11.12.2007 10:19
Folgende(r) Empfänger kann/können nicht erreicht werden:
xxxxxx@wwff.gv.at am 13.01.2008 15:34 Das E-Mail-System konnte diese Nachricht nicht übermitteln, ohne einen besonderen Grund dafür anzugeben. Überprüfen Sie die Adresse, und versuchen Sie es dann erneut. Wenn die Übermittlung nochmals fehlschlägt, wenden Sie sich an den Systemadministrator. <xxxxxxxx.xxxxxxx.at #4.0.0 smtp;421 temporary envelope failure (#4.3.0)>
--------------------------------------------------------------------------------------------------
Ihre Nachricht hat einige oder alle Empfänger nicht erreicht.
Betreff: xxxxxxxxxxxxxxxxxxxxxxxxxxx Gesendet am: 06.12.2007 10:05
Folgende(r) Empfänger kann/können nicht erreicht werden:
xxxxxxxxx@genion-clipping.si am 13.01.2008 15:33 Sie sind nicht berechtigt, Nachrichten an diesen Empfänger zu senden. Wenden Sie sich an den Systemadministrator. <xxxxx.xxxxxx.at #4.7.1 smtp;450 4.7.1 <xxxxxxxx@genion-clipping.si>: Recipient address rejected: temporary blocked due to greylisting; http://www.amis.net/greylisting/>
--------------------------------------------------------------------------------------------------------
Ihre Nachricht hat einige oder alle Empfänger nicht erreicht.
Betreff: Ihre »OBSERVER« STORNIERUNG zum Auftrag 'Media Focus' Nr.4993 Gesendet am: 20.12.2007 11:24
Folgende(r) Empfänger kann/können nicht erreicht werden:
xxxxxxxx@focusmr.com am 13.01.2008 15:33 Diese Nachricht hat das E-Mail-System des Empfängers erreicht, die Übermittlung der Nachricht wurde jedoch verweigert. Versuchen Sie nochmals, diese Nachricht zu senden. Wenn die Übermittlung erneut fehlschlägt, wenden Sie sich an den Systemadministrator. <xxxxxx.xxxxxx.at #4.2.0 smtp;450 4.2.0 <xxxxxx@focusmr.com>: Recipient address rejected: Greylisted, see http://postgrey.schweikert.ch/help/focusmr.com.html>
----------------------------------------------------------------------------------------------
The hotfix provided by Microsoft won´t help us since it is for Exchange 2003 only. Also we have read that it won´t really fix the problem anyway. We will implement a daily SMTP service restart.
Since the server will be replaced by an Exchange 2007 Server soon anyway we won´t do much more or even open a PSS case with Microsoft but I am very disappointed that an almost 2 year old known bug is not acknowledged by Microsoft.
A guy on the Partner newsgroups even sent me on a detour exploring temp tables and suggested deleting them but this is no real solution because it will kill the messages inside and also this cannot be scripted because there is no way to find out if Temptables are corrupted or not.
He also implied that Third party software is know to cause this problem. We do use GFI mailessentials on that server. We did contact GFI about this but according to posts in this thread this problem also happens on freshly installed Exchange Servers so I doubt that Thirdparty Software is the real cause for this.
No problem if Microsoft does not know the answer (no one is perfect) but just blaming others is lame ... :-(
Bye, Alex
Actionxp - 22 Jan 2008 08:36 GMT Agree. We all hate TempleTables ~~
That issue happened a lot and got many complains.
Gary James - 15 Apr 2008 20:31 GMT We are also having this issue. We have started case number srx080415601330 with MS. We have not tried any of the suggested workarounds in hopes that MS will address our issue directly.
> Agree. We all hate TempleTables ~~ > > That issue happened a lot and got many complains. Christina Tracey - 20 May 2008 14:30 GMT We are experiencing this problem too. We're running Exchange 2003 SP2 on Windows Server 2000 SP4. We also have Symantec Mail Security installed and doing our anti-virus and anti-spam for us. I can't believe there isn't a fix for this yet! I have not opened a case with Microsoft yet but will certainly reference your EMEA case #SRZ060302000872 when I do. I was relieved to find we weren't the only ones experiencing this odd Exchange behavior.
Erich Fritz - 20 May 2008 20:49 GMT I am currently having this problem. Exchange 2003 SP2 Windows Server 2003 Standard SP2 Symantec Client Security 3.1 and Symantec Mail Security 6.xx I have not opened a case, I will call in and reference your case number. No logs yet. I will enable SMTP logging.
I have applied the "workarounds" that have been suggested. So far things have been fine. For the past week however the SMTP service will not restart properly. It hangs on the startup after the scheduled task runs. All searching I've done on the issue to this point has not brought me any results. No recent software or MSFT patches have been applied that might cause the issue.
Thanks, Erich Fritz
Christina Tracey - 20 May 2008 21:03 GMT Erich,
I found that I could not implement the daily restart of SMTPSVC because whenever I tried to stop the SMTP service it would hang -- it couldn't be stopped. I tried the GlitchRetrySeconds setting workaround (set to 120) and that hasn't helped. I opened a case with Microsoft this morning and after a lot of runaround and being on the phone for almost two hours, they finally admitted my problem was indeed the "known bug" and that someone else (a TSL - not sure what that is) would be getting back to me. I'll let you know if I ever get a working fix.
Thanks, Christina Tracey
> I am currently having this problem. > Exchange 2003 SP2 [quoted text clipped - 12 lines] > Thanks, > Erich Fritz Yizhar Hurwitz - 31 Jul 2008 14:34 GMT HI.
Finally, Hopefully, beter late the never, etc...
MS acknoledge the problem, and now provides the bug fix.
Here it is:
E-mail senders do not receive an indication that some messages have been held by Exchange Server 2003 until the SMTP service, the Microsoft Exchange Information Store service, or the Exchange server is restarted http://support.microsoft.com/kb/950757/en-us
Almost 3 years after the bug was introduced at October 2005, hmmmm.....
Please reply here and write if you installed the hotfix KB950757 and if it solved the problem.
 Signature Yizhar Hurwitz http://yizhar.mvps.org
> HI. > [quoted text clipped - 61 lines] > > Bye,
|
|
|