Intelligently processing deferred mail

Question - how to go about intelligently processing deferred mail?

Currently I have Timeout.queuereturn=30h. In a typical 24hr processing
period, I have 4,000-7,000 deferred emails, but these generate nearly
1million delivery attempts. (by comparison I'm delivering about
400,000 successful emails daily).

The deferred generally fall into these categories
1 incorrect domain (majority)
2 user unknown at destination, or user mbox full at destination
3 remote MTA temp down for some legit reason, should wait for it to
come up
4 greylisted

In particular, I would like to be able to just dump (and log) all of
the #1 items.
Do mail software other than sendmail (like postfix, etc that I have no
experience with) have ways to handle this more intelligently? Is this
a us for Milter?

Ideas, suggestions?

TIA. phil

Re: Intelligently processing deferred mail

On Mar 31, 4:07=A0pm, Phil wrote:
> Question - how to go about intelligently processing deferred mail?
>
> Currently I have Timeout.queuereturn=3D30h. In a typical 24hr processing
> period, I have 4,000-7,000 deferred emails, but these generate nearly
> 1million delivery attempts. (by comparison I'm delivering about
> 400,000 successful emails daily).
>
> The deferred generally fall into these categories
> 1 incorrect domain (majority)
> 2 user unknown at destination, or user mbox full at destination
> 3 remote MTA temp down for some legit reason, should wait for it to
> come up
> 4 greylisted
>
> In particular, I would like to be able to just dump (and log) all of
> the #1 items.
> Do mail software other than sendmail (like postfix, etc that I have no
> experience with) have ways to handle this more intelligently? Is this
> a us for Milter?
>
> Ideas, suggestions?
>
> TIA.phil

Does anyone have any suggestions or thoughts on this?

philc

Re: Intelligently processing deferred mail

Phil wrote:
> On Mar 31, 4:07 pm, Phil wrote:
>> Question - how to go about intelligently processing deferred mail?
>>
>> Currently I have Timeout.queuereturn=30h. In a typical 24hr processing
>> period, I have 4,000-7,000 deferred emails, but these generate nearly
>> 1million delivery attempts. (by comparison I'm delivering about
>> 400,000 successful emails daily).
>>
>> The deferred generally fall into these categories
>> 1 incorrect domain (majority)

Solution: don't mail to incorrect domains.

>> 2 user unknown at destination, or user mbox full at destination

Solution: don't mail to non existing users

>> 3 remote MTA temp down for some legit reason, should wait for it to
>> come up

A remote system being down is out of your control. The only thing you
can do is wait and retry. This shouldn't be a big issue with normal mail
traffic.

>> 4 greylisted

Just retry after some time.

>>
>> In particular, I would like to be able to just dump (and log) all of
>> the #1 items.

You should not have the #1 (and #2) items in the first place.
Why do you want to try sending mail in hughe amounts to "incorrect
domains" and dump those message if that fails?

>> Do mail software other than sendmail (like postfix, etc that I have no
>> experience with) have ways to handle this more intelligently? Is this
>> a us for Milter?
>>
>> Ideas, suggestions?

See above.

It seems to me that you accept messages that you can't (unknown user) or
don't want (spam) to deliver to your users and that you're sending
bounce messages in those cases. As virtually all spam has an invalid or
a spoofed sender address this is the wrong thing to do; you're just
annoying innocent people (the spoofed 'senders') and stressing your own
server with undeliverable messages.

The proper solution would be to check what you can and want to deliver
before you accept the responsibility for a message and just reject
anything you don't want during the smtp phase.

Regards,

Kees.

--
Kees Theunissen.

Re: Intelligently processing deferred mail

Phil schrieb:
> On Mar 31, 4:07 pm, Phil wrote:
>> Question - how to go about intelligently processing deferred mail?
>>
>> Currently I have Timeout.queuereturn=30h. In a typical 24hr processing
>> period, I have 4,000-7,000 deferred emails, but these generate nearly
>> 1million delivery attempts. (by comparison I'm delivering about
>> 400,000 successful emails daily).
>>
>> The deferred generally fall into these categories
>> 1 incorrect domain (majority)
>> 2 user unknown at destination, or user mbox full at destination
>> 3 remote MTA temp down for some legit reason, should wait for it to
>> come up
>> 4 greylisted
>>
>> In particular, I would like to be able to just dump (and log) all of
>> the #1 items.
>> Do mail software other than sendmail (like postfix, etc that I have no
>> experience with) have ways to handle this more intelligently? Is this
>> a us for Milter?
>>
>> Ideas, suggestions?
>>
>> TIA.phil
>
> Does anyone have any suggestions or thoughts on this?

Ok, since you asked for it - my2c, IMHO, YMMV etc. etc.:

This looks wrong. Instead of trying to get rid of the deferred mails
after the fact, I'd check why they get deferred in the first place.

Category 1 shouldn't end up deferred at all, except for the rare cases
where the incorrect domain hits a temporary DNS failure. A regular
non-existent domain in either sender or recipient address should lead
to an immediate rejection of the mail during submission, so it won't
even enter your system, much less get deferred.

Category 2, subcategory 1 shouldn't get deferred either. Unknown
recipients in existing non-local domains cannot be rejected during
submission because your server doesn't know which addresses exist in
the destination domain, but the first delivery attempt to a
non-existent recipient address will already result in a 5xx permanent
error and the mail will be bounced, not deferred.

Category 2, subcategory 2 is indeed a nuisance. Treating a full
mailbox as a temporary error condition is a widespread practice and
IMHO totally wrong, because that condition does not go away without
human intervention. That said, it's the decision of the recipient
server's administrator, and you cannot do much about it, other than
retrying delivery as requested.

Category 3 and 4 are legitimate deferrals and should be retried.

Your numbers look strange, too. If 5.000 undeliverable mails generate
1.000.000 delivery attempts in 30 hours, your effective mean delivery
retry interval is 9 minutes. If some of these mails get delivered
eventually, it's even less. So it seems you are running your queue
substantially more frequently than the customary 30 mins.

And finally: why do you even care? Do those unsuccessful delivery
attempts impair your server operation?

HTH
T.

--
Please excuse my bad English/German/French/Greek/Cantonese/Klingon/...

Re: Intelligently processing deferred mail

Kees Theunissen wrote:
> Phil wrote:
>> On Mar 31, 4:07 pm, Phil wrote:
>>> Question - how to go about intelligently processing deferred mail?
>>>
>>> Currently I have Timeout.queuereturn=30h. In a typical 24hr processing
>>> period, I have 4,000-7,000 deferred emails, but these generate nearly
>>> 1million delivery attempts. (by comparison I'm delivering about
>>> 400,000 successful emails daily).
>>>
>>> The deferred generally fall into these categories
>>> 1 incorrect domain (majority)
>
> Solution: don't mail to incorrect domains.
>
>>> 2 user unknown at destination, or user mbox full at destination
>
> Solution: don't mail to non existing users
>
>>> 3 remote MTA temp down for some legit reason, should wait for it to
>>> come up
>
> A remote system being down is out of your control. The only thing you
> can do is wait and retry. This shouldn't be a big issue with normal mail
> traffic.
>
>>> 4 greylisted
>
> Just retry after some time.
>
>>>
>>> In particular, I would like to be able to just dump (and log) all of
>>> the #1 items.
>
> You should not have the #1 (and #2) items in the first place.
> Why do you want to try sending mail in hughe amounts to "incorrect
> domains" and dump those message if that fails?
>
>>> Do mail software other than sendmail (like postfix, etc that I have no
>>> experience with) have ways to handle this more intelligently? Is this
>>> a us for Milter?
>>>
>>> Ideas, suggestions?
>
> See above.
>
> It seems to me that you accept messages that you can't (unknown user) or
> don't want (spam) to deliver to your users and that you're sending
> bounce messages in those cases. As virtually all spam has an invalid or
> a spoofed sender address this is the wrong thing to do; you're just
> annoying innocent people (the spoofed 'senders') and stressing your own
> server with undeliverable messages.
>
> The proper solution would be to check what you can and want to deliver
> before you accept the responsibility for a message and just reject
> anything you don't want during the smtp phase.

Sorry, I should have taken a better look at your figures before
replying. 4,000-7,000 deferred messages on a total of 400,000 delivered
messages is not an indication that you're bouncing/backscattering
big parts of your spam.
But still #1 and #2 look like backscatter.

Do you have any indication of the nature/origin of those messages?
I'm thinking about things as:
-- mail that you relay to back-end systems or customers and that is
rejected by the downstream server,
-- Out of office replies triggered by a spam message,
-- Possibly spam, forwarded by some of your users to an external
address and rejected by the external servers.
Knowing the source of these messages might help preventing them.

Furthermore you cold move messages that stay in your queue for
too long to an other -less frequently processed- queue.

Regards,

Kees.

--
Kees Theunissen.