Intentionally slow down queue connections?

I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued items
(say 7000) all going to the same domain (samedomain.com). When sendmail
tries to run the queue (every 15 minutes or whatever), it quickly (very
quickly) spawns the processes and ties to connect to the domain to
deliver the messages as expected.

The problem is because the connection to this server is so fast theres
no delay anywhere, it rips through like a 100 per second, so of course
the receiving domain (samedomain.com) will throttle it (Connection rate
limit exceeded).

This actually makes it impossible to deliver these messages normally.
Only like 3 messages per 15 minutes actually get delivered because the
connection rate is exceeded for the rest, stopping them. How can I fix
this? I'm sure if I could tell sendmail to simply pause for a second
between each message attempt the connection rate would be fine and it
would (more slowly) go through each message in the queue delivering just
fine.

Any ideas? What am I doing wrong?
kenw232@yahoo.com

Example (rate limit exceeded because all within 1 second):
Jan 26 14:11:12 machine02 sendmail[637]: m0PJPKnJ009435:
to=, delay=23:45:52, xdelay=00:00:03,
mailer=esmtp, pri=8693889, relay=samedomain.com. [21.1.1.1], dsn=4.0.0,
stat=Deferred: 421 4.3.2 Connection rate limit exceeded.

Jan 26 14:11:12 machine02 sendmail[640]: m0PJHsTI007706:
to=, delay=23:53:18, xdelay=00:00:03,
mailer=esmtp, pri=8783928, relay=samedomain.com. [21.1.1.1], dsn=4.0.0,
stat=Deferred: 421 4.3.2 Connection rate limit exceeded.

Jan 26 14:11:12 machine02 sendmail[643]: m0PJKPJI008186:
to=, delay=23:50:47, xdelay=00:00:03,
mailer=esmtp, pri=8783928, relay=samedomain.com. [21.1.1.1], dsn=4.0.0,
stat=Deferred: 421 4.3.2 Connection rate limit exceeded.

Jan 26 14:11:12 machine02 sendmail[644]: m0PJQQdx009605:
to=, delay=23:44:46, xdelay=00:00:03,
mailer=esmtp, pri=8694006, relay=samedomain.com. [21.1.1.1], dsn=4.0.0,
stat=Deferred: 421 4.3.2 Connection rate limit exceeded.
....
(and so on for like 7000 messages)

Re: Intentionally slow down queue connections?

Ken Williams writes:

> I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued
> items (say 7000) all going to the same domain (samedomain.com). When
> sendmail tries to run the queue (every 15 minutes or whatever), it
> quickly (very quickly) spawns the processes and ties to connect to the
> domain to deliver the messages as expected.
>
> The problem is because the connection to this server is so fast theres
> no delay anywhere, it rips through like a 100 per second, so of course
> the receiving domain (samedomain.com) will throttle it (Connection
> rate limit exceeded).
>
> This actually makes it impossible to deliver these messages
> normally. Only like 3 messages per 15 minutes actually get delivered
> because the connection rate is exceeded for the rest, stopping them.
> How can I fix this? I'm sure if I could tell sendmail to simply pause
> for a second between each message attempt the connection rate would be
> fine and it would (more slowly) go through each message in the queue
> delivering just fine.
>
> Any ideas? What am I doing wrong?
> kenw232@yahoo.com
> [...]

1) Avoid "at once" delivery attempts - default sendmail's delivery mode
It requires new smtp mailer with expensive flag (F=e) and
FEATURE(`mailertable')
2) Create separate queue group for deliveries to the domain

It will make sendmail send more messages over single smtp connection
[ all configured via sendmail.mc/sendmail.cf ]

Do you want more detailed description?

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
God help the troubadour who tries to be a star. The more that you try
to find success, the more that you will fail.
-- Phil Ochs, on the Second System Effect
----
http://groups.google.com/groups?selm=87abmsnqgv@tom.fsf.hobb y-site.com

Re: Intentionally slow down queue connections?

In article Ken Williams
writes:
>I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued items
>(say 7000) all going to the same domain (samedomain.com). When sendmail
>tries to run the queue (every 15 minutes or whatever), it quickly (very
>quickly) spawns the processes and ties to connect to the domain to
>deliver the messages as expected.

The default behaviour would be that a single process opens one
connection and sends all the messages through that - i.e. you shouldn't
get one process per message, and you shouldn't exceed any *connection*
rate limit. Did you perhaps set SMTP_MAILER_MAXMSGS to 1 in your .mc
file? Bad move.

--Per Hedeland
per@hedeland.org

Re: Intentionally slow down queue connections?

Andrzej Adam Filip wrote:
> Ken Williams writes:
>
>> I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued
>> items (say 7000) all going to the same domain (samedomain.com). When
>> sendmail tries to run the queue (every 15 minutes or whatever), it
>> quickly (very quickly) spawns the processes and ties to connect to the
>> domain to deliver the messages as expected.
>>
>> The problem is because the connection to this server is so fast theres
>> no delay anywhere, it rips through like a 100 per second, so of course
>> the receiving domain (samedomain.com) will throttle it (Connection
>> rate limit exceeded).
>>
>> This actually makes it impossible to deliver these messages
>> normally. Only like 3 messages per 15 minutes actually get delivered
>> because the connection rate is exceeded for the rest, stopping them.
>> How can I fix this? I'm sure if I could tell sendmail to simply pause
>> for a second between each message attempt the connection rate would be
>> fine and it would (more slowly) go through each message in the queue
>> delivering just fine.
>>
>> Any ideas? What am I doing wrong?
>> kenw232@yahoo.com
>> [...]
>
> 1) Avoid "at once" delivery attempts - default sendmail's delivery mode
> It requires new smtp mailer with expensive flag (F=e) and
> FEATURE(`mailertable')
> 2) Create separate queue group for deliveries to the domain
>
> It will make sendmail send more messages over single smtp connection
> [ all configured via sendmail.mc/sendmail.cf ]
>
> Do you want more detailed description?
>

yes, a more detailed description would be good. If I setup a queue
group, I would have to do that for other domains too when this happens
right? I'm trying to automate this somehow. Like manually added a
mailertable entry everytime this happens is unrealistic for me.

I use confSEPARATE_PROC as true? is that bad? heres my entire sendmail.mc.


LOCAL_CONFIG

define(`confDONT_PROBE_INTERFACES',`True')dnl
define(`ALIAS_FILE', ``/etc/mail/aliases, /etc/mail/aliases_listmgr'')dnl
define(`confDONT_BLAME_SENDMAIL',`forwardfileingroupwritable dirpath')dnl
define(`confPRIVACY_FLAGS', ``authwarnings,noexpn,novrfy'')dnl
define(`STATUS_FILE', `/etc/mail/statistics')dnl
define(`DATABASE_MAP_TYPE',`hash')dnl
define(`confCW_FILE', `/etc/sendmail.cw')dnl
define(`SMTP_MAILER_FLAGS', `A')dnl
define(`confDELAY_LA', `22')dnl
define(`confREFUSE_LA', `32')dnl
define(`confQUEUE_LA', `42')dnl
define(`confTO_QUEUERETURN', `2d')dnl
define(`confTO_QUEUERETURN_NORMAL', `1d')dnl
define(`confTO_QUEUERETURN_URGENT', `6h')dnl
define(`confTO_QUEUERETURN_NONURGENT', `1d')dnl
define(`confTO_QUEUERETURN_DSN', `6h')dnl
define(`confTO_QUEUEWARN', `30d')dnl
define(`confTO_QUEUEWARN_NORMAL', `30d')dnl
define(`confTO_QUEUEWARN_URGENT', `30d')dnl
define(`confTO_QUEUEWARN_NONURGENT', `30d')dnl
define(`confTO_QUEUEWARN_DSN', `30d')dnl
define(`confTO_ICONNECT', `20s')dnl
define(`confTO_CONNECT', `2m')dnl
define(`confTO_ACONNECT', `2m')dnl
define(`confDEF_USER_ID', `root')dnl
define(`confSMTP_LOGIN_MSG', `$j $b (machine02)')dnl
define(`confBIND_OPTS', `WorkAroundBrokenAAAA')dnl
define(`confMAX_MESSAGE_SIZE', `5000000')dnl
define(`confBAD_RCPT_THROTTLE', `3')dnl
define(`confMAX_RCPTS_PER_MESSAGE', `15')dnl
define(`confMAX_QUEUE_CHILDREN', `20')dnl
define(`confMAX_DAEMON_CHILDREN', `40')dnl
define(`confMAX_RUNNERS_PER_QUEUE', `40')dnl
define(`confMAX_QUEUE_RUN_SIZE', `500')dnl
define(`confQUEUE_SORT_ORDER', `modification')dnl
define(`confMIN_FREE_BLOCKS', `500240')dnl
define(`confCONNECTION_RATE_THROTTLE', `14')dnl
define(`confMILTER_LOG_LEVEL', `1')dnl
define(`confDEAD_LETTER_DROP',`/dev/null')dnl
define(`confDF_BUFFER_SIZE',`8192')dnl
define(`confSEPARATE_PROC',`true')dnl
define(`QUEUE_DIR', `/var/spool/mqueue/q*')dnl
define(`confPID_FILE', `/var/run/sendmail.pid')

FEATURE(`masquerade_envelope')dnl
FEATURE(`delay_checks')dnl
FEATURE(`nouucp', `reject')dnl
FEATURE(use_cw_file)dnl
FEATURE(`access_db', `hash -T /etc/mail/access')dnl
FEATURE(`greet_pause',`200')dnl
FEATURE(blacklist_recipients)
FEATURE(`virtusertable', `hash /etc/mail/virtusertable')dnl
FEATURE(`genericstable', `hash /etc/mail/revvirtusertable')dnl
FEATURE(`mailertable', `hash /etc/mail/mailertable')dnl
FEATURE(`no_default_msa')dnl

GENERICS_DOMAIN_FILE(`/etc/sendmail.cg')dnl

FEATURE(local_procmail)dnl
MAILER(procmail)dnl
MAILER(smtp)dnl

Re: Intentionally slow down queue connections?

Post removed (X-No-Archive: yes)

Re: Intentionally slow down queue connections?

In article Res
writes:
>On Sat, 26 Jan 2008, Ken Williams wrote:
>
>> I use confSEPARATE_PROC as true? is that bad?

It's definitely the cause of your queue runner forking off 7000
processes, each making a connection to the same server to deliver a
single message. And it seems that both you and that server's admin think
this is bad (and I agree).

>No, for busy servers you need it, Sendmails default is just plain crazy
>for large mail servers, and if you run any sort of mailing list you
>definately want it regardless of what anyone else says, unles your members
>enjoy waiting 7 hours to get mail, one of my list servers in with over
>1700 members, now sends all msgs in under 1 minute, without that setting,
>it takes 7 hours :),

Well, I can see the problem, and why using confSEPARATE_PROC helps with
it in your case, but I really think that's the wrong hammer even for
that nail - and close to disaster in Ken's case. You don't want to fork
*each* job, just have enough queue runners that you don't get a lot of
your (presumably) carefully envelope-split mailing list messages sitting
in the queue waiting for a queue runner to pick them up. Probably on the
order of "a few per queue group" if you have "enough" queue groups.

I guess the ideal would be if sendmail itself could take a message with
a lot of recipients and fork an SMTP client for each actual destination
(not entirely possible since the exact destination host can't be known
ahead of time with multiple MXen, but using "same first MX" would
probably be OK). Maybe it already can and I have missed it, but I don't
think so. Queue groups help, but can of course never reach this ideal.

--Per Hedeland
per@hedeland.org

Re: Intentionally slow down queue connections?

Ken Williams writes:

> I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued
> items (say 7000) all going to the same domain (samedomain.com). When
> sendmail tries to run the queue (every 15 minutes or whatever), it
> quickly (very quickly) spawns the processes and ties to connect to the
> domain to deliver the messages as expected.
>
> The problem is because the connection to this server is so fast theres
> no delay anywhere, it rips through like a 100 per second, so of course
> the receiving domain (samedomain.com) will throttle it (Connection
> rate limit exceeded).
>
> This actually makes it impossible to deliver these messages
> normally. Only like 3 messages per 15 minutes actually get delivered
> because the connection rate is exceeded for the rest, stopping them.
> How can I fix this? I'm sure if I could tell sendmail to simply pause
> for a second between each message attempt the connection rate would be
> fine and it would (more slowly) go through each message in the queue
> delivering just fine.
>
> Any ideas? What am I doing wrong?
> kenw232@yahoo.com
> [...]

Just one more question before (possibly) giving *long* suboptimal advice:
What generates the messages?
Is it some "personalized mailing list" software?

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
'Course, I haven't weighed in yet. :-)
-- Larry Wall in <199710281816.KAA29614@wall.org>
----
http://groups.google.com/groups?selm=87fxwkm3xu@john.fsf.hob by-site.com

Re: Intentionally slow down queue connections?

per@hedeland.org (Per Hedeland) writes:

> In article Ken Williams
> writes:
>>I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued items
>>(say 7000) all going to the same domain (samedomain.com). When sendmail
>>tries to run the queue (every 15 minutes or whatever), it quickly (very
>>quickly) spawns the processes and ties to connect to the domain to
>>deliver the messages as expected.
>
> The default behaviour would be that a single process opens one
> connection and sends all the messages through that - i.e. you shouldn't
> get one process per message, and you shouldn't exceed any *connection*
> rate limit. Did you perhaps set SMTP_MAILER_MAXMSGS to 1 in your .mc
> file? Bad move.

One possible explanation would be that *new* messages to the domain are
still submitted. Their "at once" delivery attempts give queue run little
chance to "miss" connection rate limit set by the receiving side.

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
If we can ever make red tape nutritional, we can feed the world.
-- R. Schaeberle, "Management Accounting"
----
http://groups.google.com/groups?selm=878x2cm3p6@marcia.fsf.h obby-site.com

Re: Intentionally slow down queue connections?

Andrzej Adam Filip wrote:
> Ken Williams writes:
>
>> I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued
>> items (say 7000) all going to the same domain (samedomain.com). When
>> sendmail tries to run the queue (every 15 minutes or whatever), it
>> quickly (very quickly) spawns the processes and ties to connect to the
>> domain to deliver the messages as expected.
>>
>> The problem is because the connection to this server is so fast theres
>> no delay anywhere, it rips through like a 100 per second, so of course
>> the receiving domain (samedomain.com) will throttle it (Connection
>> rate limit exceeded).
>>
>> This actually makes it impossible to deliver these messages
>> normally. Only like 3 messages per 15 minutes actually get delivered
>> because the connection rate is exceeded for the rest, stopping them.
>> How can I fix this? I'm sure if I could tell sendmail to simply pause
>> for a second between each message attempt the connection rate would be
>> fine and it would (more slowly) go through each message in the queue
>> delivering just fine.
>>
>> Any ideas? What am I doing wrong?
>> kenw232@yahoo.com
>> [...]
>
> Just one more question before (possibly) giving *long* suboptimal advice:
> What generates the messages?
> Is it some "personalized mailing list" software?

Yes, more or less some personalized mailing list software I made will
give a bunch (say 10000) of messages to this sendmail server. The
sendmail server typically will deliver them fine. But if all the
messages are to the same domain it only takes one issue (destination
server is down, connection rate too high, etc.) and all (or a bunch) of
them will be queued up instead for later delivery. This is also usually
fine, but thats now my problem, sendmail on the next queue run will try
to connect 500 (whatever my confMAX_QUEUE_RUN_SIZE) times and instantly
try to deliver all 500 individually, easily clearing the connection rate
of the destination server therefor stopping like 495 of the messages.

Re: Intentionally slow down queue connections?

Post removed (X-No-Archive: yes)

Re: Intentionally slow down queue connections?

Ken Williams writes:

> Andrzej Adam Filip wrote:
>> Ken Williams writes:
>>
>>> I'm running sendmail 8.14.1 on linux 2.4. I have a lot of queued
>>> items (say 7000) all going to the same domain (samedomain.com). When
>>> sendmail tries to run the queue (every 15 minutes or whatever), it
>>> quickly (very quickly) spawns the processes and ties to connect to the
>>> domain to deliver the messages as expected.
>>>
>>> The problem is because the connection to this server is so fast theres
>>> no delay anywhere, it rips through like a 100 per second, so of course
>>> the receiving domain (samedomain.com) will throttle it (Connection
>>> rate limit exceeded).
>>>
>>> This actually makes it impossible to deliver these messages
>>> normally. Only like 3 messages per 15 minutes actually get delivered
>>> because the connection rate is exceeded for the rest, stopping them.
>>> How can I fix this? I'm sure if I could tell sendmail to simply pause
>>> for a second between each message attempt the connection rate would be
>>> fine and it would (more slowly) go through each message in the queue
>>> delivering just fine.
>>>
>>> Any ideas? What am I doing wrong?
>>> kenw232@yahoo.com
>>> [...]
>>
>> Just one more question before (possibly) giving *long* suboptimal advice:
>> What generates the messages?
>> Is it some "personalized mailing list" software?
>
> Yes, more or less some personalized mailing list software I made will
> give a bunch (say 10000) of messages to this sendmail server. The
> sendmail server typically will deliver them fine. But if all the
> messages are to the same domain it only takes one issue (destination
> server is down, connection rate too high, etc.) and all (or a bunch)
> of them will be queued up instead for later delivery. This is also
> usually fine, but thats now my problem, sendmail on the next queue run
> will try to connect 500 (whatever my confMAX_QUEUE_RUN_SIZE) times and
> instantly try to deliver all 500 individually, easily clearing the
> connection rate of the destination server therefor stopping like 495
> of the messages.

You may test moving queue processing from sendmail daemon (remove -bd
startup option) to cron jobs like the one below executed ever 2-5 minutes:
..../sendmail -q -OSingleThreadDelivery -OMinQueueAge=30m -OTimeout.hoststatus=10m

SingleThreadDelivery requires HostStatusDirectory
http://www.sendmail.org/m4/tweaking_config.html#confHOST_STA TUS_DIRECTORY

P.S.
You may also
1) limit number of messages sent over single smtp connection.
2) use a few hoststatus directories selected via cron job command line
to allow more than one connection to given destination host

--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
I've finally learned what "upward compatible" means. It means we get to
keep all our old mistakes.
-- Dennie van Tassel
----
http://groups.google.com/groups?selm=87k5lv9prn@william.fsf. hobby-site.com