How the fuck can I force sendmail to efficiently process its queue
How the fuck can I force sendmail to efficiently process its queue
am 21.11.2007 18:35:31 von Ignoramus4770
I have a machine with quad CPU and 4 gigs of RAM.
And yet for the life of me, I cannot figure out why I have so much
crap for local users that cannot be processed.
My sendmail is 8.13.
All I want is to run, maybe 15 queue runners at the same time, so that
undeliverable and slow junk does not slow down delivery of more
valuable things.
dnl FEATURE(`delay_checks')dnl
dnl FEATURE(`enhdnsbl', `relays.osirusoft.com', `DNSBL4: You are a rogue listserver', `t', `127.0.0.7.')
dnl FEATURE(dnsbl, `korea.services.net', `DNSBL5: No Mail from Korea is accepted.')dnl
dnl FEATURE(dnsbl, `relays.visi.com', `DNSBL6: visi.com: you are an open relay.')dnl
dnl FEATURE(dnsbl, `relays.ordb.org', `DNSBL7: relays.ordb.org: you are an open relay.')dnl
dnl FEATURE(dnsbl, `dynablock.wirehub.net', `DNSBL8: dynablock.wirehub.net says you are a dynamic IP.')dnl
dnl FEATURE(dnsbl, `nigeria.blackholes.us', `DNSBL9: We do not talk to Nigeria.')dnl
dnl FEATURE(dnsbl, `japan.blackholes.us', `DNSBL10: We do not talk to Japan.')dnl
dnl FEATURE(dnsbl, `argentina.blackholes.us ', `DNSBL11: We do not talk to argentina.')dnl
dnl FEATURE(dnsbl, `china.blackholes.us ', `DNSBL12: We do not talk to China.')dnl
dnl FEATURE(dnsbl, `brazil.blackholes.us ', `DNSBL13: We do not talk to Brazil.')dnl
dnl FEATURE(dnsbl, `malaysia.blackholes.us ', `DNSBL14: We do not talk to Malaysia.')dnl
dnl FEATURE(dnsbl, `taiwan.blackholes.us ', `DNSBL15: We do not talk to Taiwan.')dnl
dnl FEATURE(dnsbl, `hongkong.blackholes.us ', `DNSBL16: We do not talk to hongkong.')dnl
define(`PROCMAIL_MAILER_ARGS',`procmail -m $h $f $u -a $@x')dnl
SCheckFrom
R $+ @ xxx . net $#error $: 553 xxx.net does not send mail
R $+ @ homebiz . com $#error $: 553 homebiz.com does not exist
R $+ @ something . net $#error $: 553 something.net does not send mail
R $+ @ bar $#error $: 553 and which bar might that be?
R petlover @ $#error $: 553 No e-mails from petlovers (try petlover+real)
SCheckTo
R Friend @ public . com $#error $: 553 no friends at Public.com
D{MPat}Important Message From
D{MMsg}This message may contain the Melissa virus.
D{UDPat}UNIVERSITY DIPLOMAS FAST
SCheck_Subject
R${MPat} $* $#error $: 553 ${MMsg}
RRe: ${MPat} $* $#error $: 553 ${MMsg}
R${UDPat} $* $#error $: 551 Keep your fake diplomas, spammer
#H?M?X-Relay-IP: ${client_addr}
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@
# Igor Chudov's rules for algebra.com to video-collage.com
# conversion for certain addreses (cypherpunks, scrm etc).
LOCAL_RULESETS
SLocal_check_mail
#R < > $# error $@ 5.7.0 $: "554 Temporary rejecting error messages. My domain is forged in thousands of spams."
Re: How the fuck can I force sendmail to efficiently process its queue
am 21.11.2007 22:45:42 von Bill Cole
In article <3eednYYYNNf-8dnanZ2dnUVZ_tOtnZ2d@giganews.com>,
Ignoramus4770 wrote:
> I have a machine with quad CPU and 4 gigs of RAM.
>
> And yet for the life of me, I cannot figure out why I have so much
> crap for local users that cannot be processed.
>
> My sendmail is 8.13.
>
> All I want is to run, maybe 15 queue runners at the same time, so that
> undeliverable and slow junk does not slow down delivery of more
> valuable things.
>
> How can I do it?
Read the documentation? Provide relevant information? Local delivery
slowness is usually not a sendmail config problem. If the problem you
have really is slow local delivery, you need to look at things like your
global or individual procmail rules (since you are using procmail for
local delivery) and at the performance of whatever storage you are using
for local delivery. One thing that does look wrong below is your QueueLA
setting. Unless you are trying to protect something else on the box from
Sendmail, 12 or even 20 would make a lot more sense than 3.
FWIW, I suspect that your bigger problem is that you have not paying
attention to your use of third-party DNSBL's for a long time. Some of
those have been dead for YEARS and that means you are doing a blocking
DNS lookup on every connection that will fail only by timing out. In
addition to doing your own mail server performance damage, this also
makes you one of the many people who take an active part in an ongoing
DDoS of the people who own domains that formerly ran DNSBL's. I expect
that EasyNet can handle the traffic, but I know that Joe Jared has said
that he is essentially unable to do anything with osirusoft.com because
of the continued blind flood of DNS queries from people who do not pay
attention to their own systems. Of the ones that are still theoretically
functional in your config, the bulk are part of a chronically unreliable
operation that has a history of collapsing frequently for hours to days
at a time.
> divert(-1)
>
> include(`/usr/share/sendmail-cf/m4/cf.m4')dnl
>
> divert(0)dnl
> OSTYPE(linux)dnl
> DOMAIN(generic)dnl
>
> DAEMON_OPTIONS(`Name=MTA, Addr=65.182.171.162')
> DAEMON_OPTIONS(`Name=MTA1, Addr=127.0.0.1')
>
> define(`LOCAL_HACK', `PUSHDIVERT(-1)define(`_ARG_', `$2')define(`_ARGS_',
> `shift($@)')include(check_local-4.2/hack/$1.m4)POPDIVERT`'')
>
> define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn')dnl
> define( `confTO_QUEUEWARN_NORMAL', `2d' )
> define(`confTO_QUIT',`8m')
> define(`confEIGHT_BIT_HANDLING',`pass')
> # our numerous local host names
>
> # http://www.moshkow.kulichki.com/SENDMAIL/base64fiature.txt
> define(`confSMTP_MAILER', `smtp8')dnl
>
> define(confQUEUE_LA, 3)dnl
> define(confREFUSE_LA, 30)dnl
> define(confMAX_DAEMON_CHILDREN, 62)dnl
> define(confCONNECTION_RATE_THROTTLE, 13)dnl
>
> dnl define(conf_MAX_RUNNERS_PER_QUEUE, 35)dnl
> O MaxQueueChildren
> O MaxRunnersPerQueue=30
>
>
> dnl define(confDAEMON_OPTIONS, Address=208.233.99.160)dnl
>
> FEATURE(use_cw_file)dnl
> FEATURE(mailertable)dnl
> FEATURE(domaintable)dnl
> FEATURE(access_db)dnl
> FEATURE(`blacklist_recipients')dnl
> FEATURE(virtusertable)dnl
> FEATURE(`no_default_msa')dnl
>
> dnl ################################################## ANTISPAM
>
> dnl FEATURE(`delay_checks')dnl
> dnl FEATURE(`enhdnsbl', `relays.osirusoft.com', `DNSBL4: You are a rogue
> listserver', `t', `127.0.0.7.')
> dnl FEATURE(dnsbl, `korea.services.net', `DNSBL5: No Mail from Korea is
> accepted.')dnl
> dnl FEATURE(dnsbl, `relays.visi.com', `DNSBL6: visi.com: you are an open
> relay.')dnl
> dnl FEATURE(dnsbl, `relays.ordb.org', `DNSBL7: relays.ordb.org: you are an
> open relay.')dnl
> dnl FEATURE(dnsbl, `dynablock.wirehub.net', `DNSBL8: dynablock.wirehub.net
> says you are a dynamic IP.')dnl
> dnl FEATURE(dnsbl, `nigeria.blackholes.us', `DNSBL9: We do not talk to
> Nigeria.')dnl
> dnl FEATURE(dnsbl, `japan.blackholes.us', `DNSBL10: We do not talk to
> Japan.')dnl
> dnl FEATURE(dnsbl, `argentina.blackholes.us ', `DNSBL11: We do not talk to
> argentina.')dnl
> dnl FEATURE(dnsbl, `china.blackholes.us ', `DNSBL12: We do not talk to
> China.')dnl
> dnl FEATURE(dnsbl, `brazil.blackholes.us ', `DNSBL13: We do not talk to
> Brazil.')dnl
> dnl FEATURE(dnsbl, `malaysia.blackholes.us ', `DNSBL14: We do not talk to
> Malaysia.')dnl
> dnl FEATURE(dnsbl, `taiwan.blackholes.us ', `DNSBL15: We do not talk to
> Taiwan.')dnl
> dnl FEATURE(dnsbl, `hongkong.blackholes.us ', `DNSBL16: We do not talk to
> hongkong.')dnl
>
>
> define(`PROCMAIL_MAILER_ARGS',`procmail -m $h $f $u -a $@x')dnl
>
> Tichudov
> #Dmak47.algebra.com
> #Djak47.algebra.com
> Cwstump.algebra.com
>
> dnl FEATURE(local_procmail)dnl
>
> MAILER(smtp)dnl
> MAILER(procmail)dnl
>
> LOCAL_CONFIG
>
> LOCAL_RULESETS
> SLocal_check_mail_misha
> # check address against various regex checks
> R$* $: $>Parse0 $>3 $1
>
> #HMessage-Id: $>CheckMessageId
> HFrom: $>CheckFrom
> HTo: $>CheckTo
> HSubject: $>Check_Subject
>
> SCheckMessageId
> R< $+ @ $+ > $@ OK
> R$* $#error $: 553 Illegal Message-ID
>
> ### dnl define(`_READ_X_SPAM_FILT_',`dnl')dnl
> ### dnl LOCAL_HACK(`check_local')dnl
> ### dnl LOCAL_HACK(`check_header',`Message-Id',`',`',`',`',`',`',`1' )dnl
> ### dnl LOCAL_HACK(`check_header_end')dnl
>
>
>
> SCheckFrom
> R $+ @ xxx . net $#error $: 553 xxx.net does not send mail
> R $+ @ homebiz . com $#error $: 553 homebiz.com does not exist
> R $+ @ something . net $#error $: 553 something.net does not send mail
> R $+ @ bar $#error $: 553 and which bar might that be?
> R petlover @ $#error $: 553 No e-mails from petlovers (try petlover+real)
>
> SCheckTo
> R Friend @ public . com $#error $: 553 no friends at Public.com
>
>
> D{MPat}Important Message From
> D{MMsg}This message may contain the Melissa virus.
> D{UDPat}UNIVERSITY DIPLOMAS FAST
> SCheck_Subject
> R${MPat} $* $#error $: 553 ${MMsg}
> RRe: ${MPat} $* $#error $: 553 ${MMsg}
> R${UDPat} $* $#error $: 551 Keep your fake diplomas, spammer
>
> #H?M?X-Relay-IP: ${client_addr}
>
>
> # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@
> # Igor Chudov's rules for algebra.com to video-collage.com
> # conversion for certain addreses (cypherpunks, scrm etc).
>
> LOCAL_RULE_0
>
> ############################################################ scrm
> R scrm < @ algebra.com . > scrm < @ localhost . >
> R scrm-board < @ algebra.com . > scrm < @ localhost . >
> R scrm-mods < @ algebra.com . > scrm < @ localhost . >
> R scrm-approved < @ algebra.com . > scrm < @ localhost . >
> R scrm-rejected < @ algebra.com . > scrm < @ localhost . >
> R scrm-approval-key < @ algebra.com . > scrm < @ localhost . >
> R scrm-admin < @ algebra.com . > scrm < @ localhost . >
> R devnull < @ algebra.com . > devnull < @ localhost . >
> R passat-approval < @ algebra.com . > klm < @ cs.jhu.edu . >
> R ichudov < @ algebra.com . > ichudov-both < @ localhost . >
> R dasha < @ algebra.com . > dasha-both < @ localhost . >
> #R dasha < @ algebra.com . > pavlovd < @ ics.uci.edu . >
>
> ############################################################ Cypher Punks
> R cypherpunks < @ algebra.com . > cypherpunks < @ localhost . >
> R owner-cypherpunks < @ algebra.com . > devnull < @ localhost . >
>
> R cypherpunks-hosts < @ algebra.com . > cypherpunks-hosts < @ localhost . >
> R stump-users < @ algebra.com . > stump-users < @ localhost . >
> R jobs-discussion < @ algebra.com . > jobs-discussion < @ localhost . >
> R majordomo < @ algebra.com . > majordomo < @ localhost . >
>
> LOCAL_RULESETS
> SLocal_check_mail
> #R < > $# error $@ 5.7.0 $: "554 Temporary rejecting error messages. My
> domain is forged in thousands of spams."
--
Now where did I hide that website...
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 22.11.2007 04:45:33 von Scott Grayban
Ignoramus4770 wrote:
> I have a machine with quad CPU and 4 gigs of RAM.
>
> And yet for the life of me, I cannot figure out why I have so much
> crap for local users that cannot be processed.
>
> My sendmail is 8.13.
>
> All I want is to run, maybe 15 queue runners at the same time, so that
> undeliverable and slow junk does not slow down delivery of more
> valuable things.
>
> How can I do it?
>
Anyone that uses fowl language in the subject is much to immature to
understand anything we say.
Re: How the fuck can I force sendmail to efficiently process its queue
am 22.11.2007 18:59:23 von Ignoramus689
On 2007-11-22, Scott Grayban wrote:
> Ignoramus4770 wrote:
>> I have a machine with quad CPU and 4 gigs of RAM.
>>
>> And yet for the life of me, I cannot figure out why I have so much
>> crap for local users that cannot be processed.
>>
>> My sendmail is 8.13.
>>
>> All I want is to run, maybe 15 queue runners at the same time, so that
>> undeliverable and slow junk does not slow down delivery of more
>> valuable things.
>>
>> How can I do it?
>>
>
> Anyone that uses fowl language in the subject is much to immature to
> understand anything we say.
That's "foul". Fowl means certain kinds of birds.
i
Re: How the fuck can I force sendmail to efficiently process its queue
am 22.11.2007 19:03:24 von Ignoramus689
On 2007-11-21, Bill Cole wrote:
> In article <3eednYYYNNf-8dnanZ2dnUVZ_tOtnZ2d@giganews.com>,
> Ignoramus4770 wrote:
>
>> I have a machine with quad CPU and 4 gigs of RAM.
>>
>> And yet for the life of me, I cannot figure out why I have so much
>> crap for local users that cannot be processed.
>>
>> My sendmail is 8.13.
>>
>> All I want is to run, maybe 15 queue runners at the same time, so that
>> undeliverable and slow junk does not slow down delivery of more
>> valuable things.
>>
>> How can I do it?
>
> Read the documentation? Provide relevant information? Local delivery
> slowness is usually not a sendmail config problem. If the problem you
> have really is slow local delivery, you need to look at things like your
> global or individual procmail rules (since you are using procmail for
> local delivery) and at the performance of whatever storage you are using
> for local delivery. One thing that does look wrong below is your QueueLA
> setting. Unless you are trying to protect something else on the box from
> Sendmail, 12 or even 20 would make a lot more sense than 3.
I increased my QueueLA setting. Thanks. I also set up spamd
(spamassassin's daemon program) to maximum of 10 children instead of
5. Spamassassin makes a number od DNS queries and generally waits a
lot, so it makes sense to allow for more of its children.
> FWIW, I suspect that your bigger problem is that you have not paying
> attention to your use of third-party DNSBL's for a long time.
OK, maybe I am missing something, but I thought that I had them all
dnl'ed , kind of like commented out? (see quoted below)
> Some of those have been dead for YEARS and that means you are doing
> a blocking DNS lookup on every connection that will fail only by
> timing out. In addition to doing your own mail server performance
> damage, this also makes you one of the many people who take an
> active part in an ongoing DDoS of the people who own domains that
> formerly ran DNSBL's. I expect that EasyNet can handle the traffic,
> but I know that Joe Jared has said that he is essentially unable to
> do anything with osirusoft.com because of the continued blind flood
> of DNS queries from people who do not pay attention to their own
> systems. Of the ones that are still theoretically functional in your
> config, the bulk are part of a chronically unreliable operation that
> has a history of collapsing frequently for hours to days at a time.
I agree, but I thought that I took al lot them out. I will play some
more. Thanks.
i
>
>> divert(-1)
>>
>> include(`/usr/share/sendmail-cf/m4/cf.m4')dnl
>>
>> divert(0)dnl
>> OSTYPE(linux)dnl
>> DOMAIN(generic)dnl
>>
>> DAEMON_OPTIONS(`Name=MTA, Addr=65.182.171.162')
>> DAEMON_OPTIONS(`Name=MTA1, Addr=127.0.0.1')
>>
>> define(`LOCAL_HACK', `PUSHDIVERT(-1)define(`_ARG_', `$2')define(`_ARGS_',
>> `shift($@)')include(check_local-4.2/hack/$1.m4)POPDIVERT`'')
>>
>> define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn')dnl
>> define( `confTO_QUEUEWARN_NORMAL', `2d' )
>> define(`confTO_QUIT',`8m')
>> define(`confEIGHT_BIT_HANDLING',`pass')
>> # our numerous local host names
>>
>> # http://www.moshkow.kulichki.com/SENDMAIL/base64fiature.txt
>> define(`confSMTP_MAILER', `smtp8')dnl
>>
>> define(confQUEUE_LA, 3)dnl
>> define(confREFUSE_LA, 30)dnl
>> define(confMAX_DAEMON_CHILDREN, 62)dnl
>> define(confCONNECTION_RATE_THROTTLE, 13)dnl
>>
>> dnl define(conf_MAX_RUNNERS_PER_QUEUE, 35)dnl
>> O MaxQueueChildren
>> O MaxRunnersPerQueue=30
>>
>>
>> dnl define(confDAEMON_OPTIONS, Address=208.233.99.160)dnl
>>
>> FEATURE(use_cw_file)dnl
>> FEATURE(mailertable)dnl
>> FEATURE(domaintable)dnl
>> FEATURE(access_db)dnl
>> FEATURE(`blacklist_recipients')dnl
>> FEATURE(virtusertable)dnl
>> FEATURE(`no_default_msa')dnl
>>
>> dnl ################################################## ANTISPAM
>>
>> dnl FEATURE(`delay_checks')dnl
>> dnl FEATURE(`enhdnsbl', `relays.osirusoft.com', `DNSBL4: You are a rogue
>> listserver', `t', `127.0.0.7.')
>> dnl FEATURE(dnsbl, `korea.services.net', `DNSBL5: No Mail from Korea is
>> accepted.')dnl
>> dnl FEATURE(dnsbl, `relays.visi.com', `DNSBL6: visi.com: you are an open
>> relay.')dnl
>> dnl FEATURE(dnsbl, `relays.ordb.org', `DNSBL7: relays.ordb.org: you are an
>> open relay.')dnl
>> dnl FEATURE(dnsbl, `dynablock.wirehub.net', `DNSBL8: dynablock.wirehub.net
>> says you are a dynamic IP.')dnl
>> dnl FEATURE(dnsbl, `nigeria.blackholes.us', `DNSBL9: We do not talk to
>> Nigeria.')dnl
>> dnl FEATURE(dnsbl, `japan.blackholes.us', `DNSBL10: We do not talk to
>> Japan.')dnl
>> dnl FEATURE(dnsbl, `argentina.blackholes.us ', `DNSBL11: We do not talk to
>> argentina.')dnl
>> dnl FEATURE(dnsbl, `china.blackholes.us ', `DNSBL12: We do not talk to
>> China.')dnl
>> dnl FEATURE(dnsbl, `brazil.blackholes.us ', `DNSBL13: We do not talk to
>> Brazil.')dnl
>> dnl FEATURE(dnsbl, `malaysia.blackholes.us ', `DNSBL14: We do not talk to
>> Malaysia.')dnl
>> dnl FEATURE(dnsbl, `taiwan.blackholes.us ', `DNSBL15: We do not talk to
>> Taiwan.')dnl
>> dnl FEATURE(dnsbl, `hongkong.blackholes.us ', `DNSBL16: We do not talk to
>> hongkong.')dnl
>>
>>
>> define(`PROCMAIL_MAILER_ARGS',`procmail -m $h $f $u -a $@x')dnl
>>
>> Tichudov
>> #Dmak47.algebra.com
>> #Djak47.algebra.com
>> Cwstump.algebra.com
>>
>> dnl FEATURE(local_procmail)dnl
>>
>> MAILER(smtp)dnl
>> MAILER(procmail)dnl
>>
>> LOCAL_CONFIG
>>
>> LOCAL_RULESETS
>> SLocal_check_mail_misha
>> # check address against various regex checks
>> R$* $: $>Parse0 $>3 $1
>>
>> #HMessage-Id: $>CheckMessageId
>> HFrom: $>CheckFrom
>> HTo: $>CheckTo
>> HSubject: $>Check_Subject
>>
>> SCheckMessageId
>> R< $+ @ $+ > $@ OK
>> R$* $#error $: 553 Illegal Message-ID
>>
>> ### dnl define(`_READ_X_SPAM_FILT_',`dnl')dnl
>> ### dnl LOCAL_HACK(`check_local')dnl
>> ### dnl LOCAL_HACK(`check_header',`Message-Id',`',`',`',`',`',`',`1' )dnl
>> ### dnl LOCAL_HACK(`check_header_end')dnl
>>
>>
>>
>> SCheckFrom
>> R $+ @ xxx . net $#error $: 553 xxx.net does not send mail
>> R $+ @ homebiz . com $#error $: 553 homebiz.com does not exist
>> R $+ @ something . net $#error $: 553 something.net does not send mail
>> R $+ @ bar $#error $: 553 and which bar might that be?
>> R petlover @ $#error $: 553 No e-mails from petlovers (try petlover+real)
>>
>> SCheckTo
>> R Friend @ public . com $#error $: 553 no friends at Public.com
>>
>>
>> D{MPat}Important Message From
>> D{MMsg}This message may contain the Melissa virus.
>> D{UDPat}UNIVERSITY DIPLOMAS FAST
>> SCheck_Subject
>> R${MPat} $* $#error $: 553 ${MMsg}
>> RRe: ${MPat} $* $#error $: 553 ${MMsg}
>> R${UDPat} $* $#error $: 551 Keep your fake diplomas, spammer
>>
>> #H?M?X-Relay-IP: ${client_addr}
>>
>>
>> # @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@
>> # Igor Chudov's rules for algebra.com to video-collage.com
>> # conversion for certain addreses (cypherpunks, scrm etc).
>>
>> LOCAL_RULE_0
>>
>> ############################################################ scrm
>> R scrm < @ algebra.com . > scrm < @ localhost . >
>> R scrm-board < @ algebra.com . > scrm < @ localhost . >
>> R scrm-mods < @ algebra.com . > scrm < @ localhost . >
>> R scrm-approved < @ algebra.com . > scrm < @ localhost . >
>> R scrm-rejected < @ algebra.com . > scrm < @ localhost . >
>> R scrm-approval-key < @ algebra.com . > scrm < @ localhost . >
>> R scrm-admin < @ algebra.com . > scrm < @ localhost . >
>> R devnull < @ algebra.com . > devnull < @ localhost . >
>> R passat-approval < @ algebra.com . > klm < @ cs.jhu.edu . >
>> R ichudov < @ algebra.com . > ichudov-both < @ localhost . >
>> R dasha < @ algebra.com . > dasha-both < @ localhost . >
>> #R dasha < @ algebra.com . > pavlovd < @ ics.uci.edu . >
>>
>> ############################################################ Cypher Punks
>> R cypherpunks < @ algebra.com . > cypherpunks < @ localhost . >
>> R owner-cypherpunks < @ algebra.com . > devnull < @ localhost . >
>>
>> R cypherpunks-hosts < @ algebra.com . > cypherpunks-hosts < @ localhost . >
>> R stump-users < @ algebra.com . > stump-users < @ localhost . >
>> R jobs-discussion < @ algebra.com . > jobs-discussion < @ localhost . >
>> R majordomo < @ algebra.com . > majordomo < @ localhost . >
>>
>> LOCAL_RULESETS
>> SLocal_check_mail
>> #R < > $# error $@ 5.7.0 $: "554 Temporary rejecting error messages. My
>> domain is forged in thousands of spams."
>
Re: How the fsck can I force sendmail to efficiently process its queue
am 22.11.2007 19:38:17 von Ignoramus689
OK, I have changed my config. Set Queue_LA to higher value, and
removed mentions of all DNSBLs. I also went through my /etc/procmailrc
to see if there is a global recipe that has one lockfile, but no,
calls to spamd are not lockfiled.
Spamd is now being started with -m10 argument, which allows up to 10
children to run.
However, it does not help much. Here's the ps output
oot 29112 0.0 0.0 4080 1976 ? Ss 12:30 0:00 sendmail: accepting connections
root 29113 0.0 0.0 5460 2664 ? S 12:30 0:00 sendmail: ./lAMIOEhZ027950 mxpool01.netaddress.usa.net.: client DATA 354
root 30019 0.0 0.0 4400 2372 ? S 12:34 0:00 sendmail: server dsl88-226-55470.ttnet.net.tr [88.226.216.174] (may be forged) cmd read
root 30465 0.0 0.0 5368 2736 ? S 12:36 0:00 sendmail: ./lAMIaeHW030365 from queue
root 30513 0.1 0.0 5472 2684 ? Ss 12:37 0:00 sendmail: ./lAMG8xpu030414 gateway.mailrover.net.: user open
root 30517 0.0 0.0 5472 2644 ? Ss 12:37 0:00 sendmail: ./lAMF8LVv017617 apperception.com.: user open
root 30519 0.0 0.0 4780 2464 ? Ss 12:37 0:00 sendmail: ./lAMHOLIS015189 mail-kr5.bigfoot.com.: user open
root 30531 0.0 0.0 4800 2428 ? Ss 12:37 0:00 sendmail: ./lAMEInNB008825 mail-kr5.bigfoot.com.: user open
ichudov 30575 0.0 0.0 1660 472 pts/12 S+ 12:37 0:00 grep sendmail
# @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @@@@@@@@@@
# Igor Chudov's rules for algebra.com to video-collage.com
# conversion for certain addreses (cypherpunks, scrm etc).
LOCAL_RULESETS
SLocal_check_mail
#R < > $# error $@ 5.7.0 $: "554 Temporary rejecting error messages. My domain is forged in thousands of spams."
Re: How the fuck can I force sendmail to efficiently process its queue
am 23.11.2007 18:19:47 von Bill Cole
In article <_-udnXxLgYXxWdjanZ2dnUVZ_s7inZ2d@giganews.com>,
Ignoramus689 wrote:
> On 2007-11-21, Bill Cole wrote:
> > In article <3eednYYYNNf-8dnanZ2dnUVZ_tOtnZ2d@giganews.com>,
> > Ignoramus4770 wrote:
> >
> >> I have a machine with quad CPU and 4 gigs of RAM.
> >>
> >> And yet for the life of me, I cannot figure out why I have so much
> >> crap for local users that cannot be processed.
> >>
> >> My sendmail is 8.13.
> >>
> >> All I want is to run, maybe 15 queue runners at the same time, so that
> >> undeliverable and slow junk does not slow down delivery of more
> >> valuable things.
> >>
> >> How can I do it?
> >
> > Read the documentation? Provide relevant information? Local delivery
> > slowness is usually not a sendmail config problem. If the problem you
> > have really is slow local delivery, you need to look at things like your
> > global or individual procmail rules (since you are using procmail for
> > local delivery) and at the performance of whatever storage you are using
> > for local delivery. One thing that does look wrong below is your QueueLA
> > setting. Unless you are trying to protect something else on the box from
> > Sendmail, 12 or even 20 would make a lot more sense than 3.
>
> I increased my QueueLA setting. Thanks. I also set up spamd
> (spamassassin's daemon program) to maximum of 10 children instead of
> 5. Spamassassin makes a number od DNS queries and generally waits a
> lot, so it makes sense to allow for more of its children.
*Now* you mention it....
You really need to be looking from the physical mailstore backwards:
disk, filesystem, mailbox access (e.g. locking), filtering, delivery
agent, MTA. Starting with the MTA is not really the best troubleshooting
approach.
> > FWIW, I suspect that your bigger problem is that you have not paying
> > attention to your use of third-party DNSBL's for a long time.
>
> OK, maybe I am missing something, but I thought that I had them all
> dnl'ed , kind of like commented out? (see quoted below)
Yep. Sorry, visual oops on my part.
--
Now where did I hide that website...
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 23.11.2007 22:30:28 von unknown
Post removed (X-No-Archive: yes)
Re: How the fuck can I force sendmail to efficiently process its queue
am 24.11.2007 04:53:28 von Ignoramus24248
On 2007-11-23, Bill Cole wrote:
> In article <_-udnXxLgYXxWdjanZ2dnUVZ_s7inZ2d@giganews.com>,
> Ignoramus689 wrote:
>
>> On 2007-11-21, Bill Cole wrote:
>> > In article <3eednYYYNNf-8dnanZ2dnUVZ_tOtnZ2d@giganews.com>,
>> > Ignoramus4770 wrote:
>> >
>> >> I have a machine with quad CPU and 4 gigs of RAM.
>> >>
>> >> And yet for the life of me, I cannot figure out why I have so much
>> >> crap for local users that cannot be processed.
>> >>
>> >> My sendmail is 8.13.
>> >>
>> >> All I want is to run, maybe 15 queue runners at the same time, so that
>> >> undeliverable and slow junk does not slow down delivery of more
>> >> valuable things.
>> >>
>> >> How can I do it?
>> >
>> > Read the documentation? Provide relevant information? Local delivery
>> > slowness is usually not a sendmail config problem. If the problem you
>> > have really is slow local delivery, you need to look at things like your
>> > global or individual procmail rules (since you are using procmail for
>> > local delivery) and at the performance of whatever storage you are using
>> > for local delivery. One thing that does look wrong below is your QueueLA
>> > setting. Unless you are trying to protect something else on the box from
>> > Sendmail, 12 or even 20 would make a lot more sense than 3.
>>
>> I increased my QueueLA setting. Thanks. I also set up spamd
>> (spamassassin's daemon program) to maximum of 10 children instead of
>> 5. Spamassassin makes a number od DNS queries and generally waits a
>> lot, so it makes sense to allow for more of its children.
>
> *Now* you mention it....
>
> You really need to be looking from the physical mailstore backwards:
> disk, filesystem, mailbox access (e.g. locking), filtering, delivery
> agent, MTA. Starting with the MTA is not really the best troubleshooting
> approach.
Right.
Anyway, the changes that I made, along with sendmail -qR... started
REPEATEDLY, did seem to finally bring the count of local queued
messages to zero.
>
>> > FWIW, I suspect that your bigger problem is that you have not paying
>> > attention to your use of third-party DNSBL's for a long time.
>>
>> OK, maybe I am missing something, but I thought that I had them all
>> dnl'ed , kind of like commented out? (see quoted below)
>
> Yep. Sorry, visual oops on my part.
>
No problem. I took them out a year or two ago, when I set up
spamassassin, as its docs told me to stop using blacklists other than
through spamassassin.
i
Re: How the fuck can I force sendmail to efficiently process its queue
am 24.11.2007 05:52:25 von Bill Cole
In article ,
Ignoramus24248 wrote:
> On 2007-11-23, Bill Cole wrote:
> > In article <_-udnXxLgYXxWdjanZ2dnUVZ_s7inZ2d@giganews.com>,
> > Ignoramus689 wrote:
> >
> >> On 2007-11-21, Bill Cole wrote:
> >> > In article <3eednYYYNNf-8dnanZ2dnUVZ_tOtnZ2d@giganews.com>,
> >> > Ignoramus4770 wrote:
> >> >
> >> >> I have a machine with quad CPU and 4 gigs of RAM.
> >> >>
> >> >> And yet for the life of me, I cannot figure out why I have so much
> >> >> crap for local users that cannot be processed.
> >> >>
> >> >> My sendmail is 8.13.
> >> >>
> >> >> All I want is to run, maybe 15 queue runners at the same time, so that
> >> >> undeliverable and slow junk does not slow down delivery of more
> >> >> valuable things.
> >> >>
> >> >> How can I do it?
> >> >
> >> > Read the documentation? Provide relevant information? Local delivery
> >> > slowness is usually not a sendmail config problem. If the problem you
> >> > have really is slow local delivery, you need to look at things like your
> >> > global or individual procmail rules (since you are using procmail for
> >> > local delivery) and at the performance of whatever storage you are using
> >> > for local delivery. One thing that does look wrong below is your QueueLA
> >> > setting. Unless you are trying to protect something else on the box from
> >> > Sendmail, 12 or even 20 would make a lot more sense than 3.
> >>
> >> I increased my QueueLA setting. Thanks. I also set up spamd
> >> (spamassassin's daemon program) to maximum of 10 children instead of
> >> 5. Spamassassin makes a number od DNS queries and generally waits a
> >> lot, so it makes sense to allow for more of its children.
> >
> > *Now* you mention it....
> >
> > You really need to be looking from the physical mailstore backwards:
> > disk, filesystem, mailbox access (e.g. locking), filtering, delivery
> > agent, MTA. Starting with the MTA is not really the best troubleshooting
> > approach.
>
> Right.
>
> Anyway, the changes that I made, along with sendmail -qR... started
> REPEATEDLY, did seem to finally bring the count of local queued
> messages to zero.
That implies that you were running in queue-only mode a lot and were
never spawning queue runners.
The queue-only mode is a function of the QueueLA setting. When you had
it set to 3, hitting that should not have been uncommon. The other bit
begs the question: does your main sendmail daemon have a -q
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 24.11.2007 05:57:42 von unknown
Post removed (X-No-Archive: yes)
Re: How the fuck can I force sendmail to efficiently process its queue
am 24.11.2007 06:27:18 von Ignoramus24248
On 2007-11-24, Bill Cole wrote:
> In article ,
> Ignoramus24248 wrote:
>
>> On 2007-11-23, Bill Cole wrote:
>> > In article <_-udnXxLgYXxWdjanZ2dnUVZ_s7inZ2d@giganews.com>,
>> > Ignoramus689 wrote:
>> >
>> >> On 2007-11-21, Bill Cole wrote:
>> >> > In article <3eednYYYNNf-8dnanZ2dnUVZ_tOtnZ2d@giganews.com>,
>> >> > Ignoramus4770 wrote:
>> >> >
>> >> >> I have a machine with quad CPU and 4 gigs of RAM.
>> >> >>
>> >> >> And yet for the life of me, I cannot figure out why I have so much
>> >> >> crap for local users that cannot be processed.
>> >> >>
>> >> >> My sendmail is 8.13.
>> >> >>
>> >> >> All I want is to run, maybe 15 queue runners at the same time, so that
>> >> >> undeliverable and slow junk does not slow down delivery of more
>> >> >> valuable things.
>> >> >>
>> >> >> How can I do it?
>> >> >
>> >> > Read the documentation? Provide relevant information? Local delivery
>> >> > slowness is usually not a sendmail config problem. If the problem you
>> >> > have really is slow local delivery, you need to look at things like your
>> >> > global or individual procmail rules (since you are using procmail for
>> >> > local delivery) and at the performance of whatever storage you are using
>> >> > for local delivery. One thing that does look wrong below is your QueueLA
>> >> > setting. Unless you are trying to protect something else on the box from
>> >> > Sendmail, 12 or even 20 would make a lot more sense than 3.
>> >>
>> >> I increased my QueueLA setting. Thanks. I also set up spamd
>> >> (spamassassin's daemon program) to maximum of 10 children instead of
>> >> 5. Spamassassin makes a number od DNS queries and generally waits a
>> >> lot, so it makes sense to allow for more of its children.
>> >
>> > *Now* you mention it....
>> >
>> > You really need to be looking from the physical mailstore backwards:
>> > disk, filesystem, mailbox access (e.g. locking), filtering, delivery
>> > agent, MTA. Starting with the MTA is not really the best troubleshooting
>> > approach.
>>
>> Right.
>>
>> Anyway, the changes that I made, along with sendmail -qR... started
>> REPEATEDLY, did seem to finally bring the count of local queued
>> messages to zero.
>
> That implies that you were running in queue-only mode a lot and were
> never spawning queue runners.
Quite possibly (though I am not sure of the exact meaning of what you
said).
> The queue-only mode is a function of the QueueLA setting. When you had
> it set to 3, hitting that should not have been uncommon. The other bit
> begs the question: does your main sendmail daemon have a -q
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 24.11.2007 06:30:22 von Ignoramus24248
On 2007-11-24, Res wrote:
> On Fri, 23 Nov 2007, Ignoramus24248 wrote:
>
>
>
>
>> No problem. I took them out a year or two ago, when I set up
>> spamassassin, as its docs told me to stop using blacklists other than
>> through spamassassin.
>
> I think id rather stop the scum at MTA than accept the message and
> then run it through spamassassin
True, but the blacklists are usually transient and most go away after
their owners get fed up or move on to other things.
> If you have a busy network, you'd want to reverse that, enable at
> MTA, and DISABLE all blacklist lookups in SA
>
> I use these and have very little trouble
>
> dnl #cn kr tw hk
> FEATURE(`enhdnsbl', `zz.countries.nerd.dk', `"553 rejected"',`',`127.0.0.156.',`127.0.1.154.',`127.0.0.158.',` 127.0.1.88.')dnl
> FEATURE(`enhdnsbl', `dnsbl.sorbs.net', `"553 rejected - see http://www.sorbs.net/lookup.shtml?"$&{client_addr}', `')dnl
> FEATURE(`enhdnsbl', `bl.spamcop.net', `"553 rejected - see http://spamcop.net/bl.shtml?"$&{client_addr}', `')dnl
> FEATURE(`enhdnsbl', `combined.njabl.org',`"553 rejected - see http://njabl.org/lookup?"$&{client_addr}', `')dnl
>
> (yes I'm aware combined.njabl will soon cease to exist probably and we
> should use zen, but I disagree with spamhaus's policy on 'we'll block you
> from requests, pay us and you can rsync the DB', because so many people
> are blocked LONG before spamhaus reckons they should be, and some are not,
> so, either someone deliberately mucks with the big red button cause they
> are bored or they are technically incompetant in setting up a working
> automation of acl's.
>
> Very few hits make it to njabl as SORBS grabs most of it and
> Spamcop the few left overs :)
I do like spamassassing scoring policy, where a DNS lookup usually is
only advisory and affects the score. Some of my email contacts,
unfortunately, have been blocklisted by some blacklists, usually due
to either being clueless and running viruses, or due to inheriting a
bad IP. In any case, missing their emails would mean loss of $$ and
goodwill.
> Some blacklists being looked up in SA *may* be outdated, you'd need to
> check that as well if you insist on using SA doing the scumbag lookups
I update SA every night (SA rules).
i
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 24.11.2007 14:46:48 von per
In article Res writes:
>On Fri, 23 Nov 2007, Ignoramus24248 wrote:
>
>> No problem. I took them out a year or two ago, when I set up
>> spamassassin, as its docs told me to stop using blacklists other than
>> through spamassassin.
>
>I think id rather stop the scum at MTA than accept the message and
>then run it through spamassassin
>
>If you have a busy network, you'd want to reverse that, enable at
>MTA, and DISABLE all blacklist lookups in SA
If you run SA from a milter (e.g. MimeDefang), you don't need to
*accept* the message first - though you do need to receive it. So I
guess the ideal for someone that has a lot of time to spend babysitting
the anti-spam setup is to use a combination - sendmail rule check for
blacklists that you trust/value enough to let them make a final
judgement about your mail on their own (if any), SA for the others where
you want to apply scoring. Though of course if you really have a *lot*
of time, you could implement blacklist scoring in sendmail rules...
--Per Hedeland
per@hedeland.org
Re: How the fuck can I force sendmail to efficiently process its queue
am 24.11.2007 15:06:44 von per
In article
Ignoramus24248 writes:
>On 2007-11-24, Bill Cole wrote:
>>
>> That implies that you were running in queue-only mode a lot and were
>> never spawning queue runners.
>
>Quite possibly (though I am not sure of the exact meaning of what you
>said).
When your load average is above QueueLA, sendmail will only queue
messages, not deliver, even locally - i.e. "queue-only mode" - on the
assumption that delivery will raise the load further. Which is fine,
except that if your RefuseLA is *higher* and you have a busy server, you
will keep accepting more mail and queue it, which will keep the load
high (it will hover around RefuseLA), and you never get around to
delivering the queued mail, and the queue just grows and grows, which
makes queueing more and more expensive...
In short the default of having RefuseLA higher than QueueLA is basically
only appropriate for a server where mail processing is a low-intensity,
low-priority task - you really want to receive it if at all possible,
but other tasks are more important, so it's OK to let the received mail
sit in the queue until they're finished. This was probably the most
common use case back when those values were chosen, but for a dedicated,
busy mail server they are a disaster.
>> The queue-only mode is a function of the QueueLA setting. When you had
>> it set to 3, hitting that should not have been uncommon. The other bit
>> begs the question: does your main sendmail daemon have a -q
Re: How the fsck can I force sendmail to efficiently process its queue
am 24.11.2007 17:53:42 von Bill Cole
In article ,
Ignoramus24248 wrote:
> On 2007-11-24, Bill Cole wrote:
[...]
> > That implies that you were running in queue-only mode a lot and were
> > never spawning queue runners.
>
> Quite possibly (though I am not sure of the exact meaning of what you
> said).
You said this is on a 4-cpu machine, and originally your QueueLA setting
was 3. Whenever the load average went over 3 (roughly equivalent to a
load average of 0.75 on a 1-cpu machine, i.e. significantly idle)
Sendmail went into queue-only mode: rather than delivering mail as it
comes in (normal operation) Sendmail was dropping mail into the queue
rather than delivering it.
Why you had that and why you only increased the QueueLA to 6 after I
suggested much higher numbers is something only you can explain to
yourself. If you want to understand that setting, I suggest the Bat
Book. If you want a rule of thumb, I suggest 3-5 times the CPU count on
a modern system where your delivery stack (e.g. procmail+spamc/spamd in
your case) is hoggy. Setting it to 3 or 6 on a machine where you want
speedy delivery is self-defeating. The default is 8, and I'm pretty sure
that's chosen to be reasonable for single-cpu machines with very
lightweight local delivery.
> > The queue-only mode is a function of the QueueLA setting. When you had
> > it set to 3, hitting that should not have been uncommon. The other bit
> > begs the question: does your main sendmail daemon have a -q
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 24.11.2007 23:21:39 von unknown
Post removed (X-No-Archive: yes)
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 24.11.2007 23:49:58 von unknown
Post removed (X-No-Archive: yes)
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 25.11.2007 03:09:11 von per
In article Res writes:
>On Sat, 24 Nov 2007, Per Hedeland wrote:
>
>>> I think id rather stop the scum at MTA than accept the message and
>>> then run it through spamassassin
>>>
>>> If you have a busy network, you'd want to reverse that, enable at
>>> MTA, and DISABLE all blacklist lookups in SA
>>
>> If you run SA from a milter (e.g. MimeDefang), you don't need to
>> *accept* the message first - though you do need to receive it. So I
>
>Thats the thing, for DATA content you need to accept it all fopr testing,
>MIMEDefang and other milters do this then reject it,
It's of course up to the milter whether to reject, allow as-is, allow
with an added spam-warning, forward to a different recipient, or
whatever. All of those are doable with MIMEDefang, based on SA results
or other checks.
> the problem is with
>SA there can be many false alarms, the method used with say MailScanner
>is it also accepts it, then will silently discard high scored spam,
>but spam of low scrore is attached to a warning, that way if its
>"debateable spam" the user has the choice read it or delete it.
Silently discarding anything is generally a bad idea, but it's your
system - in any case the exact same behaviour can be achieved with
MIMEDefang + SA.
>Both of these options are admin configurable so deliver, delete store
>forward to some address or whatever you want. So in this case I can still
>get my ciscowire newsleters and solaris weekly and my network brokers
>specials, MIMEDefang would mark these as spam and I'd never get them,
Only if that's the way you set things up.
>MailScanner, although not blocking (and lets face it, since when has a 5xx
>response to spam *ever* stopped spam to that adress from the spammers)
You don't reject in the SMTP session to get rid of the spammers, but to
make sure notification reaches a legitimate sender in case of a false
positive, while still avoiding the risk of sending spam bounces.
>has the advantage of batch processing which from all tests I've done left
>MIMEDefang, the spamd and clam milters completely in its wake, it also is
>much nicer on resources, load wise and not holding an smtp connection open
>whilst milters do their thing.
Sure, if you don't care that legitimate mail may get lost, you can do
much more efficient processing.
--Per Hedeland
per@hedeland.org
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 25.11.2007 05:10:22 von unknown
Post removed (X-No-Archive: yes)
Re: How the fsck can I force sendmail to efficiently process itsqueue
am 25.11.2007 06:18:17 von Ignoramus11731
On 2007-11-24, Res wrote:
> On Sat, 24 Nov 2007, Per Hedeland wrote:
>
>> There are lots of different knobs to turn, and that particular one may
>> not even be the optimal one for you. In many/most cases it will be self-
>
> Has he tried using....
>
> define(`confSEPARATE_PROC',`True')
> define(`confQUEUE_SORT_ORDER', `modification')
>
> ...also one must ask, if he has a powerful box why oh why (unless hes
> on a home dsl/cable/isdn/dial connection) has he set:
> define(confMAX_DAEMON_CHILDREN, 62)dnl ????
> define(confCONNECTION_RATE_THROTTLE, 13)dnl
>
>
> On dual cpu HP dl380 G3/4's with same amount of ram I use -
> define(`confCONNECTION_RATE_THROTTLE', `150')
> define(`confMAX_DAEMON_CHILDREN',`500')
> define(`confMAX_QUEUE_CHILDREN',`500')
>
> ....the server usually does about 20-30 connections constantly,
> the load rarely goes above 3, and thats all thanks to spamassassin
>
>
Very nice. I want to, first, report that local queue has been at zero
since I made my changes a couple of days ago, and people do not report
stuck mail anymore.
For the record, some low settings I had, were inherited from a POS
linux box that died in '2000. (power supply failed) That's the
history.
There are no other performance related settings. I set QUEUE_LA low
because I also have a webserver, which actually makes money for me, I
would lose money if my webserver performance decreased (algebra.com).
But if emails are queued temporarily, it's OK. The key word here is
"temporarily".
Do these settings make sense to you?
thanks
i
Re: How the fuck can I force sendmail to efficiently process its queue
am 25.11.2007 06:20:54 von Ignoramus11731
On 2007-11-24, Per Hedeland wrote:
> In article
> Ignoramus24248 writes:
>>On 2007-11-24, Bill Cole wrote:
>>>
>>> That implies that you were running in queue-only mode a lot and were
>>> never spawning queue runners.
>>
>>Quite possibly (though I am not sure of the exact meaning of what you
>>said).
>
> When your load average is above QueueLA, sendmail will only queue
> messages, not deliver, even locally - i.e. "queue-only mode" - on the
> assumption that delivery will raise the load further. Which is fine,
> except that if your RefuseLA is *higher* and you have a busy server, you
> will keep accepting more mail and queue it, which will keep the load
> high (it will hover around RefuseLA), and you never get around to
> delivering the queued mail, and the queue just grows and grows, which
> makes queueing more and more expensive...
>
> In short the default of having RefuseLA higher than QueueLA is basically
> only appropriate for a server where mail processing is a low-intensity,
> low-priority task - you really want to receive it if at all possible,
> but other tasks are more important, so it's OK to let the received mail
> sit in the queue until they're finished. This was probably the most
> common use case back when those values were chosen, but for a dedicated,
> busy mail server they are a disaster.
Well, it is a webserver also (algebra.com), I definitely assign
webserving a higher priority. That's how I decide on settings.
>>> The queue-only mode is a function of the QueueLA setting. When you had
>>> it set to 3, hitting that should not have been uncommon. The other bit
>>> begs the question: does your main sendmail daemon have a -q
Re: How the fsck can I force sendmail to efficiently process its queue
am 25.11.2007 06:26:23 von Ignoramus11731
On 2007-11-24, Bill Cole wrote:
> In article ,
> Ignoramus24248 wrote:
>
>> On 2007-11-24, Bill Cole wrote:
> [...]
>> > That implies that you were running in queue-only mode a lot and were
>> > never spawning queue runners.
>>
>> Quite possibly (though I am not sure of the exact meaning of what you
>> said).
>
> You said this is on a 4-cpu machine, and originally your QueueLA setting
> was 3. Whenever the load average went over 3 (roughly equivalent to a
> load average of 0.75 on a 1-cpu machine, i.e. significantly idle)
> Sendmail went into queue-only mode: rather than delivering mail as it
> comes in (normal operation) Sendmail was dropping mail into the queue
> rather than delivering it.
The reason was stupid, I had these settings on a single CPU POS
machine that died in 2000.
> Why you had that and why you only increased the QueueLA to 6 after I
> suggested much higher numbers is something only you can explain to
> yourself. If you want to understand that setting, I suggest the Bat
> Book. If you want a rule of thumb, I suggest 3-5 times the CPU count on
> a modern system where your delivery stack (e.g. procmail+spamc/spamd in
> your case) is hoggy. Setting it to 3 or 6 on a machine where you want
> speedy delivery is self-defeating. The default is 8, and I'm pretty sure
> that's chosen to be reasonable for single-cpu machines with very
> lightweight local delivery.
OK, that makes total sense. I adjusted my settings (see below).
>> > The queue-only mode is a function of the QueueLA setting. When you had
>> > it set to 3, hitting that should not have been uncommon. The other bit
>> > begs the question: does your main sendmail daemon have a -q
Re: How the fsck can I force sendmail to efficiently process itsqueue
am 25.11.2007 09:39:32 von unknown
Post removed (X-No-Archive: yes)
Re: How the fsck can I force sendmail to efficiently process itsqueue
am 25.11.2007 15:35:51 von Ignoramus19423
On 2007-11-25, Res wrote:
> On Sat, 24 Nov 2007, Ignoramus11731 wrote:
>
>> For the record, some low settings I had, were inherited from a POS
>> linux box that died in '2000. (power supply failed) That's the
>> history.
>
> That explains it..
>
>> There are no other performance related settings. I set QUEUE_LA low
>> because I also have a webserver, which actually makes money for me, I
>> would lose money if my webserver performance decreased (algebra.com).
>
> Do you have a lot of hoggy cgi's? webserver traffic is normally very
> light, and in my experiences if its slow, its because of disk IO, not
> CPU/memory.
Yes, I do, I have scripts that solve mathematical equations and
simplify expressions, and draw a shipload of formulas as JPEGs and
animated GIFs (math cartoons). Plus I have a homegrown content
management system.
>> Do these settings make sense to you?
>
> They look a LOT better than you had :) and probably suffice well enough
> so you dont have your problem again, hopefully.
OK, thanks, I will keep them then, and will keep an eye on queues.
Thanks to all!
i
Re: How the fuck can I force sendmail to efficiently process its queue
am 25.11.2007 22:43:30 von per
In article
Ignoramus11731 writes:
>On 2007-11-24, Per Hedeland wrote:
>>
>> When your load average is above QueueLA, sendmail will only queue
>> messages, not deliver, even locally - i.e. "queue-only mode" - on the
>> assumption that delivery will raise the load further. Which is fine,
>> except that if your RefuseLA is *higher* and you have a busy server, you
>> will keep accepting more mail and queue it, which will keep the load
>> high (it will hover around RefuseLA), and you never get around to
>> delivering the queued mail, and the queue just grows and grows, which
>> makes queueing more and more expensive...
>>
>> In short the default of having RefuseLA higher than QueueLA is basically
>> only appropriate for a server where mail processing is a low-intensity,
>> low-priority task - you really want to receive it if at all possible,
>> but other tasks are more important, so it's OK to let the received mail
>> sit in the queue until they're finished. This was probably the most
>> common use case back when those values were chosen, but for a dedicated,
>> busy mail server they are a disaster.
>
>Well, it is a webserver also (algebra.com), I definitely assign
>webserving a higher priority. That's how I decide on settings.
OK, but I wrote *low-intensity* too...
>Thanks. I did make some more changes (see my other post), and also I
>will spawn queue runners every 10 minutes.
>
>define(confQUEUE_LA, 8)dnl
>define(confREFUSE_LA, 30)dnl
....and if the load on your server isn't always below 8 when mail is
coming in, those settings will drive it up towards 30, which may (or may
not) be problematic for your web service. Per above, I would suggest
that you set confREFUSE_LA to the maximum load you want to allow mail to
cause, and then set confQUEUE_LA to something slightly higher.
--Per Hedeland
per@hedeland.org
Re: How the fuck can I force sendmail to efficiently process its queue
am 26.11.2007 05:01:13 von Ignoramus19423
On 2007-11-25, Per Hedeland wrote:
>
>>Thanks. I did make some more changes (see my other post), and also I
>>will spawn queue runners every 10 minutes.
>>
>>define(confQUEUE_LA, 8)dnl
>>define(confREFUSE_LA, 30)dnl
>
> ...and if the load on your server isn't always below 8 when mail is
> coming in, those settings will drive it up towards 30, which may (or may
> not) be problematic for your web service. Per above, I would suggest
> that you set confREFUSE_LA to the maximum load you want to allow mail to
> cause, and then set confQUEUE_LA to something slightly higher.
>
> per@hedeland.org
OK, thanks, I set refuse to 15 and queue to 20. I think that I am all
set now. Thanks guys.
To the poster who frowned on my use of word "fuck", I asked exactly
the same question politely a couple days before asking it not so
politely. The polite question elicited zero responses. This one, did.
i
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 26.11.2007 07:46:10 von per
In article Res writes:
>On Sun, 25 Nov 2007, Per Hedeland wrote:
>
>>> has the advantage of batch processing which from all tests I've done left
>>> MIMEDefang, the spamd and clam milters completely in its wake, it also is
>>> much nicer on resources, load wise and not holding an smtp connection open
>>> whilst milters do their thing.
>>
>> Sure, if you don't care that legitimate mail may get lost, you can do
>> much more efficient processing.
>
>Never lost legitmate mail yet :)
Or you just don't know it.:-)
> I've run that setup on multiple sendmail
>and qmail servers, doing 2-3 million emails a day p/box, it works
>out rather nicely, and the boxes still sit idling saying "bring it on" :)
>
>Silently discarding mail is a very big no-no, you're quite correct, _BUT_
>I don't regard high scored spam as mail, its just pure trash that tried to
>invade the privacy of the average netizen sent by sombeboy who is just as
>much a piece of gutter trash as the trash they sent.
Yeah, there's probably some score level at which you can safely discard,
but what about the almost-certainly-spam below that? By accepting
everything and batch-processing afterwards you have to either discard
that too (which you apparently don't), or leave the users to deal with
it (with some help from your spam-warning header/flag). This means that
your users either have to wade throuh a lot of spam or (I would assume
this is what happens) set up their own filters which discards the
probably-spam unseen (or at best files it in a trash box which is only
looked at if they get some separate indication that they are missing a
message). If they don't get a lot of spam, your high score is obviously
not high enough to guarantee spam.
So, legitimate mail still runs a high risk of being effectively
discarded, just that you have moved the "responsibility" on to the
users. In comparison with such a setup, rejecting in the SMTP session
means that you can be a lot more aggressive - it's not the end of a
world if a legitimate message is rejected, because the sender will be
promptly notified, and your users won't have to see *or* discard the
spam with semi-high scores.
--Per Hedeland
per@hedeland.org
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 27.11.2007 05:14:11 von Bill Cole
In article ,
Res wrote:
> On Sat, 24 Nov 2007, Per Hedeland wrote:
>
> >> I think id rather stop the scum at MTA than accept the message and
> >> then run it through spamassassin
> >>
> >> If you have a busy network, you'd want to reverse that, enable at
> >> MTA, and DISABLE all blacklist lookups in SA
> >
> > If you run SA from a milter (e.g. MimeDefang), you don't need to
> > *accept* the message first - though you do need to receive it. So I
>
> Thats the thing, for DATA content you need to accept it all fopr testing,
> MIMEDefang and other milters do this then reject it, the problem is with
> SA there can be many false alarms, the method used with say MailScanner
> is it also accepts it, then will silently discard high scored spam,
> but spam of low scrore is attached to a warning, that way if its
> "debateable spam" the user has the choice read it or delete it.
Are you really familiar with how MIMEDefang works with SA?
The usual approach is to do the SA scan at the ends of DATA, rejecting
or accepting mail at that point and for accepted mail maybe tagging the
subject, adding a score header, or even accepting mail but not
delivering it (maybe delivering to a quarantine or 'spam folder' while
telling the MTA to discard its copy)
If you want to pass everything through using MD for MUA filtering, it is
trivially simple to modify the example filter script to do that. That
has been the case for all versions of MD I've used, dating back to 2004.
> Both of these options are admin configurable so deliver, delete store
> forward to some address or whatever you want. So in this case I can still
> get my ciscowire newsleters and solaris weekly and my network brokers
> specials, MIMEDefang would mark these as spam and I'd never get them,
> which I must first to do know teh addresses to whitelist.
No, you have to configure your spam filtering to do what you want it to
do, no matter what tool you pick. The variations in personal preferences
vary widely enough on enough axes that having a tool do what you like in
one way by default is pretty much a random coincidence and doesn't mean
that it will do what you like in other ways.
> MailScanner, although not blocking (and lets face it, since when has a 5xx
> response to spam *ever* stopped spam to that adress from the spammers)
From looking at corporate mail systems as large as 1 million+
messages/day, I see a lot of examples where 5xx at the end of data has
caused that sender to stop sending to the particular recipient. Most
importantly, I have seen that happen with both high-end spammers (i.e.
not the ones using zombied systems) and with their close relatives, the
'legitimate' bulk email marketers who send extremely spammy (as scored
by SA) mail to people who actually want those messages.
By making rejections explicit at the border, you make it possible for
senders to know that they should stop sending your system mail that you
don't want to deliver. If you accept mail and silently discard it, you
are acting as a sink for that mail and giving senders (spammers or
"false positive" victims) the impression that the message was delivered.
I've seen enough unbudging ranting on both/all sides of that issue to no
longer think that there is any hope for a rational consensus on best
practice...
> has the advantage of batch processing which from all tests I've done left
> MIMEDefang, the spamd and clam milters completely in its wake, it also is
> much nicer on resources, load wise and not holding an smtp connection open
> whilst milters do their thing.
>
> We used to have up to 500 connections active with that, now with MailScanner
> we'd have no more than 20 continous active per box.
The connection issue is a serious one, but as far as other resources are
concerned, all you are doing is spreading out the demand over time. In a
perverse way, the spiking can be a positive thing because spammers tend
to be less patient than legitimate senders and you end up shedding a
significant amount of the spike volume by spammers not waiting around.
> > blacklists that you trust/value enough to let them make a final
>
> I work on the basis that if you're in an RBL, you're in it for a reason
> and in 99% of the times not just because you sent one message somebody
> somewhere thought was spam, its usually a bit more then that.
>
> I do agree you must make your choice of RBLs carefuly, there are some
> cluless idiots out there like APEWS that have no clue, but reputable ones
> generally get it right, and thats less probable spam that SA has to sift
> through.
Those two paragraphs reflect a remarkable capacity cognitive dissonance.
There are a lot of DNSBL's that simply suck.
--
Now where did I hide that website...
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 27.11.2007 06:32:49 von Ignoramus23492
On 2007-11-27, Bill Cole wrote:
> Most importantly, I have seen that happen with both high-end
> spammers (i.e. not the ones using zombied systems)
Can you clarify this sentence?
> By making rejections explicit at the border, you make it possible for
> senders to know that they should stop sending your system mail that you
> don't want to deliver. If you accept mail and silently discard it, you
> are acting as a sink for that mail and giving senders (spammers or
> "false positive" victims) the impression that the message was delivered.
I actually agree. At the very least, I would like to learn how to 5xx
the really high score spam.
i
Re: How the fuck can I force sendmail to efficiently process itsqueue
am 29.11.2007 07:54:21 von unknown
Post removed (X-No-Archive: yes)
Re: How the fuck can I force sendmail to efficiently process itsqueue