My access map is configured for hash and it is basically at 300 K
entries. It takes about 15 minutes to build as I suspect that the
number of elements (fixed at 4096) is too small. When there were only
150 K entries it built in about 5 seconds. Since the number of elements
is not configurable, would I be better off switching to the btree map
type or changing the number of elements in makemap are re-building it?
Doug Hardie
> My access map is configured for hash and it is basically at 300 K
> entries. It takes about 15 minutes to build as I suspect that the
> number of elements (fixed at 4096) is too small. When there were only
> 150 K entries it built in about 5 seconds. Since the number of elements
> is not configurable, would I be better off switching to the btree map
> type or changing the number of elements in makemap are re-building it?
1) Use "-c size" option to make makemap use bigger cache.
The default cache size is 1048576 (1 Mega Byte).
Try "-c 4194304" (4MB) and find out if it makes a difference.
Remember: it will help to decrease creation times.
Lookup times may be "longer than usually".
2) If "1" does not help then consider creating btree map from
*presorted* input file.
[ from ancient discussion about virtusertable with 25E6 entries :-) ]
PS
I personally suggest "2" with some (perl) script doing presorting.
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
We are not loved by our friends for what we are; rather, we are loved in
spite of what we are.
-- Victor Hugo
----
http://groups.google.com/groups?selm=87fxxotfw5@francisco.fs f.hobby-site.com
In article <87fxxotfw5@francisco.fsf.hobby-site.com>,
Andrzej Adam Filip
> Doug Hardie
>
> > My access map is configured for hash and it is basically at 300 K
> > entries. It takes about 15 minutes to build as I suspect that the
> > number of elements (fixed at 4096) is too small. When there were only
> > 150 K entries it built in about 5 seconds. Since the number of elements
> > is not configurable, would I be better off switching to the btree map
> > type or changing the number of elements in makemap are re-building it?
>
> 1) Use "-c size" option to make makemap use bigger cache.
> The default cache size is 1048576 (1 Mega Byte).
> Try "-c 4194304" (4MB) and find out if it makes a difference.
>
> Remember: it will help to decrease creation times.
> Lookup times may be "longer than usually".
>
> 2) If "1" does not help then consider creating btree map from
> *presorted* input file.
> [ from ancient discussion about virtusertable with 25E6 entries :-) ]
>
> PS
> I personally suggest "2" with some (perl) script doing presorting.
Wow - what a difference. I ran the following tests on an otherwise idle
machine.
The reference test - the production approach.
time /usr/sbin/makemap hash /etc/mail/accessnew < /etc/mail/access.tmp
8.490u 43.003s 5:28.96 15.6% 76+1558k 1497+336io 0pf+0w
Changing the cache size.
time /usr/sbin/makemap -c 4194304 hash /etc/mail/accessnew <
/etc/mail/access.tmp
7.916u 31.640s 5:10.50 12.7% 76+5441k 1321+356io 0pf+0w
Using btree with unsorted input.
time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.tmp
6.161u 10.630s 2:10.71 12.8% 76+2810k 0+676io 0pf+0w
Using btree with sorted input.
time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.sort
2.317u 1.217s 0:04.19 84.0% 77+2789k 0+368io 0pf+0w
Btree with sorted input is incredibly fast. Since I am generating most
of the access file from a database it becomes real easy to have it
sorted. Even without the sort its faster though. I believe all I need
to do to switch to btree format is recreate the access file and then
change sendmail.cf from
Kaccess hash /etc/mail/access
to
Kaccess btree /etc/mail/access
Doug Hardie
> In article <87fxxotfw5@francisco.fsf.hobby-site.com>,
> Andrzej Adam Filip
>
>> Doug Hardie
>>
>> > My access map is configured for hash and it is basically at 300 K
>> > entries. It takes about 15 minutes to build as I suspect that the
>> > number of elements (fixed at 4096) is too small. When there were only
>> > 150 K entries it built in about 5 seconds. Since the number of elements
>> > is not configurable, would I be better off switching to the btree map
>> > type or changing the number of elements in makemap are re-building it?
>>
>> 1) Use "-c size" option to make makemap use bigger cache.
>> The default cache size is 1048576 (1 Mega Byte).
>> Try "-c 4194304" (4MB) and find out if it makes a difference.
>>
>> Remember: it will help to decrease creation times.
>> Lookup times may be "longer than usually".
>>
>> 2) If "1" does not help then consider creating btree map from
>> *presorted* input file.
>> [ from ancient discussion about virtusertable with 25E6 entries :-) ]
>>
>> PS
>> I personally suggest "2" with some (perl) script doing presorting.
>
> Wow - what a difference. I ran the following tests on an otherwise idle
> machine.
>
> The reference test - the production approach.
>
> time /usr/sbin/makemap hash /etc/mail/accessnew < /etc/mail/access.tmp
> 8.490u 43.003s 5:28.96 15.6% 76+1558k 1497+336io 0pf+0w
>
> Changing the cache size.
>
> time /usr/sbin/makemap -c 4194304 hash /etc/mail/accessnew <
> /etc/mail/access.tmp
> 7.916u 31.640s 5:10.50 12.7% 76+5441k 1321+356io 0pf+0w
>
> Using btree with unsorted input.
>
> time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.tmp
> 6.161u 10.630s 2:10.71 12.8% 76+2810k 0+676io 0pf+0w
>
> Using btree with sorted input.
>
> time /usr/sbin/makemap btree /etc/mail/accessnew < /etc/mail/access.sort
> 2.317u 1.217s 0:04.19 84.0% 77+2789k 0+368io 0pf+0w
>
> Btree with sorted input is incredibly fast. Since I am generating most
> of the access file from a database it becomes real easy to have it
> sorted. Even without the sort its faster though. I believe all I need
> to do to switch to btree format is recreate the access file and then
> change sendmail.cf from
>
> Kaccess hash /etc/mail/access
>
> to
>
> Kaccess btree /etc/mail/access
FYI: I have suggested (once) to allow (as an option) to make access use
keys format optimized for btree performance (increasing chances of
hitting cached page):
connect:xxx.example.com -> com.example.xxx:connect
xxx.example.com -> com.example.xxx:
connect:1.2.3.4 -> 1.2.3.4:connect
connect:5.6.7 -> 5.6.7:connect
8.9 -> 8.9:
Sendmail can do 20-30 access lookups when processing single incoming
message so reduction of number of I/O operations for *BIG* maps make
difference.
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
"We dedicated ourselves to a powerful idea -- organic law rather than naked
power. There seems to be universal acceptance of that idea in the nation."
-- Supreme Court Justice Potter Steart
----
http://groups.google.com/groups?selm=871w95q4hi@victoria.fsf .hobby-site.com
Doug Hardie
> [...] I believe all I need to do to switch to btree format is
> recreate the access file and then change sendmail.cf from
>
> Kaccess hash /etc/mail/access
>
> to
>
> Kaccess btree /etc/mail/access
You may simply change default db format used by sendmail from hash to btree:
define(`DATABASE_MAP_TYPE',`btree')dnl
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
Technology is a constand battle between manufacturers producing bigger and
more idiot-proof systems and nature producing bigger and better idiots.
-- Slashdot signature
----
http://groups.google.com/groups?selm=87tzm1opn6@wesley.fsf.h obby-site.com
In article <87tzm1opn6@wesley.fsf.hobby-site.com>,
Andrzej Adam Filip
> Doug Hardie
>
> > [...] I believe all I need to do to switch to btree format is
> > recreate the access file and then change sendmail.cf from
> >
> > Kaccess hash /etc/mail/access
> >
> > to
> >
> > Kaccess btree /etc/mail/access
>
> You may simply change default db format used by sendmail from hash to btree:
> define(`DATABASE_MAP_TYPE',`btree')dnl
Unfortunately, our sendmail.cf is much older than the mc stuff. There
is no way to create a mc file that makes our cf file.
Doug Hardie
> In article <87tzm1opn6@wesley.fsf.hobby-site.com>,
> Andrzej Adam Filip
>
>> Doug Hardie
>>
>> > [...] I believe all I need to do to switch to btree format is
>> > recreate the access file and then change sendmail.cf from
>> >
>> > Kaccess hash /etc/mail/access
>> >
>> > to
>> >
>> > Kaccess btree /etc/mail/access
>>
>> You may simply change default db format used by sendmail from hash to btree:
>> define(`DATABASE_MAP_TYPE',`btree')dnl
>
> Unfortunately, our sendmail.cf is much older than the mc stuff. There
> is no way to create a mc file that makes our cf file.
I hope you are going to fix it (sometime)?
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
Repel them. Repel them. Induce them to relinquish the spheroid.
-- Indiana University fans' chant for their perennially bad football team
----
http://groups.google.com/groups?selm=877iiwo9zb@theodore.fsf .hobby-site.com
In article <877iiwo9zb@theodore.fsf.hobby-site.com>,
Andrzej Adam Filip
> Doug Hardie
>
> > In article <87tzm1opn6@wesley.fsf.hobby-site.com>,
> > Andrzej Adam Filip
> >
> >> Doug Hardie
> >>
> >> > [...] I believe all I need to do to switch to btree format is
> >> > recreate the access file and then change sendmail.cf from
> >> >
> >> > Kaccess hash /etc/mail/access
> >> >
> >> > to
> >> >
> >> > Kaccess btree /etc/mail/access
> >>
> >> You may simply change default db format used by sendmail from hash to
> >> btree:
> >> define(`DATABASE_MAP_TYPE',`btree')dnl
> >
> > Unfortunately, our sendmail.cf is much older than the mc stuff. There
> > is no way to create a mc file that makes our cf file.
>
> I hope you are going to fix it (sometime)?
Not likely. The maintainer is quite comfortable with the cf file and
has thousands of lines of additional features incorporated.
Doug Hardie
> In article <877iiwo9zb@theodore.fsf.hobby-site.com>,
> Andrzej Adam Filip
>
>> Doug Hardie
>>
>> > In article <87tzm1opn6@wesley.fsf.hobby-site.com>,
>> > Andrzej Adam Filip
>> >
>> >> Doug Hardie
>> >>
>> >> > [...] I believe all I need to do to switch to btree format is
>> >> > recreate the access file and then change sendmail.cf from
>> >> >
>> >> > Kaccess hash /etc/mail/access
>> >> >
>> >> > to
>> >> >
>> >> > Kaccess btree /etc/mail/access
>> >>
>> >> You may simply change default db format used by sendmail from hash to
>> >> btree:
>> >> define(`DATABASE_MAP_TYPE',`btree')dnl
>> >
>> > Unfortunately, our sendmail.cf is much older than the mc stuff. There
>> > is no way to create a mc file that makes our cf file.
>>
>> I hope you are going to fix it (sometime)?
>
> Not likely. The maintainer is quite comfortable with the cf file and
> has thousands of lines of additional features incorporated.
I am unable to resist an urge to (miss)represent it as an non confidence
vote in sendmail.org development of mc files ;-)
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
A truth that's told with bad intent
Beats all the lies you can invent.
-- William Blake
----
http://groups.google.com/groups?selm=87ejd3a127@jess.fsf.hob by-site.com
Andrzej Adam Filip wrote:
> Doug Hardie
>> Not likely. The maintainer is quite comfortable with the cf file and
>> has thousands of lines of additional features incorporated.
> I am unable to resist an urge to (miss)represent it as an non confidence
> vote in sendmail.org development of mc files ;-)
No, I think it's a sysadmin ensuring job security...
Regards,
David.
"David F. Skoll"
> Andrzej Adam Filip wrote:
>
>> Doug Hardie
>>> Not likely. The maintainer is quite comfortable with the cf file and
>>> has thousands of lines of additional features incorporated.
>
>> I am unable to resist an urge to (miss)represent it as an non confidence
>> vote in sendmail.org development of mc files ;-)
>
> No, I think it's a sysadmin ensuring job security...
Recommendation to use m4 files to generate sendmail.cf lasts too many
years to make me capable to share (fully) your interpretation :-)
P.S.
1) Difference of opinions is something normal
2) Happy new 2008 year!
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
Knocked, you weren't in.
-- Opportunity
----
http://groups.google.com/groups?selm=87prwm67n8@albert.fsf.h obby-site.com
In article <87ejd3a127@jess.fsf.hobby-site.com>,
Andrzej Adam Filip
> Doug Hardie
>
>> > Not likely. The maintainer is quite comfortable with the cf file
and
> > has thousands of lines of additional features incorporated.
>
> I am unable to resist an urge to (miss)represent it as an non confidence
> vote in sendmail.org development of mc files ;-)
Nah its not that or job security. I doubt if anyone else would want to
spend 4 or so hours a day maintaing our spam blocking as a volunteer.
He has never used the m4 stuff and is quite comfortable writing his own
cf rules. He doesn't want to learn m4 and have to rewrite thousands of
lines of rules.
Post removed (X-No-Archive: yes)
Doug Hardie
> In article <87ejd3a127@jess.fsf.hobby-site.com>,
> Andrzej Adam Filip
>
>> Doug Hardie
>>
>> > Not likely. The maintainer is quite comfortable with the cf file
>> > and has thousands of lines of additional features incorporated.
>>
>> I am unable to resist an urge to (miss)represent it as an non confidence
>> vote in sendmail.org development of mc files ;-)
>
> Nah its not that or job security. I doubt if anyone else would want to
> spend 4 or so hours a day maintaing our spam blocking as a volunteer.
> He has never used the m4 stuff and is quite comfortable writing his own
> cf rules. He doesn't want to learn m4 and have to rewrite thousands of
> lines of rules.
One person doing something for one site only directly in sendmail.cf
*usually* means code that can not be maintained by anybody else.
--
[pl>en: Andrew] Andrzej Adam Filip : anfi@priv.onet.pl : anfi@xl.wp.pl
Open-Sendmail: http://open-sendmail.sourceforge.net/
'Martyrdom' is the only way a person can become famous without ability.
-- George Bernard Shaw
----
http://groups.google.com/groups?selm=87bq85kjab@thomas.fsf.h obby-site.com
In article <87bq85kjab@thomas.fsf.hobby-site.com>,
Andrzej Adam Filip
> Doug Hardie
>
> > In article <87ejd3a127@jess.fsf.hobby-site.com>,
> > Andrzej Adam Filip
> >
> >> Doug Hardie
> >>
> >> > Not likely. The maintainer is quite comfortable with the cf file
> >> > and has thousands of lines of additional features incorporated.
> >>
> >> I am unable to resist an urge to (miss)represent it as an non confidence
> >> vote in sendmail.org development of mc files ;-)
> >
> > Nah its not that or job security. I doubt if anyone else would want to
> > spend 4 or so hours a day maintaing our spam blocking as a volunteer.
> > He has never used the m4 stuff and is quite comfortable writing his own
> > cf rules. He doesn't want to learn m4 and have to rewrite thousands of
> > lines of rules.
>
> One person doing something for one site only directly in sendmail.cf
> *usually* means code that can not be maintained by anybody else.
Absolutely correct. I have tried to go through the added rules, and its
too much work. Its probably been this way since the late 80's. I don't
know when this setup was first created. It was ported to our server
from other servers he maintains.