Regular intermittent Kerberos failures

Hi guys,

This is a last desperate call for help. About once a week, for between
2 and 10 minutes, users are unable to log in to our main web
application (ASP based). They get the following message:

'Failed to generate SSPI context'

Looking at the System Log on the web server displays the following
messages for the web site and SQL SPNs:

'The Security System detected an authentication error for the server
HTTP/. The failure code from authentication protocol
Kerberos was "The time at the Primary Domain Controller is different
than the time at the Backup Domain Controller or member server by too
large an amount.
(0xc0000133)".'

' The Security System detected an authentication error for the server
MSSQLSvc/S05010010.corp.dnsdom.net:1433. The failure code from
authentication protocol Kerberos was "The time at the Primary Domain
Controller is different than the time at the Backup Domain Controller
or member server by too large an amount.
(0xc0000133)".'

I have used net time to check the times on the Domain Controller, web
server and db server. Can't see any problems. Our system guys have
been through the 'Failed to generate SSPI context' knowledge base
articles.

I haven't seen anything referring to this as a regularly repeating
intermittent problem. We are getting worried cos there is always the
chance it won't come back up!

Any help very gratefully received.

Cheers,

James

RE: Regular intermittent Kerberos failures

Hi James

At a guess this could be a network failure, although if there is a pattern
to the times this occur it would point to something which is scheduled such
as AV or IDS software.

To eliminate the time difference being an issue you may want to try
syncronising both servers with an external time source and not rely on the AD.

John

"JimLad" wrote:

> Hi guys,
>
> This is a last desperate call for help. About once a week, for between
> 2 and 10 minutes, users are unable to log in to our main web
> application (ASP based). They get the following message:
>
> 'Failed to generate SSPI context'
>
> Looking at the System Log on the web server displays the following
> messages for the web site and SQL SPNs:
>
> 'The Security System detected an authentication error for the server
> HTTP/. The failure code from authentication protocol
> Kerberos was "The time at the Primary Domain Controller is different
> than the time at the Backup Domain Controller or member server by too
> large an amount.
> (0xc0000133)".'
>
> ' The Security System detected an authentication error for the server
> MSSQLSvc/S05010010.corp.dnsdom.net:1433. The failure code from
> authentication protocol Kerberos was "The time at the Primary Domain
> Controller is different than the time at the Backup Domain Controller
> or member server by too large an amount.
> (0xc0000133)".'
>
> I have used net time to check the times on the Domain Controller, web
> server and db server. Can't see any problems. Our system guys have
> been through the 'Failed to generate SSPI context' knowledge base
> articles.
>
> I haven't seen anything referring to this as a regularly repeating
> intermittent problem. We are getting worried cos there is always the
> chance it won't come back up!
>
> Any help very gratefully received.
>
> Cheers,
>
> James
>
>

Re: Regular intermittent Kerberos failures

The messages you posted indicate an Active Directory configuration
problem rather than a SQL Server problem.

>From the information you've provided, its impossible to diagnose what
the problem is without knowing the architecture of your active
directory forest -- whether the HTTP server that logs into your SQL
Server is a member of the domain (which it sounds like it is), and
whether it goes thorough a firewall or any proxy servers that maybe
caching old records.

While Active Directory identifies clients connecting to servers,
Kerberos (which is a layer that runs ontop of active directory for
Microsoft platforms) also authenticates a server to the client. If
the servers are farmed, or there are many secondary domain
controllers, kerberos will check that they are all true mirrors of
each other to prevent somebody from setting up an unauthorized
secondary domain controller to spoof your forest (and thereby allow
unauthorized access via bogus active directory account entries on the
spoofed controller).

Re: Regular intermittent Kerberos failures

On Aug 28, 10:01 am, JimLad wrote:
> On Aug 24, 5:43 pm, Andy wrote:
>
>
>
>
>
> > The messages you posted indicate an Active Directory configuration
> > problem rather than a SQL Server problem.
>
> > >From the information you've provided, its impossible to diagnose what
>
> > the problem is without knowing the architecture of your active
> > directory forest -- whether the HTTP server that logs into your SQL
> > Server is a member of the domain (which it sounds like it is), and
> > whether it goes thorough a firewall or any proxy servers that maybe
> > caching old records.
>
> > While Active Directory identifies clients connecting to servers,
> > Kerberos (which is a layer that runs ontop of active directory for
> > Microsoft platforms) also authenticates a server to the client. If
> > the servers are farmed, or there are many secondary domain
> > controllers, kerberos will check that they are all true mirrors of
> > each other to prevent somebody from setting up an unauthorized
> > secondary domain controller to spoof your forest (and thereby allow
> > unauthorized access via bogus active directory account entries on the
> > spoofed controller).
>
> Hi Andy,
>
> Thanks for that. The messages indicate a timing problem: given that
> Kerberos only requires servers to be within 5 minutes is this a case
> of a misleading error message or is it that I am not using net time on
> enough domain controllers? I also notice that the Kerberos group
> policy "Maximum Tolerance for
> Computer Clock Synchronization" is 'Not Defined'. Does this need to
> be
> defined or will it automatically use the default of 5 minutes?
>
> Would turning on Kerberos event logging help to diagnose this? Would
> turning it on on the web server be sufficient or would it need to be
> enabled on dcs and db server as well? And would turning it on be a bad
> idea on a production system?
>
> More info: there is a single web server and a single db server, based
> in London, no proxy or firewall between them. There are 4 domain
> controllers in London. All of these are in the same domain. There are
> other domain controllers at other locations in the same domain.
>
> Any ideas on how to diagnose the problem would be extremely welcome.
>
> Many thanks.
>
> Cheers,
>
> James- Hide quoted text -
>
> - Show quoted text -

Hi,

We turned on Kerberos tracing and in the 16 seconds that it didn't
work this week we got the following messages on the web server:

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 3
Date: 30/08/2007
Time: 17:01:38
User: N/A
Computer: S05010072
Description:

A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 16:1:39.0000 8/30/2007 Z
Error Code: 0xb KDC_ERR_NEVER_VALID
Extended Error: 0xc0000133 KLIN(0)
Client Realm:
Client Name:
Server Realm: CORP.DNSDOM.NET
Server Name: MSSQLSvc/S05010010.corp.dnsdom.net:1433
Target Name: MSSQLSvc/S05010010.corp.dnsdom.net:1433@CORP.DNSDOM.NET
Error Text:
File: 9
Line: ae0
Error Data is in record data.

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 3
Date: 30/08/2007
Time: 17:01:47
User: N/A
Computer: S05010072
Description:

A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 16:1:49.0000 8/30/2007 Z
Error Code: 0xb KDC_ERR_NEVER_VALID
Extended Error: 0xc0000133 KLIN(0)
Client Realm:
Client Name:
Server Realm: CORP.DNSDOM.NET
Server Name: HTTP/
Target Name: HTTP/@CORP.DNSDOM.NET
Error Text:
File: 9
Line: ae0
Error Data is in record data.

0xB - KDC_ERR_NEVER_VALID: Requested start time is later than end time
Associated internal Windows error codes
=B7None
Corresponding debug output messages
=B7DebugLog("Client asked for endtime before starttime\n")
Possible Cause and Resolution
=B7There is a time difference between the KDC and the client.
Resolution
For Kerberos authentication to work, you must synchronize clocks on
the client and on the server. For more information about this error
and how to resolve it, see Time Synchronization (Clock Skew) earlier
in this white paper.

Any ideas why we would get this error message once a week for a window
of between a few seconds and 10 minutes?

Is there any way of knowing where the KDC is? I assume it's one of the
domain controllers, but as we have several is there a way of knowing
which is being used?

We have also been getting non-fatal Kerberos messages (0x25
KRB_AP_ERR_SKEW) about the time on file server S20. This isn't a DC
and isn't involved in the authentication so I'm not sure why we are
getting this message, even though that server is indeed 6 minutes
fast.

Outside this time window we get lots of the following messages:
0x34 KRB_ERR_RESPONSE_TOO_BIG
0xd KDC_ERR_BADOPTION
0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
0x25 KRB_AP_ERR_SKEW

Cheers,

James

Re: Regular intermittent Kerberos failures

Hi,

To answer some questions:

KDC runs on all Domain Controllers by default. You need to use a tool like
KerbTray or KList to see where the Kerberos tickets in question are coming
from


> We have also been getting non-fatal Kerberos messages (0x25
> KRB_AP_ERR_SKEW) about the time on file server S20. This isn't a DC
> and isn't involved in the authentication so I'm not sure why we are
> getting this message, even though that server is indeed 6 minutes
> fast.

Well, machines also authenticate to each other.

> 0x34 KRB_ERR_RESPONSE_TOO_BIG

Generally means that the packet was too big to be transmitted and was
fragmented. Should generally be OK, because Kerberos can be sent over TCP
rather than just UDP.

> 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN

An SPN is missing from within your Active Directory


> 0x25 KRB_AP_ERR_SKEW

Time is out by more than the permitted deviation.

It looks like you have some time sync issues in your organisation. Are you
using the default Windows time sync heirachy (by default all DCs sync time
with the PDCe FSMO role holder, and all clients sync with their
authenticating DCs), or have you overriden this in some way?

Cheers
Ken


"JimLad" wrote in message
news:1188491704.797545.33500@50g2000hsm.googlegroups.com...
On Aug 28, 10:01 am, JimLad wrote:
> On Aug 24, 5:43 pm, Andy wrote:
>
>
>
>
>
> > The messages you posted indicate an Active Directory configuration
> > problem rather than a SQL Server problem.
>
> > >From the information you've provided, its impossible to diagnose what
>
> > the problem is without knowing the architecture of your active
> > directory forest -- whether the HTTP server that logs into your SQL
> > Server is a member of the domain (which it sounds like it is), and
> > whether it goes thorough a firewall or any proxy servers that maybe
> > caching old records.
>
> > While Active Directory identifies clients connecting to servers,
> > Kerberos (which is a layer that runs ontop of active directory for
> > Microsoft platforms) also authenticates a server to the client. If
> > the servers are farmed, or there are many secondary domain
> > controllers, kerberos will check that they are all true mirrors of
> > each other to prevent somebody from setting up an unauthorized
> > secondary domain controller to spoof your forest (and thereby allow
> > unauthorized access via bogus active directory account entries on the
> > spoofed controller).
>
> Hi Andy,
>
> Thanks for that. The messages indicate a timing problem: given that
> Kerberos only requires servers to be within 5 minutes is this a case
> of a misleading error message or is it that I am not using net time on
> enough domain controllers? I also notice that the Kerberos group
> policy "Maximum Tolerance for
> Computer Clock Synchronization" is 'Not Defined'. Does this need to
> be
> defined or will it automatically use the default of 5 minutes?
>
> Would turning on Kerberos event logging help to diagnose this? Would
> turning it on on the web server be sufficient or would it need to be
> enabled on dcs and db server as well? And would turning it on be a bad
> idea on a production system?
>
> More info: there is a single web server and a single db server, based
> in London, no proxy or firewall between them. There are 4 domain
> controllers in London. All of these are in the same domain. There are
> other domain controllers at other locations in the same domain.
>
> Any ideas on how to diagnose the problem would be extremely welcome.
>
> Many thanks.
>
> Cheers,
>
> James- Hide quoted text -
>
> - Show quoted text -

Hi,

We turned on Kerberos tracing and in the 16 seconds that it didn't
work this week we got the following messages on the web server:

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 3
Date: 30/08/2007
Time: 17:01:38
User: N/A
Computer: S05010072
Description:

A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 16:1:39.0000 8/30/2007 Z
Error Code: 0xb KDC_ERR_NEVER_VALID
Extended Error: 0xc0000133 KLIN(0)
Client Realm:
Client Name:
Server Realm: CORP.DNSDOM.NET
Server Name: MSSQLSvc/S05010010.corp.dnsdom.net:1433
Target Name: MSSQLSvc/S05010010.corp.dnsdom.net:1433@CORP.DNSDOM.NET
Error Text:
File: 9
Line: ae0
Error Data is in record data.

Event Type: Error
Event Source: Kerberos
Event Category: None
Event ID: 3
Date: 30/08/2007
Time: 17:01:47
User: N/A
Computer: S05010072
Description:

A Kerberos Error Message was received:
on logon session
Client Time:
Server Time: 16:1:49.0000 8/30/2007 Z
Error Code: 0xb KDC_ERR_NEVER_VALID
Extended Error: 0xc0000133 KLIN(0)
Client Realm:
Client Name:
Server Realm: CORP.DNSDOM.NET
Server Name: HTTP/
Target Name: HTTP/@CORP.DNSDOM.NET
Error Text:
File: 9
Line: ae0
Error Data is in record data.

0xB - KDC_ERR_NEVER_VALID: Requested start time is later than end time
Associated internal Windows error codes
·None
Corresponding debug output messages
·DebugLog("Client asked for endtime before starttime\n")
Possible Cause and Resolution
·There is a time difference between the KDC and the client.
Resolution
For Kerberos authentication to work, you must synchronize clocks on
the client and on the server. For more information about this error
and how to resolve it, see Time Synchronization (Clock Skew) earlier
in this white paper.

Any ideas why we would get this error message once a week for a window
of between a few seconds and 10 minutes?

Is there any way of knowing where the KDC is? I assume it's one of the
domain controllers, but as we have several is there a way of knowing
which is being used?

We have also been getting non-fatal Kerberos messages (0x25
KRB_AP_ERR_SKEW) about the time on file server S20. This isn't a DC
and isn't involved in the authentication so I'm not sure why we are
getting this message, even though that server is indeed 6 minutes
fast.

Outside this time window we get lots of the following messages:
0x34 KRB_ERR_RESPONSE_TOO_BIG
0xd KDC_ERR_BADOPTION
0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
0x25 KRB_AP_ERR_SKEW

Cheers,

James

Re: Regular intermittent Kerberos failures

On Sep 4, 6:50 am, "Ken Schaefer"
wrote:
> Hi,
>
> To answer some questions:
>
> KDC runs on all Domain Controllers by default. You need to use a tool like
> KerbTray or KList to see where the Kerberos tickets in question are coming
> from
>
> > We have also been getting non-fatal Kerberos messages (0x25
> > KRB_AP_ERR_SKEW) about the time on file server S20. This isn't a DC
> > and isn't involved in the authentication so I'm not sure why we are
> > getting this message, even though that server is indeed 6 minutes
> > fast.
>
> Well, machines also authenticate to each other.
>
> > 0x34 KRB_ERR_RESPONSE_TOO_BIG
>
> Generally means that the packet was too big to be transmitted and was
> fragmented. Should generally be OK, because Kerberos can be sent over TCP
> rather than just UDP.
>
> > 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
>
> An SPN is missing from within your Active Directory
>
> > 0x25 KRB_AP_ERR_SKEW
>
> Time is out by more than the permitted deviation.
>
> It looks like you have some time sync issues in your organisation. Are you
> using the default Windows time sync heirachy (by default all DCs sync time
> with the PDCe FSMO role holder, and all clients sync with their
> authenticating DCs), or have you overriden this in some way?
>
> Cheers
> Ken
>
> "JimLad" wrote in message
>
> news:1188491704.797545.33500@50g2000hsm.googlegroups.com...
> On Aug 28, 10:01 am, JimLad wrote:
>
>
>
>
>
> > On Aug 24, 5:43 pm, Andy wrote:
>
> > > The messages you posted indicate an Active Directory configuration
> > > problem rather than a SQL Server problem.
>
> > > >From the information you've provided, its impossible to diagnose what
>
> > > the problem is without knowing the architecture of your active
> > > directory forest -- whether the HTTP server that logs into your SQL
> > > Server is a member of the domain (which it sounds like it is), and
> > > whether it goes thorough a firewall or any proxy servers that maybe
> > > caching old records.
>
> > > While Active Directory identifies clients connecting to servers,
> > > Kerberos (which is a layer that runs ontop of active directory for
> > > Microsoft platforms) also authenticates a server to the client. If
> > > the servers are farmed, or there are many secondary domain
> > > controllers, kerberos will check that they are all true mirrors of
> > > each other to prevent somebody from setting up an unauthorized
> > > secondary domain controller to spoof your forest (and thereby allow
> > > unauthorized access via bogus active directory account entries on the
> > > spoofed controller).
>
> > Hi Andy,
>
> > Thanks for that. The messages indicate a timing problem: given that
> > Kerberos only requires servers to be within 5 minutes is this a case
> > of a misleading error message or is it that I am not using net time on
> > enough domain controllers? I also notice that the Kerberos group
> > policy "Maximum Tolerance for
> > Computer Clock Synchronization" is 'Not Defined'. Does this need to
> > be
> > defined or will it automatically use the default of 5 minutes?
>
> > Would turning on Kerberos event logging help to diagnose this? Would
> > turning it on on the web server be sufficient or would it need to be
> > enabled on dcs and db server as well? And would turning it on be a bad
> > idea on a production system?
>
> > More info: there is a single web server and a single db server, based
> > in London, no proxy or firewall between them. There are 4 domain
> > controllers in London. All of these are in the same domain. There are
> > other domain controllers at other locations in the same domain.
>
> > Any ideas on how to diagnose the problem would be extremely welcome.
>
> > Many thanks.
>
> > Cheers,
>
> > James- Hide quoted text -
>
> > - Show quoted text -
>
> Hi,
>
> We turned on Kerberos tracing and in the 16 seconds that it didn't
> work this week we got the following messages on the web server:
>
> Event Type: Error
> Event Source: Kerberos
> Event Category: None
> Event ID: 3
> Date: 30/08/2007
> Time: 17:01:38
> User: N/A
> Computer: S05010072
> Description:
>
> A Kerberos Error Message was received:
> on logon session
> Client Time:
> Server Time: 16:1:39.0000 8/30/2007 Z
> Error Code: 0xb KDC_ERR_NEVER_VALID
> Extended Error: 0xc0000133 KLIN(0)
> Client Realm:
> Client Name:
> Server Realm: CORP.DNSDOM.NET
> Server Name: MSSQLSvc/S05010010.corp.dnsdom.net:1433
> Target Name: MSSQLSvc/S05010010.corp.dnsdom.net:1...@CORP.DNSDOM.NET
> Error Text:
> File: 9
> Line: ae0
> Error Data is in record data.
>
> Event Type: Error
> Event Source: Kerberos
> Event Category: None
> Event ID: 3
> Date: 30/08/2007
> Time: 17:01:47
> User: N/A
> Computer: S05010072
> Description:
>
> A Kerberos Error Message was received:
> on logon session
> Client Time:
> Server Time: 16:1:49.0000 8/30/2007 Z
> Error Code: 0xb KDC_ERR_NEVER_VALID
> Extended Error: 0xc0000133 KLIN(0)
> Client Realm:
> Client Name:
> Server Realm: CORP.DNSDOM.NET
> Server Name: HTTP/
> Target Name: HTTP/@CORP.DNSDOM.NET
> Error Text:
> File: 9
> Line: ae0
> Error Data is in record data.
>
> 0xB - KDC_ERR_NEVER_VALID: Requested start time is later than end time
> Associated internal Windows error codes
> =B7None
> Corresponding debug output messages
> =B7DebugLog("Client asked for endtime before starttime\n")
> Possible Cause and Resolution
> =B7There is a time difference between the KDC and the client.
> Resolution
> For Kerberos authentication to work, you must synchronize clocks on
> the client and on the server. For more information about this error
> and how to resolve it, see Time Synchronization (Clock Skew) earlier
> in this white paper.
>
> Any ideas why we would get this error message once a week for a window
> of between a few seconds and 10 minutes?
>
> Is there any way of knowing where the KDC is? I assume it's one of the
> domain controllers, but as we have several is there a way of knowing
> which is being used?
>
> We have also been getting non-fatal Kerberos messages (0x25
> KRB_AP_ERR_SKEW) about the time on file server S20. This isn't a DC
> and isn't involved in the authentication so I'm not sure why we are
> getting this message, even though that server is indeed 6 minutes
> fast.
>
> Outside this time window we get lots of the following messages:
> 0x34 KRB_ERR_RESPONSE_TOO_BIG
> 0xd KDC_ERR_BADOPTION
> 0x7 KDC_ERR_S_PRINCIPAL_UNKNOWN
> 0x25 KRB_AP_ERR_SKEW
>
> Cheers,
>
> James- Hide quoted text -
>
> - Show quoted text -

Thanks Ken. Useful stuff.

I've posted a new subject based on a message I found in the security
log on the DC.

http://groups.google.com/group/comp.protocols.kerberos/brows e_thread/thread=
/ce62e8b04e3cddad/5af3d0b03cee0927#5af3d0b03cee0927

Cheers,

James