Help Analyze Crash Dump

I'm running a .NET2.0 web app that is causing the workerprocess w3wp.exe to
crash several times a day.

Looks like the external dll file dmvclient.dll that I use in my application
is the cause. Unfortunately I don't have access to the .pdb right now.

Please tell me if I'm looking at the dump correctly when I say the apparent
call to the function dmvGetVarCount3 in dmvclient is causing this crash.

I have these lines in my code:
[DllImport("dmvclient.dll")]
public static extern int dmvGetVarCount3(int handle);

but there is no actual call to dmvGetVarCount3() in my code ...

If you can, please give me some pointers to find out more information.
Thanks!!
Victor


IIS DebugDiag output:
==============
Type of Analysis Performed Crash Analysis
Machine Name XXXXXXX
Operating System Windows Server 2003 Service Pack 2
Number Of Processors 4
Process ID 904
Process Image c:\WINNT\system32\inetsrv\w3wp.exe
System Up-Time 5 day(s) 13:45:31
Process Up-Time 0 day(s) 18:57:16


Thread 32 - System ID 4988
Entry point msvcrt!_endthread+3b
Create time 10/25/2007 6:47:10 AM
Time spent in user mode 0 Days 0:0:0.0
Time spent in kernel mode 0 Days 0:0:0.0

Function Arg 1 Arg 2 Arg 3 Source
msvcrt!_output+66b 1bb4fe18 1b36b0e4 1bb4fe48
msvcrt!sprintf+31 1bb4fe74 1b36b0e4 019281e8
dmvclient!dmvGetVarCount3+22c 0d20312e 000a0d0a 00000000
3020312e 0d20312e 000a0d0a 00000000
3020312e 00000000 00000000 00000000

MSVCRT!_OUTPUT+66BIn
w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Chance_Exception_C0000005.dmp
the assembly instruction at msvcrt!_output+66b in
C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused an access
violation exception (0xC0000005) when trying to read from memory location
0x019281e8 on thread 32

Module Information
Image Name: C:\WINNT\system32\msvcrt.dll Symbol Type: PDB
Base address: 0x77ba0000 Time Stamp: Sat Feb 17 09:02:46 2007
Checksum: 0x000635ef Comments:
COM DLL: False Company Name: Microsoft Corporation
ISAPIExtension: False File Description: Windows NT CRT DLL
ISAPIFilter: False File Version: 7.0.3790.3959 (srv03_sp2_rtm.070216-1710)
Managed DLL: False Internal Name: msvcrt.dll
VB DLL: False Legal Copyright: © Microsoft Corporation. All rights
reserved.
Loaded Image Name: msvcrt.dll Legal Trademarks:
Mapped Image Name: Original filename: msvcrt.dll
Module name: msvcrt Private Build:
Single Threaded: False Product Name: Microsoft® Windows® Operating System
Module Size: 360.00 KBytes Product Version: 7.0.3790.3959
Symbol File Name:
c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pdb Special
Build: &


WinDbg output:
==========
0:031> ~32s
^ Illegal thread error in '~32s'

0:031> kb
ChildEBP RetAddr Args to Child
1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
*** WARNING: Unable to verify checksum for dmvclient.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
dmvclient.dll -
1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
WARNING: Stack unwind information not available. Following frames may be
wrong.
1c3efe44 01948040 1bb77ba0 00000001 00000000 dmvclient!dmvGetVarCount3+0x22c
1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
1c3efe4c 00000000 00000000 1c3effb8 00000000 dmvclient!dmvGetVarCount3+0x659b0

Re: Help Analyze Crash Dump

You need symbols - the function names you see are the closest exported
functions that came before the instruction that failed. The +xxx after the
function name is the instruction offset which is really quite large, which
implies that it is happening well after the function listed. The MSVCRT
function wrong too - it is almost always a string compare, memory move, etc.
that crashes due to bad parameters being passed.

Pat


"Victor" wrote in message
news:B0A52819-E926-44A3-AF95-CD5C65EAD45E@microsoft.com...
> I'm running a .NET2.0 web app that is causing the workerprocess w3wp.exe
> to
> crash several times a day.
>
> Looks like the external dll file dmvclient.dll that I use in my
> application
> is the cause. Unfortunately I don't have access to the .pdb right now.
>
> Please tell me if I'm looking at the dump correctly when I say the
> apparent
> call to the function dmvGetVarCount3 in dmvclient is causing this crash.
>
> I have these lines in my code:
> [DllImport("dmvclient.dll")]
> public static extern int dmvGetVarCount3(int handle);
>
> but there is no actual call to dmvGetVarCount3() in my code ...
>
> If you can, please give me some pointers to find out more information.
> Thanks!!
> Victor
>
>
> IIS DebugDiag output:
> ==============
> Type of Analysis Performed Crash Analysis
> Machine Name XXXXXXX
> Operating System Windows Server 2003 Service Pack 2
> Number Of Processors 4
> Process ID 904
> Process Image c:\WINNT\system32\inetsrv\w3wp.exe
> System Up-Time 5 day(s) 13:45:31
> Process Up-Time 0 day(s) 18:57:16
>
>
> Thread 32 - System ID 4988
> Entry point msvcrt!_endthread+3b
> Create time 10/25/2007 6:47:10 AM
> Time spent in user mode 0 Days 0:0:0.0
> Time spent in kernel mode 0 Days 0:0:0.0
>
> Function Arg 1 Arg 2 Arg 3 Source
> msvcrt!_output+66b 1bb4fe18 1b36b0e4 1bb4fe48
> msvcrt!sprintf+31 1bb4fe74 1b36b0e4 019281e8
> dmvclient!dmvGetVarCount3+22c 0d20312e 000a0d0a 00000000
> 3020312e 0d20312e 000a0d0a 00000000
> 3020312e 00000000 00000000 00000000
>
> MSVCRT!_OUTPUT+66BIn
> w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Chance_Exception_C0000005.dmp
> the assembly instruction at msvcrt!_output+66b in
> C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused an
> access
> violation exception (0xC0000005) when trying to read from memory location
> 0x019281e8 on thread 32
>
> Module Information
> Image Name: C:\WINNT\system32\msvcrt.dll Symbol Type: PDB
> Base address: 0x77ba0000 Time Stamp: Sat Feb 17 09:02:46 2007
> Checksum: 0x000635ef Comments:
> COM DLL: False Company Name: Microsoft Corporation
> ISAPIExtension: False File Description: Windows NT CRT DLL
> ISAPIFilter: False File Version: 7.0.3790.3959
> (srv03_sp2_rtm.070216-1710)
> Managed DLL: False Internal Name: msvcrt.dll
> VB DLL: False Legal Copyright: © Microsoft Corporation. All rights
> reserved.
> Loaded Image Name: msvcrt.dll Legal Trademarks:
> Mapped Image Name: Original filename: msvcrt.dll
> Module name: msvcrt Private Build:
> Single Threaded: False Product Name: Microsoft® Windows® Operating
> System
> Module Size: 360.00 KBytes Product Version: 7.0.3790.3959
> Symbol File Name:
> c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pdb
> Special
> Build: &
>
>
> WinDbg output:
> ==========
> 0:031> ~32s
> ^ Illegal thread error in '~32s'
>
> 0:031> kb
> ChildEBP RetAddr Args to Child
> 1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
> *** WARNING: Unable to verify checksum for dmvclient.dll
> *** ERROR: Symbol file could not be found. Defaulted to export symbols
> for
> dmvclient.dll -
> 1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> 1c3efe44 01948040 1bb77ba0 00000001 00000000
> dmvclient!dmvGetVarCount3+0x22c
> 1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
> 1c3efe4c 00000000 00000000 1c3effb8 00000000
> dmvclient!dmvGetVarCount3+0x659b0

Re: Help Analyze Crash Dump

Pat,

That's great information! It helps me understand better.
I will try to get my hands on the pdb for the symbols.

Can you confirm that the crash is indeed happening in the dmvclient.dll (by
bad parameters or bad code, doesn't matter to me at this point) or could that
be wrong too?

Thanks!!
Victor


"Pat [MSFT]" wrote:

> You need symbols - the function names you see are the closest exported
> functions that came before the instruction that failed. The +xxx after the
> function name is the instruction offset which is really quite large, which
> implies that it is happening well after the function listed. The MSVCRT
> function wrong too - it is almost always a string compare, memory move, etc.
> that crashes due to bad parameters being passed.
>
> Pat
>
>
> "Victor" wrote in message
> news:B0A52819-E926-44A3-AF95-CD5C65EAD45E@microsoft.com...
> > I'm running a .NET2.0 web app that is causing the workerprocess w3wp.exe
> > to
> > crash several times a day.
> >
> > Looks like the external dll file dmvclient.dll that I use in my
> > application
> > is the cause. Unfortunately I don't have access to the .pdb right now.
> >
> > Please tell me if I'm looking at the dump correctly when I say the
> > apparent
> > call to the function dmvGetVarCount3 in dmvclient is causing this crash.
> >
> > I have these lines in my code:
> > [DllImport("dmvclient.dll")]
> > public static extern int dmvGetVarCount3(int handle);
> >
> > but there is no actual call to dmvGetVarCount3() in my code ...
> >
> > If you can, please give me some pointers to find out more information.
> > Thanks!!
> > Victor
> >
> >
> > IIS DebugDiag output:
> > ==============
> > Type of Analysis Performed Crash Analysis
> > Machine Name XXXXXXX
> > Operating System Windows Server 2003 Service Pack 2
> > Number Of Processors 4
> > Process ID 904
> > Process Image c:\WINNT\system32\inetsrv\w3wp.exe
> > System Up-Time 5 day(s) 13:45:31
> > Process Up-Time 0 day(s) 18:57:16
> >
> >
> > Thread 32 - System ID 4988
> > Entry point msvcrt!_endthread+3b
> > Create time 10/25/2007 6:47:10 AM
> > Time spent in user mode 0 Days 0:0:0.0
> > Time spent in kernel mode 0 Days 0:0:0.0
> >
> > Function Arg 1 Arg 2 Arg 3 Source
> > msvcrt!_output+66b 1bb4fe18 1b36b0e4 1bb4fe48
> > msvcrt!sprintf+31 1bb4fe74 1b36b0e4 019281e8
> > dmvclient!dmvGetVarCount3+22c 0d20312e 000a0d0a 00000000
> > 3020312e 0d20312e 000a0d0a 00000000
> > 3020312e 00000000 00000000 00000000
> >
> > MSVCRT!_OUTPUT+66BIn
> > w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Chance_Exception_C0000005.dmp
> > the assembly instruction at msvcrt!_output+66b in
> > C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused an
> > access
> > violation exception (0xC0000005) when trying to read from memory location
> > 0x019281e8 on thread 32
> >
> > Module Information
> > Image Name: C:\WINNT\system32\msvcrt.dll Symbol Type: PDB
> > Base address: 0x77ba0000 Time Stamp: Sat Feb 17 09:02:46 2007
> > Checksum: 0x000635ef Comments:
> > COM DLL: False Company Name: Microsoft Corporation
> > ISAPIExtension: False File Description: Windows NT CRT DLL
> > ISAPIFilter: False File Version: 7.0.3790.3959
> > (srv03_sp2_rtm.070216-1710)
> > Managed DLL: False Internal Name: msvcrt.dll
> > VB DLL: False Legal Copyright: © Microsoft Corporation. All rights
> > reserved.
> > Loaded Image Name: msvcrt.dll Legal Trademarks:
> > Mapped Image Name: Original filename: msvcrt.dll
> > Module name: msvcrt Private Build:
> > Single Threaded: False Product Name: Microsoft® Windows® Operating
> > System
> > Module Size: 360.00 KBytes Product Version: 7.0.3790.3959
> > Symbol File Name:
> > c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pdb
> > Special
> > Build: &
> >
> >
> > WinDbg output:
> > ==========
> > 0:031> ~32s
> > ^ Illegal thread error in '~32s'
> >
> > 0:031> kb
> > ChildEBP RetAddr Args to Child
> > 1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
> > *** WARNING: Unable to verify checksum for dmvclient.dll
> > *** ERROR: Symbol file could not be found. Defaulted to export symbols
> > for
> > dmvclient.dll -
> > 1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
> > WARNING: Stack unwind information not available. Following frames may be
> > wrong.
> > 1c3efe44 01948040 1bb77ba0 00000001 00000000
> > dmvclient!dmvGetVarCount3+0x22c
> > 1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
> > 1c3efe4c 00000000 00000000 1c3effb8 00000000
> > dmvclient!dmvGetVarCount3+0x659b0
>

Re: Help Analyze Crash Dump

The module names are not wrong. Just the actual function names, since
you are missing symbols, so you just get huge offsets from the last
public export.

While one can speculate on what is going on, you really need the right
symbols to know what API call is being made and make sense of the
parameters for the best possible diagnosis. And if you want to assign
responsibility, you need to have the best diagnosis.

You need that info to distinguish between memory corruption or just a
bug passing bad pointer around. Culprits are often different in those
cases, and responsibility can shift.

In short, I would first concentrate on getting the msvcrt symbols
(perhaps these guys copied their own version from Microsoft) to make
sense of which CRT function call is made, which then makes sense of
the parameters, and one can work backwards to see if it looks like
memory corruption, programming error, etc.


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//





On Jan 8, 1:45=A0pm, Victor wrote:
> Pat,
>
> That's great information! It helps me understand better.
> I will try to get my hands on the pdb for the symbols.
>
> Can you confirm that the crash is indeed happening in the dmvclient.dll (b=
y
> bad parameters or bad code, doesn't matter to me at this point) or could t=
hat
> be wrong too?
>
> Thanks!!
> Victor
>
>
>
> "Pat [MSFT]" wrote:
> > You need symbols - the function names you see are the closest exported
> > functions that came before the instruction that failed. =A0The +xxx afte=
r the
> > function name is the instruction offset which is really quite large, whi=
ch
> > implies that it is happening well after the function listed. =A0The MSVC=
RT
> > function wrong too - it is almost always a string compare, memory move, =
etc.
> > that crashes due to bad parameters being passed.
>
> > Pat
>
> > "Victor" wrote in message
> >news:B0A52819-E926-44A3-AF95-CD5C65EAD45E@microsoft.com...
> > > I'm running a .NET2.0 web app that is causing the workerprocess w3wp.e=
xe
> > > to
> > > crash several times a day.
>
> > > Looks like the external dll file dmvclient.dll that I use in my
> > > application
> > > is the cause. Unfortunately I don't have access to the .pdb right now.=

>
> > > Please tell me if I'm looking at the dump correctly when I say the
> > > apparent
> > > call to the function dmvGetVarCount3 in dmvclient is causing this cras=
h.
>
> > > I have these lines in my code:
> > > [DllImport("dmvclient.dll")]
> > > public static extern int dmvGetVarCount3(int handle);
>
> > > but there is no actual call to dmvGetVarCount3() in my code ...
>
> > > If you can, please give me some pointers to find out more information.=

> > > Thanks!!
> > > Victor
>
> > > IIS DebugDiag output:
> > > ==============
> > > Type of Analysis Performed =A0 Crash Analysis
> > > Machine Name =A0 XXXXXXX
> > > Operating System =A0 Windows Server 2003 Service Pack 2
> > > Number Of Processors =A0 4
> > > Process ID =A0 904
> > > Process Image =A0 c:\WINNT\system32\inetsrv\w3wp.exe
> > > System Up-Time =A0 5 day(s) 13:45:31
> > > Process Up-Time =A0 0 day(s) 18:57:16
>
> > > Thread 32 - System ID 4988
> > > Entry point =A0 msvcrt!_endthread+3b
> > > Create time =A0 10/25/2007 6:47:10 AM
> > > Time spent in user mode =A0 0 Days 0:0:0.0
> > > Time spent in kernel mode =A0 0 Days 0:0:0.0
>
> > > Function =A0 =A0 Arg 1 =A0 =A0 Arg 2 =A0 =A0 Arg 3 =A0 Source
> > > msvcrt!_output+66b =A0 =A0 1bb4fe18 =A0 =A0 1b36b0e4 =A0 =A0 1bb4fe48
> > > msvcrt!sprintf+31 =A0 =A0 1bb4fe74 =A0 =A0 1b36b0e4 =A0 =A0 019281e8
> > > dmvclient!dmvGetVarCount3+22c =A0 =A0 0d20312e =A0 =A0 000a0d0a =A0 =
=A0 00000000
> > > 3020312e =A0 =A0 0d20312e =A0 =A0 000a0d0a =A0 =A0 00000000
> > > 3020312e =A0 =A0 00000000 =A0 =A0 00000000 =A0 =A0 00000000
>
> > > MSVCRT!_OUTPUT+66BIn
> > > w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Chance_=
Excep=ADtion_C0000005.dmp
> > > the assembly instruction at msvcrt!_output+66b in
> > > C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused an
> > > access
> > > violation exception (0xC0000005) when trying to read from memory locat=
ion
> > > 0x019281e8 on thread 32
>
> > > Module Information
> > > Image Name: C:\WINNT\system32\msvcrt.dll =A0 Symbol Type: =A0PDB
> > > Base address: 0x77ba0000 =A0 Time Stamp: =A0Sat Feb 17 09:02:46 2007
> > > Checksum: 0x000635ef =A0 Comments:
> > > COM DLL: False =A0 Company Name: =A0Microsoft Corporation
> > > ISAPIExtension: False =A0 File Description: =A0Windows NT CRT DLL
> > > ISAPIFilter: False =A0 File Version: =A07.0.3790.3959
> > > (srv03_sp2_rtm.070216-1710)
> > > Managed DLL: False =A0 Internal Name: =A0msvcrt.dll
> > > VB DLL: False =A0 Legal Copyright:  © Microsoft Corporation. All r=
ights
> > > reserved.
> > > Loaded Image Name: =A0msvcrt.dll =A0 Legal Trademarks:
> > > Mapped Image Name: =A0 =A0 Original filename: =A0msvcrt.dll
> > > Module name: =A0msvcrt =A0 Private Build:
> > > Single Threaded: =A0False =A0 Product Name: =A0Microsoft=AE Windows=AE=
Operating
> > > System
> > > Module Size: =A0360.00 KBytes =A0 Product Version: =A07.0.3790.3959
> > > Symbol File Name:
> > > c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pdb
> > > Special
> > > Build: =A0&
>
> > > WinDbg output:
> > > ==========
> > > 0:031> ~32s
> > > =A0 =A0 =A0 =A0 ^ Illegal thread error in '~32s'
>
> > > 0:031> kb
> > > ChildEBP RetAddr =A0Args to Child
> > > 1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
> > > *** WARNING: Unable to verify checksum for dmvclient.dll
> > > *** ERROR: Symbol file could not be found. =A0Defaulted to export symb=
ols
> > > for
> > > dmvclient.dll -
> > > 1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
> > > WARNING: Stack unwind information not available. Following frames may =
be
> > > wrong.
> > > 1c3efe44 01948040 1bb77ba0 00000001 00000000
> > > dmvclient!dmvGetVarCount3+0x22c
> > > 1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
> > > 1c3efe4c 00000000 00000000 1c3effb8 00000000
> > > dmvclient!dmvGetVarCount3+0x659b0- Hide quoted text -
>
> - Show quoted text -

Re: Help Analyze Crash Dump

Thanks for the great information!

The strange thing is that I'm setting the symbol path to the microsoft
download site, I deleted all local symbol caches but the msvcrt still shows
offset. They have a standard msvcrt.dll on their Win2003 server.

The crashes are happening in staging as well as in production so I'm ruling
out memory on those servers that way although the file dmvclient.dll connects
to a third server where the memory corruption could occur.

Right now I'm waiting on the pdb file for dmvclient to find out more..

Thanks!

"David Wang" wrote:

> The module names are not wrong. Just the actual function names, since
> you are missing symbols, so you just get huge offsets from the last
> public export.
>
> While one can speculate on what is going on, you really need the right
> symbols to know what API call is being made and make sense of the
> parameters for the best possible diagnosis. And if you want to assign
> responsibility, you need to have the best diagnosis.
>
> You need that info to distinguish between memory corruption or just a
> bug passing bad pointer around. Culprits are often different in those
> cases, and responsibility can shift.
>
> In short, I would first concentrate on getting the msvcrt symbols
> (perhaps these guys copied their own version from Microsoft) to make
> sense of which CRT function call is made, which then makes sense of
> the parameters, and one can work backwards to see if it looks like
> memory corruption, programming error, etc.
>
>
> //David
> http://w3-4u.blogspot.com
> http://blogs.msdn.com/David.Wang
> //
>
>
>
>
>
> On Jan 8, 1:45 pm, Victor wrote:
> > Pat,
> >
> > That's great information! It helps me understand better.
> > I will try to get my hands on the pdb for the symbols.
> >
> > Can you confirm that the crash is indeed happening in the dmvclient.dll (by
> > bad parameters or bad code, doesn't matter to me at this point) or could that
> > be wrong too?
> >
> > Thanks!!
> > Victor
> >
> >
> >
> > "Pat [MSFT]" wrote:
> > > You need symbols - the function names you see are the closest exported
> > > functions that came before the instruction that failed. The +xxx after the
> > > function name is the instruction offset which is really quite large, which
> > > implies that it is happening well after the function listed. The MSVCRT
> > > function wrong too - it is almost always a string compare, memory move, etc.
> > > that crashes due to bad parameters being passed.
> >
> > > Pat
> >
> > > "Victor" wrote in message
> > >news:B0A52819-E926-44A3-AF95-CD5C65EAD45E@microsoft.com...
> > > > I'm running a .NET2.0 web app that is causing the workerprocess w3wp.exe
> > > > to
> > > > crash several times a day.
> >
> > > > Looks like the external dll file dmvclient.dll that I use in my
> > > > application
> > > > is the cause. Unfortunately I don't have access to the .pdb right now.
> >
> > > > Please tell me if I'm looking at the dump correctly when I say the
> > > > apparent
> > > > call to the function dmvGetVarCount3 in dmvclient is causing this crash.
> >
> > > > I have these lines in my code:
> > > > [DllImport("dmvclient.dll")]
> > > > public static extern int dmvGetVarCount3(int handle);
> >
> > > > but there is no actual call to dmvGetVarCount3() in my code ...
> >
> > > > If you can, please give me some pointers to find out more information.
> > > > Thanks!!
> > > > Victor
> >
> > > > IIS DebugDiag output:
> > > > ==============
> > > > Type of Analysis Performed Crash Analysis
> > > > Machine Name XXXXXXX
> > > > Operating System Windows Server 2003 Service Pack 2
> > > > Number Of Processors 4
> > > > Process ID 904
> > > > Process Image c:\WINNT\system32\inetsrv\w3wp.exe
> > > > System Up-Time 5 day(s) 13:45:31
> > > > Process Up-Time 0 day(s) 18:57:16
> >
> > > > Thread 32 - System ID 4988
> > > > Entry point msvcrt!_endthread+3b
> > > > Create time 10/25/2007 6:47:10 AM
> > > > Time spent in user mode 0 Days 0:0:0.0
> > > > Time spent in kernel mode 0 Days 0:0:0.0
> >
> > > > Function Arg 1 Arg 2 Arg 3 Source
> > > > msvcrt!_output+66b 1bb4fe18 1b36b0e4 1bb4fe48
> > > > msvcrt!sprintf+31 1bb4fe74 1b36b0e4 019281e8
> > > > dmvclient!dmvGetVarCount3+22c 0d20312e 000a0d0a 00000000
> > > > 3020312e 0d20312e 000a0d0a 00000000
> > > > 3020312e 00000000 00000000 00000000
> >
> > > > MSVCRT!_OUTPUT+66BIn
> > > > w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Chance_Excep­tion_C0000005.dmp
> > > > the assembly instruction at msvcrt!_output+66b in
> > > > C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused an
> > > > access
> > > > violation exception (0xC0000005) when trying to read from memory location
> > > > 0x019281e8 on thread 32
> >
> > > > Module Information
> > > > Image Name: C:\WINNT\system32\msvcrt.dll Symbol Type: PDB
> > > > Base address: 0x77ba0000 Time Stamp: Sat Feb 17 09:02:46 2007
> > > > Checksum: 0x000635ef Comments:
> > > > COM DLL: False Company Name: Microsoft Corporation
> > > > ISAPIExtension: False File Description: Windows NT CRT DLL
> > > > ISAPIFilter: False File Version: 7.0.3790.3959
> > > > (srv03_sp2_rtm.070216-1710)
> > > > Managed DLL: False Internal Name: msvcrt.dll
> > > > VB DLL: False Legal Copyright: © Microsoft Corporation. All rights
> > > > reserved.
> > > > Loaded Image Name: msvcrt.dll Legal Trademarks:
> > > > Mapped Image Name: Original filename: msvcrt.dll
> > > > Module name: msvcrt Private Build:
> > > > Single Threaded: False Product Name: Microsoft® Windows® Operating
> > > > System
> > > > Module Size: 360.00 KBytes Product Version: 7.0.3790.3959
> > > > Symbol File Name:
> > > > c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pdb
> > > > Special
> > > > Build: &
> >
> > > > WinDbg output:
> > > > ==========
> > > > 0:031> ~32s
> > > > ^ Illegal thread error in '~32s'
> >
> > > > 0:031> kb
> > > > ChildEBP RetAddr Args to Child
> > > > 1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
> > > > *** WARNING: Unable to verify checksum for dmvclient.dll
> > > > *** ERROR: Symbol file could not be found. Defaulted to export symbols
> > > > for
> > > > dmvclient.dll -
> > > > 1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
> > > > WARNING: Stack unwind information not available. Following frames may be
> > > > wrong.
> > > > 1c3efe44 01948040 1bb77ba0 00000001 00000000
> > > > dmvclient!dmvGetVarCount3+0x22c
> > > > 1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
> > > > 1c3efe4c 00000000 00000000 1c3effb8 00000000
> > > > dmvclient!dmvGetVarCount3+0x659b0- Hide quoted text -
> >
> > - Show quoted text -
>
>

Re: Help Analyze Crash Dump

If you can post the new stack w/the symbols we might be able to get a better
view. If you do a disassembly (u eip-32 eip+32) it will show what exactly
was happening. It looks like a deref failed.


Pat



"Victor" wrote in message
news:71B491EC-292C-4DC1-BD85-D72952BD5477@microsoft.com...
> Thanks for the great information!
>
> The strange thing is that I'm setting the symbol path to the microsoft
> download site, I deleted all local symbol caches but the msvcrt still
> shows
> offset. They have a standard msvcrt.dll on their Win2003 server.
>
> The crashes are happening in staging as well as in production so I'm
> ruling
> out memory on those servers that way although the file dmvclient.dll
> connects
> to a third server where the memory corruption could occur.
>
> Right now I'm waiting on the pdb file for dmvclient to find out more..
>
> Thanks!
>
> "David Wang" wrote:
>
>> The module names are not wrong. Just the actual function names, since
>> you are missing symbols, so you just get huge offsets from the last
>> public export.
>>
>> While one can speculate on what is going on, you really need the right
>> symbols to know what API call is being made and make sense of the
>> parameters for the best possible diagnosis. And if you want to assign
>> responsibility, you need to have the best diagnosis.
>>
>> You need that info to distinguish between memory corruption or just a
>> bug passing bad pointer around. Culprits are often different in those
>> cases, and responsibility can shift.
>>
>> In short, I would first concentrate on getting the msvcrt symbols
>> (perhaps these guys copied their own version from Microsoft) to make
>> sense of which CRT function call is made, which then makes sense of
>> the parameters, and one can work backwards to see if it looks like
>> memory corruption, programming error, etc.
>>
>>
>> //David
>> http://w3-4u.blogspot.com
>> http://blogs.msdn.com/David.Wang
>> //
>>
>>
>>
>>
>>
>> On Jan 8, 1:45 pm, Victor wrote:
>> > Pat,
>> >
>> > That's great information! It helps me understand better.
>> > I will try to get my hands on the pdb for the symbols.
>> >
>> > Can you confirm that the crash is indeed happening in the dmvclient.dll
>> > (by
>> > bad parameters or bad code, doesn't matter to me at this point) or
>> > could that
>> > be wrong too?
>> >
>> > Thanks!!
>> > Victor
>> >
>> >
>> >
>> > "Pat [MSFT]" wrote:
>> > > You need symbols - the function names you see are the closest
>> > > exported
>> > > functions that came before the instruction that failed. The +xxx
>> > > after the
>> > > function name is the instruction offset which is really quite large,
>> > > which
>> > > implies that it is happening well after the function listed. The
>> > > MSVCRT
>> > > function wrong too - it is almost always a string compare, memory
>> > > move, etc.
>> > > that crashes due to bad parameters being passed.
>> >
>> > > Pat
>> >
>> > > "Victor" wrote in message
>> > >news:B0A52819-E926-44A3-AF95-CD5C65EAD45E@microsoft.com...
>> > > > I'm running a .NET2.0 web app that is causing the workerprocess
>> > > > w3wp.exe
>> > > > to
>> > > > crash several times a day.
>> >
>> > > > Looks like the external dll file dmvclient.dll that I use in my
>> > > > application
>> > > > is the cause. Unfortunately I don't have access to the .pdb right
>> > > > now.
>> >
>> > > > Please tell me if I'm looking at the dump correctly when I say the
>> > > > apparent
>> > > > call to the function dmvGetVarCount3 in dmvclient is causing this
>> > > > crash.
>> >
>> > > > I have these lines in my code:
>> > > > [DllImport("dmvclient.dll")]
>> > > > public static extern int dmvGetVarCount3(int handle);
>> >
>> > > > but there is no actual call to dmvGetVarCount3() in my code ...
>> >
>> > > > If you can, please give me some pointers to find out more
>> > > > information.
>> > > > Thanks!!
>> > > > Victor
>> >
>> > > > IIS DebugDiag output:
>> > > > ==============
>> > > > Type of Analysis Performed Crash Analysis
>> > > > Machine Name XXXXXXX
>> > > > Operating System Windows Server 2003 Service Pack 2
>> > > > Number Of Processors 4
>> > > > Process ID 904
>> > > > Process Image c:\WINNT\system32\inetsrv\w3wp.exe
>> > > > System Up-Time 5 day(s) 13:45:31
>> > > > Process Up-Time 0 day(s) 18:57:16
>> >
>> > > > Thread 32 - System ID 4988
>> > > > Entry point msvcrt!_endthread+3b
>> > > > Create time 10/25/2007 6:47:10 AM
>> > > > Time spent in user mode 0 Days 0:0:0.0
>> > > > Time spent in kernel mode 0 Days 0:0:0.0
>> >
>> > > > Function Arg 1 Arg 2 Arg 3 Source
>> > > > msvcrt!_output+66b 1bb4fe18 1b36b0e4 1bb4fe48
>> > > > msvcrt!sprintf+31 1bb4fe74 1b36b0e4 019281e8
>> > > > dmvclient!dmvGetVarCount3+22c 0d20312e 000a0d0a
>> > > > 00000000
>> > > > 3020312e 0d20312e 000a0d0a 00000000
>> > > > 3020312e 00000000 00000000 00000000
>> >
>> > > > MSVCRT!_OUTPUT+66BIn
>> > > > w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Chance_Excep­tion_C0000005.dmp
>> > > > the assembly instruction at msvcrt!_output+66b in
>> > > > C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused
>> > > > an
>> > > > access
>> > > > violation exception (0xC0000005) when trying to read from memory
>> > > > location
>> > > > 0x019281e8 on thread 32
>> >
>> > > > Module Information
>> > > > Image Name: C:\WINNT\system32\msvcrt.dll Symbol Type: PDB
>> > > > Base address: 0x77ba0000 Time Stamp: Sat Feb 17 09:02:46 2007
>> > > > Checksum: 0x000635ef Comments:
>> > > > COM DLL: False Company Name: Microsoft Corporation
>> > > > ISAPIExtension: False File Description: Windows NT CRT DLL
>> > > > ISAPIFilter: False File Version: 7.0.3790.3959
>> > > > (srv03_sp2_rtm.070216-1710)
>> > > > Managed DLL: False Internal Name: msvcrt.dll
>> > > > VB DLL: False Legal Copyright: © Microsoft Corporation. All
>> > > > rights
>> > > > reserved.
>> > > > Loaded Image Name: msvcrt.dll Legal Trademarks:
>> > > > Mapped Image Name: Original filename: msvcrt.dll
>> > > > Module name: msvcrt Private Build:
>> > > > Single Threaded: False Product Name: Microsoft® Windows®
>> > > > Operating
>> > > > System
>> > > > Module Size: 360.00 KBytes Product Version: 7.0.3790.3959
>> > > > Symbol File Name:
>> > > > c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pdb
>> > > > Special
>> > > > Build: &
>> >
>> > > > WinDbg output:
>> > > > ==========
>> > > > 0:031> ~32s
>> > > > ^ Illegal thread error in '~32s'
>> >
>> > > > 0:031> kb
>> > > > ChildEBP RetAddr Args to Child
>> > > > 1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
>> > > > *** WARNING: Unable to verify checksum for dmvclient.dll
>> > > > *** ERROR: Symbol file could not be found. Defaulted to export
>> > > > symbols
>> > > > for
>> > > > dmvclient.dll -
>> > > > 1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
>> > > > WARNING: Stack unwind information not available. Following frames
>> > > > may be
>> > > > wrong.
>> > > > 1c3efe44 01948040 1bb77ba0 00000001 00000000
>> > > > dmvclient!dmvGetVarCount3+0x22c
>> > > > 1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
>> > > > 1c3efe4c 00000000 00000000 1c3effb8 00000000
>> > > > dmvclient!dmvGetVarCount3+0x659b0- Hide quoted text -
>> >
>> > - Show quoted text -
>>
>>

Re: Help Analyze Crash Dump

FYI: You cannot rule out memory corruption with your logic that it
happens in staging and production. Getting a good symbols is key to
debugging, period.

Scan the entire disk for msvcrt*.dll because anyone can drop their own
private copy to use. Make sure it's not like some ancient copy of
msvcrt.dll -- what are the dates of the msvcrt*.dll that you find on
your server?


//David
http://w3-4u.blogspot.com
http://blogs.msdn.com/David.Wang
//



On Jan 9, 8:26=A0am, Victor wrote:
> Thanks for the great information!
>
> The strange thing is that I'm setting the symbol path to the microsoft
> download site, I deleted all local symbol caches but the msvcrt =A0still s=
hows
> offset. They have a standard msvcrt.dll on their Win2003 server.
>
> The crashes are happening in staging as well as in production so I'm rulin=
g
> out memory on those servers that way although the file dmvclient.dll conne=
cts
> to a third server where the memory corruption could occur.
>
> Right now I'm waiting on the pdb file for dmvclient to find out more..
>
> Thanks!
>
>
>
> "David Wang" wrote:
> > The module names are not wrong. Just the actual function names, since
> > you are missing symbols, so you just get huge offsets from the last
> > public export.
>
> > While one can speculate on what is going on, you really need the right
> > symbols to know what API call is being made and make sense of the
> > parameters for the best possible diagnosis. And if you want to assign
> > responsibility, you need to have the best diagnosis.
>
> > You need that info to distinguish between memory corruption or just a
> > bug passing bad pointer around. Culprits are often different in those
> > cases, and responsibility can shift.
>
> > In short, I would first concentrate on getting the msvcrt symbols
> > (perhaps these guys copied their own version from Microsoft) to make
> > sense of which CRT function call is made, which then makes sense of
> > the parameters, and one can work backwards to see if it looks like
> > memory corruption, programming error, etc.
>
> > //David
> >http://w3-4u.blogspot.com
> >http://blogs.msdn.com/David.Wang
> > //
>
> > On Jan 8, 1:45 pm, Victor wrote:
> > > Pat,
>
> > > That's great information! It helps me understand better.
> > > I will try to get my hands on the pdb for the symbols.
>
> > > Can you confirm that the crash is indeed happening in the dmvclient.dl=
l (by
> > > bad parameters or bad code, doesn't matter to me at this point) or cou=
ld that
> > > be wrong too?
>
> > > Thanks!!
> > > Victor
>
> > > "Pat [MSFT]" wrote:
> > > > You need symbols - the function names you see are the closest export=
ed
> > > > functions that came before the instruction that failed. =A0The +xxx =
after the
> > > > function name is the instruction offset which is really quite large,=
which
> > > > implies that it is happening well after the function listed. =A0The =
MSVCRT
> > > > function wrong too - it is almost always a string compare, memory mo=
ve, etc.
> > > > that crashes due to bad parameters being passed.
>
> > > > Pat
>
> > > > "Victor" wrote in message
> > > >news:B0A52819-E926-44A3-AF95-CD5C65EAD45E@microsoft.com...
> > > > > I'm running a .NET2.0 web app that is causing the workerprocess w3=
wp.exe
> > > > > to
> > > > > crash several times a day.
>
> > > > > Looks like the external dll file dmvclient.dll that I use in my
> > > > > application
> > > > > is the cause. Unfortunately I don't have access to the .pdb right =
now.
>
> > > > > Please tell me if I'm looking at the dump correctly when I say the=

> > > > > apparent
> > > > > call to the function dmvGetVarCount3 in dmvclient is causing this =
crash.
>
> > > > > I have these lines in my code:
> > > > > [DllImport("dmvclient.dll")]
> > > > > public static extern int dmvGetVarCount3(int handle);
>
> > > > > but there is no actual call to dmvGetVarCount3() in my code ...
>
> > > > > If you can, please give me some pointers to find out more informat=
ion.
> > > > > Thanks!!
> > > > > Victor
>
> > > > > IIS DebugDiag output:
> > > > > ==============
> > > > > Type of Analysis Performed =A0 Crash Analysis
> > > > > Machine Name =A0 XXXXXXX
> > > > > Operating System =A0 Windows Server 2003 Service Pack 2
> > > > > Number Of Processors =A0 4
> > > > > Process ID =A0 904
> > > > > Process Image =A0 c:\WINNT\system32\inetsrv\w3wp.exe
> > > > > System Up-Time =A0 5 day(s) 13:45:31
> > > > > Process Up-Time =A0 0 day(s) 18:57:16
>
> > > > > Thread 32 - System ID 4988
> > > > > Entry point =A0 msvcrt!_endthread+3b
> > > > > Create time =A0 10/25/2007 6:47:10 AM
> > > > > Time spent in user mode =A0 0 Days 0:0:0.0
> > > > > Time spent in kernel mode =A0 0 Days 0:0:0.0
>
> > > > > Function =A0 =A0 Arg 1 =A0 =A0 Arg 2 =A0 =A0 Arg 3 =A0 Source
> > > > > msvcrt!_output+66b =A0 =A0 1bb4fe18 =A0 =A0 1b36b0e4 =A0 =A0 1bb4f=
e48
> > > > > msvcrt!sprintf+31 =A0 =A0 1bb4fe74 =A0 =A0 1b36b0e4 =A0 =A0 019281=
e8
> > > > > dmvclient!dmvGetVarCount3+22c =A0 =A0 0d20312e =A0 =A0 000a0d0a =
=A0 =A0 00000000
> > > > > 3020312e =A0 =A0 0d20312e =A0 =A0 000a0d0a =A0 =A0 00000000
> > > > > 3020312e =A0 =A0 00000000 =A0 =A0 00000000 =A0 =A0 00000000
>
> > > > > MSVCRT!_OUTPUT+66BIn
> > > > > w3wp__PID__904__Date__10_25_2007__Time_06_47_10AM__237__Seco nd_Cha=
nce_Excep­­tion_C0000005.dmp
> > > > > the assembly instruction at msvcrt!_output+66b in
> > > > > C:\WINNT\system32\msvcrt.dll from Microsoft Corporation has caused=
an
> > > > > access
> > > > > violation exception (0xC0000005) when trying to read from memory l=
ocation
> > > > > 0x019281e8 on thread 32
>
> > > > > Module Information
> > > > > Image Name: C:\WINNT\system32\msvcrt.dll =A0 Symbol Type: =A0PDB
> > > > > Base address: 0x77ba0000 =A0 Time Stamp: =A0Sat Feb 17 09:02:46 20=
07
> > > > > Checksum: 0x000635ef =A0 Comments:
> > > > > COM DLL: False =A0 Company Name: =A0Microsoft Corporation
> > > > > ISAPIExtension: False =A0 File Description: =A0Windows NT CRT DLL
> > > > > ISAPIFilter: False =A0 File Version: =A07.0.3790.3959
> > > > > (srv03_sp2_rtm.070216-1710)
> > > > > Managed DLL: False =A0 Internal Name: =A0msvcrt.dll
> > > > > VB DLL: False =A0 Legal Copyright:  © Microsoft Corporation. A=
ll rights
> > > > > reserved.
> > > > > Loaded Image Name: =A0msvcrt.dll =A0 Legal Trademarks:
> > > > > Mapped Image Name: =A0 =A0 Original filename: =A0msvcrt.dll
> > > > > Module name: =A0msvcrt =A0 Private Build:
> > > > > Single Threaded: =A0False =A0 Product Name: =A0Microsoft=AE Window=
s=AE Operating
> > > > > System
> > > > > Module Size: =A0360.00 KBytes =A0 Product Version: =A07.0.3790.395=
9
> > > > > Symbol File Name:
> > > > > c:\symcache\msvcrt.pdb\A7F38CEE7E684B94B7AA9FFFCAB446851\msv crt.pd=
b
> > > > > Special
> > > > > Build: =A0&
>
> > > > > WinDbg output:
> > > > > ==========
> > > > > 0:031> ~32s
> > > > > =A0 =A0 =A0 =A0 ^ Illegal thread error in '~32s'
>
> > > > > 0:031> kb
> > > > > ChildEBP RetAddr =A0Args to Child
> > > > > 1c3efdfc 77bd08b9 1c3efe18 1bb6b0e4 1c3efe48 msvcrt!_output+0x66b
> > > > > *** WARNING: Unable to verify checksum for dmvclient.dll
> > > > > *** ERROR: Symbol file could not be found. =A0Defaulted to export =
symbols
> > > > > for
> > > > > dmvclient.dll -
> > > > > 1c3efe38 1bb1241c 1c3efe74 1bb6b0e4 01948040 msvcrt!sprintf+0x31
> > > > > WARNING: Stack unwind information not available. Following frames =
may be
> > > > > wrong.
> > > > > 1c3efe44 01948040 1bb77ba0 00000001 00000000
> > > > > dmvclient!dmvGetVarCount3+0x22c
> > > > > 1c3efe48 1bb77ba0 00000001 00000000 1c3effb8 0x1948040
> > > > > 1c3efe4c 00000000 00000000 1c3effb8 00000000
> > > > > dmvclient!dmvGetVarCount3+0x659b0- Hide quoted text -
>
> > > - Show quoted text -- Hide quoted text -
>
> - Show quoted text -