Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl +Activestate Pelr 5.8.8 ErrorID 26

Hi I'm trying to debug a persistent error that we've had for a long time, but
has now become acute.

We always "use strict" and do extensive testing, but this problem has eluded
us so far.

Every now and again (4 or 5 times per day) a process would error with a
Windows Error 26 generating a Windows Error Reporting Popup on a terminal
server console. The user data and process looks like it has completed
without any error at the Perl and DB level: it's only the garbage collection
that's showing an error.

In the past this wasn't so bad, as the Apache2 process died and could be
restarted and no harm was done. Now that the code has been moved from a real
to a virtual server (which is allegedly identical but clearly isn't) the
httpd.exe process is hanging awaiting user input to confirm the error popup
window before terminating.

We have searched high and low and tried all sorts of registry hacks.

We've also written more watchdog code to regularly fetch pages from the
server and if they time out to call taskkill to recycle the process, which
although it works, isn't ideal.

Question 1) Does anyone have a reliable way of ignoring this Windows popup
error and letting the httpd.exe process terminate silently so that our
standard process watchdog can restart it?


We have also been looking at the output of a crash dump that we triggered
manually with MS userdump.exe. We already have symbols files for Apache2,
other modiles and mod_perl installed. The error is occurring during a string
copy during the general destructor process of mod_perl after our code has
successfully completed.

Although the Windows error is clearly being triggered by mod_perl, we
suspect that the root cause lies much deeper and is probably something to do
with win32::OLE and not being thread safe.

We can't find any symbols files for perl itself.

Question 2) Can anyone tell us how to reliably dump trace information at the
perl level so we can trace precisely which perl program (and hopefully which
subroutine or method) is triggering the problem?

Again we've searched high and low for a perl58.pdb and perl.pdb symbol table
for the Activestate Perl build but we cannot find one anywhere. Would this
mean we have to recompile our own custom Perl kernel to get this info? If so
that would be a real pain,as then we would have altered something on the
core system.

So can anyone point me at a ready built version of symbol tables
(perl58.pdb) for Activestate Perl Win32?

If you have any other constructive tips for how to dig into this further,
please share.

Thanks
--
View this message in context: http://old.nabble.com/Help-Debugging-Windows-Server-2003-Win 32-%2B-Apache2.2-%2B-mod_perl-%2B-Activestate-Pelr-5.8.8-Err orID-26-tp32004421p32004421.html
Sent from the mod_perl - General mailing list archive at Nabble.com.

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

Q1 : are you running Apache as a Windows Service ?
Q2 : if yes, in the Service properties, is the "allow service to interact with the
desktop" checkbox checked ?
Q3 : still if yes, under what user-id ? is it "Local System", or another local user, or a
domain user ?
Q4 : for such an error, there should be some message in the Windows Event Logs.
What does it say there ?


HappyPerlUser wrote:
> Hi I'm trying to debug a persistent error that we've had for a long time, but
> has now become acute.
>
> We always "use strict" and do extensive testing, but this problem has eluded
> us so far.
>
> Every now and again (4 or 5 times per day) a process would error with a
> Windows Error 26 generating a Windows Error Reporting Popup on a terminal
> server console. The user data and process looks like it has completed
> without any error at the Perl and DB level: it's only the garbage collection
> that's showing an error.
>
> In the past this wasn't so bad, as the Apache2 process died and could be
> restarted and no harm was done. Now that the code has been moved from a real
> to a virtual server (which is allegedly identical but clearly isn't) the
> httpd.exe process is hanging awaiting user input to confirm the error popup
> window before terminating.
>
> We have searched high and low and tried all sorts of registry hacks.
>
> We've also written more watchdog code to regularly fetch pages from the
> server and if they time out to call taskkill to recycle the process, which
> although it works, isn't ideal.
>
> Question 1) Does anyone have a reliable way of ignoring this Windows popup
> error and letting the httpd.exe process terminate silently so that our
> standard process watchdog can restart it?
>
>
> We have also been looking at the output of a crash dump that we triggered
> manually with MS userdump.exe. We already have symbols files for Apache2,
> other modiles and mod_perl installed. The error is occurring during a string
> copy during the general destructor process of mod_perl after our code has
> successfully completed.
>
> Although the Windows error is clearly being triggered by mod_perl, we
> suspect that the root cause lies much deeper and is probably something to do
> with win32::OLE and not being thread safe.
>
> We can't find any symbols files for perl itself.
>
> Question 2) Can anyone tell us how to reliably dump trace information at the
> perl level so we can trace precisely which perl program (and hopefully which
> subroutine or method) is triggering the problem?
>
> Again we've searched high and low for a perl58.pdb and perl.pdb symbol table
> for the Activestate Perl build but we cannot find one anywhere. Would this
> mean we have to recompile our own custom Perl kernel to get this info? If so
> that would be a real pain,as then we would have altered something on the
> core system.
>
> So can anyone point me at a ready built version of symbol tables
> (perl58.pdb) for Activestate Perl Win32?
>
> If you have any other constructive tips for how to dig into this further,
> please share.
>
> Thanks

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

Thanks. Answers below.


awarnier wrote:
>
> Q1 : are you running Apache as a Windows Service ?
> Q2 : if yes, in the Service properties, is the "allow service to interact
> with the
> desktop" checkbox checked ?
> Q3 : still if yes, under what user-id ? is it "Local System", or another
> local user, or a
> domain user ?
> Q4 : for such an error, there should be some message in the Windows Event
> Logs.
> What does it say there ?
>
>
A1. Yes Apache 2.2 running as a service
Apache/2.2.19 (Win32) mod_auth_sspi/1.0.4 mod_perl/2.0.4 Perl/v5.8.8

A2. yes. interactive checkbox is checked

A3. Local System account

A4. Exact text is
The instruction at "0x2808625d" referenced memory at "0x13715268". The
memory could not be "read".
Click on OK to terminate the program

The instruction is usually the same, although sometimes it is "0x2802627f"
The referenced memory changes every time.


http://old.nabble.com/file/p32004847/httpd-error.png

Further stack debug that we have below:

0:999> .reload;!analyze -v;r;kv;lmnt;.logclose
............................................................ .....
.................
Loading unloaded module list
.................
*** WARNING: Unable to verify checksum for perl58.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
perl58.dll -
*** WARNING: Unable to verify checksum for mod_perl.so
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
mod_perl.so -
************************************************************ *******************
*
*
* Exception Analysis
*
*
*
************************************************************ *******************

*** WARNING: Unable to verify checksum for libapr-1.dll
*** WARNING: Unable to verify checksum for libhttpd.dll
*** WARNING: Unable to verify checksum for httpd.exe
*** WARNING: Unable to verify checksum for libaprutil-1.dll
GetPageUrlData failed, server returned HTTP status 404
URL requested:
http://watson.microsoft.com/StageOne/httpd_exe/2_2_19_0/perl 58_dll/5_8_8_822/0008627f.htm?Retriage=1

FAULTING_IP:
perl58!Perl_my_strlcpy+91f
2808627f 83250000000000 and dword ptr ds:[0],0

EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
..exr 0xffffffffffffffff
ExceptionAddress: 2808627f (perl58!Perl_my_strlcpy+0x0000091f)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter[0]: 00000001
Parameter[1]: 00000000
Attempt to write to address 00000000

DEFAULT_BUCKET_ID: NULL_POINTER_WRITE

PROCESS_NAME: httpd.exe

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced
memory at "0x%08lx". The memory could not be "%s".

EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
referenced memory at "0x%08lx". The memory could not be "%s".

EXCEPTION_PARAMETER1: 00000001

EXCEPTION_PARAMETER2: 00000000

WRITE_ADDRESS: 00000000

FOLLOWUP_IP:
perl58!Perl_my_strlcpy+91f
2808627f 83250000000000 and dword ptr ds:[0],0

MOD_LIST:

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

FAULTING_THREAD: 00002814

PRIMARY_PROBLEM_CLASS: NULL_POINTER_WRITE

BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_WRITE

LAST_CONTROL_TRANSFER: from 1000ada9 to 2808627f

STACK_TEXT:
WARNING: Stack unwind information not available. Following frames may be
wrong.
10b5fe68 1000ada9 00d360f4 145ce478 00c46da8 perl58!Perl_my_strlcpy+0x91f
10b5fe80 10001d3b 00d360f4 00d360f4 00d360f4
mod_perl!modperl_perl_destruct+0x5f
10b5fe98 10001dc6 00c46da8 1000268b 00bdeed0
mod_perl!modperl_interp_destroy+0x1f
10b5fea0 1000268b 00bdeed0 00bdeec0 00c46da8
mod_perl!modperl_interp_pool_destroy+0x47
10b5febc 100025a4 00bdeed0 00e8e7e0 00c46da8
mod_perl!modperl_tipool_putback_data+0xfa
10b5ff00 6eec7fdb 01046b18 01046b48 0105d0d0
mod_perl!modperl_tipool_putback_data+0x13
10b5ff24 6ff0aa33 01046b08 00000001 00bdeb2c libapr_1!apr_pool_destroy+0x3b
10b5ff3c 6ff04d61 0105d0d0 0105d0d0 0105d0d0
libhttpd!ap_process_http_connection+0x83
10b5ff54 6ff05023 0105d0d0 008d3550 10b5ff84
libhttpd!ap_run_process_connection+0x21
10b5ff64 6ff1da2c 0105d0d0 00eeb568 00000000
libhttpd!ap_process_connection+0x33
10b5ff84 77bcb530 0105d0c8 00000000 00000000 libhttpd!worker_main+0x9c
10b5ffb8 77e6482f 00e9e5b8 00000000 00000000 msvcrt!_endthreadex+0xa3
10b5ffec 00000000 77bcb4bc 00e9e5b8 00000000 kernel32!BaseThreadStart+0x34


SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: perl58!Perl_my_strlcpy+91f

FOLLOWUP_NAME: MachineOwner

MODULE_NAME: perl58

IMAGE_NAME: perl58.dll

DEBUG_FLR_IMAGE_TIMESTAMP: 46aff16d

STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt
ntdll!LdrpFailureData ; ~999s; .ecxr ; kb

FAILURE_BUCKET_ID: NULL_POINTER_WRITE_c0000005_perl58.dll!Perl_my_strlcpy

BUCKET_ID: APPLICATION_FAULT_NULL_POINTER_WRITE_perl58!Perl_my_strlcpy+ 91f

WATSON_STAGEONE_URL:
http://watson.microsoft.com/StageOne/httpd_exe/2_2_19_0/4dd6 eda8/perl58_dll/5_8_8_822/46aff16d/c0000005/0008627f.htm?Ret riage=1

Followup: MachineOwner
---------

eax=00000000 ebx=12ba03b8 ecx=00d3d7e8 edx=00d3d7e8 esi=12bccff8
edi=00d96de0
eip=2808627f esp=10b5fdb4 ebp=10b5fe68 iopl=0 nv up ei pl nz na pe
nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000
efl=00010206
perl58!Perl_my_strlcpy+0x91f:
2808627f 83250000000000 and dword ptr ds:[0],0
ds:0023:00000000=????????
ChildEBP RetAddr Args to Child
WARNING: Stack unwind information not available. Following frames may be
wrong.
10b5fe68 1000ada9 00d360f4 145ce478 00c46da8 perl58!Perl_my_strlcpy+0x91f
10b5fe80 10001d3b 00d360f4 00d360f4 00d360f4
mod_perl!modperl_perl_destruct+0x5f
10b5fe98 10001dc6 00c46da8 1000268b 00bdeed0
mod_perl!modperl_interp_destroy+0x1f
10b5fea0 1000268b 00bdeed0 00bdeec0 00c46da8
mod_perl!modperl_interp_pool_destroy+0x47
10b5febc 100025a4 00bdeed0 00e8e7e0 00c46da8
mod_perl!modperl_tipool_putback_data+0xfa
10b5ff00 6eec7fdb 01046b18 01046b48 0105d0d0
mod_perl!modperl_tipool_putback_data+0x13
10b5ff24 6ff0aa33 01046b08 00000001 00bdeb2c libapr_1!apr_pool_destroy+0x3b
(CONV: stdcall)
10b5ff3c 6ff04d61 0105d0d0 0105d0d0 0105d0d0
libhttpd!ap_process_http_connection+0x83 (CONV: cdecl)
10b5ff54 6ff05023 0105d0d0 008d3550 10b5ff84
libhttpd!ap_run_process_connection+0x21 (CONV: stdcall)
10b5ff64 6ff1da2c 0105d0d0 00eeb568 00000000
libhttpd!ap_process_connection+0x33 (CONV: stdcall)
10b5ff84 77bcb530 0105d0c8 00000000 00000000 libhttpd!worker_main+0x9c
(CONV: stdcall)
10b5ffb8 77e6482f 00e9e5b8 00000000 00000000 msvcrt!_endthreadex+0xa3 (FPO:
[Non-Fpo])
10b5ffec 00000000 77bcb4bc 00e9e5b8 00000000 kernel32!BaseThreadStart+0x34
(FPO: [Non-Fpo])
start end module name
003c0000 003c7000 mod_auth_sspi mod_auth_sspi.so Wed May 17 20:19:58 2006
(446B694E)
003e0000 003e5000 Cwd Cwd.dll Wed Aug 01 04:35:58 2007
(46AFF18E)
003f0000 003f5000 Const Const.dll Fri Aug 22 03:14:08 2008
(48AE12E0)
00400000 00405000 httpd httpd.exe Sat May 21 00:39:36 2011
(4DD6EDA8)
00ca0000 00ca7000 Connection Connection.dll Fri Aug 22 03:12:18 2008
(48AE1272)
00cb0000 00cb6000 ServerUtil ServerUtil.dll Fri Aug 22 03:13:21 2008
(48AE12B1)
00cc0000 00cc8000 RequestUtil RequestUtil.dll Fri Aug 22 03:13:09 2008
(48AE12A5)
00cd0000 00cd5000 Util Util.dll Fri Aug 22 03:13:47 2008
(48AE12CB)
10000000 1001f000 mod_perl mod_perl.so Fri Aug 22 03:03:26 2008
(48AE105E)
11430000 11436000 Response Response.dll Fri Aug 22 03:13:13 2008
(48AE12A9)
11440000 1144a000 RequestRec RequestRec.dll Fri Aug 22 03:13:04 2008
(48AE12A0)
11450000 11457000 RequestIO RequestIO.dll Fri Aug 22 03:13:00 2008
(48AE129C)
11460000 11466000 Log Log.dll Fri Aug 22 03:12:43 2008
(48AE128B)
11470000 11476000 Access Access.dll Fri Aug 22 03:11:37 2008
(48AE1249)
11480000 11487000 Table Table.dll Fri Aug 22 03:09:48 2008
(48AE11DC)
11490000 11495000 Status Status.dll Fri Aug 22 03:09:23 2008
(48AE11C3)
114a0000 114a5000 Global Global.dll Fri Aug 22 03:13:43 2008
(48AE12C7)
114b0000 114c7000 DBI DBI.dll Wed Mar 05 06:55:04 2008
(47CE35B8)
114d0000 114d7000 Util_114d0000 Util.dll Wed Aug 01 04:38:45 2007
(46AFF235)
114e0000 114e6000 Pool Pool.dll Fri Aug 22 03:08:39 2008
(48AE1197)
114f0000 11504000 Calc Calc.dll Tue Mar 14 01:16:51 2006
(44160B73)
11510000 11516000 HiRes HiRes.dll Tue Jan 06 00:08:34 2009
(496292F2)
11520000 11526000 Fcntl Fcntl.dll Wed Aug 01 04:38:15 2007
(46AFF217)
11530000 11543000 POSIX POSIX.dll Wed Aug 01 04:39:04 2007
(46AFF248)
11550000 1155a000 Win32 Win32.dll Wed Aug 01 04:40:19 2007
(46AFF293)
11560000 11566000 Service Service.dll Wed Aug 01 05:07:46 2007
(46AFF902)
11570000 11575000 perlchartdir58mt perlchartdir58mt.dll Mon Jan 14
16:52:00 2008 (478B8520)
11580000 117b5000 chartdir chartdir.dll Sat Mar 28 16:59:31 2009
(49CE4963)
118d0000 118d5000 Base64 Base64.dll Tue Jul 01 02:46:41 2008
(48697E71)
118e0000 118e6000 MD5 MD5.dll Wed Aug 01 04:36:30 2007
(46AFF1AE)
118f0000 118f7000 Socket Socket.dll Wed Aug 01 04:39:35 2007
(46AFF267)
11900000 11905000 Hostname Hostname.dll Wed Aug 01 04:39:47 2007
(46AFF273)
11910000 11916000 IO IO.dll Wed Aug 01 04:38:39 2007
(46AFF22F)
11920000 11946000 Registry Registry.dll Wed Aug 01 05:06:09 2007
(46AFF8A1)
12950000 12964000 WinError WinError.dll Wed Aug 01 05:08:04 2007
(46AFF914)
12970000 12ace000 mysql mysql.dll Thu Apr 19 05:37:28 2007
(4626E3F8)
12ad0000 12ad5000 Util_12ad0000 Util.dll Fri Aug 22 03:11:16 2008
(48AE1234)
12ae0000 12ae5000 Error Error.dll Fri Aug 22 03:07:33 2008
(48AE1155)
28000000 280c5000 perl58 perl58.dll Wed Aug 01 04:35:25 2007
(46AFF16D)
5a4c0000 5a4d3000 zlib1 zlib1.dll Mon Mar 15 02:51:58 2010
(4B9D92BE)
5f270000 5f2ca000 hnetcfg hnetcfg.dll Sat Feb 17 14:59:38 2007
(45D70A4A)
68000000 68035000 rsaenh rsaenh.dll Sat Feb 17 06:49:58 2007
(45D69786)
6ee50000 6ee59000 libapriconv_1 libapriconv-1.dll Sat May 21 00:38:31 2011
(4DD6ED67)
6ee60000 6ee90000 libaprutil_1 libaprutil-1.dll Sat May 21 00:39:22 2011
(4DD6ED9A)
6eec0000 6eee3000 libapr_1 libapr-1.dll Sat May 21 00:38:29 2011
(4DD6ED65)
6fb00000 6fb06000 mod_authz_host mod_authz_host.so Sat May 21 00:39:43
2011 (4DD6EDAF)
6fba0000 6fba7000 mod_deflate mod_deflate.so Sat May 21 00:39:52 2011
(4DD6EDB8)
6fc00000 6fc06000 mod_setenvif mod_setenvif.so Sat May 21 00:40:06 2011
(4DD6EDC6)
6fc20000 6fc27000 mod_mime mod_mime.so Sat May 21 00:39:57 2011
(4DD6EDBD)
6fc30000 6fc37000 mod_log_config mod_log_config.so Sat May 21 00:39:58
2011 (4DD6EDBE)
6fc60000 6fc67000 mod_imagemap mod_imagemap.so Sat May 21 00:40:01 2011
(4DD6EDC1)
6fc80000 6fc86000 mod_dir mod_dir.so Sat May 21 00:40:00 2011
(4DD6EDC0)
6fc90000 6fc97000 mod_cgi mod_cgi.so Sat May 21 00:39:56 2011
(4DD6EDBC)
6fcd0000 6fcd6000 mod_alias mod_alias.so Sat May 21 00:39:59 2011
(4DD6EDBF)
6ff00000 6ff46000 libhttpd libhttpd.dll Sat May 21 00:39:36 2011
(4DD6EDA8)
71ae0000 71ae8000 wshtcpip wshtcpip.dll Sat Feb 17 15:02:27 2007
(45D70AF3)
71b20000 71b61000 mswsock mswsock.dll Fri Jun 20 20:32:38 2008
(485BF7C6)
71bb0000 71bb9000 wsock32 wsock32.dll Tue Mar 25 10:43:26 2003
(3E8024BE)
71bf0000 71bf8000 ws2help ws2help.dll Sat Feb 17 15:02:18 2007
(45D70AEA)
71c00000 71c17000 ws2_32 ws2_32.dll Sat Feb 17 15:02:17 2007
(45D70AE9)
71f60000 71f64000 security security.dll Tue Mar 25 10:43:24 2003
(3E8024BC)
76290000 762ad000 imm32 imm32.dll Sat Feb 17 14:59:59 2007
(45D70A5F)
76b70000 76b7b000 psapi psapi.dll Sat Feb 17 15:01:25 2007
(45D70AB5)
76ed0000 76efa000 dnsapi dnsapi.dll Thu Mar 03 08:06:05 2011
(4D6F3DDD)
76f10000 76f3e000 wldap32 wldap32.dll Sat Feb 17 15:01:57 2007
(45D70AD5)
76f50000 76f63000 secur32 secur32.dll Wed Feb 04 12:41:02 2009
(49897ECE)
76f70000 76f77000 winrnr winrnr.dll Sat Feb 17 15:01:43 2007
(45D70AC7)
76f80000 76f85000 rasadhlp rasadhlp.dll Sat Feb 17 15:01:01 2007
(45D70A9D)
77380000 77411000 user32 user32.dll Fri Mar 02 07:38:46 2007
(45E7C676)
77420000 77523000 comctl32 comctl32.dll Tue Sep 07 14:08:31 2010
(4C862B3F)
77670000 777a9000 ole32 ole32.dll Mon Jul 26 12:01:18 2010
(4C4D5CEE)
77b90000 77b98000 version version.dll Sat Feb 17 15:01:44 2007
(45D70AC8)
77ba0000 77bfa000 msvcrt msvcrt.dll Sat Feb 17 15:02:46 2007
(45D70B06)
77c00000 77c49000 gdi32 gdi32.dll Thu Oct 23 13:43:54 2008
(4900637A)
77c50000 77cf0000 rpcrt4 rpcrt4.dll Wed Aug 18 11:27:25 2010
(4C6BA77D)
77e40000 77f42000 kernel32 kernel32.dll Sat Mar 21 18:08:26 2009
(49C51F0A)
7c800000 7c8c3000 ntdll ntdll.dll Fri Oct 22 14:27:14 2010
(4CC18322)
7c8d0000 7d0cf000 shell32 shell32.dll Fri Jan 21 22:59:50 2011
(4D3A01D6)
7d180000 7d1d2000 shlwapi shlwapi.dll Thu Oct 15 22:19:54 2009
(4AD783EA)
7d1e0000 7d27c000 advapi32 advapi32.dll Mon Feb 09 12:02:55 2009
(49900D5F)

Unloaded modules:
5a4c0000 5a4d3000 zlib1.dll
Timestamp: Mon Mar 15 02:51:58 2010 (4B9D92BE)
Checksum: 00000000
ImageSize: 00013000
6fba0000 6fba7000 mod_deflate.so
Timestamp: Sat May 21 00:39:52 2011 (4DD6EDB8)
Checksum: 00000000
ImageSize: 00007000
6fcd0000 6fcd6000 mod_alias.so
Timestamp: Sat May 21 00:39:59 2011 (4DD6EDBF)
Checksum: 00000000
ImageSize: 00006000
6fb00000 6fb06000 mod_authz_host.so
Timestamp: Sat May 21 00:39:43 2011 (4DD6EDAF)
Checksum: 00000000
ImageSize: 00006000
6fc90000 6fc97000 mod_cgi.so
Timestamp: Sat May 21 00:39:56 2011 (4DD6EDBC)
Checksum: 00000000
ImageSize: 00007000
6fc80000 6fc86000 mod_dir.so
Timestamp: Sat May 21 00:40:00 2011 (4DD6EDC0)
Checksum: 00000000
ImageSize: 00006000
6fc60000 6fc67000 mod_imagemap.so
Timestamp: Sat May 21 00:40:01 2011 (4DD6EDC1)
Checksum: 00000000
ImageSize: 00007000
6fc30000 6fc37000 mod_log_config.so
Timestamp: Sat May 21 00:39:58 2011 (4DD6EDBE)
Checksum: 00000000
ImageSize: 00007000
6fc20000 6fc27000 mod_mime.so
Timestamp: Sat May 21 00:39:57 2011 (4DD6EDBD)
Checksum: 00000000
ImageSize: 00007000
6fc00000 6fc06000 mod_setenvif.so
Timestamp: Sat May 21 00:40:06 2011 (4DD6EDC6)
Checksum: 00000000
ImageSize: 00006000
28000000 280c5000 perl58.dll
Timestamp: Wed Aug 01 04:35:25 2007 (46AFF16D)
Checksum: 00000000
ImageSize: 000C5000
10000000 1001f000 mod_perl.so
Timestamp: Fri Aug 22 03:03:26 2008 (48AE105E)
Checksum: 00000000
ImageSize: 0001F000
77670000 777a9000 ole32.dll
Timestamp: Mon Jul 26 12:01:18 2010 (4C4D5CEE)
Checksum: 0013CFD9
ImageSize: 00139000
003c0000 003c7000 mod_auth_sspi.so
Timestamp: Wed May 17 20:19:58 2006 (446B694E)
Checksum: 00009E3B
ImageSize: 00007000
71f60000 71f64000 SECURITY.DLL
Timestamp: Tue Mar 25 10:43:24 2003 (3E8024BC)
Checksum: 0000FD45
ImageSize: 00004000
00cd0000 00cd5000 Util.dll
Timestamp: Fri Aug 22 03:13:47 2008 (48AE12CB)
Checksum: 00000000
ImageSize: 00005000
Closing open log file c:\debuglog.txt
--
View this message in context: http://old.nabble.com/Help-Debugging-Windows-Server-2003-Win 32-%2B-Apache2.2-%2B-mod_perl-%2B-Activestate-Pelr-5.8.8-Err orID-26-tp32004421p32004847.html
Sent from the mod_perl - General mailing list archive at Nabble.com.

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

HappyPerlUser wrote:
> Thanks. Answers below.
>
>
> awarnier wrote:
>> Q1 : are you running Apache as a Windows Service ?
>> Q2 : if yes, in the Service properties, is the "allow service to interact
>> with the
>> desktop" checkbox checked ?
>> Q3 : still if yes, under what user-id ? is it "Local System", or another
>> local user, or a
>> domain user ?
>> Q4 : for such an error, there should be some message in the Windows Event
>> Logs.
>> What does it say there ?
>>
>>
> A1. Yes Apache 2.2 running as a service
> Apache/2.2.19 (Win32) mod_auth_sspi/1.0.4 mod_perl/2.0.4 Perl/v5.8.8
>
> A2. yes. interactive checkbox is checked

That is why you get this error box.
Have you tried unchecking that option ?
(I'm not saying that it will solve the underlying problem, but it may remove the annoying
symptom).

On a separate note : since you are running this on virtual machines, you may want to try
running Apache in a command window, from the command-line. There may be error messages
appearing then (the ones written to STDERR by perl e.g.), which otherwise you would not
get to see.
Since it is a virtual machine, you can leave the command window open, and do not need to
log off. You can just connect and disconnect from the virtual console as needed.
Be aware that if you do that, you will be running Apache under the user-id under which you
are currently logged-in on the server, so watch file permissions etc..


>
> A3. Local System account

That's interesting, considering you are logged-in as LocalSystem, which is not a domain
account. mod_auth_sspi is to do Windows domain authentication, isn't it ?
I am curious to know how that works when the server is not running as a domain account.
(But this has probably nothing to do with the issue at hand).

>
> A4. Exact text is
> The instruction at "0x2808625d" referenced memory at "0x13715268". The
> memory could not be "read".
> Click on OK to terminate the program
>
> The instruction is usually the same, although sometimes it is "0x2802627f"
> The referenced memory changes every time.
>
>
> http://old.nabble.com/file/p32004847/httpd-error.png
>
> Further stack debug that we have below:
>
> 0:999> .reload;!analyze -v;r;kv;lmnt;.logclose
> ............................................................ ....
> ................
> Loading unloaded module list
> ................
> *** WARNING: Unable to verify checksum for perl58.dll
> *** ERROR: Symbol file could not be found. Defaulted to export symbols for
> perl58.dll -
> *** WARNING: Unable to verify checksum for mod_perl.so
> *** ERROR: Symbol file could not be found. Defaulted to export symbols for
> mod_perl.so -
> ************************************************************ *******************
> *
> *
> * Exception Analysis
> *
> *
> *
> ************************************************************ *******************
>
> *** WARNING: Unable to verify checksum for libapr-1.dll
> *** WARNING: Unable to verify checksum for libhttpd.dll
> *** WARNING: Unable to verify checksum for httpd.exe
> *** WARNING: Unable to verify checksum for libaprutil-1.dll
> GetPageUrlData failed, server returned HTTP status 404
> URL requested:
> http://watson.microsoft.com/StageOne/httpd_exe/2_2_19_0/perl 58_dll/5_8_8_822/0008627f.htm?Retriage=1
>
> FAULTING_IP:
> perl58!Perl_my_strlcpy+91f
> 2808627f 83250000000000 and dword ptr ds:[0],0
>
> EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff)
> .exr 0xffffffffffffffff
> ExceptionAddress: 2808627f (perl58!Perl_my_strlcpy+0x0000091f)
> ExceptionCode: c0000005 (Access violation)
> ExceptionFlags: 00000000
> NumberParameters: 2
> Parameter[0]: 00000001
> Parameter[1]: 00000000
> Attempt to write to address 00000000
>
> DEFAULT_BUCKET_ID: NULL_POINTER_WRITE
>
> PROCESS_NAME: httpd.exe
>
> ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx" referenced
> memory at "0x%08lx". The memory could not be "%s".
>
> EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at "0x%08lx"
> referenced memory at "0x%08lx". The memory could not be "%s".
>
> EXCEPTION_PARAMETER1: 00000001
>
> EXCEPTION_PARAMETER2: 00000000
>
> WRITE_ADDRESS: 00000000
>
> FOLLOWUP_IP:
> perl58!Perl_my_strlcpy+91f
> 2808627f 83250000000000 and dword ptr ds:[0],0
>
> MOD_LIST:
>
> NTGLOBALFLAG: 0
>
> APPLICATION_VERIFIER_FLAGS: 0
>
> FAULTING_THREAD: 00002814
>
> PRIMARY_PROBLEM_CLASS: NULL_POINTER_WRITE
>
> BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_WRITE
>
> LAST_CONTROL_TRANSFER: from 1000ada9 to 2808627f
>
> STACK_TEXT:
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> 10b5fe68 1000ada9 00d360f4 145ce478 00c46da8 perl58!Perl_my_strlcpy+0x91f
> 10b5fe80 10001d3b 00d360f4 00d360f4 00d360f4
> mod_perl!modperl_perl_destruct+0x5f
> 10b5fe98 10001dc6 00c46da8 1000268b 00bdeed0
> mod_perl!modperl_interp_destroy+0x1f
> 10b5fea0 1000268b 00bdeed0 00bdeec0 00c46da8
> mod_perl!modperl_interp_pool_destroy+0x47
> 10b5febc 100025a4 00bdeed0 00e8e7e0 00c46da8
> mod_perl!modperl_tipool_putback_data+0xfa
> 10b5ff00 6eec7fdb 01046b18 01046b48 0105d0d0
> mod_perl!modperl_tipool_putback_data+0x13
> 10b5ff24 6ff0aa33 01046b08 00000001 00bdeb2c libapr_1!apr_pool_destroy+0x3b
> 10b5ff3c 6ff04d61 0105d0d0 0105d0d0 0105d0d0
> libhttpd!ap_process_http_connection+0x83
> 10b5ff54 6ff05023 0105d0d0 008d3550 10b5ff84
> libhttpd!ap_run_process_connection+0x21
> 10b5ff64 6ff1da2c 0105d0d0 00eeb568 00000000
> libhttpd!ap_process_connection+0x33
> 10b5ff84 77bcb530 0105d0c8 00000000 00000000 libhttpd!worker_main+0x9c
> 10b5ffb8 77e6482f 00e9e5b8 00000000 00000000 msvcrt!_endthreadex+0xa3
> 10b5ffec 00000000 77bcb4bc 00e9e5b8 00000000 kernel32!BaseThreadStart+0x34
>
>
> SYMBOL_STACK_INDEX: 0
>
> SYMBOL_NAME: perl58!Perl_my_strlcpy+91f
>
> FOLLOWUP_NAME: MachineOwner
>
> MODULE_NAME: perl58
>
> IMAGE_NAME: perl58.dll
>
> DEBUG_FLR_IMAGE_TIMESTAMP: 46aff16d
>
> STACK_COMMAND: dt ntdll!LdrpLastDllInitializer BaseDllName ; dt
> ntdll!LdrpFailureData ; ~999s; .ecxr ; kb
>
> FAILURE_BUCKET_ID: NULL_POINTER_WRITE_c0000005_perl58.dll!Perl_my_strlcpy
>
> BUCKET_ID: APPLICATION_FAULT_NULL_POINTER_WRITE_perl58!Perl_my_strlcpy+ 91f
>
> WATSON_STAGEONE_URL:
> http://watson.microsoft.com/StageOne/httpd_exe/2_2_19_0/4dd6 eda8/perl58_dll/5_8_8_822/46aff16d/c0000005/0008627f.htm?Ret riage=1
>
> Followup: MachineOwner
> ---------
>
> eax=00000000 ebx=12ba03b8 ecx=00d3d7e8 edx=00d3d7e8 esi=12bccff8
> edi=00d96de0
> eip=2808627f esp=10b5fdb4 ebp=10b5fe68 iopl=0 nv up ei pl nz na pe
> nc
> cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000
> efl=00010206
> perl58!Perl_my_strlcpy+0x91f:
> 2808627f 83250000000000 and dword ptr ds:[0],0
> ds:0023:00000000=????????
> ChildEBP RetAddr Args to Child
> WARNING: Stack unwind information not available. Following frames may be
> wrong.
> 10b5fe68 1000ada9 00d360f4 145ce478 00c46da8 perl58!Perl_my_strlcpy+0x91f
> 10b5fe80 10001d3b 00d360f4 00d360f4 00d360f4
> mod_perl!modperl_perl_destruct+0x5f
> 10b5fe98 10001dc6 00c46da8 1000268b 00bdeed0
> mod_perl!modperl_interp_destroy+0x1f
> 10b5fea0 1000268b 00bdeed0 00bdeec0 00c46da8
> mod_perl!modperl_interp_pool_destroy+0x47
> 10b5febc 100025a4 00bdeed0 00e8e7e0 00c46da8
> mod_perl!modperl_tipool_putback_data+0xfa
> 10b5ff00 6eec7fdb 01046b18 01046b48 0105d0d0
> mod_perl!modperl_tipool_putback_data+0x13
> 10b5ff24 6ff0aa33 01046b08 00000001 00bdeb2c libapr_1!apr_pool_destroy+0x3b
> (CONV: stdcall)
> 10b5ff3c 6ff04d61 0105d0d0 0105d0d0 0105d0d0
> libhttpd!ap_process_http_connection+0x83 (CONV: cdecl)
> 10b5ff54 6ff05023 0105d0d0 008d3550 10b5ff84
> libhttpd!ap_run_process_connection+0x21 (CONV: stdcall)
> 10b5ff64 6ff1da2c 0105d0d0 00eeb568 00000000
> libhttpd!ap_process_connection+0x33 (CONV: stdcall)
> 10b5ff84 77bcb530 0105d0c8 00000000 00000000 libhttpd!worker_main+0x9c
> (CONV: stdcall)
> 10b5ffb8 77e6482f 00e9e5b8 00000000 00000000 msvcrt!_endthreadex+0xa3 (FPO:
> [Non-Fpo])
> 10b5ffec 00000000 77bcb4bc 00e9e5b8 00000000 kernel32!BaseThreadStart+0x34
> (FPO: [Non-Fpo])
> start end module name
> 003c0000 003c7000 mod_auth_sspi mod_auth_sspi.so Wed May 17 20:19:58 2006
> (446B694E)
> 003e0000 003e5000 Cwd Cwd.dll Wed Aug 01 04:35:58 2007
> (46AFF18E)
> 003f0000 003f5000 Const Const.dll Fri Aug 22 03:14:08 2008
> (48AE12E0)
> 00400000 00405000 httpd httpd.exe Sat May 21 00:39:36 2011
> (4DD6EDA8)
> 00ca0000 00ca7000 Connection Connection.dll Fri Aug 22 03:12:18 2008
> (48AE1272)
> 00cb0000 00cb6000 ServerUtil ServerUtil.dll Fri Aug 22 03:13:21 2008
> (48AE12B1)
> 00cc0000 00cc8000 RequestUtil RequestUtil.dll Fri Aug 22 03:13:09 2008
> (48AE12A5)
> 00cd0000 00cd5000 Util Util.dll Fri Aug 22 03:13:47 2008
> (48AE12CB)
> 10000000 1001f000 mod_perl mod_perl.so Fri Aug 22 03:03:26 2008
> (48AE105E)
> 11430000 11436000 Response Response.dll Fri Aug 22 03:13:13 2008
> (48AE12A9)
> 11440000 1144a000 RequestRec RequestRec.dll Fri Aug 22 03:13:04 2008
> (48AE12A0)
> 11450000 11457000 RequestIO RequestIO.dll Fri Aug 22 03:13:00 2008
> (48AE129C)
> 11460000 11466000 Log Log.dll Fri Aug 22 03:12:43 2008
> (48AE128B)
> 11470000 11476000 Access Access.dll Fri Aug 22 03:11:37 2008
> (48AE1249)
> 11480000 11487000 Table Table.dll Fri Aug 22 03:09:48 2008
> (48AE11DC)
> 11490000 11495000 Status Status.dll Fri Aug 22 03:09:23 2008
> (48AE11C3)
> 114a0000 114a5000 Global Global.dll Fri Aug 22 03:13:43 2008
> (48AE12C7)
> 114b0000 114c7000 DBI DBI.dll Wed Mar 05 06:55:04 2008
> (47CE35B8)
> 114d0000 114d7000 Util_114d0000 Util.dll Wed Aug 01 04:38:45 2007
> (46AFF235)
> 114e0000 114e6000 Pool Pool.dll Fri Aug 22 03:08:39 2008
> (48AE1197)
> 114f0000 11504000 Calc Calc.dll Tue Mar 14 01:16:51 2006
> (44160B73)
> 11510000 11516000 HiRes HiRes.dll Tue Jan 06 00:08:34 2009
> (496292F2)
> 11520000 11526000 Fcntl Fcntl.dll Wed Aug 01 04:38:15 2007
> (46AFF217)
> 11530000 11543000 POSIX POSIX.dll Wed Aug 01 04:39:04 2007
> (46AFF248)
> 11550000 1155a000 Win32 Win32.dll Wed Aug 01 04:40:19 2007
> (46AFF293)
> 11560000 11566000 Service Service.dll Wed Aug 01 05:07:46 2007
> (46AFF902)
> 11570000 11575000 perlchartdir58mt perlchartdir58mt.dll Mon Jan 14
> 16:52:00 2008 (478B8520)
> 11580000 117b5000 chartdir chartdir.dll Sat Mar 28 16:59:31 2009
> (49CE4963)
> 118d0000 118d5000 Base64 Base64.dll Tue Jul 01 02:46:41 2008
> (48697E71)
> 118e0000 118e6000 MD5 MD5.dll Wed Aug 01 04:36:30 2007
> (46AFF1AE)
> 118f0000 118f7000 Socket Socket.dll Wed Aug 01 04:39:35 2007
> (46AFF267)
> 11900000 11905000 Hostname Hostname.dll Wed Aug 01 04:39:47 2007
> (46AFF273)
> 11910000 11916000 IO IO.dll Wed Aug 01 04:38:39 2007
> (46AFF22F)
> 11920000 11946000 Registry Registry.dll Wed Aug 01 05:06:09 2007
> (46AFF8A1)
> 12950000 12964000 WinError WinError.dll Wed Aug 01 05:08:04 2007
> (46AFF914)
> 12970000 12ace000 mysql mysql.dll Thu Apr 19 05:37:28 2007
> (4626E3F8)
> 12ad0000 12ad5000 Util_12ad0000 Util.dll Fri Aug 22 03:11:16 2008
> (48AE1234)
> 12ae0000 12ae5000 Error Error.dll Fri Aug 22 03:07:33 2008
> (48AE1155)
> 28000000 280c5000 perl58 perl58.dll Wed Aug 01 04:35:25 2007
> (46AFF16D)
> 5a4c0000 5a4d3000 zlib1 zlib1.dll Mon Mar 15 02:51:58 2010
> (4B9D92BE)
> 5f270000 5f2ca000 hnetcfg hnetcfg.dll Sat Feb 17 14:59:38 2007
> (45D70A4A)
> 68000000 68035000 rsaenh rsaenh.dll Sat Feb 17 06:49:58 2007
> (45D69786)
> 6ee50000 6ee59000 libapriconv_1 libapriconv-1.dll Sat May 21 00:38:31 2011
> (4DD6ED67)
> 6ee60000 6ee90000 libaprutil_1 libaprutil-1.dll Sat May 21 00:39:22 2011
> (4DD6ED9A)
> 6eec0000 6eee3000 libapr_1 libapr-1.dll Sat May 21 00:38:29 2011
> (4DD6ED65)
> 6fb00000 6fb06000 mod_authz_host mod_authz_host.so Sat May 21 00:39:43
> 2011 (4DD6EDAF)
> 6fba0000 6fba7000 mod_deflate mod_deflate.so Sat May 21 00:39:52 2011
> (4DD6EDB8)
> 6fc00000 6fc06000 mod_setenvif mod_setenvif.so Sat May 21 00:40:06 2011
> (4DD6EDC6)
> 6fc20000 6fc27000 mod_mime mod_mime.so Sat May 21 00:39:57 2011
> (4DD6EDBD)
> 6fc30000 6fc37000 mod_log_config mod_log_config.so Sat May 21 00:39:58
> 2011 (4DD6EDBE)
> 6fc60000 6fc67000 mod_imagemap mod_imagemap.so Sat May 21 00:40:01 2011
> (4DD6EDC1)
> 6fc80000 6fc86000 mod_dir mod_dir.so Sat May 21 00:40:00 2011
> (4DD6EDC0)
> 6fc90000 6fc97000 mod_cgi mod_cgi.so Sat May 21 00:39:56 2011
> (4DD6EDBC)
> 6fcd0000 6fcd6000 mod_alias mod_alias.so Sat May 21 00:39:59 2011
> (4DD6EDBF)
> 6ff00000 6ff46000 libhttpd libhttpd.dll Sat May 21 00:39:36 2011
> (4DD6EDA8)
> 71ae0000 71ae8000 wshtcpip wshtcpip.dll Sat Feb 17 15:02:27 2007
> (45D70AF3)
> 71b20000 71b61000 mswsock mswsock.dll Fri Jun 20 20:32:38 2008
> (485BF7C6)
> 71bb0000 71bb9000 wsock32 wsock32.dll Tue Mar 25 10:43:26 2003
> (3E8024BE)
> 71bf0000 71bf8000 ws2help ws2help.dll Sat Feb 17 15:02:18 2007
> (45D70AEA)
> 71c00000 71c17000 ws2_32 ws2_32.dll Sat Feb 17 15:02:17 2007
> (45D70AE9)
> 71f60000 71f64000 security security.dll Tue Mar 25 10:43:24 2003
> (3E8024BC)
> 76290000 762ad000 imm32 imm32.dll Sat Feb 17 14:59:59 2007
> (45D70A5F)
> 76b70000 76b7b000 psapi psapi.dll Sat Feb 17 15:01:25 2007
> (45D70AB5)
> 76ed0000 76efa000 dnsapi dnsapi.dll Thu Mar 03 08:06:05 2011
> (4D6F3DDD)
> 76f10000 76f3e000 wldap32 wldap32.dll Sat Feb 17 15:01:57 2007
> (45D70AD5)
> 76f50000 76f63000 secur32 secur32.dll Wed Feb 04 12:41:02 2009
> (49897ECE)
> 76f70000 76f77000 winrnr winrnr.dll Sat Feb 17 15:01:43 2007
> (45D70AC7)
> 76f80000 76f85000 rasadhlp rasadhlp.dll Sat Feb 17 15:01:01 2007
> (45D70A9D)
> 77380000 77411000 user32 user32.dll Fri Mar 02 07:38:46 2007
> (45E7C676)
> 77420000 77523000 comctl32 comctl32.dll Tue Sep 07 14:08:31 2010
> (4C862B3F)
> 77670000 777a9000 ole32 ole32.dll Mon Jul 26 12:01:18 2010
> (4C4D5CEE)
> 77b90000 77b98000 version version.dll Sat Feb 17 15:01:44 2007
> (45D70AC8)
> 77ba0000 77bfa000 msvcrt msvcrt.dll Sat Feb 17 15:02:46 2007
> (45D70B06)
> 77c00000 77c49000 gdi32 gdi32.dll Thu Oct 23 13:43:54 2008
> (4900637A)
> 77c50000 77cf0000 rpcrt4 rpcrt4.dll Wed Aug 18 11:27:25 2010
> (4C6BA77D)
> 77e40000 77f42000 kernel32 kernel32.dll Sat Mar 21 18:08:26 2009
> (49C51F0A)
> 7c800000 7c8c3000 ntdll ntdll.dll Fri Oct 22 14:27:14 2010
> (4CC18322)
> 7c8d0000 7d0cf000 shell32 shell32.dll Fri Jan 21 22:59:50 2011
> (4D3A01D6)
> 7d180000 7d1d2000 shlwapi shlwapi.dll Thu Oct 15 22:19:54 2009
> (4AD783EA)
> 7d1e0000 7d27c000 advapi32 advapi32.dll Mon Feb 09 12:02:55 2009
> (49900D5F)
>
> Unloaded modules:
> 5a4c0000 5a4d3000 zlib1.dll
> Timestamp: Mon Mar 15 02:51:58 2010 (4B9D92BE)
> Checksum: 00000000
> ImageSize: 00013000
> 6fba0000 6fba7000 mod_deflate.so
> Timestamp: Sat May 21 00:39:52 2011 (4DD6EDB8)
> Checksum: 00000000
> ImageSize: 00007000
> 6fcd0000 6fcd6000 mod_alias.so
> Timestamp: Sat May 21 00:39:59 2011 (4DD6EDBF)
> Checksum: 00000000
> ImageSize: 00006000
> 6fb00000 6fb06000 mod_authz_host.so
> Timestamp: Sat May 21 00:39:43 2011 (4DD6EDAF)
> Checksum: 00000000
> ImageSize: 00006000
> 6fc90000 6fc97000 mod_cgi.so
> Timestamp: Sat May 21 00:39:56 2011 (4DD6EDBC)
> Checksum: 00000000
> ImageSize: 00007000
> 6fc80000 6fc86000 mod_dir.so
> Timestamp: Sat May 21 00:40:00 2011 (4DD6EDC0)
> Checksum: 00000000
> ImageSize: 00006000
> 6fc60000 6fc67000 mod_imagemap.so
> Timestamp: Sat May 21 00:40:01 2011 (4DD6EDC1)
> Checksum: 00000000
> ImageSize: 00007000
> 6fc30000 6fc37000 mod_log_config.so
> Timestamp: Sat May 21 00:39:58 2011 (4DD6EDBE)
> Checksum: 00000000
> ImageSize: 00007000
> 6fc20000 6fc27000 mod_mime.so
> Timestamp: Sat May 21 00:39:57 2011 (4DD6EDBD)
> Checksum: 00000000
> ImageSize: 00007000
> 6fc00000 6fc06000 mod_setenvif.so
> Timestamp: Sat May 21 00:40:06 2011 (4DD6EDC6)
> Checksum: 00000000
> ImageSize: 00006000
> 28000000 280c5000 perl58.dll
> Timestamp: Wed Aug 01 04:35:25 2007 (46AFF16D)
> Checksum: 00000000
> ImageSize: 000C5000
> 10000000 1001f000 mod_perl.so
> Timestamp: Fri Aug 22 03:03:26 2008 (48AE105E)
> Checksum: 00000000
> ImageSize: 0001F000
> 77670000 777a9000 ole32.dll
> Timestamp: Mon Jul 26 12:01:18 2010 (4C4D5CEE)
> Checksum: 0013CFD9
> ImageSize: 00139000
> 003c0000 003c7000 mod_auth_sspi.so
> Timestamp: Wed May 17 20:19:58 2006 (446B694E)
> Checksum: 00009E3B
> ImageSize: 00007000
> 71f60000 71f64000 SECURITY.DLL
> Timestamp: Tue Mar 25 10:43:24 2003 (3E8024BC)
> Checksum: 0000FD45
> ImageSize: 00004000
> 00cd0000 00cd5000 Util.dll
> Timestamp: Fri Aug 22 03:13:47 2008 (48AE12CB)
> Checksum: 00000000
> ImageSize: 00005000
> Closing open log file c:\debuglog.txt

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

awarnier wrote:
>
>
>>
>> A2. yes. interactive checkbox is checked
>
> That is why you get this error box.
> Have you tried unchecking that option ?
> (I'm not saying that it will solve the underlying problem, but it may
> remove the annoying
> symptom).
>
> On a separate note : since you are running this on virtual machines, you
> may want to try
> running Apache in a command window, from the command-line.
>
>>
>> A3. Local System account
>
> That's interesting, considering you are logged-in as LocalSystem, which is
> not a domain
> account. mod_auth_sspi is to do Windows domain authentication, isn't it ?
>
Have now unchecked the enable interact with desktop checkbox, although I
thought we needed that for some automation scripts in the past (that ran
external windows programs on the server).

May also try running Apache from the command line, but then again as you say
it is then a different user, and quite different from running as a service.

This server is not part of a domain. Maybe we just need to remove the
mod_auth_sspi module {probably a hang over from another system}.
--
View this message in context: http://old.nabble.com/Help-Debugging-Windows-Server-2003-Win 32-%2B-Apache2.2-%2B-mod_perl-%2B-Activestate-Pelr-5.8.8-Err orID-26-tp32004421p32006592.html
Sent from the mod_perl - General mailing list archive at Nabble.com.

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

HappyPerlUser wrote:
>
> Have now unchecked the enable interact with desktop checkbox, although I
> thought we needed that for some automation scripts in the past (that ran
> external windows programs on the server).
>
> May also try running Apache from the command line, but then again as you
> say it is then a different user, and quite different from running as a
> service.
>
> This server is not part of a domain. Maybe we just need to remove the
> mod_auth_sspi module {probably a hang over from another system}.
>

Removed mod_auth_sspi module. It was a hang over. No change in behavior.

Despite the enable interact with desktop being unchecked, the popup messages
still seems to be appearing on the admin console (need to confirm this)

Running Apache from the command line and disabling our watchog did give us
some further information.

Stared httpd.exe: no output and normal service started. Requests were
serviced normally

Initially there were 2 windows processes: PID 3748 & 9000

After the server stalled, the popup appeared on the terminal server and no
further requests were serviced.

Upon hitting the OK button on the popup, there were still 2 windows
processes, but now they were PID 3748 1580

Again server stalled and a popup appeared. No web requests serviced. After
clicking OK requests were serviced again normally. Now the PIDs are 3748 and
3556.

No messages ever appeared on the Windows cmd window running the httpd.exe
process (presume this meant that nothing was sent to stderr or stdout).

No crash dump was ever created, even though I had userdump configured to
monitor the process.

So it looks like something inside Apache is causing the popup and then
attempting to recover internally by regenerating a new worker process.

Could this be related to config parameters???
# Turn on library reload mechanism
PerlModule Apache2::Reload
PerlInitHandler Apache2::Reload

I'd like mod_perl to auto-reload, but I'd like it to do that without waiting
for the OK button on the windows popup.

regards,
--
View this message in context: http://old.nabble.com/Help-Debugging-Windows-Server-2003-Win 32-%2B-Apache2.2-%2B-mod_perl-%2B-Activestate-Pelr-5.8.8-Err orID-26-tp32004421p32011207.html
Sent from the mod_perl - General mailing list archive at Nabble.com.

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

HappyPerlUser wrote:
>
>
> HappyPerlUser wrote:
>> Have now unchecked the enable interact with desktop checkbox, although I
>> thought we needed that for some automation scripts in the past (that ran
>> external windows programs on the server).
>>
>> May also try running Apache from the command line, but then again as you
>> say it is then a different user, and quite different from running as a
>> service.
>>
>> This server is not part of a domain. Maybe we just need to remove the
>> mod_auth_sspi module {probably a hang over from another system}.
>>
>
> Removed mod_auth_sspi module. It was a hang over. No change in behavior.
>
> Despite the enable interact with desktop being unchecked, the popup messages
> still seems to be appearing on the admin console (need to confirm this)
>
> Running Apache from the command line and disabling our watchog did give us
> some further information.
>
> Stared httpd.exe: no output and normal service started. Requests were
> serviced normally
>
> Initially there were 2 windows processes: PID 3748 & 9000
>
> After the server stalled, the popup appeared on the terminal server and no
> further requests were serviced.
>
> Upon hitting the OK button on the popup, there were still 2 windows
> processes, but now they were PID 3748 1580
>
> Again server stalled and a popup appeared. No web requests serviced. After
> clicking OK requests were serviced again normally. Now the PIDs are 3748 and
> 3556.
>
> No messages ever appeared on the Windows cmd window running the httpd.exe
> process (presume this meant that nothing was sent to stderr or stdout).
>
> No crash dump was ever created, even though I had userdump configured to
> monitor the process.
>
> So it looks like something inside Apache is causing the popup and then
> attempting to recover internally by regenerating a new worker process.
>
> Could this be related to config parameters???
> # Turn on library reload mechanism
> PerlModule Apache2::Reload
> PerlInitHandler Apache2::Reload
>
> I'd like mod_perl to auto-reload, but I'd like it to do that without waiting
> for the OK button on the windows popup.
>

You probably know this, but just in case : the Apache2::Reload module /should/ have
nothing to do with the fact that the Apache server restarts after a crash. It's function
is to trigger a reload of a perl library module, if it notices that the module file has
changed on the disk since it was first loaded.
I am not sure how exactly it does that, but it is possible that in the case of Windows, it
does that by forcing a global "Apache reload" (aka kill the Apache children/threads and
restart them, which will force a reload of the Perl interpreter(s) etc..).

Also, as you might imagine, doing this means that it has to check the module files on disk
regularly, so it will cost some overhead. I would use this on a development server (where
modules might change frequently), but not on a production server (where they should change
only occasionally, and when they do, you should know and can do a manual Apache reload).

Under Windows, Apache starts 2 processes : one is the "main" Apache (similar to the Linux
case), and the other one is the "children" (in the forms of threads) which actually
process the requests. That's why you see 2 PID's.

But then, there are the following notes :
http://search.cpan.org/~phred/mod_perl-2.0.5/docs/api/Apache 2/Reload.pod#Threaded_MPM_and_Multiple_Perl_Interpreters

These notes lead me to suspect that, perhaps, there /is/ a relationship between your usage
of Apache2::Reload and the errors you are seeing. Such as perhaps if one of the
applications uses something like these "pseudo-hashes" they are talking about.
In any case, it gives me the general impression that using Apache2::Reload on a Windows
machine may not be the most stable/reliable thing to do.

What happens if you disable Apache2::Reload ?

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

awarnier wrote:
>
>
>
>
> What happens if you disable Apache2::Reload ?
>
>

Didn't think it was related, and we aren't in the habit of changing the perl
libraries, so I've disabled it for now. This does seem to have a significant
contributing effect. I haven't been able to trigger the problem now for 30
minutes and before I could hammer the server and cause the popup in a matter
of minutes. I can't explain this, and can only assume it's down to some sort
of race condition. I doubt this is the root cause, but it could reduce the
pain. Thanks.
--
View this message in context: http://old.nabble.com/Help-Debugging-Windows-Server-2003-Win 32-%2B-Apache2.2-%2B-mod_perl-%2B-Activestate-Pelr-5.8.8-Err orID-26-tp32004421p32012115.html
Sent from the mod_perl - General mailing list archive at Nabble.com.

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl+ Activestate Pelr 5.8.8 ErrorID

awarnier wrote:
>
>
>
>
> What happens if you disable Apache2::Reload ?
>
>
48 hours continuous operations now without a single glitch at all. Almost
certain that Apache2::Reload was not the root cause, but it was certainly
heavily implicated in the problem and resultant instability. If anyone's
interested in a crash dump I can re-enable this module temporarily and try
and generate one for you.

--
View this message in context: http://old.nabble.com/Help-Debugging-Windows-Server-2003-Win 32-%2B-Apache2.2-%2B-mod_perl-%2B-Activestate-Pelr-5.8.8-Err orID-26-tp32004421p32026697.html
Sent from the mod_perl - General mailing list archive at Nabble.com.

Re: Help Debugging Windows Server 2003 Win32 + Apache2.2 + mod_perl + Activestate Pelr 5.8.8 ErrorID

> awarnier wrote:
>
> >
> >
> > What happens if you disable Apache2::Reload ?
>
> 48 hours continuous operations now without a single glitch at all. Almost
> certain that Apache2::Reload was not the root cause, but it was certainly
> heavily implicated in the problem and resultant instability. If anyone's
> interested in a crash dump I can re-enable this module temporarily and try
> and generate one for you.

I experienced similar problems many years ago with Apache2::Reload
in a Microsoft Windows environment. Although disabling
Apache2::Reload did reduce the problem, it didn't completely
eliminate it and instability continued, although much less frequently
and after much longer time periods.

After having additional troubles with .DLL files and the
"perl.exe.manifest" file, I finally abandoned Windows and now run
everything under NetBSD Unix which simply doesn't exhibit these
problems for me...

I know this probably isn't the most constructive solution for you,
but it's good to know that there are other options. I suspect that
there was some problem either with Apache HTTPd or one of its modules
(and I haven't ruled out ModPerl 2 or APR 2), or MS-Windows itself.

The main reason I gave up on Windows is that each OS version
exhibited different patterns of when and how the crashing occurred.
On one system I saw blue screens, on another the system would just
stop responding (and there were less than 10 simultaneous connections
at any one time due to limits on the external firewall, so this
wasn't it either), and then on another I had problems with Apache
HTTPd just crashing with a variety of different error codes in the
Windows Event Viewer thingy. I could not narrow down specific
results to a particular version of Windows, but Unix did solve the
problem for me while also yielding better performance so I decided to
just ignore this problem and [mostly] forget about it.

Randolf Richardson - randolf@inter-corporate.com
Inter-Corporate Computer & Network Services, Inc.
Vancouver, British Columbia, Canada
http://www.inter-corporate.com