Strategies for dealing with general protection faults from non-debuggable threads.

[tstanley]

I'm having a problem with my (rather large) application.  It seems that randomly the application will just crash in the debugger with the dreaded "Unknown source position, thread id 0x00000FEC.  A non-debuggable thread caused a 'General Protection' fault at address 0x764BC99E."  While the thread id changes, the address is always the same.  This can seemingly happen at any time, though generally it's either when I have a pop up installed (using InstallPopup(...)) that has some plots that are being updated with data I'm acquiring from an instrument, or changing tabs on a tab control.  I have seen it when when a FileSelectPopupEx(..) is up, when resizing the application window, and other times.  Looking at where the main thread is in the application, it seemingly can be stopped just about anywhere.

I'm assuming it's one of the libraries that my application depends on.  However, this address is well outside the addresses in the map file.  It's also not in a range from the "Modules" window.  I have WLDAP32.dll at 0x75810000-0x75855000 and then kernel32.dll from 0x76610000-0x76720000.

I've tried adding the .c/.h files to my project from instrument files and running debuggable versions of DLLs in my project that I have control over that were built with CVI.  This had no effect, except for changing the address of the fault slightly (such as 0x7710C99E) presumably because memory got mapped out differently.  However, the address still isn't in a known module from the "Modules" window.

I had thought it might have to do with the toolbar library (no real reason but a hunch) so I added the .c/.h files from the toolbar library from CVI 7.1 (the source doesn't seem to be available for later versions?) and that made no difference.

I'm kind of running out of ideas on how to track down what is causing this.  About the only thing I have going right now is that so far, is that I haven't seen the error so far running the same code on a Windows XP/CVI 2012 computer, though I haven't run it as a much as I have on my Windows 7 64-bit/CVI 2012SP1 machine so I can't be sure.  It also doesn't seem to affect a "release" build, though I haven't discounted the possibility that memory is silently getting corrupted in the background which will eventually bring the application down.

At this point, any thoughts/ideas would be much appreciated.  Thanks.

[DocRabbit]

Hallo,

I've been also in this situation (not only once 😉 - e.g. here) - there are my hints:

- enable generation of map file (checkbox on Target Settings)

- install some debugger (WinDbg?)

- enable generation of memory dump (like Dr. Watson ...)

- after crash try to analyze ...

Good luck!

George

[tstanley]

Thanks for the response.

I should have mentioned that I do have the map file being generated.  The map file has proven itself to be invaluable in other situations, but in this case the address is not anywhere in the map file, so unless there is something I'm missing, it doesn't seem to help in this situation.

As for other debuggers, I'm not sure how to invoke them.  If I run the debuggable through CVI, then the CVI debugger seems to catch the problem, things like Dr. Watson don't seem to get invoked, and I just have CVI's tools at my disposal.  I suppose I could try running the debuggable .exe outside of CVI and seeing if it crashes.

It's interesting that you had problems with the tooltips, since that just reminded me that I do have some of the old-style tooltips in my project.  I should convert them over to the new-style.  It's worth a shot at least.  

[tstanley]

Unfortunently changing the tooltip style did not help.

I'm a bit of a newbie at Windbg, but I have generated a crash dump per https://decibel.ni.com/content/docs/DOC-1496.  

It would appear that something in NetworkItemFactory.dll (from Microsoft) is trying to access a pointer at 0xfeeefeee, where 0xfeeefeee is Microsoft's way of marking unallocated memory on the heap.  Other than that, I'm going to have to try to dig deeper to make sense of this.

[tstanley]

It looks like it may be related to the initialization of the COM library, from this post on MSDN:

http://social.msdn.microsoft.com/Forums/en-US/vcmfcatl/thread/5037519a-78e2-42f4-94cd-bbe88e0f16d6

And thusly related to this forum post I made a few months ago (this is the same application):

http://forums.ni.com/t5/LabWindows-CVI/FileSelectPopup-hang-with-restrict-extension-option-and/td-p/...

That also seemingly explains why the problem hasn't cropped up in Windows XP.  I'm a bit unsure on all of this, so I'm trying different things to see if the problem goes away, but if anyone has any advice on how to set things up it would be most helpful.  Thanks.

[tstanley]

I think I've eliminated the crash.  It would appear to be related to the new "-Ex" file popups.  That is, FileSelectPopupEx(...) and it's cousins.  It would appear that my application is okay until I open up one of the new file/directory pop ups.  After this, the application will crash in a few minutes.  Here's the things I tried:

• Replaced CA_InitActiveXThreadStyleForCurrentThread(...) with the Windows' CoInitializeEx(..) function.  I realize that the CVI function is just a wrapper around the Windows function (at least according to the documentation), however, I wanted to try calling it before initializing the CVI Runtime, and it would appear that there is no CVI wrapper around the CoUninitialize(..) function which seems like it should be called.  This had no effect.
• Changing to COINIT_MULTITHREADED from COINIT_APARTMENTTHREADED.  This resulted in a non-fatal runtime UIEFileDialogWrongConcurrencyModel error with the various file pop ups.
• Added the COINIT_DISABLE_OLE1DDE flag.  This had no effect.
• Put the IMAQ functions in their own thread, so that I could eliminate the calls to CA_InitActiveXThreadStyleForCurrentThread/CoInitializeEx.  This solved the problem I was having with the IMAQ functions and file popups previously, but this had no effect on the random general protection faults.
• Replaced all instances of the new -Ex file/directory popups with the old non-Ex versions.  So far this has eliminated the random application crashes.

While it's relieving to have figured this out.  It's kind of disappointing to be stuck with the old dialogs.  I would like to be able to use the new ones.    Perhaps an NI engineer can comment on this?

[tstanley]

Attached is a small application that shows the problem on my system.  Instructions for use:

1.  Run application in CVI debugger on Windows 7 (probably Vista and 8 will work too)

2.  Open the FileSelectPopup and cancel out of the pop up a time or two.

3.  Wait.

4.  Within a couple of minutes the application will crash on my system with a fatal runtime error, unknown source position, general protection fault at 0x7710C99E.

[tstanley]

And here's the "experimental" hotfix from Microsoft that addresses this problem.  So far it seems to work:

http://support.microsoft.com/kb/2718841/en-us

Also, according to Microsoft this does not affect Windows Vista or 8.  

[SteveD123]

I see you are using IMAQ. I know this sounds obvious, but have you checked for memory leaks? I found out (the hard way) that you have to be very careful to correctly de-allocate memory that IMAQ may have allocated when you invoking some of the image processing functions (e.g. imaqEdgeTool4). For some reason the CVI resource monitor does not catch this issue. Looking at the memory consumption of the process in task manager will indicate a problem. I know that in my case I would eventually receive an error message similar to yours.

[tstanley]

Thanks for the thought.  I've certainly had a few memory leaks with the IMAQ libraries, but in this case I've got it to the point where I can duplicate the problem without calling any of the IMAQ stuff.  I'm about 90% sure I've got the solution, but given the sporadic nature of the problem I'm going to give it a few more days to be sure.