DAQmxbase blocking issue (mac os)

[Ageradi]

I understand. You had me worried for a time  Thanks for your reply.

Let's imagine DAQmxBase driver installs the labview runtime, how can I check for its presence? 

[AdamKemp]

You will find the LabVIEW Runtime in /Library/Frameworks. I'm not sure which version DAQmxBase uses. If you can load it and call functions in it then the RTE must be installed.

[Ageradi]

You are right.

I have two versions of LabVIEW X.Y Runtime.framework in /Library/Frameworks

Version 9.0 and version 11.0

Should I remove version 9.0? 

[AdamKemp]

No, don't remove any of those. DAQmxBase may have been built with version 9, and even if not then perhaps some other component on your system was.

[Ageradi]

Ok, I won't. I imagine they came with successive updates of DAQmxBase.

---

@AdamKemp wrote:

I just wanted to follow up on this issue because I am currently working on a potential fix. This is a bug in the LabVIEW runtime which is unique to OS X.

So, are you part of NI Developers, or is the source code of the driver available ? I am also interested in fixing this.

When I load a task for the first time, some initialization occurs sowhere and results in a large amount of new threads being spawned. These thread are idle. When exiting, for some reason, one of the threads seems to be waiting for someting. Indefinitely. Do you know:

1. What is the purpose of all the spawned threads (apparently, they sit there for the lifetime of the app).

2. What is the waiting thread looking for? An event? Other threads to exit? And why does it wait forever when the initialization wasn't done on the main thread?

Thanks for your help.

[AdamKemp]

I work on the LabVIEW team. The hang bug was in the LabVIEW runtime engine, and that is why I was working on it.

The threads are part of the LabVIEW execution system. LabVIEW executes VI code using a set of thread pools ("execution systems") rather than just executing everything in a single thread. Even though in your case you are calling the VI through a C entrypoint in a DLL we still execute the code using those thread pools so that we can do things in parallel.

The hang was caused by some code that I had added in a previous release to prevent a crash when a process which had loaded a LV-built DLL exited from its main routine before the LV-built DLL was unloaded. In that case the C runtime would start running the normal uninitialization/cleanup routines, but because LabVIEW's runtime engine still had threads going which used those resources being destroyed we would crash. The code I added caused the uninitialization routines to notify the LabVIEW runtime engine to kill those threads and then block until they were finished. That fixed the crash. The problem on OS X is that the thread which handles the "kill everything" event is whichever thread first called into LabVIEW code (not a thread we created), and that thread in your case no longer exists. In that case we would block indefinitely because the message was never received. Calling into LabVIEW from the main thread first ensured that the thread that is wanting to send that message is the same thread that would handle that message, and thus you could do it directly from that thread instead of hanging.

[Ageradi]

Thank you very much for this detailed answer.

Since the bug appears when the application is exiting, and since I know for a fact I will no longer need DAQmxBase services, Aas a temporary fix, can I:

- Send the kill everything message myself, and notify the waiting thread? If yes, How do I do that using C/C++ calls?

- Kill the waiting thread?

- Kill all threads?

[AdamKemp]

- Send the kill everything message myself, and notify the waiting thread? If yes, How do I do that using C/C++ calls?

The thread to send the message to no longer exists so you can't send it yourself any better than we can.

- Kill the waiting thread?

The waiting thread is the main thread, hence the hang.

- Kill all threads?

The other threads that are still running are not the problem. The problem is the main thread blocking while waiting for a thread which no longer exists to handle a message.

The way to work around it is to convince LabVIEW that the thread which should handle that message is the main thread itself, not some other thread. The way you do that is to call into a LabVIEW-built DLL from the main thread first (before calling into any others). If you do that you shouldn't have any problems. If you want to avoid a long delay then I suggest you do this:

1. Create a LV-built .framework built with the same version of LabVIEW as DAQmxBase with a single entrypoint which does nothing (i.e., a blank VI).
   
2. Load that .framework when you launch so that any cost of loading the runtime is considered part of the launch process, which is usually more acceptable to be a bit slower. The user doesn't usually expect the app to be responsive during the app's initial launch process.
   
3. Call into that entrypoint from the main thread at the end of your initialization. This should be fast because the .framework is already loaded, and the runtime engine should also already be loaded. If it is still slow then at least it's still part of your launch process.
   
4. Continue using DAQmxBase however you were using it before.
   

In my tests while investigating this issue I found that the above steps reliably prevented the hang from happening.

[Ageradi]

1. Create a LV-built .framework built with the same version of LabVIEW as DAQmxBase with a single entrypoint which does nothing (i.e., a blank VI).
2. Load that .framework when you launch so that any cost of loading the runtime is considered part of the launch process, which is usually more acceptable to be a bit slower. The user doesn't usually expect the app to be responsive during the app's initial launch process.

True. But the user doesn't expect the app to be that slow to launch, to be frozen, unresponsve, to have the mouse blinking and changing aspect, with random appearances from the spinning coloured wheel... Plus, I already load the labview runtime in my initialization, with a message explaining what's happening.

I don't understand everything – key points are missing – but sounds like the issue is that the thread that first called in down the labview runtime is long dead when the cleanup occurs. And that, indirectly is caussing al the issue.

The problem is solved if I frist call from the main thread, because the main thread is still alive when the cleanup occurs.

So, naturally, I tried to solve the issue by keeping the calling thread alive for the entiere app's life time. Unfortunately, this did not solve the issue.

Plus, the blocking thread is never the main thread. It is one of the threads that were spawned when I first called a DAQmxBaseLoad function.

So what exactly is the blocking threas waiting for? Is it a message, a signal, some variable becoming true? A callback?

[AdamKemp]

There must be some kind of disconnect here because calling a VI which is blank should not be even noticeable by the user if you have already loaded the runtime engine. Doing the steps I listed above should not be that slow. Have you tried doing exactly what I listed above (with the blank VI in a dummy .framework)? What exactly is happening when you do that?

Also, how exactly are you determining that some other thread is hanging? Keep in mind that if you see a thread waiting for a mutex for a long time that doesn't mean it is hung. That behavior is normal for the LabVIEW execution system threads. They spend most of their time doing nothing but waiting for something to do. They're not hung. They're just not doing anything.

It doesn't make sense to say that a hang on exit is occurring in some thread other than the main thread. Once the main thread stops running the app is dead. At some point during exit any thread that's not the main thread is mercilessly killed. The only way to hang on exit is if the main thread is hung. In this case it is hung because it is waiting for something to happen which will never happen. Other threads may still be running while the main thread is hung, and they may be active or just sitting there waiting for a mutex, but the hang symptom is caused by the main thread's inability to continue, not those other threads.